Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replace dependency on broken node-unzipper with native zlib #5714

Merged
merged 16 commits into from
May 2, 2023

Conversation

Durisvk
Copy link
Contributor

@Durisvk Durisvk commented Apr 20, 2023

Description

Fixes:
#5614
#5677
firebase/firebase-tools-ui#939
firebase/firebase-tools-ui#940
firebase/firebase-tools-ui#942

The node-unzipper package is generating incorrect files when unpacking the ui and pubsub emulator archive files (ZJONSSON/node-unzipper#271). It generates a Syntax error and the emulator fails to execute. The issue happens on the latest NodeJS 18 version.

Scenarios Tested

Automated: src/test/unzip.spec.ts

Manual:

Running the command before the fix

firebase emulators:start --only database,auth,ui --import ./firebase-emulator --export-on-exit

Output:

i  emulators: Starting emulators: auth, database
⚠  ui: Not starting the ui emulator, make sure you have run firebase init.
i  database: Database Emulator logging to database-debug.log
i  database: Importing data from /Users/jurajcarnogursky/ooo/vette/monorepo/firebase-emulator/database_export/vette-dev-default-rtdb.json
i  database: Importing data from /Users/jurajcarnogursky/ooo/vette/monorepo/firebase-emulator/database_export/vette-staging-default-rtdb.json
i  auth: Importing config from /Users/jurajcarnogursky/ooo/vette/monorepo/firebase-emulator/auth_export/config.json
i  auth: Importing accounts from /Users/jurajcarnogursky/ooo/vette/monorepo/firebase-emulator/auth_export/accounts.json
i  ui: Emulator UI logging to ui-debug.log
⚠  ui: Fatal error occurred: 
   Emulator UI has exited with code: 1, 
   stopping all running emulators
i  ui: Stopping Emulator UI
⚠  ui: Error stopping Emulator UI
i  database: Stopping Database Emulator
i  auth: Stopping Authentication Emulator
i  hub: Stopping emulator hub
i  logging: Stopping Logging Emulator

Then running rm -rf ~/.cache/firebase/emulators, linking my new version of firebase-tools and rerunning the same command again:

firebase emulators:start --only database,auth,ui --import ./firebase-emulator --export-on-exit

Output:

i  emulators: Starting emulators: auth, database
⚠  ui: Not starting the ui emulator, make sure you have run firebase init.
i  database: downloading firebase-database-emulator-v4.11.0.jar...
Progress: ==================================================================================================================================================================================================================================> (100% of 35MB
i  database: Database Emulator logging to database-debug.log
i  database: Importing data from /Users/jurajcarnogursky/ooo/vette/monorepo/firebase-emulator/database_export/vette-dev-default-rtdb.json
i  database: Importing data from /Users/jurajcarnogursky/ooo/vette/monorepo/firebase-emulator/database_export/vette-staging-default-rtdb.json
i  auth: Importing config from /Users/jurajcarnogursky/ooo/vette/monorepo/firebase-emulator/auth_export/config.json
i  auth: Importing accounts from /Users/jurajcarnogursky/ooo/vette/monorepo/firebase-emulator/auth_export/accounts.json
i  ui: downloading ui-v1.11.5.zip...
Progress: ===================================================================================================================================================================================================================================> (100% of 4MB
i  ui: Emulator UI logging to ui-debug.log

┌─────────────────────────────────────────────────────────────┐
│ ✔  All emulators ready! It is now safe to connect your app. │
│ i  View Emulator UI at http://127.0.0.1:5002/               │
└─────────────────────────────────────────────────────────────┘

┌────────────────┬────────────────┬────────────────────────────────┐
│ Emulator       │ Host:Port      │ View in Emulator UI            │
├────────────────┼────────────────┼────────────────────────────────┤
│ Authentication │ 127.0.0.1:9099 │ http://127.0.0.1:5002/auth     │
├────────────────┼────────────────┼────────────────────────────────┤
│ Database       │ 127.0.0.1:9000 │ http://127.0.0.1:5002/database │
└────────────────┴────────────────┴────────────────────────────────┘
  Emulator Hub running at 127.0.0.1:4400
  Other reserved ports: 4500

Issues? Report them at https://github.com/firebase/firebase-tools/issues and attach the *-debug.log files.

Sample Commands

N/A

IMPORTANT ❗

Is it possible to automatically execute rm -rf ~/.cache/firebase for users after updating the package? If not then the fix will just not work because the corrupted cached version will be preferred.

One workaround I can think of is introducing a cache versioning as a text file in the ~/.cache/firebase folder. Whenever that number doesn't match the sourcecode or the file doesn't exist the cache is forcefully regenerated and the cache version is adjusted to match the one in the sourcecode.

@google-cla
Copy link

google-cla bot commented Apr 20, 2023

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

Copy link
Contributor

@christhompsongoogle christhompsongoogle Apr 24, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Curious: What is this broken.zip file for?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sorry, it was copied over from the fixtures in unzipper and forgot to delete it since it's not used in the tests. Going to remove it.

I've also included LICENSE file from node-unzipper package since I added some of the fixtures from there - https://github.com/ZJONSSON/node-unzipper/blob/master/LICENSE . I am not an expert on license policies - is it okay to keep it in there or should I remove it?

@christhompsongoogle
Copy link
Contributor

christhompsongoogle commented Apr 24, 2023

Thanks for contributing to the firebase-tools repo !

Is it possible to automatically execute rm -rf ~/.cache/firebase for users after updating the package? If not then the fix will just not work because the corrupted cached version will be preferred.

We can force a re-download of the corrupted emulators in the cache by bumping their versions.

@codecov-commenter
Copy link

codecov-commenter commented Apr 24, 2023

Codecov Report

Patch coverage: 58.42% and project coverage change: +0.01 🎉

Comparison is base (e3d1c99) 55.20% compared to head (f052c5c) 55.22%.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #5714      +/-   ##
==========================================
+ Coverage   55.20%   55.22%   +0.01%     
==========================================
  Files         328      329       +1     
  Lines       22417    22500      +83     
  Branches     4576     4586      +10     
==========================================
+ Hits        12375    12425      +50     
- Misses       8941     8971      +30     
- Partials     1101     1104       +3     
Impacted Files Coverage Δ
src/extensions/extensionsHelper.ts 50.95% <50.00%> (ø)
src/unzip.ts 58.13% <58.13%> (ø)
src/emulator/download.ts 19.35% <100.00%> (+0.89%) ⬆️

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

@Durisvk
Copy link
Contributor Author

Durisvk commented Apr 24, 2023

Thanks for contributing to the firebase-tools repo !

Is it possible to automatically execute rm -rf ~/.cache/firebase for users after updating the package? If not then the fix will just not work because the corrupted cached version will be preferred.

We can force a re-download of the corrupted emulators in the cache by bumping their versions.

awesome, @christhompsongoogle , in case this gets released can you keep track of bumping the version of ui emulator and pubsub emulator?

@christhompsongoogle
Copy link
Contributor

awesome, @christhompsongoogle , in case this gets released can you keep track of bumping the version of ui emulator and pubsub emulator?

Sure, I'd be happy to bump the versions once this is in

@Durisvk Durisvk force-pushed the master branch 2 times, most recently from cd0b797 to 8a159e2 Compare April 24, 2023 19:50
@christhompsongoogle
Copy link
Contributor

christhompsongoogle commented Apr 25, 2023

After doing some testing on Windows it seems that there's an issue with the mkdir (see below for details).

If I were to guess I think this has to do with the fact that windows uses backslash instead of forward slashes for their directory names, so outputFilePath.lastIndexOf will likely return an empty result.

i  ui: downloading ui-v1.11.5.zip... {"metadata":{"emulator":{"name":"ui"},"message":"downloading ui-v1.11.5.zip..."}}
[2023-04-25T02:11:36.192Z] >>> [apiv2][query] GET https://storage.googleapis.com/firebase-preview-drop/emulator/ui-v1.11.5.zip
[2023-04-25T02:11:36.220Z] <<< [apiv2][status] GET https://storage.googleapis.com/firebase-preview-drop/emulator/ui-v1.11.5.zip 200
[2023-04-25T02:11:36.221Z] <<< [apiv2][body] GET https://storage.googleapis.com/firebase-preview-drop/emulator/ui-v1.11.5.zip [stream]
Progress: ==============================================================================================> (100% of 4MB)
i  emulators: Shutting down emulators. {"metadata":{"emulator":{"name":"hub"},"message":"Shutting down emulators."}}
i  ui: Stopping Emulator UI {"metadata":{"emulator":{"name":"ui"},"message":"Stopping Emulator UI"}}
i  database: Stopping Database Emulator {"metadata":{"emulator":{"name":"database"},"message":"Stopping Database Emulator"}}
!  Database Emulator has exited upon receiving signal: SIGINT
i  firestore: Stopping Firestore Emulator {"metadata":{"emulator":{"name":"firestore"},"message":"Stopping Firestore Emulator"}}
!  Firestore Emulator has exited upon receiving signal: SIGINT
i  auth: Stopping Authentication Emulator {"metadata":{"emulator":{"name":"auth"},"message":"Stopping Authentication Emulator"}}
i  storage: Stopping Storage Emulator {"metadata":{"emulator":{"name":"storage"},"message":"Stopping Storage Emulator"}}
i  hub: Stopping emulator hub {"metadata":{"emulator":{"name":"hub"},"message":"Stopping emulator hub"}}
i  logging: Stopping Logging Emulator {"metadata":{"emulator":{"name":"logging"},"message":"Stopping Logging Emulator"}}
[2023-04-25T02:11:36.443Z] Error: ENOENT: no such file or directory, mkdir ''

Copy link
Contributor

@joehan joehan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Really appreciate the contribution! Some changes to make, but this is a great start.

I also pulled this locally and did some testing on various versions of Node on a Mac w/ Intel chips, seems to work correctly AFAICT.

CHANGELOG.md Outdated
@@ -5,3 +5,5 @@
- Lift GCF 2nd gen naming restrictions (#5690)
- Fixes a bug where `ext:install` and `ext:configure` would error on extensions with no params.
- Fixed an issue with Vite and Angular integrations using a obsolete NPM command (#5710)
- Fix bugs with UI emulator and PubSub emulator not starting correctly (#5714)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- Fix bugs with UI emulator and PubSub emulator not starting correctly (#5714)
- Fixes bug were emulators would not starting correctly due to corrupted ZIP files. (#5614, #5677)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rewording this a lil bit.

CHANGELOG.md Outdated
@@ -5,3 +5,5 @@
- Lift GCF 2nd gen naming restrictions (#5690)
- Fixes a bug where `ext:install` and `ext:configure` would error on extensions with no params.
- Fixed an issue with Vite and Angular integrations using a obsolete NPM command (#5710)
- Fix bugs with UI emulator and PubSub emulator not starting correctly (#5714)
- Should resolve #5614 #5677 firebase/firebase-tools-ui#939 firebase/firebase-tools-ui#940 firebase/firebase-tools-ui#942
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- Should resolve #5614 #5677 firebase/firebase-tools-ui#939 firebase/firebase-tools-ui#940 firebase/firebase-tools-ui#942

@@ -0,0 +1,25 @@
Copyright (c) 2012 - 2013 Near Infinity Corporation
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please remove this file - we aren't able to accept PRs that add new licenses to this repo.

@@ -0,0 +1,165 @@
import { expect } from "chai";
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This thorough testing is great! However, these seem like integration tests to me. They are longer running than is ideal for our unit tests, and they also occasionally flake due to timeouts when run locally.

I think these tests would be appropriate as integration tests. Could you separate them into a different npm command test:unzip, and ensure that they are not run as part of npm run test ?

});

after(async () => {
// await fs.promises.rmdir(ZIP_TEMPORARY_PATH, { recursive: true });
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems like it should not be commented out

before(function (done) {
// eslint-disable-next-line @typescript-eslint/no-invalid-this
this.timeout(5000);
(async () => {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unclear to me why we need this needs to be set up as a self executing function. Could this just be:

before(async () => {
 ...
}

});

it("should unzip a ui emulator zip file", async function () {
// eslint-disable-next-line @typescript-eslint/no-invalid-this
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can remove this linter violation by refactoring to:

it("should unzip...", async () => {}).timeout(2000);

});

it("should unzip a pubsub emulator zip file", async function () {
// eslint-disable-next-line @typescript-eslint/no-invalid-this
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can remove this linter violation by refactoring to:

it("should unzip...", async () => {}).timeout(2000);

@Durisvk
Copy link
Contributor Author

Durisvk commented Apr 25, 2023

outputFilePath

Hi, unfortunately I have no Windows device, I've pushed a fix replacing / with path.sep which should match the operating system, could you please retry?

@Durisvk
Copy link
Contributor Author

Durisvk commented Apr 25, 2023

Hi @joehan, thank you for the great points. I've addressed them.

Regarding moving the integration tests for unzip into a separate test - is it okay that I've moved them under npm test:emulator command since the logic is used within the emulators?

@christhompsongoogle
Copy link
Contributor

could you please retry?

Here's the latest issue I encountered

Progress: ==============================================================================================> (100% of 4MB)
i  emulators: Shutting down emulators. {"metadata":{"emulator":{"name":"hub"},"message":"Shutting down emulators."}}
i  ui: Stopping Emulator UI {"metadata":{"emulator":{"name":"ui"},"message":"Stopping Emulator UI"}}
i  database: Stopping Database Emulator {"metadata":{"emulator":{"name":"database"},"message":"Stopping Database Emulator"}}
!  Database Emulator has exited upon receiving signal: SIGINT
i  firestore: Stopping Firestore Emulator {"metadata":{"emulator":{"name":"firestore"},"message":"Stopping Firestore Emulator"}}
!  Firestore Emulator has exited upon receiving signal: SIGINT
i  auth: Stopping Authentication Emulator {"metadata":{"emulator":{"name":"auth"},"message":"Stopping Authentication Emulator"}}
i  storage: Stopping Storage Emulator {"metadata":{"emulator":{"name":"storage"},"message":"Stopping Storage Emulator"}}
i  hub: Stopping emulator hub {"metadata":{"emulator":{"name":"hub"},"message":"Stopping emulator hub"}}
i  logging: Stopping Logging Emulator {"metadata":{"emulator":{"name":"logging"},"message":"Stopping Logging Emulator"}}
[2023-04-26T21:46:51.161Z] Error: EISDIR: illegal operation on a directory, open 'C:\Users\chris\.cache\firebase\emulators\ui-v1.11.5\client'

The directory has a client directory but it's empty - and no server directory exists there

@Durisvk
Copy link
Contributor Author

Durisvk commented Apr 27, 2023

Hey @christhompsongoogle, sorry for the trouble. I've pushed a new version with potential fix and additional console logs in case the fix doesn't work. Once I've manage to fix that I'll go ahead and remove those console logs.

In case you have time, can you test it? I'll have my brother over at my place on Saturday so I could borrow his Windows laptop to debug this but currently I don't have access to any windows machine.

Copy link
Contributor

@joehan joehan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few more minor tweaks, but this is very close to LGTM.

CHANGELOG.md Outdated
@@ -5,5 +5,4 @@
- Lift GCF 2nd gen naming restrictions (#5690)
- Fixes a bug where `ext:install` and `ext:configure` would error on extensions with no params.
- Fixed an issue with Vite and Angular integrations using a obsolete NPM command (#5710)
- Fix bugs with UI emulator and PubSub emulator not starting correctly (#5714)
- Should resolve #5614 #5677 firebase/firebase-tools-ui#939 firebase/firebase-tools-ui#940 firebase/firebase-tools-ui#942
- Fixes bug were emulators would not starting correctly due to corrupted ZIP files. (#5614, #5677)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- Fixes bug were emulators would not starting correctly due to corrupted ZIP files. (#5614, #5677)
- Fixes bug were emulators would not start correctly due to corrupted ZIP files. (#5614, #5677)

@@ -0,0 +1,12 @@
module.exports = (() => {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you run npm run format? This file is failing whitespace linting

src/unzip.ts Outdated
const compressionMethod = entryHeader.readUInt16LE(8);
if (compressionMethod === 0) {
// Store (no compression)
console.log(`Writing file: \${outputFilePath}`);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please switch this to logger.debug(). That way, the logs will only appear in --debug mode.

src/unzip.ts Outdated

const outputFilePath = path.normalize(path.join(outputDir, entry.fileName));

console.log(`Processing entry: \${entry.fileName}`);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please switch this to logger.debug(). That way, the logs will only appear in --debug mode.

@Durisvk
Copy link
Contributor Author

Durisvk commented Apr 28, 2023

@joehan I've adjusted the code based on your comments. I've also rebased on the current master.

@christhompsongoogle, once you'll be retesting this, can you include a --debug flag? (I've changed console.logs to logger.debugs)

Copy link
Contributor

@joehan joehan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM once @christhompsongoogle confirms that it looks good on Windows

@christhompsongoogle
Copy link
Contributor

christhompsongoogle commented Apr 28, 2023

can you test it

Just retested on Windows and everything looks great. Thank you for the windows fixes !

LGTM on this one

@Durisvk
Copy link
Contributor Author

Durisvk commented Apr 28, 2023

Hey should I remove the unnecessary debug logs? in case someone wants to debug something so that they don't see these unnecessary logs.

@christhompsongoogle
Copy link
Contributor

christhompsongoogle commented Apr 28, 2023

Hey should I remove the unnecessary debug logs? in case someone wants to debug something so that they don't see these unnecessary logs.

I don't mind leaving them in - they'll require some formatting to satisfy lint before we merge.

@Durisvk
Copy link
Contributor Author

Durisvk commented Apr 30, 2023

Hey should I remove the unnecessary debug logs? in case someone wants to debug something so that they don't see these unnecessary logs.

I don't mind leaving them in - they'll require some formatting to satisfy lint before we merge.

I'm having a really tough time understanding the linting rules (there's plenty of warnings popping up). Can you advise what needs to be changed before merging?

@joehan
Copy link
Contributor

joehan commented May 1, 2023

Hey @Durisvk, I did some more testing this weekend and realized that the extensions emulator isn't working under this PR. I'm going to try to debug/fix it when I have some time this week, but if you want to take a crack at it, the easiest way to exercise the path is the integration test:

FBTOOLS_TARGET_PROJECT= npm run test:extensions-emulator

@joehan
Copy link
Contributor

joehan commented May 2, 2023

So, it seems like this implementation doesn't handle zips that have the data descriptor bit set: https://en.wikipedia.org/wiki/ZIP_(file_format)#:~:text=the%20compressed%20data.-,Data%20descriptor,-%5Bedit%5D.

Incidentally, Extensions zips all have this bit set. Looking into how to add support for this now

src/unzip.ts Outdated
await fs.promises.mkdir(parentDir, { recursive: true });

const compressionMethod = entryHeader.readUInt16LE(8);
if (compressionMethod === 0) {
if (entry.compressedSize === 0 || compressionMethod === 0) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't love that we're using the compressedSize to make decisions - though that might legitimately be part of the header. Do you have a link to the header breakdown (or the bitFlag === 8 details) we could leave as a comment?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch - this actually does not need to be here, was leftover from a different approach i tried.

@joehan joehan enabled auto-merge (squash) May 2, 2023 23:30
@joehan joehan disabled auto-merge May 2, 2023 23:30
@joehan joehan merged commit 19a8384 into firebase:master May 2, 2023
@cupidchan
Copy link

@Durisvk @christhompsongoogle @joehan do you have a timeline when this fix will be available in the official NPM? Thanks!

ProfHercules pushed a commit to ProfHercules/firebase-tools that referenced this pull request May 5, 2023
…#5714)

* Replace dependency on broken node-unzipper with native zlib

Fixes:
firebase#5614
firebase#5677
firebase/firebase-tools-ui#939
firebase/firebase-tools-ui#940

* remove unused broken.zip fixture

* add changelog record

* fix timing out test

* fix use operating system file delimiter

* addressing code review comments from @joehan

* better support for Windows path separators

TODO: remove console logs after @christhompsongoogle tries it out.

* addressing code review comments from @joehan

* Add support for zip files that use signed data descriptors instead of a full entry header

* Formats, fixes uneeded async/await

* Remove unnecessary handling for empty files

* Remove console.log

---------

Co-authored-by: joehan <joehanley@google.com>
@christhompsongoogle
Copy link
Contributor

@Durisvk @christhompsongoogle @joehan do you have a timeline when this fix will be available in the official NPM? Thanks!

This should be available now, with firebase-tools v12.0 (and possibly 11.30 before that)

tonyjhuang pushed a commit that referenced this pull request May 22, 2023
* Replace dependency on broken node-unzipper with native zlib

Fixes:
#5614
#5677
firebase/firebase-tools-ui#939
firebase/firebase-tools-ui#940

* remove unused broken.zip fixture

* add changelog record

* fix timing out test

* fix use operating system file delimiter

* addressing code review comments from @joehan

* better support for Windows path separators

TODO: remove console logs after @christhompsongoogle tries it out.

* addressing code review comments from @joehan

* Add support for zip files that use signed data descriptors instead of a full entry header

* Formats, fixes uneeded async/await

* Remove unnecessary handling for empty files

* Remove console.log

---------

Co-authored-by: joehan <joehanley@google.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants