Win 10 - symlink - operation not permitted, symlink

[Fizcko]

Current behavior:

Unable to install cypress on Windows 10.

Desired behavior:

Computers on my company cannot run cmd as administrator to resolve symlink problems.
Is it possible to provide a way to install Cypress without global cache directory ?

Test code to reproduce

> npm install --save cypress
  √  Downloaded Cypress
  ×  Unzipping Cypress        4% 130s
    → Cypress Version: 5.1.0
     Finishing Installation
The Cypress App could not be unzipped.

Search for an existing issue or open a GitHub issue at

https://github.com/cypress-io/cypress/issues

----------

Error: EPERM: operation not permitted, symlink '..\..\..\esprima\bin\esvalidate.js' -> 'C:\_cache\cypress\5.1.0\Cypress.app\Contents\Resources\app\packages\rewriter\node_modules\recast\node_modules\.bin\esvalidate'

----------

Platform: win32 (10.0.18363)
Cypress Version: 5.1.0

Versions

Node JS: 12.13.0
Cypress: 5.1.0
Operating system: Windows 10 (10.0.18363)

[jonnathan-ls]

I have the same "operation not allowed" problem, but for the Electron path

✔  Downloaded Cypress
✖ The Cypress App could not be unzipped.

Search for an existing issue or open a GitHub issue at

https://github.com/cy…
The Cypress App could not be unzipped.

Search for an existing issue or open a GitHub issue at

https://github.com/cypress-io/cypress/issues

----------

Error: EPERM: operation not permitted, symlink 'Versions\Current\Electron Framework' -> 'C:\Users\X252185\AppData\Local\Cypress\Cache\7.5.0\Cypress.app\Contents\Frameworks\Electron Framework.framework\Electron Framework'

----------

Platform: win32 (10.0.18363)
Cypress Version: 7.5.0

[cheesestringer]

Same error as @Fizcko but on version 7.5.0

Error: EPERM: operation not permitted, symlink '..\..\..\esprima\bin\esvalidate.js' -> 'C:\Users\xyz\AppData\Local\Cypress\Cache\7.5.0\Cypress.app\Contents\Resources\app\packages\rewriter\node_modules\recast\node_modules\.bin\esvalidate'


Platform: win32 (10.0.18363)
Cypress Version: 7.5.0

[cypress-app-bot]

This issue has not had any activity in 180 days. Cypress evolves quickly and the reported behavior should be tested on the latest version of Cypress to verify the behavior is still occurring. It will be closed in 14 days if no updates are provided.

[aarowman]

I'm having this issue today, on latest version 12.13.0. Please do not close the issue, but investigate it

[MikeMcC399]

@aarowman

I'm having this issue today, on latest version 12.13.0. Please do not close the issue, but investigate it

Have you referred to the npm documentation?

I cannot guarantee it, but perhaps
https://docs.npmjs.com/cli/v9/commands/npm-install?v=true#bin-links

npm install cypress --save-dev --no-bin-links

will help.

[aarowman]

@MikeMcC399 Why is this necessary here? Since Cypress downloads the binary to a separate folder, wouldn't we still need the links?

I didn't have to use it before. I had cypress 12.7.0 installed fine, but then when upgrading (simply updating version in package.json), then I get this error.

I am currently clearing my npm cache and deleting node_modules to try a fresh install, see if it helps anything. I see what you're referencing and can try it out if that doesn't work.

[MikeMcC399]

@aarowman

Are you using a corporate Windows 10 system which has security policies applied or do you have full administrator rights on your system?

Could you post your error message, just to make sure that we are looking at the same issue?
The previous report was for Cypress 7.5.0 which was released almost 2 years ago in 2021.

[aarowman]

I am not running as administrator. The cache is in AppData/local. Will this only work if running as admin?

I would hope that wouldn't be required, but I do have the access if needed.

Yes, we are on a corporate system, but this worked on previous installs of Cypress... I am running into this today for some reason.

The error matches the description above. Here is the error:

npm ERR! Platform: win32-x64 (10.0.18363)
npm ERR! Cypress Version: 12.13.0
npm ERR! [FAILED] The Cypress App could not be unzipped.
npm ERR! [FAILED]
npm ERR! [FAILED] Search for an existing issue or open a GitHub issue at
npm ERR! [FAILED]
npm ERR! [FAILED] https://github.com/cypress-io/cypress/issues
npm ERR! [FAILED]
npm ERR! [FAILED] ----------
npm ERR! [FAILED]
npm ERR! [FAILED] Error: EPERM: operation not permitted, symlink '..\..\..\update-browserslist-db\cli.js' -> 'C:\Users\myUser\AppData\Local\Cypress\Cache\12.13.0\Cypress.app\Contents\Resources\app\node_modules\browserslist\node_modules\.bin\browserslist-lint'
npm ERR! [FAILED]
npm ERR! [FAILED] ----------
npm ERR! [FAILED]
npm ERR! [FAILED] Platform: win32-x64 (10.0.18363)
npm ERR! [FAILED] Cypress Version: 12.13.0

[MikeMcC399]

@aarowman

Thanks for your exact error log. This does look like it's a Windows permissions problem. Are you able to run npm install from a terminal window as Administrator?

(Your Windows version 10.0.18363 looks quite old and https://learn.microsoft.com/en-us/lifecycle/products/windows-10-home-and-pro says support for Windows 10.0.18363 aka 1909 or 19H2 ended on May 11, 2021. I don't know if that would have any influence on the issue. I tried to reproduce on Windows 11 but without success. It all worked for me.)

[MikeMcC399]

@aarowman

Which version of Node.js and npm are you using? I will try to reproduce this on a Windows 10 system instead of a Windows 11 system.

Going back to your error message, there are actually two parts:

• The Cypress App could not be unzipped.
• Error: EPERM: operation not permitted, symlink

Edit: This turns out to be one problem.

[aarowman]

@MikeMcC399 As mentioned, npm install worked fine in the past. It also works fine for all other dependencies (I have done it hundreds of times). I do have the ability to use admin to run it, but is that actually necessary? Shouldn't Cypress work the same as the rest of the npm downloads? The Cypress cache is in my user AppData folder, so I shouldn't need admin to access that.

$ node -v
v18.15.0
$ npm -v
9.5.0

[MikeMcC399]

@aarowman

Thanks for passing on your node / npm versions!

As a troubleshooting step you could attempt to install as Admin. If that works, then that would point to your non-admin user not having sufficient permissions / rights, which could be separately looked at. You could also check to see if you can still re-install cypress@12.7.0 with your non-admin user, as you said, this previously was no problem.

You might possibly be able to get some additional information through
npm install cypress --loglevel verbose
although I'm not too hopeful that would pinpoint the issue.

I can't really say what should or shouldn't work in terms of non-admin. The Cypress documentation does not discuss this and there was no troubleshooting done on the original issue as far as I can see. I am however willing to dig deeper into this one.

Your official Cypress.io assignee is however @AtofStryker. I am just one of the community contributors responding to you here.

[aarowman]

@MikeMcC399 @AtofStryker The install does appear to work when running as admin (using Git Bash), so it seems like a permission issue. The rest of the npm install works fine for other dependencies. The cache is pointing to user AppData, which should be owned by the local user.

Why does Cypress require admin to unzip the file?

I see in this code: https://github.com/cypress-io/cypress/blob/7ef4300a6b1a367f980e5d6a0182dac292694d01/cli/lib/tasks/install.js#LL345C2-L345C34
const downloadDir = os.tmpdir()
Doesn't that also download to AppData by default? So the download should be in a location that the local user can use to unzip.

[AtofStryker]

@MikeMcC399 I think you are going a fantastic job responding to this issue.

@MikeMcC399 @AtofStryker The install does appear to work when running as admin (using Git Bash), so it seems like a permission issue. The rest of the npm install works fine for other dependencies. The cache is pointing to user AppData, which should be owned by the local user.

That is good at least since this could function as a workaround.

Why does Cypress require admin to unzip the file?

I wouldn't say Cypress requires admin to unzip the file, but your system depending on the directory might require it to link the file. In the post above, do you know who owns this directory: ..\..\..\update-browserslist-db\cli.js and what the location is since its a relative path? If you try to symlink that yourself in gitbash, what are the results?

I see in this code: https://github.com/cypress-io/cypress/blob/7ef4300a6b1a367f980e5d6a0182dac292694d01/cli/lib/tasks/install.js#LL345C2-L345C34 const downloadDir = os.tmpdir() Doesn't that also download to AppData by default? So the download should be in a location that the local user can use to unzip.

That should be correct assuming another temp directory is configured on your machine. I don't think its an issue with the actual access of the directory, but the permissions to symlink to/from it. It's hard to say, but we might have a better idea depending on who owns each directory.

[MikeMcC399]

@aarowman

I found the missing debug log instructions! If you could execute the following in a Git Bash terminal, it should show exactly what is happening and where it is failing:

export DEBUG='cypress:*'
npm install cypress@latest --loglevel verbose --foreground-scripts

If you have previously been successful installing then you may need to execute the following first:

npx cypress cache clear
npm uninstall cypress

@AtofStryker

I think you are going a fantastic job responding to this issue.

Thanks for the compliment and for your participation! As always it's a teamwork thing to get to the bottom of these issues. No single person has all the answers!

[MikeMcC399]

@aarowman

Alternatively to debug, it may be sufficient just to execute

export DEBUG='cypress:*'
npx cypress install

[aarowman]

@AtofStryker I'm not sure we can confidently say the admin permission workaround is sufficient. Not all users may have the ability to use admin, and there should be no requirement to use it.

The relative path ..\..\..\update-browserslist-db\cli.js is being hit during the cypress script, I believe it's during/after unzip behavior, so I'm not sure where that's coming from. That is only happening because the cypress script is trying to access it (or unzip is, at least).

I'm not sure how the temp dir is defined. Again, I'm just running npm install and then cypress runs the post install script, which pulls the binary and unzips it to the cache. Since the script is run from a local user file (workspace), and the cache is local to the user, I don't understand why the file permissions are having trouble, unless the way the temp dir is configured requires admin, or all symlinks require admin permissions for some reason.

Maybe I'm misunderstanding why we even need symlinks at all, but I don't really know everything about cypress is running the post install to get the binary working. It seems like something is broken on Windows if not using admin, and I think Cypress should detect this scenario and work appropriately without the user needing to change the environment/permissions.

[MikeMcC399]

@aarowman

If you run the debug commands you will see which directories are defined, including the os-specific temp directory. We would need to see the output from your system before making any generic judgements.

[aarowman]

@MikeMcC399 Yes, I'm working on that now. For some reason npm uninstall cypress tried to download the binary again, so it failed and I can't uninstall. I guess I need admin to do that too :-/

[MikeMcC399]

@aarowman

Yes, I'm working on that now.

Good! That should shed some light on the steps.

For some reason npm uninstall cypress tried to download the binary again, so it failed and I can't uninstall. I guess I need admin to do that too

I can't reproduce that behavior. You can also use the same debug parameters for uninstall i.e.

export DEBUG='cypress:*'
npm uninstall cypress --loglevel verbose --foreground-scripts

[aarowman]

@MikeMcC399 @AtofStryker Here you go... I did all of this WITHOUT admin mode.

I deleted node_modules and package-lock.json, then ran npm install for my entire project WITHOUT cypress. Everything's good.

Next, run:

export DEBUG='cypress:*'
npm install cypress@latest --loglevel verbose --foreground-scripts

Here is the output (sanitized):

npm info run cypress@12.13.0 postinstall node_modules/cypress node index.js --exec install

> cypress@12.13.0 postinstall
> node index.js --exec install

2023-06-02T22:55:12.288Z cypress:cli installing Cypress from NPM
2023-06-02T22:55:15.802Z cypress:cli installing with options {}
2023-06-02T22:55:15.815Z cypress:cli detecting arch { osPlatform: 'win32', osArch: 'x64' }
2023-06-02T22:55:15.819Z cypress:cli version in package.json is 12.13.0, version to install is 12.13.0
2023-06-02T22:55:15.853Z cypress:cli Reading binary package.json from: C:\Users\myUser\AppData\Local\Cypress\Cache\12.13.0\Cypress\resources\app\package.json
2023-06-02T22:55:15.857Z cypress:cli no binary installed under cli version
2023-06-02T22:55:15.859Z cypress:cli checking local file C:\Users\myUser\myWorkingDir\12.13.0 cwd C:\Users\myUser\myWorkingDir\node_modules\cypress
2023-06-02T22:55:15.861Z cypress:cli preparing to download and unzip version  12.13.0 to path C:\Users\myUser\AppData\Local\Cypress\Cache\12.13.0
Installing Cypress (version: 12.13.0)

[STARTED] Task without title.
2023-06-02T22:55:15.890Z cypress:cli Using CYPRESS_DOWNLOAD_MIRROR from environment variable
2023-06-02T22:55:15.891Z cypress:cli Using CYPRESS_DOWNLOAD_MIRROR from environment variable
2023-06-02T22:55:15.903Z cypress:cli needed Cypress version: 12.13.0
2023-06-02T22:55:15.903Z cypress:cli source url https://myCypressRemote.com/desktop/12.13.0?platform=win32&arch=x64
2023-06-02T22:55:15.903Z cypress:cli downloading cypress.zip to "C:\Users\myUser\AppData\Local\Temp\cypress-2276.zip"
2023-06-02T22:55:15.929Z cypress:cli Downloading package {
  url: 'https://myCypressRemote.com/desktop/12.13.0?platform=win32&arch=x64',
  proxy: 'myProxy.com',
  downloadDestination: 'C:\\Users\\myUser\\AppData\\Local\\Temp\\cypress-2276.zip'
}
2023-06-02T22:55:15.931Z cypress:cli using custom CA details from npm config
2023-06-02T22:55:17.858Z cypress:cli expected file size 168186226
[TITLE]  Downloading Cypress      0% 0s
[TITLE]  Downloading Cypress      1% 102s
[TITLE]  Downloading Cypress      1% 113s
[TITLE]  Downloading Cypress      1% 123s
[TITLE]  Downloading Cypress      1% 133s


....


[TITLE]  Downloading Cypress      97% 0s
[TITLE]  Downloading Cypress      98% 0s
[TITLE]  Downloading Cypress      99% 0s
[TITLE]  Downloading Cypress      100% 0s
2023-06-02T22:55:34.356Z cypress:cli downloading finished
2023-06-02T22:55:34.357Z cypress:cli only checking expected file size 168186226
2023-06-02T22:55:34.363Z cypress:cli downloaded file has the expected size ✅
2023-06-02T22:55:34.363Z cypress:cli verified
2023-06-02T22:55:34.364Z cypress:cli finished downloading file: C:\Users\myUser\AppData\Local\Temp\cypress-2276.zip
[TITLE]  Downloaded Cypress
[SUCCESS]  Downloaded Cypress
[STARTED] Task without title.
2023-06-02T22:55:34.378Z cypress:cli:unzip removing existing unzipped binary C:\Users\myUser\AppData\Local\Cypress\Cache\12.13.0
2023-06-02T22:55:43.575Z cypress:cli:unzip unzipping from C:\Users\myUser\AppData\Local\Temp\cypress-2276.zip
2023-06-02T22:55:43.575Z cypress:cli:unzip into C:\Users\myUser\AppData\Local\Cypress\Cache\12.13.0
2023-06-02T22:55:43.601Z cypress:cli:unzip zipFile entries count 21583
2023-06-02T22:55:43.603Z cypress:cli:unzip unzipping with node.js (slow)
2023-06-02T22:55:43.612Z cypress:cli:unzip calling Node extract tool C:\Users\myUser\AppData\Local\Temp\cypress-2276.zip { dir: 'C:\\Users\\myUser\\AppData\\Local\\Cypress\\Cache\\12.13.0', onEntry: [Function: tick] }
[TITLE]  Unzipping Cypress        0% 0s
[TITLE]  Unzipping Cypress        1% 429s


....


[TITLE]  Unzipping Cypress        7% 875s
[TITLE]  Unzipping Cypress        7% 876s
[TITLE]  Unzipping Cypress        7% 877s
2023-06-02T22:56:49.695Z cypress:cli:unzip error EPERM: operation not permitted, symlink '..\..\..\update-browserslist-db\cli.js' -> 'C:\Users\myUser\AppData\Local\Cypress\Cache\12.13.0\Cypress.app\Contents\Resources\app\node_modules\browserslist\node_modules\.bin\browserslist-lint'
[FAILED] The Cypress App could not be unzipped.
[FAILED]
[FAILED] Search for an existing issue or open a GitHub issue at
[FAILED]
[FAILED] https://github.com/cypress-io/cypress/issues
[FAILED]
[FAILED] ----------
[FAILED]
[FAILED] Error: EPERM: operation not permitted, symlink '..\..\..\update-browserslist-db\cli.js' -> 'C:\Users\myUser\AppData\Local\Cypress\Cache\12.13.0\Cypress.app\Contents\Resources\app\node_modules\browserslist\node_modules\.bin\browserslist-lint'
[FAILED]
[FAILED] ----------
[FAILED]
[FAILED] Platform: win32-x64 (10.0.18363)
[FAILED] Cypress Version: 12.13.0
The Cypress App could not be unzipped.

Search for an existing issue or open a GitHub issue at

https://github.com/cypress-io/cypress/issues

----------

Error: EPERM: operation not permitted, symlink '..\..\..\update-browserslist-db\cli.js' -> 'C:\Users\myUser\AppData\Local\Cypress\Cache\12.13.0\Cypress.app\Contents\Resources\app\node_modules\browserslist\node_modules\.bin\browserslist-lint'

----------

Platform: win32-x64 (10.0.18363)
Cypress Version: 12.13.0
npm info run cypress@12.13.0 postinstall { code: 1, signal: null }
npm WARN cleanup Failed to remove some directories [
npm WARN cleanup   [
npm WARN cleanup     'C:\\Users\\myUser\\myWorkingDir\\node_modules\\@cypress\\xvfb',
npm WARN cleanup     [Error: EPERM: operation not permitted, rmdir 'C:\Users\myUser\myWorkingDir\node_modules\@cypress\xvfb\node_modules'] {
npm WARN cleanup       errno: -4048,
npm WARN cleanup       code: 'EPERM',
npm WARN cleanup       syscall: 'rmdir',
npm WARN cleanup       path: 'C:\\Users\\myUser\\myWorkingDir\\node_modules\\@cypress\\xvfb\\node_modules'
npm WARN cleanup     }
npm WARN cleanup   ],
npm WARN cleanup   [
npm WARN cleanup     'C:\\Users\\myUser\\myWorkingDir\\node_modules\\cypress\\node_modules',
npm WARN cleanup     [Error: EPERM: operation not permitted, rmdir 'C:\Users\myUser\myWorkingDir\node_modules\cypress\node_modules\@types\node'] {
npm WARN cleanup       errno: -4048,
npm WARN cleanup       code: 'EPERM',
npm WARN cleanup       syscall: 'rmdir',
npm WARN cleanup       path: 'C:\\Users\\myUser\\myWorkingDir\\node_modules\\cypress\\node_modules\\@types\\node'
npm WARN cleanup     }
npm WARN cleanup   ],
npm WARN cleanup   [
npm WARN cleanup     'C:\\Users\\myUser\\myWorkingDir\\node_modules\\cypress',
npm WARN cleanup     [Error: EPERM: operation not permitted, rmdir 'C:\Users\myUser\myWorkingDir\node_modules\cypress\node_modules\@types\node'] {
npm WARN cleanup       errno: -4048,
npm WARN cleanup       code: 'EPERM',
npm WARN cleanup       syscall: 'rmdir',
npm WARN cleanup       path: 'C:\\Users\\myUser\\myWorkingDir\\node_modules\\cypress\\node_modules\\@types\\node'
npm WARN cleanup     }
npm WARN cleanup   ]
npm WARN cleanup ]
npm verb stack Error: command failed
npm verb stack     at ChildProcess.<anonymous> (C:\Program Files\nodejs\node_modules\npm\node_modules\@npmcli\promise-spawn\lib\index.js:53:27)
npm verb stack     at ChildProcess.emit (node:events:513:28)
npm verb stack     at maybeClose (node:internal/child_process:1091:16)
npm verb stack     at ChildProcess._handle.onexit (node:internal/child_process:302:5)
npm verb pkgid cypress@12.13.0
npm verb cwd C:\Users\myUser\myWorkingDir
npm verb Windows_NT 10.0.18363
npm verb node v18.15.0
npm verb npm  v9.5.0
npm ERR! code 1
npm ERR! path C:\Users\myUser\myWorkingDir\node_modules\cypress
npm ERR! command failed
npm ERR! command C:\Windows\system32\cmd.exe /d /s /c node index.js --exec install
npm verb exit 1
npm verb unfinished npm timer reify 1685746443642
npm verb unfinished npm timer reify:build 1685746504505
npm verb unfinished npm timer build 1685746504508
npm verb unfinished npm timer build:deps 1685746504510
npm verb unfinished npm timer build:run:postinstall 1685746504784
npm verb unfinished npm timer build:run:postinstall:node_modules/cypress 1685746504786
npm verb code 1

npm ERR! A complete log of this run can be found in:

Let me know if any of this is helpful. Even after the unzip error, it couldn't delete some of the node dependencies for cleanup, but that might be unrelated.

I wonder if it's related to the npm unzip tool having issues. I noted these lines:

2023-06-02T22:55:43.603Z cypress:cli:unzip unzipping with node.js (slow)
2023-06-02T22:55:43.612Z cypress:cli:unzip calling Node extract tool

Or maybe the symlink command is having trouble? I'm not sure how it tries to create that on Windows. Does it use a symlink cli ? If so, what if that cli doesn't exist? I'm not sure why admin mode would work though..

[MikeMcC399]

@aarowman

Let me know if any of this is helpful.

Yes, definitely! Thanks for the logs! It confirms that unzip is failing on your system trying to create a symlink.

Even after the unzip error, it couldn't delete some of the node dependencies for cleanup, but that might be unrelated.

I would concentrate first on the symlink issue, but you might want to try to do a manual clean up and see what happens. The log doesn't say why the rmdir operations were not permitted.

I wonder if it's related to the npm unzip tool having issues. I noted these lines:
2023-06-02T22:55:43.603Z cypress:cli:unzip unzipping with node.js (slow)
2023-06-02T22:55:43.612Z cypress:cli:unzip calling Node extract tool
Or maybe the symlink command is having trouble? I'm not sure how it tries to create that on Windows. Does it use a symlink cli ? If so, what if that cli doesn't exist? I'm not sure why admin mode would work though..

It is using the npm module extract-zip. A non-admin cannot normally create symlinks on Windows 10. There is a developer mode to allow this, however I don't think that symlinks should be created, so it shouldn't be necessary to enable this mode.

The problem is not reproducible for me on a standard Windows 10 system. I tested installing Cypress 12.13.0 on Windows 10 Version 22H2 (OS Build 19045.2965), node 18.16.0, npm 9.5.1, local account, Account type: Standard User, no special adjustments to any permissions / rights.

Cypress/resources/app/node_modules/update-browserslist-db/cli.js is simply copied to
C:\Users\{Username}\AppData\Local\Cypress\Cache\12.13.0\Cypress\resources\app\node_modules\update-browserslist-db\cli.js without any symlink being created.

zipinfo cypress.zip **/update-browserslist-db/* executed in a bash terminal on the win32 / x64 zip file also doesn't look like there is a symlink involved.

So it is quite suspicious that in your system it is trying to create a symlink (and then failing).

My log file showed:

cypress:cli expected checksum d9904bd2ef12894bd0584aa533a40a0f1ddb9e65ad6bf3c7d3d17486252357a179d0339d3fe4e274f38bab94a1217180a420a42e87ecd3d6183507ae171eba86
cypress:cli expected file size 181048759

This is different to your log which has a different file size and says:

2023-06-02T22:55:15.890Z cypress:cli Using CYPRESS_DOWNLOAD_MIRROR from environment variable
2023-06-02T22:55:15.891Z cypress:cli Using CYPRESS_DOWNLOAD_MIRROR from environment variable
2023-06-02T22:55:15.903Z cypress:cli needed Cypress version: 12.13.0
2023-06-02T22:55:15.903Z cypress:cli source url https://myCypressRemote.com/desktop/12.13.0?platform=win32&arch=x64
2023-06-02T22:55:15.903Z cypress:cli downloading cypress.zip to "C:\Users\myUser\AppData\Local\Temp\cypress-2276.zip"
2023-06-02T22:55:15.929Z cypress:cli Downloading package {
  url: 'https://myCypressRemote.com/desktop/12.13.0?platform=win32&arch=x64',
  proxy: 'myProxy.com',
  downloadDestination: 'C:\\Users\\myUser\\AppData\\Local\\Temp\\cypress-2276.zip'
}
2023-06-02T22:55:15.931Z cypress:cli using custom CA details from npm config
2023-06-02T22:55:17.858Z cypress:cli expected file size 168186226

I'm wondering if your copy of Cypress 12.13.0 is corrupt or something has gone wrong with the download mirroring?

You might want to try installing a fresh copy of Cypress 12.13.0 or try the previous version Cypress 12.12.0. You could also perhaps try on a different Windows 10 system.

If you want to get an even more detailed log you can add DEBUG=extract-zip, for instance:

export DEBUG='cypress:*,extract-zip'
npm install cypress@latest --loglevel verbose --foreground-scripts

[aarowman]

It is using the npm module extract-zip. A non-admin cannot normally create symlinks on Windows 10. There is a developer mode to allow this, however I don't think that symlinks should be created, so it shouldn't be necessary to enable this mode.

If that's the case, how do we find why this is even happening? To me, that's the root of the issue - something is happening that shouldn't be needed, and it's causing issues on Windows (at least this version) without admin. I can try the extract-zip debug to see if I get more info.

I'm wondering if your copy of Cypress 12.13.0 is corrupt or something has gone wrong with the download mirroring?

I do not believe it is an issue with my copy of the file or a download mirror issue, because when running with admin, it works fine. If the file or mirror were an issue, it would still fail with admin.

If you want to get an even more detailed log you can add DEBUG=extract-zip

I'll give this a shot to get more info.

[aarowman]

Here you go:

}
2023-06-06T16:07:34.733Z extract-zip finished processing Cypress.app/Contents/Resources/app/node_modules/lodash.flatten/
2023-06-06T16:07:34.734Z extract-zip zipfile entry Cypress.app/Contents/Resources/app/node_modules/lodash.flatten/LICENSE
2023-06-06T16:07:34.745Z extract-zip extracting entry {
  filename: 'Cypress.app/Contents/Resources/app/node_modules/lodash.flatten/LICENSE',
  isDir: false,
  isSymlink: false
}
2023-06-06T16:07:34.746Z extract-zip mkdir {
  dir: 'C:\\Users\\myUser\\AppData\\Local\\Cypress\\Cache\\12.13.0\\Cypress.app\\Contents\\Resources\\app\\node_modules\\lodash.flatten',
  recursive: true
}
2023-06-06T16:07:34.755Z extract-zip opening read stream C:\Users\myUser\AppData\Local\Cypress\Cache\12.13.0\Cypress.app\Contents\Resources\app\node_modules\lodash.flatten\LICENSE
2023-06-06T16:07:34.771Z extract-zip finished processing Cypress.app/Contents/Resources/app/node_modules/lodash.flatten/LICENSE
2023-06-06T16:07:34.772Z extract-zip zipfile entry Cypress.app/Contents/Resources/app/node_modules/lodash.flatten/package.json
[TITLE]  Unzipping Cypress        7% 630s
2023-06-06T16:07:34.784Z extract-zip extracting entry {
  filename: 'Cypress.app/Contents/Resources/app/node_modules/lodash.flatten/package.json',
  isDir: false,
  isSymlink: false
}
2023-06-06T16:07:34.784Z extract-zip mkdir {
  dir: 'C:\\Users\\myUser\\AppData\\Local\\Cypress\\Cache\\12.13.0\\Cypress.app\\Contents\\Resources\\app\\node_modules\\lodash.flatten',
  recursive: true
}
2023-06-06T16:07:34.796Z extract-zip opening read stream C:\Users\myUser\AppData\Local\Cypress\Cache\12.13.0\Cypress.app\Contents\Resources\app\node_modules\lodash.flatten\package.json
2023-06-06T16:07:34.811Z extract-zip finished processing Cypress.app/Contents/Resources/app/node_modules/lodash.flatten/package.json
2023-06-06T16:07:34.812Z extract-zip zipfile entry Cypress.app/Contents/Resources/app/node_modules/browserslist/
2023-06-06T16:07:34.824Z extract-zip extracting entry {
  filename: 'Cypress.app/Contents/Resources/app/node_modules/browserslist/',
  isDir: true,
  isSymlink: false
}
2023-06-06T16:07:34.825Z extract-zip mkdir {
  dir: 'C:\\Users\\myUser\\AppData\\Local\\Cypress\\Cache\\12.13.0\\Cypress.app\\Contents\\Resources\\app\\node_modules\\browserslist\\',
  recursive: true,
  mode: 493
}
2023-06-06T16:07:34.837Z extract-zip finished processing Cypress.app/Contents/Resources/app/node_modules/browserslist/
2023-06-06T16:07:34.838Z extract-zip zipfile entry Cypress.app/Contents/Resources/app/node_modules/browserslist/LICENSE
2023-06-06T16:07:34.850Z extract-zip extracting entry {
  filename: 'Cypress.app/Contents/Resources/app/node_modules/browserslist/LICENSE',
  isDir: false,
  isSymlink: false
}
2023-06-06T16:07:34.851Z extract-zip mkdir {
  dir: 'C:\\Users\\myUser\\AppData\\Local\\Cypress\\Cache\\12.13.0\\Cypress.app\\Contents\\Resources\\app\\node_modules\\browserslist',
  recursive: true
}
2023-06-06T16:07:34.862Z extract-zip opening read stream C:\Users\myUser\AppData\Local\Cypress\Cache\12.13.0\Cypress.app\Contents\Resources\app\node_modules\browserslist\LICENSE
2023-06-06T16:07:34.884Z extract-zip finished processing Cypress.app/Contents/Resources/app/node_modules/browserslist/LICENSE
2023-06-06T16:07:34.884Z extract-zip zipfile entry Cypress.app/Contents/Resources/app/node_modules/browserslist/node_modules/
[TITLE]  Unzipping Cypress        7% 631s
2023-06-06T16:07:34.899Z extract-zip extracting entry {
  filename: 'Cypress.app/Contents/Resources/app/node_modules/browserslist/node_modules/',
  isDir: true,
  isSymlink: false
}
2023-06-06T16:07:34.899Z extract-zip mkdir {
  dir: 'C:\\Users\\myUser\\AppData\\Local\\Cypress\\Cache\\12.13.0\\Cypress.app\\Contents\\Resources\\app\\node_modules\\browserslist\\node_modules\\',
  recursive: true,
  mode: 493
}
2023-06-06T16:07:34.911Z extract-zip finished processing Cypress.app/Contents/Resources/app/node_modules/browserslist/node_modules/
2023-06-06T16:07:34.912Z extract-zip zipfile entry Cypress.app/Contents/Resources/app/node_modules/browserslist/node_modules/.bin/
2023-06-06T16:07:34.924Z extract-zip extracting entry {
  filename: 'Cypress.app/Contents/Resources/app/node_modules/browserslist/node_modules/.bin/',
  isDir: true,
  isSymlink: false
}
2023-06-06T16:07:34.925Z extract-zip mkdir {
  dir: 'C:\\Users\\myUser\\AppData\\Local\\Cypress\\Cache\\12.13.0\\Cypress.app\\Contents\\Resources\\app\\node_modules\\browserslist\\node_modules\\.bin\\',
  recursive: true,
  mode: 493
}
2023-06-06T16:07:34.937Z extract-zip finished processing Cypress.app/Contents/Resources/app/node_modules/browserslist/node_modules/.bin/
2023-06-06T16:07:34.937Z extract-zip zipfile entry Cypress.app/Contents/Resources/app/node_modules/browserslist/node_modules/.bin/browserslist-lint
[TITLE]  Unzipping Cypress        7% 632s
2023-06-06T16:07:34.951Z extract-zip extracting entry {
  filename: 'Cypress.app/Contents/Resources/app/node_modules/browserslist/node_modules/.bin/browserslist-lint',
  isDir: false,
  isSymlink: true
}
2023-06-06T16:07:34.951Z extract-zip mkdir {
  dir: 'C:\\Users\\myUser\\AppData\\Local\\Cypress\\Cache\\12.13.0\\Cypress.app\\Contents\\Resources\\app\\node_modules\\browserslist\\node_modules\\.bin',
  recursive: true
}
2023-06-06T16:07:34.963Z extract-zip opening read stream C:\Users\myUser\AppData\Local\Cypress\Cache\12.13.0\Cypress.app\Contents\Resources\app\node_modules\browserslist\node_modules\.bin\browserslist-lint
2023-06-06T16:07:34.976Z extract-zip creating symlink ../../../update-browserslist-db/cli.js C:\Users\myUser\AppData\Local\Cypress\Cache\12.13.0\Cypress.app\Contents\Resources\app\node_modules\browserslist\node_modules\.bin\browserslist-lint
[FAILED] The Cypress App could not be unzipped.
[FAILED]
[FAILED] Search for an existing issue or open a GitHub issue at
[FAILED]
[FAILED] https://github.com/cypress-io/cypress/issues
[FAILED]
[FAILED] ----------
[FAILED]
[FAILED] Error: EPERM: operation not permitted, symlink '..\..\..\update-browserslist-db\cli.js' -> 'C:\Users\myUser\AppData\Local\Cypress\Cache\12.13.0\Cypress.app\Contents\Resources\app\node_modules\browserslist\node_modules\.bin\browserslist-lint'
[FAILED]
[FAILED] ----------
[FAILED]
[FAILED] Platform: win32-x64 (10.0.18363)
[FAILED] Cypress Version: 12.13.0
The Cypress App could not be unzipped.

Search for an existing issue or open a GitHub issue at

https://github.com/cypress-io/cypress/issues

----------

[aarowman]

Not much more info, just shows that extract-zip does in fact try to create a symlink for that specific .js file, for some reason. Does the Cypress binary have that file as a symlink in the zip? Then extract-zip is just trying to recreated it in the same format?

[MikeMcC399]

@aarowman

In my detailed log there are no instances of
isSymlink: true

My copy of Cypress is coming directly from https://download.cypress.io/desktop/12.13.0?platform=win32&arch=x64 which differs from your setup.

Unfortunately I don't think I can contribute any more to this issue since I am unable to reproduce it or to explain it.

You may want to give consideration to also trying out bypassing your mirror.

[aarowman]

@MikeMcC399 Did you look specifically at this file: Cypress.app/Contents/Resources/app/node_modules/browserslist/node_modules/.bin/browserslist-lint and see isSymlink: false on it?

I'm not sure how the mirror would be affecting the nature of the symlinks. Especially since the mirror does not edit the zip in transit, I don't see any reason that those are connected.

Also, please make sure you are NOT running as admin when you test this, since running as admin seems to allow permissions to work.

[MikeMcC399]

2023-06-03T14:03:21.497Z extract-zip finished processing Cypress/resources/app/node_modules/browserslist/node_modules/.bin/
2023-06-03T14:03:21.497Z extract-zip zipfile entry Cypress/resources/app/node_modules/browserslist/node_modules/.bin/browserslist-lint
2023-06-03T14:03:21.497Z extract-zip extracting entry {
  filename: 'Cypress/resources/app/node_modules/browserslist/node_modules/.bin/browserslist-lint',
  isDir: false,
  isSymlink: false

[aarowman]

I wonder if the zip we are downloading is different based on different Windows systems or platforms, since Cypress detects that when downloading. It's interesting why mine is being set as a symlink but yours isn't.

Note: as shown in the first logs above, the download it is using is /desktop/12.13.0?platform=win32&arch=x64 . Is yours the same?

[aarowman]

I also noticed the file path is different - yours is Cypress/resources/app/..., while mine is Cypress.app/Contents/Resources/app/... - I never configured any of that explicitly, so I'd expect it would use the Cypress defaults for Win10. What is setting these file paths?

I'm also curious what defines what is a symlink in the zip itself? I assume there is some build process on the Cypress side to create the binary zip file.

[aarowman]

Quick note: I tried v 12.12.0 and got the exact same result on the same file inside the binary. Our mirror creates a new direct copy from the source (after scanning) and we didn't have 12.12.0 cached before now (last download was 12.7 before we tried 12.13), so I know it pulled a new one. It's definitely something unique about the source zip and symlink settings for this platform.

Of course the workaround of running with admin will work, but I don't see why there's a need for symlinks on this zip file at all, or why the Windows install tries to reproduce them. Is there a way to disable that? Or force the zip to have NO symlinks for Windows? This seems like a setup issue for the packaging - the way the zip is created and/or unzipped.

(I'm also surprised we need a node_modules folder inside the zip anyway. We already have the cypress dependency inside the project, which can have any extra runtime files we need. I guess the cypress.exe needs all those extra files? Seems like a lot...)

[aarowman]

Update: It looks like running with admin doesn't actually solve the problem. It just made npm install complete without errors.
I opened a new terminal (Git bash) and tried to run cypress verify and it said the binary wasn't even there. I tried cypress install after that and it tried to download and failed again (same reason). I tried cypress install on an admin terminal, and it seemed to download/unzip this time.. but cypress verify and cypress info and cypress open all fail still.

Without trying to share all the logs here in this thread since those are likely different issues, I'd like to circle back that the "workaround" to use admin for the install may not actually be sufficient (it should also not be preferred). The binary download and unzip process is having trouble and I'm not sure where to go from here

Edit:
This is the error from cypress info:
Error: spawn C:\Users\myUser\AppData\Local\Cypress\Cache\12.13.0\Cypress\Cypress.exe ENOENT

It makes sense, because that file doesn't exist. I'm not sure why the download says it succeeds, but that doesn't exist. I only have C:\Users\myUser\AppData\Local\Cypress\Cache\12.13.0\Cypress.app, which only has \Contents. There's no .exe in there, and I don't know why it has Cypress.app as the folder instead of Cypress

[aarowman]

Is that the folder structure for a non-Windows download? Is it possible that our mirror is somehow grabbing the file for a different OS? That could explain a lot of this. I'm not sure how the mirror/cache works, but that could be getting in the way - not a corrupt version of the file, just the WRONG version of the file. I'm still not sure why this would happen because the endpoint is /desktop/12.13.0?platform=win32&arch=x64 which shouldn't have any other OS structure. That is, unless the ?x=y params are being ignored somehow.

This could be an issue with mirroring and/or the Cypress download site logic. The error message itself still aligns to the subject of this issue. I can see if the issue persists from the direct link. That will not be an acceptable workaround for my organization, but it could at least give answers.

Edit:
I did not receive issues when trying the direct link (not admin). The problem is definitely because the download from the mirror is getting the wrong zip file.
It looks like it is defaulting to MacOS in my situation, even though I'm on Windows. Not sure why that's happening

[MikeMcC399]

@aarowman

It looks like it is defaulting to MacOS in my situation, even though I'm on Windows. Not sure why that's happening

That could explain the different file sizes, different file structure and symlink differences.

The documentation doesn't describe how to set up an exact 1:1 mirror copy of https://download.cypress.io including the ability to process the query parameters ?platform=win32&arch=x64.

I suggest that you refer to the Cypress documentation Advanced Installation > Download path template to ensure that your system is pulling the Cypress binary for the correct architecture. You may need to be using CYPRESS_DOWNLOAD_PATH_TEMPLATE.

[aarowman]

@MikeMcC399 The final url it's using is still myMirror.com/desktop/12.13.0?platform=win32&arch=x64

Here is the line from the log:

2023-06-07T14:09:01.221Z cypress:cli Downloading package {
  url: 'myMirror.com/desktop/12.13.0?platform=win32&arch=x64',
  proxy: 'http://myProxy.com:9999',
  downloadDestination: 'C:\\Users\\myUser\\AppData\\Local\\Temp\\cypress-18636.zip'
}

So I don't think the url on my client end is the problem. I have no idea why the mirror is getting the MacOS values. It's possible the mirror doesn't forward the query parameters as expected. Our mirror is set up to point to https://download.cypress.io/ and there aren't any custom changes or download behavior, so I'm not sure the Download path template will change anything, because the url is being constructed as we think it should be.

Is there more documentation on how to set up the mirror? Maybe that's why there's an issue.
Also, can you confirm if MacOS is the default value, if the query params are missing?

[MikeMcC399]

@aarowman

I don't have any experience in setting up a download mirror, so I have to defer to the Cypress.io team for questions about how this is supposed to work and I don't have any private information about this. I would however assume that a mirror would be a COPY of the Cypress download site and not simply a proxy which forwards requests.

If you install without using a mirror and you are on a Windows machine, then accessing https://download.cypress.io/ will (at the time of writing) redirect the request to https://cdn.cypress.io/desktop/12.13.0/win32-x64/cypress.zip.

Probably your mirror site will not work that way and that would be the reason for the existence of the CYPRESS_DOWNLOAD_PATH_TEMPLATE parameter to ensure that a mirror copy site will provide the needed cypress.zip version. The zip files all have the same name, they are just located in different directories on the cdn server.

Are you bound to using a mirror server or do you have the option to install directly from Cypress?

[aarowman]

We need to use a mirror for security and network access reasons.

I believe the mirror acts as both a proxy and it caches the downloads so it saves the copies. It does not copy everything right up front (even if it did, that wouldn't work forever as new versions come out). If something does not exist in the mirror cache, it pulls from the source (like a proxy), scans it, then makes it available.

If download.cypress.io does a redirect to cdn.cypress.io, maybe we should point our mirror directly to cdn.cypress.io and then we can use the rest of the download path template so we can explicitly define the params. That may unfortunately be required for a mirror setup change.

I would expect that the mirror could point to download.cypress.io and the downloads would still work fine, but if the query params are being ignored, then using the full path at cdn.cypress.io may be safer. :-/

[aarowman]

The documentation says "If you choose to mirror the entire Cypress download site," when referencing download.cypress.io, so it seems that when attempting to do download.cypress.io, we would need to mirror everything, including the redirect behavior. I'm not sure if the mirror was set up that way (another team handled that).

The Advanced Installation > Download path template link you mentioned indicates that cdn.cypress.io will work a little more easily per this part:

To install the binary from a download mirror that matches the exact file structure of https://cdn.cypress.io (works for Cypress 9.3.0 or newer):

export CYPRESS_DOWNLOAD_MIRROR=https://cypress-download.local
export CYPRESS_DOWNLOAD_PATH_TEMPLATE='${endpoint}/${platform}-${arch}/cypress.zip'
# Example of a resulting URL: https://cypress-download.local/desktop/10.11.0/linux-x64/cypress.zip

Ok, I will look into requesting our mirror be updated to point to cdn.cypress.io instead (or create a new mirror for that)

Second question - I also see in the docs that it talks about

To define CYPRESS_DOWNLOAD_PATH_TEMPLATE in .npmrc, ...

It looks like we don't have to put it as a full environment variable. Will this work for the CYPRESS_DOWNLOAD_MIRROR as well? Since .npmrc values are not direct environment variables (npm will set them with npm_config_* instead), does Cypress honor both ways to set the variables?

[MikeMcC399]

https://download.cypress.io/desktop.json shows the directory structure for the current version:

{
    "name": "Cypress",
    "version": "12.13.0",
    "packages": {
        "mac": {
            "url": "https://cdn.cypress.io/desktop/12.13.0/darwin-x64/cypress.zip"
        },
        "linux64": {
            "url": "https://cdn.cypress.io/desktop/12.13.0/linux-x64/cypress.zip"
        },
        "darwin": {
            "url": "https://cdn.cypress.io/desktop/12.13.0/darwin-x64/cypress.zip"
        },
        "linux": {
            "url": "https://cdn.cypress.io/desktop/12.13.0/linux-x64/cypress.zip"
        },
        "darwin-x64": {
            "url": "https://cdn.cypress.io/desktop/12.13.0/darwin-x64/cypress.zip"
        },
        "darwin-arm64": {
            "url": "https://cdn.cypress.io/desktop/12.13.0/darwin-arm64/cypress.zip"
        },
        "linux-x64": {
            "url": "https://cdn.cypress.io/desktop/12.13.0/linux-x64/cypress.zip"
        },
        "linux-arm64": {
            "url": "https://cdn.cypress.io/desktop/12.13.0/linux-arm64/cypress.zip"
        },
        "win32-x64": {
            "url": "https://cdn.cypress.io/desktop/12.13.0/win32-x64/cypress.zip"
        }
    }
}

[MikeMcC399]

@aarowman

Have you been able to test installing and using the Windows version of Cypress in your non-admin account?

[aarowman]

@MikeMcC399 as mentioned above,

Edit:
I did not receive issues when trying the direct link (not admin). The problem is definitely because the download from the mirror is getting the wrong zip file.

We have not yet set up a new mirror to point to CDN (it may take a few days/weeks for the team to address this) to verify this works through a mirror.

I think we found the problem, though, which is not a fundamental issue in the Cypress zip or download, but in the way mirrors are set up.
I think this issue 8471 is not the cause and can likely be closed.

I would, however, suggest that we update or create better documentation around how to set up a mirror and what options there are (mirroring the entire download site - how would that work with the params, vs mirroring CDN only, requiring installers to use path template, vs custom file path setup). The current docs talk about usage on the project side, but not the mirror setup itself.

[MikeMcC399]

@aarowman

I think this issue 8471 is not the cause and can likely be closed.

Good suggestion!

I would, however, suggest that we update or create better documentation around how to set up a mirror and what options there are (mirroring the entire download site - how would that work with the params, vs mirroring CDN only, requiring installers to use path template, vs custom file path setup). The current docs talk about usage on the project side, but not the mirror setup itself.

My suggestion would be for you to open a new issue on https://github.com/cypress-io/cypress-documentation for this. Maybe Cypress.io could also supply or recommend some utility software to assist in consistently setting up and operating a mirror copy of https://download.cypress.io?

[aarowman]

@MikeMcC399 Yes, I can do that. Note: I wouldn't be the best 'user' of this since I'm not the person that actually sets up the mirror. I just think it'd be a good idea on the Cypress side to have better documentation in that area

Edit:
Created new issue cypress-io/cypress-documentation#5290

[mschile]

I'm going to go ahead and close this issue. If you're experiencing a bug similar to this in Cypress, please open a new issue with a fully reproducible example that we can run. There may be a specific edge case with the issue that we need more detail to fix.