Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dependabot is giving a 401 error trying to generate a security update PR from a private scope #5885

Closed
1 task done
scottdickerson opened this issue Oct 13, 2022 · 13 comments
Labels
L: javascript:yarn npm packages via yarn T: bug 🐞 Something isn't working

Comments

@scottdickerson
Copy link

Is there an existing issue for this?

  • I have searched the existing issues

Package ecosystem

yarn

Package manager version

v3

Language version

No response

Manifest location and content before the Dependabot update

/package.json

dependabot.yml content

No response

Updated dependency

No response

What you expected to see, versus what you actually saw

Dependabot is not able to generate a security fix PR because it gets a 401 unauthorized trying to pull dependent packages from our organization scoped private registry.

I see an obvious error about an environment variable missing in the logs below.

This is correct to access and install from my private repo I do need a GH_API_ACCESS_TOKEN environment variable to be set. I'm using it in both my .yarnrc.yml file

npmScopes:
  contrast-security-inc:
    npmAuthToken: '${GH_API_ACCESS_TOKEN}'
    npmPublishRegistry: 'https://npm.pkg.github.com/'
    npmRegistryServer: 'https://npm.pkg.github.com/'

and my .npmrc file:

always-auth = true
@contrast-security-inc:registry=https://npm.pkg.github.com/
//npm.pkg.github.com/:_authToken=${GH_API_ACCESS_TOKEN}

I do actually have a secret set for dependabot here
image

But somehow I need to pass this to dependabot as an environment variable from my secrets.
#4660

Native package manager behavior

yarn install works correctly for us.

Images of the diff or a link to the PR, issue, or logs

proxy | 2022/10/13 18:29:49 [004] GET https://npm.pkg.github.com:443/@contrast-security-inc%2fdesign-system-badges
  proxy | 2022/10/13 18:29:49 [004] 401 https://npm.pkg.github.com:443/@contrast-security-inc%2fdesign-system-badges
updater | INFO <job_483339586> The latest possible version of @xmldom/xmldom that can be installed is 0.7.5
updater | INFO <job_483339586> The earliest fixed version is 0.8.3.
updater | INFO <job_483339586> Finished job processing
updater | INFO Results:
updater | Dependabot encountered '1' error(s) during execution, please check the logs for more details.
updater | time="2022-10-13T18:29:50Z" level=info msg="task complete" container_id=job-483339586-updater exit_code=0 job_id=483339586 step=updater

https://github.com/Contrast-Security-Inc/skeletor/security/dependabot/19/update-logs/275319909

I see a really good error here
https://github.com/Contrast-Security-Inc/skeletor/network/updates/483358113

updater | ERROR <job_483358113> Usage Error: Environment variable not found (GH_API_ACCESS_TOKEN) in /home/dependabot/dependabot-updater/repo/.yarnrc.yml (in /home/dependabot/dependabot-updater/repo/.yarnrc.yml)

Smallest manifest that reproduces the issue

No response

@scottdickerson scottdickerson added the T: bug 🐞 Something isn't working label Oct 13, 2022
@scottdickerson
Copy link
Author

@jurre I'm not sure if you had enabled the newer version of dependabot in this repo that might be causing this issue:
https://github.com/Contrast-Security-Inc/skeletor

@jurre
Copy link
Member

jurre commented Oct 14, 2022

@scottdickerson yeah it is indeed enabled for that repo, it looks like there's two things you'll need to do to get Dependabot to run smoothly on this repo:

  1. Configure Dependabot so that it knows about this private registry, follow these docs
  2. Provide fallback variables in your yarnrc.yaml1 for the npmAuthToken: '${GH_API_ACCESS_TOKEN}' value:
npmAuthToken: '${GH_API_ACCESS_TOKEN}:-bypass'

For context, Dependabot does not have direct access to these credentials, instead they are injected by a job-specific proxy that these requests are routed through. We'll publish some documentation about the yarn environment variables, because it's not immediately clear that this is required, but hopefully this should get you going for now.

Footnotes

  1. https://yarnpkg.com/configuration/yarnrc

@scottdickerson
Copy link
Author

scottdickerson commented Oct 14, 2022

@jurre I THINK I set the configuration as you're suggesting but now it's failing earlier for me trying to fetch the public registry packages:
https://github.com/Contrast-Security-Inc/skeletor/security/dependabot/19/update-logs/275373157

proxy | 2022/10/14 14:31:36 [024] GET https://npm.pkg.github.com:443/ansi-regex
  proxy | 2022/10/14 14:31:36 [024] * authenticating npm registry request (host: npm.pkg.github.com, token auth)
  proxy | 2022/10/14 14:31:36 [024] 404 https://npm.pkg.github.com:443/ansi-regex

That package should be in the main registry not in the github.com npm registry
Here is my yarnrc.yml

nodeLinker: node-modules

npmScopes:
  contrast-security-inc:
    npmAuthToken: '${GH_API_ACCESS_TOKEN}:-bypass'
    npmPublishRegistry: 'https://npm.pkg.github.com/'
    npmRegistryServer: 'https://npm.pkg.github.com/'

plugins:
  - path: .yarn/plugins/@yarnpkg/plugin-workspace-tools.cjs
    spec: '@yarnpkg/plugin-workspace-tools'

yarnPath: .yarn/releases/yarn-3.2.3.cjs

And here is my dependabot.yml

# Basic dependabot.yml file with
# minimum configuration for two package managers

version: 2
registries:
  npm-github:
    type: npm-registry
    url: https://npm.pkg.github.com
    token: ${{secrets.GH_API_ACCESS_TOKEN}}
updates:
  # Enable version updates for npm
  - package-ecosystem: "npm"
    # Look for `package.json` and `lock` files in the `root` directory
    directory: "/"
    registries: "*"
    # Check the npm registry for updates every day (weekdays)
    schedule:
      interval: "daily"

It seems like I need to somehow tell dependabot to use the npm-github registry only for the @contrast-security-inc scoped packages, but the npm-npmjs for everything else?

@scottdickerson
Copy link
Author

one interesting thing in the logs, is it seems smart enough to check the public and private registries for the dependency with security vulnerabilities @xmldom/xmldom. But not smart enough to check both registries for this dependency? ansi-regex

proxy | 2022/10/14 14:43:39 [016] 404 https://npm.pkg.github.com:443/@xmldom%2Fxmldom
  proxy | 2022/10/14 14:43:39 [018] GET https://registry.npmjs.org:443/@xmldom%2Fxmldom
  proxy | 2022/10/14 14:43:39 [018] 200 https://registry.npmjs.org:443/@xmldom%2Fxmldom
  proxy | 2022/10/14 14:43:39 [020] GET https://registry.npmjs.org:443/@xmldom%2Fxmldom/0.8.3
  proxy | 2022/10/14 14:43:39 [020] 200 https://registry.npmjs.org:443/@xmldom%2Fxmldom/0.8.3
updater | INFO <job_483994722> Latest version is 0.8.3
  proxy | 2022/10/14 14:43:41 [022] GET https://npm.pkg.github.com:443/ansi-regex
  proxy | 2022/10/14 14:43:41 [022] * authenticating npm registry request (host: npm.pkg.github.com, token auth)
  proxy | 2022/10/14 14:43:41 [022] 404 https://npm.pkg.github.com:443/ansi-regex
updater | INFO VulnerabilityAuditor: starting audit
updater | INFO VulnerabilityAuditor: missing lockfile
updater | INFO <job_483994722> Requirements to unlock update_not_possible
updater | INFO <job_483994722> Requirements update strategy widen_ranges
  proxy | 2022/10/14 14:43:42 [024] GET https://npm.pkg.github.com:443/ansi-regex
  proxy | 2022/10/14 14:43:42 [024] * authenticating npm registry request (host: npm.pkg.github.com, token auth)
  proxy | 2022/10/14 14:43:42 [024] 404 https://npm.pkg.github.com:443/ansi-regex
updater | INFO <job_483994722> The latest possible version of @xmldom/xmldom that can be installed is 0.7.5
updater | INFO <job_483994722> The earliest fixed version is 0.8.3.
updater | INFO <job_4839947

@scottdickerson
Copy link
Author

I also had to remove the :-bypass option above from the .yarnrc.yaml, because it was breaking my yarn install entirely.

https://github.com/Contrast-Security-Inc/skeletor/actions/runs/3251066736/jobs/5335573631

Fetch step
  ➤ YN0041: │ @contrast-security-inc/eslint-config@npm:1.2.0::__archiveUrl=https%3A%2F%2Fnpm.pkg.github.com%2Fdownload%2F%40Contrast-Security-Inc%2Feslint-config%2F1.2.0%2F19b28737f42ceffca9efe305157de072816e4a59: Invalid authentication (as an unknown user)
  ➤ YN0041: │ @contrast-security-inc/stylelint-config@npm:2.0.0::__archiveUrl=https%3A%2F%2Fnpm.pkg.github.com%2Fdownload%2F%40Contrast-Security-Inc%2Fstylelint-config%2F2.0.0%2F25493e145d32f6add9a1ce5da9609e9e9706da0a: Invalid authentication (as an unknown user)
  ➤ YN0013: │ 6 packages were already cached, 2335 had to b

@jurre
Copy link
Member

jurre commented Oct 15, 2022

Thanks @scottdickerson, I or @pavera will take a look at this next week! I can't see the Actions logs fwiw, but surprising that just providing a default value would break anything!?

Would you be able to share the package.json and yarn.lock files for this project?

@scottdickerson
Copy link
Author

Unfortunately not to security concerns @jurre @pavera

@jurre
Copy link
Member

jurre commented Oct 19, 2022

@scottdickerson any other way we could try and reproduce the issue maybe?

Curious also; why do you have the .npmrc file checked into that repo? I thought yarn doesn't use it?

@pavera pavera added the L: javascript:yarn npm packages via yarn label Oct 19, 2022
@pavera
Copy link
Contributor

pavera commented Oct 20, 2022

@scottdickerson I've deployed at least a partial fix for this, would you be able to give it another try and let me know the results?

@scottdickerson
Copy link
Author

absolutely thank you @pavera

@scottdickerson
Copy link
Author

It appears to still be failing for us @pavera when dependabot checks for upgrades: https://github.com/Contrast-Security-Inc/skeletor/network/updates/489341448

proxy | 2022/10/21 13:56:11 [210] GET https://npm.pkg.github.com:443/yargs
  proxy | 2022/10/21 13:56:11 [210] * authenticating npm registry request (host: npm.pkg.github.com, token auth)
  proxy | 2022/10/21 13:56:11 [210] 404 https://npm.pkg.github.com:443/yargs
updater | INFO <job_489341448> Requirements to unlock own
updater | INFO <job_489341448> Requirements update strategy widen_ranges
updater | INFO <job_489341448> Updating yargs from 17.5.1 to 17.6.0
updater | I, [2022-10-21T13:56:12.378479 #7]  INFO -- sentry: ** [Raven] Sending event 8657f57305e74b50ae1910dcd75a9c65 to Sentry
  proxy | 2022/10/21 13:56:12 [212] POST https://sentry.io:443/api/1451818/store/
  proxy | 2022/10/21 13:56:12 [212] 200 https://sentry.io:443/api/1451818/store/
updater | ERROR <job_489341448> Error processing yargs (Dependabot::SharedHelpers::HelperSubprocessFailed)
updater | ERROR <job_489341448> Usage Error: Environment variable not found (GH_API_ACCESS_TOKEN) in /home/dependabot/dependabot-updater/repo/.yarnrc.yml (in /home/dependabot/dependabot-updater/repo/.yarnrc.yml)
updater | <job_489341448> 
updater | <job_489341448> Yarn Package Manager - 3.2.3
updater | <job_489341448> 
updater | <job_489341448>   $ yarn <command>
updater | <job_489341448> 
updater | <job_489341448> You can also print more details about any of these commands by calling them with 
updater | <job_489341448> the `-h,--help` flag right after the command name.
updater | ERROR <job_489341448> /home/dependabot/common/lib/dependabot/shared_helpers.rb:314:in `run_shell_command'
updater | ERROR <job_489341448> /home/dependabot/npm_and_yarn/lib/dependabot/npm_and_yarn/helpers.rb:25:in `run_yarn_commands'
updater | ERROR <job_489341448> /home/dependabot/npm_and_yarn/lib/dependabot/npm_and_yarn/file_updater/yarn_lockfile_updater.rb:169:in `run_yarn_berry_top_level_updater'
updater | ERROR <job_489341448> /home/dependabot/npm_and_yarn/lib/dependabot/npm_and_yarn/file_updater/yarn_lockfile_updater.rb:112:in `block (2 levels) in run_yarn_updater'
updater | ERROR <job_489341448> /home/dependabot/npm_and_yarn/lib/dependabot/npm_and_yarn/file_updater/yarn_lockfile_updater.rb:109:in `chdir'
updater | ERROR <job_489341448>

@pavera
Copy link
Contributor

pavera commented Oct 24, 2022

OK, I think I see the issue here. The .yarnrc.yml should look like:

nodeLinker: node-modules

npmScopes:
  contrast-security-inc:
    npmAuthToken: '${GH_API_ACCESS_TOKEN:-bypass}'
    npmPublishRegistry: 'https://npm.pkg.github.com/'
    npmRegistryServer: 'https://npm.pkg.github.com/'

plugins:
  - path: .yarn/plugins/@yarnpkg/plugin-workspace-tools.cjs
    spec: '@yarnpkg/plugin-workspace-tools'

yarnPath: .yarn/releases/yarn-3.2.3.cjs

Apologies for the earlier guidance, this explains why it was breaking locally/in CI because the :-bypass needs to be inside the variable expansion brackets. You were getting Actual-access-token:-bypass with the earlier config.

If you can give it another shot with the above .yarnrc.yml we'll see what we get.

@scottdickerson
Copy link
Author

it worked, thank you for the help! @pavera

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
L: javascript:yarn npm packages via yarn T: bug 🐞 Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants