RFC: proxying credentials for private registries

[mburumaxwell]

RFC -> Request For Comments and Request For Contributions

There have been a number of issues regarding authentication to private registries/feed. When this extension was made, dependabot/GitHub was not so clear about how they should be handled. That has since evolved to be a bit more clear including support for them in the configuration file.

However, as with every other product, not many can foresee what comes next and even then, things don't always go as planned. With credentials, things have changed. Especially for NuGet/.NET. It appears that we need to use a proxy to handle credentials going forward. This can be seen in dependabot-cli and dependabot-action. To support this requires an API that speaks dependabot so that we can use the closed source proxy or build our own.

A very simple solution would be directly using the dependabot-cli but it would mean loosing: auto complete, auto approve, policy bypass, skip PR creation, etc

It would be great and ideal to hear from the community.

Reading:

• Use dotnet credential provider dependabot/dependabot-core#8927 (comment)
• Support for NuGet feed credentials with only username/password in nuget.config dependabot/dependabot-core#10360 (comment)

cc: @rhyskoedijk

[DaleMckeown]

@mburumaxwell Could you clarify whether losing the ability to auto complete, auto approve, policy bypass, skip PR creation etc would be for private feeds only, or would it also be the case for public feeds as well?

For me, losing these features is better than losing access to private registries/feeds entirely. However, if it means losing access to these features in public feeds as well, this is probably a more difficult decision.

[DaleMckeown]

In that case, this is a difficult decision with significant implications. As someone who uses both private feeds and auto-approve/complete etc in equal measure, having to drop support for either is going to be disruptive regardless of which decision is made.

Have dependabot-core confirmed that they are not able to add auto-approve/complete into the CLI directly?

Although harder and more time consuming, perhaps building a custom proxy that is able to support all cases is the best way forwards.

[rhyskoedijk]

Have dependabot-core confirmed that they are not able to add auto-approve/complete into the CLI directly?

I haven't seen any [closed] issues or [rejected] PRs to suggest they are opposed to this and I do think it is worth considering.

Looking through the dependabot-core PR history, @mburumaxwell was actually the one who contributed a lot of the current Azure DevOps support! Given that, I would assume submitting more contributions for the missing features is probably going to be welcomed.

[mburumaxwell]

I am not sure if auto complete/approve are concepts in the core aspect of dependabot. It may need to be there before the support can trickle down to the CLI. In turn that might mean all the GitHub providers having that support. Arguably they may not want to have this supported: dependabot/dependabot-core#1973

It appears the official guide for auto approve and auto complete/merge is using a separate GitHub actions workflow:

• https://docs.github.com/en/code-security/dependabot/working-with-dependabot/automating-dependabot-with-github-actions#approve-a-pull-request
• https://docs.github.com/en/code-security/dependabot/working-with-dependabot/automating-dependabot-with-github-actions#enable-auto-merge-on-a-pull-request

[DaleMckeown]

In my org, for a few other non dependabot CI/CD pipelines we utilise another DevOps extension to create branches and automate PRs. Perhaps that extension could be leveraged to do the PR creation and auto-complete?

The extension is https://marketplace.visualstudio.com/items?itemName=ShaykiAbramczyk.CreatePullRequest

[mburumaxwell]

I think the extension is meant to create pull requests not modify existing ones. I think we'd recommend Azure CLI with the DevOps extension to do the updates or using the RestAPI directly. Need to check what is supported.

[rhyskoedijk]

I like the idea of using dependabot-cli to perform the update as it seems like the most robust way to ensure update behaviour doesn't fall behind with new features or breaking changes. In theory if the DevOps extension just built a job.yml and fed it to dependabot-cli, all of the code in /updater and the need to build custom docker images could be removed right? If so, that would greatly simplify things.

The functionality for managing PRs (create/update/abandon/auto-completing/auto-approve/etc) would be moved in to the extension TypeScript code, outside of the dependabot tools.

Pros:

• Less complexity (no custom docker images)
• Less code to maintain (no ruby code needed)
• Less likely for updates to break when updating dependabot-core.
• New dependabot features become available earlier, without development effort.

Cons:

• Less freedom and control over making customisations and workarounds to the dependabot-core compared to what currently exists.
• DevOps agents will require extra tool "go", to run dependabot-cli

Proof-of-concept:

#1318

[rhyskoedijk]

I've updated the parent comment as I now realise that dependabot-cli doesn't actually create/update/delete PRs, it just does a dry-run of the update and generates scenario files of what the PR should be. This means that the custom DevOps PR logic can be moved to the extension TypeScript code, external to dependabot.

Straw-man of update process

TL;DR; The DevOps extension converts dependabot.yaml in to job.yml then runs dependabot update -f job.yaml -o update-scenario.yaml. Dependabot-CLI performs a dry-run of the update and generates outputs to update-scenario.yml. The DevOps extension parses update-scenario.yml and creates pull requests using DevOps API.

 sequenceDiagram
    participant ext as Dependabot DevOps Extension
    participant agent as DevOps Pipeline Agent
    participant devops as DevOps API
    participant cli as Dependabot CLI
    participant core as Dependabot Updater
    participant feed as Package Feed

    ext->>ext: Read and parse `dependabot.yml`
    ext->>ext: Write `job.yaml`
    ext->>agent: Download dependabot-cli from github
    ext->>+cli: Execute `dependabot update -f job.yaml -o update-scenario.yaml`
    cli->>+core: Run update for `job.yaml` with proxy and dependabot-updater docker containers
    core->>devops: Fetch source files from repository
    core->>core: Discover dependencies
    loop for each dependency
        core->>feed: Fetch latest version
        core->>core: Update dependency files
    end
    core-->>-cli: Report outputs
    cli->>cli: Write outputs to `update-sceario.yaml`
    cli-->>-ext: Update completed

    ext->>ext: Read and parse `update-sceario.yaml`
    loop for each output
      alt when output is "create_pull_request"
        ext->>devops: Create pull request source branch
        ext->>devops: Push commit to source branch
        ext->>devops: Create pull request
        ext->>devops: Set auto-approve
        ext->>devops: Set auto-complete
      end
      alt when output is "update_pull_request"
        ext->>devops: Push commit to pull request
        ext->>devops: Update pull request description
        ext->>devops: Set auto-approve
        ext->>devops: Set auto-complete
      end
      alt when output is "close_pull_request"
        ext->>devops: Create comment thread on pull request with close reason
        ext->>devops: Abandon pull request
        ext->>devops: Delete source branch
      end
    end

Loading

This straw-man is very high level and skips a bunch of extra logic that would be required to handle updates to existing PRs, tracking the "existing PRs", storing dependency list metadata, etc.

Example job file

job:
    package-manager: nuget
    allowed-updates:
        - update-type: all
    source:
        provider: azure
        repo: rhyskoedijk/Dependabot/_git/Dependabot-Tests
        directory: /WebApplicationNetCore
        commit: 940d4d7302c1e0590ef3b9bdea498316e34d1b7a
credentials:
    - host: dev.azure.com
      password: 12345
      type: git_source
      username: x-access-token
    - token: PAT:12345
      type: nuget_feed
      url: https://pkgs.dev.azure.com/rhyskoedijk/Dependabot/_packaging/Private-NuGet/nuget/v3/index.json

Example scenario file

output:
    - type: update_dependency_list
      expect:
        data:
            dependencies:
                - name: Serilog.AspNetCore
                  requirements:
                    - file: /WebApplicationNetCore/WebApplicationNetCore.csproj
                      groups:
                        - dependencies
                      requirement: 8.0.1
                      source: null
                  version: 8.0.1
            dependency_files:
                - /WebApplicationNetCore/WebApplicationNetCore.csproj
                - /WebApplicationNetCore/nuget.config
    - type: create_pull_request
      expect:
        data:
            base-commit-sha: 940d4d7302c1e0590ef3b9bdea498316e34d1b7a
            dependencies:
                - name: Serilog.AspNetCore
                  previous-requirements:
                    - file: /WebApplicationNetCore/WebApplicationNetCore.csproj
                      groups:
                        - dependencies
                      requirement: 8.0.1
                      source: null
                  previous-version: 8.0.1
                  requirements:
                    - file: /WebApplicationNetCore/WebApplicationNetCore.csproj
                      groups:
                        - dependencies
                      requirement: 8.0.2
                      source:
                        nuspec_url: https://api.nuget.org/v3-flatcontainer/serilog.aspnetcore/8.0.2/serilog.aspnetcore.nuspec
                        source_url: null
                        type: nuget_repo
                        url: https://api.nuget.org/v3/index.json
                  version: 8.0.2
            updated-dependency-files:
                - content: "<Project Sdk=\"Microsoft.NET.Sdk.Web\">...snip...</Project>\r\n"
                  content_encoding: utf-8
                  deleted: false
                  directory: /WebApplicationNetCore
                  name: WebApplicationNetCore.csproj
                  operation: update
                  support_file: false
                  type: file
            pr-title: ⬆️ Bump Serilog.AspNetCore from 8.0.1 to 8.0.2 in /WebApplicationNetCore
            pr-body: |
                Bumps [Serilog.AspNetCore](https://github.com/serilog/serilog-aspnetcore) from 8.0.1 to 8.0.2.
                <details>
                <summary>Release notes</summary>
                ...snip...
                </details>
                <br />
            commit-message: |-
                :arrow_up: Bump Serilog.AspNetCore in /WebApplicationNetCore

                Bumps [Serilog.AspNetCore](https://github.com/serilog/serilog-aspnetcore) from 8.0.1 to 8.0.2.
                - [Release notes](https://github.com/serilog/serilog-aspnetcore/releases)
                - [Commits](https://github.com/serilog/serilog-aspnetcore/compare/v8.0.1...v8.0.2)
    - type: mark_as_processed
      expect:
        data:
            base-commit-sha: 940d4d7302c1e0590ef3b9bdea498316e34d1b7a

[rhyskoedijk]

@mburumaxwell what do you think of this approach?

In hindsight, this is what I probably should have done instead of creating the update_vnext script, but I didn't know dependabot-cli existed at the time.

Proof of concept here: #1318

[rhyskoedijk]

Update on this, I got a bit carried away with the PoC and it is now at the point where it will create pull requests in DevOps using the outputs of dependabot-cli. Private feed authentication "just works" without any hacks or workarounds. There are still a lot of TODO comments remaining, but overall I don't see any roadblocks with this approach so far.

[mburumaxwell]

This is awesome!
We may not get everything at the start but it sure is a very big step.

[PieniIhme]

Documentation says under Unsupported features and configurations / Server that "Private feed/registry authentication may not work with all package ecyosystems." and link gets me here to this discussion.

We are using server since Azure DevOps extension MS Hosted Agent ran longer than the maximum time of 60 minutes (even when we splitted configuration into multiple stages using targetUpdateIds).

I have been trying to get authentication to work but not succeeded. I have tried to ignore these private packages via dependabot.yml configuration file but it is not making any difference.

I like to know is private feed (nuget) still not working when using server implementation?

[rhyskoedijk]

Unfortunately the server component does not fully support auth with private feed due to the lack of a credential proxy which is now required by Dependabot; To get private feed auth working, the server component will need to be updated to use Dependabot CLI or implement the credential proxy in some other way.

Personally, I am currently focused on getting the extension task migrated over to Dependabot CLI first. After this, I might look at upgrading the server component if that work is still outstanding.

If your only issue is the 60 minute timeout, there are ways to work around that issue, such as:

• Extend the timeout of your pipeline
• Do a manual pass of package updates first; If you get everything up to date first, then use Dependabot for incremental updates, it will run much quicker.