Create a dependabot-alike for updating thirdparty dependencies for Pants

[stuhood]

dependabot itself is not accepting support for new ecosystems, but they suggest that forking dependabot-script would be a good starting place for creating a bot for updating some other ecosystem.

[stuhood]

There are two relevant flavors of automated dependency updates:

1. Transitive dependencies:
   • To update transitive dependencies as of 2.11.x (with [python].enable_resolves=True for Python, or by default for the JVM), running ./pants generate-lockfiles and committing the result is sufficient. This will not edit any BUILD files or requirement files: only the transitive deps in your lockfiles.
2. Root dependencies:
   • Automating updates of your root dependencies is slightly more involved currently: doing it as a new Pants goal would likely make sense: see Implement a goal to automate a portion of (root) dependency upgrades #15275.

[asherf]

FWIW, internally at toolchain we are using this script:

from __future__ import annotations

import logging
from argparse import ArgumentParser, Namespace
from pathlib import Path

import httpx
import tomlkit
from packaging import version
from packaging.specifiers import SpecifierSet
from pkg_resources import Requirement

from toolchain.base.toolchain_binary import ToolchainBinary

_logger = logging.getLogger(__name__)


class UpgradePythonReqs(ToolchainBinary):
    description = "Upgrade python requirements"

    _PANTS_CFG = Path("pants.toml")
    _SKIP_TOOLS = frozenset(
        (
            "flake8",  # We can't use flake8 6.x since it doesn't support running under pythong 3.7 which we use with the TC pants plugin
        )
    )

    def __init__(self, cmd_args: Namespace) -> None:
        super().__init__(cmd_args)
        self._reqs_files = [Path(fl) for fl in cmd_args.reqs]
        self._client = httpx.Client(base_url="https://pypi.org/pypi")

    def run(self) -> int:
        for req_file in self._reqs_files:
            _logger.info(f"Processing: {req_file.as_posix()}")
            self._process_reqs_file(req_file)
        self._upgrade_python_tools(self._PANTS_CFG)
        return 0

    def _process_reqs_file(self, req_file: Path) -> bool:
        lines = []
        upgrades = []
        for line in req_file.read_text().splitlines():
            new_req = self._maybe_get_upgraded_req(line)
            if not new_req:
                lines.append(line)
            else:
                lines.append(new_req)
                upgrades.append(new_req)
        if upgrades:
            _logger.info(f"{req_file.as_posix()} upgrades: {', '.join(upgrades)}")
            req_file.write_text("\n".join(lines))
        else:
            _logger.warning(f"No Upgrades for {req_file.as_posix()}")
        return bool(upgrades)

    def _maybe_get_upgraded_req(self, req_line: str) -> str | None:
        req = Requirement.parse(req_line)  # TODO: add try-except and just copy the line as is if we can't parse it.
        if not req.specs or req.specs[0][0] != "==":
            return None
        response = self._client.get(f"{req.project_name}/json")
        response.raise_for_status()
        current_version = version.parse(req.specs[0][-1])
        latest_version = version.parse(response.json()["info"]["version"])
        if current_version == latest_version:
            return None
        req.specifier = SpecifierSet(f"=={latest_version}")  # type: ignore[attr-defined]
        return str(req)

    def _upgrade_python_tools(self, pants_cfg: Path) -> bool:
        upgrades = []
        cfg = tomlkit.parse(pants_cfg.read_text())
        for scope, scope_cfg in cfg.items():
            if scope in self._SKIP_TOOLS:
                continue
            for opt, val in scope_cfg.items():
                if opt != "version":
                    continue
                new_req = self._maybe_get_upgraded_req(val)
                if not new_req:
                    continue
                scope_cfg[opt] = new_req
                upgrades.append(new_req)
        if upgrades:
            _logger.info(f"{pants_cfg.as_posix()} upgrades: {', '.join(upgrades)}")
            pants_cfg.write_text(tomlkit.dumps(cfg))
        else:
            _logger.warning(f"No Upgrades for {pants_cfg.as_posix()}")
        return bool(upgrades)

    @classmethod
    def add_arguments(cls, parser: ArgumentParser) -> None:
        parser.add_argument("reqs", nargs="+")

In combination w/ GHA workflow:

name: Upgrade Python Requirements
# Based on https://www.oddbird.net/2022/06/01/dependabot-single-pull-request/
on: 
  workflow_dispatch: # Allow running on-demand
  schedule:

    - cron: <TBD>
jobs:
  upgrade:
    name: Upgrade Python Requirements & Open Pull Request
    runs-on: ubuntu-latest
    env:
      BRANCH_NAME:  python-reqs
    steps:
      - uses: actions/checkout@v3
      - name: Set up Python 3.9.12
        uses: actions/setup-python@v4
        with:
          python-version: "3.9.12"
      - uses: actions/setup-go@v3
        with:
          go-version: '1.18'
      - name: Cache pants
        uses: actions/cache@v3
        with:
          key: ${{ runner.os }}-${{ hashFiles('pants*toml') }}-v2
          path: |
             ~/.cache/pants/setup
      - name: Set env vars
        run: |
          echo 'PANTS_CONFIG_FILES=+["${{ github.workspace }}/pants.ci.toml"]' >> ${GITHUB_ENV}
${GITHUB_ENV}
      - name: Bootstrap Pants
        run: ./pants version
      - name: Upgrade reqs
        run: ./pants run src/python/toolchain/prod/upgrade_python_reqs.py -- 3rdparty/python/requirements.txt
      - name: generate lock files
        run: ./pants generate-lockfiles
      - name: Detect changes
        id: changes
        run:
          # This output boolean tells us if the dependencies have actually changed
          echo "::set-output name=count::$(git status --porcelain=v1 3rdparty/ 2>/dev/null | wc -l)"
      - name: Commit & push changes
        # Only push if changes exist
        if: steps.changes.outputs.count > 0
        run: |
          git config user.name github-actions
          git config user.email github-actions@github.com
          git add 3rdparty/
          git add pants.toml
          git commit -m "Automated Python Requirements upgrades"
          git push -f origin ${{ github.ref_name }}:$BRANCH_NAME
      - name: Open pull request if needed
        if: steps.changes.outputs.count > 0
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        # Only open a PR if the branch is not attached to an existing one
        run: |
          PR=$(gh pr list --head $BRANCH_NAME --json number -q '.[0].number')
          if [ -z $PR ]; then
            echo "pr description" > pr_body.txt
            gh pr create \
            --head $BRANCH_NAME \
            --title "Automated Python Requirements upgrades" \
            --body-file pr_body.txt
          else
            echo "Pull request already exists, won't create a new one."
          fi

to achieve this.

[asherf]

I tried to convert the python script into a pants goal rule but got stuck.

[jake-normal]

Looks like the dependabot team is now accepting contributions that add additional ecosystems: https://github.com/dependabot/dependabot-core/blob/main/CONTRIBUTING.md#contributing-new-ecosystems