Error with URL requirements and pip variables

[jsangmeister]

Is there an existing issue for this?

• I have searched the existing issues

Package ecosystem

pip

Package manager version

No response

Language version

No response

Manifest location and content before the Dependabot update

-r https://raw.githubusercontent.com/OpenSlides/openslides-datastore-service/${DATASTORE_COMMIT_HASH}/requirements/requirements-general.txt

DATASTORE_COMMIT_HASH is filled beforehand by a script.

dependabot.yml content

version: 2
updates:
  - package-ecosystem: "pip"
    directory: "/requirements/partial"
    schedule:
      interval: "daily"

Updated dependency

No response

What you expected to see, versus what you actually saw

Dependabot broke down with the following error and did not create any update PRs for any dependency, although some need updating:

Error during file fetching; aborting: /requirements/partial/https:/raw.githubusercontent.com/OpenSlides/openslides-datastore-service/${DATASTORE_COMMIT_HASH}/requirements/requirements-general.txt not found

Instead, such a line should simply be ignored. I'm not sure if the URL or the variable in the URL is the problem, but both should be relatively to filter out.

Native package manager behavior

No response

Images of the diff or a link to the PR, issue, or logs

No response

Smallest manifest that reproduces the issue

No response

[deivid-rodriguez]

If we ignore the url, then we'll be ignoring constraints during the update and may end up proposing updates incompatible with the project. I think we need to fetch all constraints to make sure we don't break projects.

Can you verify whether the URL without an ENV works? If it does not, then that's a bug we should fix.

Regarding the ENV itself, supporting that kind of thing would fall under #4660 I believe.

[jsangmeister]

Can you verify whether the URL without an ENV works?

I just created a test repo with the following content:

• requirements.txt:

-r https:/raw.githubusercontent.com/OpenSlides/openslides-datastore-service/main/requirements/requirements-general.txt

• .github/dependabot.yml:

version: 2
updates:
  - package-ecosystem: "pip"
    directory: "/"
    schedule:
      interval: "daily"

and the same error appears:

Error during file fetching; aborting: /https:/raw.githubusercontent.com/OpenSlides/openslides-datastore-service/main/requirements/requirements-general.txt not found

If we ignore the url, then we'll be ignoring constraints during the update and may end up proposing updates incompatible with the project. I think we need to fetch all constraints to make sure we don't break projects.

The problem is that even if you fix the URL handling, dependabot will try to fetch the URL without the replaced variable, which will obviously fail and the error in my case will persist. I don't need the variable expansion, it would already help if dependabot ignored requirements which it knows it can't fetch, like ones containing pip variables. Yes, some incompatible updates might be suggested, but that would be better than not being able to use dependabot at all. Also, in my case, I have control over both repositories, so I can update them both at once to prevent incompatible versions alltogether.

[deivid-rodriguez]

I think it's better to not create PRs at all than potentially creating incompatible PRs. People don't like it when Dependabot creates PRs that break their CI, or even worse, their apps. If we allowed this, it would certainly get reported as a bug eventually.

I think we should eventually support ENV variables to satisfy cases like yours. I recommend you watch #4660.

I'll leave this issue open to track the URL handling problem.

Thanks for reporting and testing this!