Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Gradle: generateSpdxEnums fails fetching files from ScanCode repo #5725

Closed
fviernau opened this issue Sep 1, 2022 · 3 comments
Closed

Gradle: generateSpdxEnums fails fetching files from ScanCode repo #5725

fviernau opened this issue Sep 1, 2022 · 3 comments
Assignees
@sschuberth
Labels
bug Issues that are considered to be bugs

Comments

@fviernau
Copy link
Member

fviernau commented Sep 1, 2022 •
edited
Loading

I've tried multiple times on my machine, but each time the task fails [1].
The reason for that may be [2].

A solution could be to re-write fetching data to simply download the zip archive instead of using SvnExport.
This downloads more data but can be ok as the task is not performed frequently.

[1]

./gradlew generateSpdxEnums
....

Execution failed for task ':utils:spdx-utils:importScanCodeLicenseTexts'.
> svn-export failed for https://github.com/nexB/scancode-toolkit/trunk/src/licensedcode/data/licenses
  svn: E175002: PROPFIND of '/nexB/scancode-toolkit/trunk/src/licensedcode/data/licenses': 504 Gateway Timeout (https://github.com)

[2] https://gramps.discourse.group/t/problem-downloading-gramps-source-code-from-github-using-svn-since-24-aug/750/12
@sschuberth
Copy link
Member

sschuberth commented Sep 1, 2022 •
edited
Loading

A solution could be to re-write fetching data to simply download the zip archive instead of using SvnExport.

I also once had a rewrite in mind to get rid of SvnExport, by leveraging https://docs.github.com/en/rest/repos/contents. We probably could even use our own GitHub GraphQL client for that.

@fviernau fviernau added the bug Issues that are considered to be bugs label Sep 1, 2022
@sschuberth
Copy link
Member

I'll investigate some alternatives to rewrite the code.

@sschuberth sschuberth self-assigned this Sep 1, 2022
@tsteenbe
Copy link
Member

tsteenbe commented Sep 1, 2022

Just to provide some context - I stumble upon this bug as I was trying to update ORT to SPDX license list 3.18.

@fviernau fviernau changed the title gradle: GenerateSpdxEnums fails fetching files from ScanCode repo gradle: GenerateSpdxEnums fails fetching files from ScanCode's code repository Sep 1, 2022
@sschuberth sschuberth changed the title gradle: GenerateSpdxEnums fails fetching files from ScanCode's code repository Gradle: generateSpdxEnums fails fetching files from ScanCode repo Sep 2, 2022
sschuberth added a commit that referenced this issue Sep 2, 2022
@sschuberth
Gradle: Migrate from the SvnExport to the Download plugin
e15495f 
GitHub's legacy feature of transparently exposing Git repositories as
Subversion repositories is rather slow and not very reliable. Actually,
it stopped working completely some time ago. Solve this by rewriting the
`spdx-utils` import tasks to use the `Download` plugin [1] instead on
URLs created via GitHub's tree API [2] to list directory contents (the
content API [3] does not work for ScanCode's massive license list due to
its limitation to 1000 files for a directory).

Fixes #5725.

[1]: https://github.com/michel-kraemer/gradle-download-task
[2]: https://docs.github.com/en/rest/git/trees#get-a-tree
[3]: https://docs.github.com/en/rest/repos/contents#get-repository-content

Signed-off-by: Sebastian Schuberth <sschuberth@gmail.com>
sschuberth added a commit that referenced this issue Sep 2, 2022
@sschuberth
Gradle: Migrate from the SvnExport to the Download plugin
65b1da0 
GitHub's legacy feature of transparently exposing Git repositories as
Subversion repositories is rather slow and not very reliable. Actually,
it stopped working completely some time ago. Solve this by rewriting the
`spdx-utils` import tasks to use the `Download` plugin [1] instead on
URLs created via GitHub's tree API [2] to list directory contents (the
content API [3] does not work for ScanCode's massive license list due to
its limitation to 1000 files for a directory).

Fixes #5725.

[1]: https://github.com/michel-kraemer/gradle-download-task
[2]: https://docs.github.com/en/rest/git/trees#get-a-tree
[3]: https://docs.github.com/en/rest/repos/contents#get-repository-content

Signed-off-by: Sebastian Schuberth <sschuberth@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@sschuberth sschuberth

Labels
bug Issues that are considered to be bugs
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@sschuberth @tsteenbe @fviernau