Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support custom matrices for artifact upload/download #98

Merged

Conversation

armanbilge
Copy link
Member

Fixes #74. In doing so, I hope it fixes #81 as well.

This is a fairly complex change, consisting of the following:

  1. Use all the keys in the matrix to determine the upload artifact id.
  • Currently they are always $os-$scala-$java. If the matrix is crossed on other things, such as JVM/JS, this leads to artifacts with duplicate ids being uploaded. I suspect this may be what causes the failures we've been observing in CI.
  • Now, they are named $os-$java-$scala-$key1-$key2 etc. This assumes that sensible value names are being used in the matrix (idk what the rules are for artifact upload names).
  1. Introduce a githubWorkflowArtifactDownloadExtraKeys setting (please help bikeshed name).
  • There are two kinds of keys wrt to the publishing step. Some keys (such as the Scala version) we must download artifacts for each value of that key. Other keys (such as OS or Java), we only need to download the artifacts for the primary value of that key.
  • This setting is a Set[String] of keys of the first kind above (i.e., for which we must download artifacts for each value). We use it to add the key project (with values rootJVM, rootJS and/or rootNative) when setting up the platform cross in the matrix.
  • With this information, the publish workflow "re-creates" the matrix as a sequence of steps for downloading artifacts, heeding any exclusions.

These changes are fairly internal but possibly deserve a minor version bump. In any case, before releasing I will test the snapshot on some relevant projects.

@armanbilge armanbilge changed the title Make artifact upload/download Make artifact upload/download work with complex matrices Jan 23, 2022
@armanbilge armanbilge linked an issue Jan 24, 2022 that may be closed by this pull request
@armanbilge armanbilge changed the title Make artifact upload/download work with complex matrices Support complex matrices for artifact upload/download Jan 24, 2022
@armanbilge armanbilge changed the title Support complex matrices for artifact upload/download Support custom matrices for artifact upload/download Jan 24, 2022
@rossabaker
Copy link
Member

Regarding the choice of Set, does the order of keys matter, or do they just need to be in the same arbitrary order in order to generate a unique ID?

@armanbilge
Copy link
Member Author

armanbilge commented Jan 26, 2022

Just a consistent (but otherwise arbitrary) ordering. Maybe should be a SortedSet here then?

Edit: sorry, I'm confusing with the githubWorkflowArtifactDownloadExtraKeys setting which is a Set.

Copy link
Member

@rossabaker rossabaker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, if consistent-but-arbitrary is the goal, and that makes sense, then I think it's fine.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Make artifact upload/download step cross-aware
2 participants