Add yarn build and yarn test to OUI release pipeline

[BSFishy]

For OUI's release process we need yarn build and preferably yarn test to be run sometime during the pipeline. yarn build generates important release artifacts, including generating CSS files from Sass source files. yarn test would be nice as a final sanity check before cutting a release.

I need guidance on how to properly incorporate these commands through our pipeline. I would think the best way to go about this would be putting yarn test in the Github workflow and yarn build in the Jenkins workflow, but I'm not sure how to do those things.

[gaiksaya]

Hi @BSFishy,

As a part of release pipeline we do not build, test anything on release pipeline which is Jenkins in this case. See #1234 for more details.
Since this environment is not regularly exercised and would require a lot of maintenance we decided to go with this approach. If you want to do last step of sanity testing you can add it to your release workflow here https://github.com/opensearch-project/oui/blob/main/.github/workflows/release-drafter.yml

Release pipeline is triggered only when a release is "drafted" on GitHub which is automated in above mentioned workflow. Hence you can add a step before that, that can fail the entire workflow if the checks fail.
I would like to understand what release artifacts needs to be shipped and where?
I do not see any artifacts that can be /needs to be attached on npmjs: https://www.npmjs.com/package/@opensearch-project/oui
Please let me know if I am missing something.

Also check how we publish to NPM today https://github.com/opensearch-project/opensearch-build-libraries/blob/main/vars/publishToNpm.groovy#L20

Thanks!

[BSFishy]

Specifically, the dist/ folder, es/ folder, lib/ folder, and test-env/ folder. Those can be found by exploring 1.0.0 on NPM. However, using the current pipeline configuration, those artifacts aren't included. All of them are generated when running yarn build.

[gaiksaya]

I see! We use draft release artifacts as an intern/staging location for publishing anything to anywhere. See asset section here. Can you see if you can include these artifacts as a tarball so that we can take them, sign them if needed and then publish to NPM?

[gaiksaya]

Adding @VachaShah @harshavamsi @rupal-bq @joshuali925 (maintainers of opensearch-js and reporting-cli) which is also published on NPM to see if they have more information on this.
Thanks!

[BSFishy]

These artifacts can be generated in the Github workflow. Is uploading them as artifacts all that is necessary? Could we use something like https://github.com/actions/upload-artifact?

[gaiksaya]

The action you specified only uploads the artifacts to the workflow run. This is how we can attach the artifacts to the drafted release so that it can be retained for longer time. https://github.com/opensearch-project/opensearch-py/blob/main/.github/workflows/release-drafter.yml#L37-L42
I am not yet sure how to publish those to npm using command line. Can you look into that a bit?

Thanks!

[BSFishy]

According to item b) of https://docs.npmjs.com/cli/v9/commands/npm-publish?v=true#description, a tarball can be specified as an argument to the command to upload.

Will work on adding the tarball as an artifact to the release workflow

Update: added tarball to release workflow: opensearch-project/oui#636

[gaiksaya]

Hi @BSFishy ,
We would also need to add this functionality to jenkins groovy library https://github.com/opensearch-project/opensearch-build-libraries/blob/main/vars/publishToNpm.groovy#L20
Is there any release scheduled by which we would need to get it in?
cc: @bbarani

[zelinh]

Bring @AMoo-Miki into the discussion here.
We need to understand why we are doing this since we are not doing this for any other components. This could be a good practice for other components in the future.

[BSFishy]

Hi @BSFishy , We would also need to add this functionality to jenkins groovy library https://github.com/opensearch-project/opensearch-build-libraries/blob/main/vars/publishToNpm.groovy#L20 Is there any release scheduled by which we would need to get it in? cc: @bbarani

We want to get a release out for 2.7, so we want to work on having a release around/maybe a little after March 31. Also, this will help a lot for Pink Elephant, so the sooner the better for that.

[gaiksaya]

Discussed with team offline. Targeting April 6th for this change to get in and then release.

[BSFishy]

Bring @AMoo-Miki into the discussion here. We need to understand why we are doing this since we are not doing this for any other components. This could be a good practice for other components in the future.

The difference here is that we have TS -> JS and SCSS -> CSS transformations happening as part of our build process that are consumed downstream. Other JS repos in OpenSearch org are JS native, and simply point to their source files. OUI is unable to do that because we have an extensive build system that performs a lot of transformations that aren't immediately present in the repo, they only show up when yarn build is run. This would probably be a good feature to have for other repos too in case they want to incorporate some sort of bundling or transformation or build pipeline in the future.

[jordarlu]

Hi, @BSFishy , thanks for the elaboration above.
Once the opensearch-project/oui#636 is merged, can we assume that all the needed files/artifacts ( artifacts generated only after 'yarn build' ) will be tar.gz into artifacts.tar.gz ? Thanks

[BSFishy]

Hi, @BSFishy , thanks for the elaboration above. Once the opensearch-project/oui#636 is merged, can we assume that all the needed files/artifacts ( artifacts generated only after 'yarn build' ) will be tar.gz into artifacts.tar.gz ? Thanks

Yes, all files will be in artifacts.tar.gz. The structure will look something like this:

artifacts.tar.gz/
├─ oui.tar.gz/
│  ├─ dist/
│  │  ├─ oui.min.js
│  │  ├─ oui_dark_theme.css
│  │  ├─ ...
│  ├─ lib/
│  │  ├─ ...
│  ├─ es/
│  │  ├─ ...

[gaiksaya]

Keeping it open until OUI PR is merged.
Thanks!