Support Windows runner for CI workflows #1275

RyanL1997 · 2022-12-20T01:32:11Z

Description

Support Windows runner for CI workflows of integration test and Cypress test

Issues Resolved

Resolved Passing Integration tests CI on Windows #1171

Testing

GitHub Actions

Check List

New functionality includes testing
~~New functionality has been documented~~
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

RyanL1997 · 2022-12-20T13:56:26Z

This PR is still a WIP. The major blockers are:

The SAML Integration test is failing due to unknown reason.[1]
For the Cypress test, the yarn start --no-base-path & command cannot make the dashboards server up running after the step "Configure and Run OpenSearch Dashboards".[2]

Reference

[1]: https://github.com/RyanL1997/security-dashboards-plugin/actions/runs/3716119921/jobs/6333900876
[2]: https://github.com/RyanL1997/security-dashboards-plugin/actions/runs/3732417653/jobs/6331827892#step:13:84

cwperks · 2022-12-20T14:22:24Z

Is the test getting stuck somewhere? It looks like its reaching the 10min timeout limit:

Timeout - Async callback was not invoked within the 600000 ms timeout specified by jest.setTimeout.Timeout - Async callback was not invoked within the 600000 ms timeout specified by jest.setTimeout.Error:

on this test:

> 287 |   it('Tenancy persisted after Logout in SAML', async () => {

RyanL1997 · 2022-12-23T15:19:49Z

Updates

Here is the log of using:

...
Start-Process yarn -ArgumentList 'pretest:jest_server'
node .\test\run_jest_tests.js --config .\test\jest.config.server.js
...

https://github.com/RyanL1997/security-dashboards-plugin/actions/runs/3752295912/jobs/6374283816#step:17:101.
I think by running like this the IDP server is not running it up, because me and Stephen received the same log when we only used the command: node .\test\run_jest_tests.js --config .\test\jest.config.server.js

RyanL1997 · 2023-01-06T23:35:43Z

Updates on 01/06/2023

There are two major issues blocking this PR:

IDP server of SAML integration test couldn't be started on windows runner so that will lead to the failing of integration test, and the workflow will stuck on the running of integration test without any logs by using yarn command. [1]
OS Dashboards server couldn't be started on windows runner, so that will lead to the failing of Cypress test. [2]

For the first issue of IDP server of integration test workflow, instead of using yarn, I also tried to use node .\test\jest_integration\runIdpServer.js & with node .\test\run_jest_tests.js --config .\test\jest.config.server.js [3] and it seems like it will performs the same, but with logs to show that SAML test is the only test case failed [4]. According to these, I will first try to refactor the initialization of IDP server into the SAML test case. If this approach is not feasible, I will set a bypass of the SAML test case in window runner only, and create a follow up issue and documentation to this for future fix.

For the second issue of Cypress workflow, I was assuming that it may need longer time to set up OSD on windows runner. However, according to the experiments me and @scrawfor99 did for using the original yarn command, even we have extended the sleep time to 800 seconds [5], the dashboard server still could't be started on windows runner (see [2]).

On behalf the similarity of the above findings, I guess the way we are using of starting severs on windows runner is wrong. I'm still doing research about this and I will also keep updating the findings.

Reference

[1] : https://github.com/opensearch-project/security-dashboards-plugin/actions/runs/3859072362/jobs/6578260549
[2] : https://github.com/opensearch-project/security-dashboards-plugin/actions/runs/3859072379/jobs/6578260760#step:13:62
[3] : https://github.com/RyanL1997/security-dashboards-plugin/actions/runs/3716119921/workflow#L109-L110
[4] : https://github.com/RyanL1997/security-dashboards-plugin/actions/runs/3716119921/jobs/6333900876#step:17:61
[5] : https://github.com/opensearch-project/security-dashboards-plugin/actions/runs/3859072379/workflow#L89

peternied · 2023-01-09T15:48:58Z

As written the SAML IDP is started before the tests run, but this seems backwards to me. The SAML server should be started by the tests that need it so endpoint, state, and other properties can be reviewed and confirmed by the tests. Can we migrate the IDP start up logic into the tests - this side-steps issues of how to start the server in Windows vs Linux environments.

As there is considerable progress towards getting the rest of the tests running, can we handle the SAML tests running on windows in a separate PR to keep up speed, what do you think?

stephen-crawford · 2023-01-09T19:48:06Z

@RyanL1997 thank you for filing this. I think @peternied's suggestion to migrate the server launch into the tests themselves is a good one. You and I have both tried about every other way of getting the server to launch and I think it is just a matter of something being different on the window's runner versus the Linux which may not be configurable. That being said, great job with all this.

RyanL1997 · 2023-01-10T07:11:20Z

Update 01/09/2023

@cwperks, Here is the findings after we tried to curl the http://localhost:5601 after checkout to the functional test repo [1]:

curl: (7) Failed to connect to localhost port 5601 after 2229 ms: Connection refused

It seems like we lost the server after the checkout.
I also tried to curl the http://0.0.0.0:5601, and it returned me this [2]:

curl: (7) Failed to connect to 0.0.0.0 port 5601 after 0 ms: Address not available

Reference

[1] : https://github.com/RyanL1997/security-dashboards-plugin/actions/runs/3880154925/jobs/6617977487#step:12:30.
[2] : https://github.com/RyanL1997/security-dashboards-plugin/actions/runs/3880184753/jobs/6618034164#step:12:28

RyanL1997 · 2023-01-12T19:46:09Z

@opensearch-project/security Guys I think I need your support on this. If you can take a look of this comment (#1275 (comment)) I think the server of OSD cannot be carried after the checkout of functional test repo for Cypress Test.

DarshitChanpura · 2023-01-12T20:34:41Z

Looks like both, Cypress and Integration test, CI checks failed with java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin] during Run Opensearch with Single plugin stage

stephen-crawford · 2023-01-12T20:39:23Z

@DarshitChanpura, I was trying to help @RyanL1997 remove the SAML test from the Windows side and saw this too. I am looking into it now and will hopefully identify a cause soon.

DarshitChanpura · 2023-01-12T20:59:22Z

Seems like this could be a potential root cause:

Caused by: java.lang.NoClassDefFoundError: com/fasterxml/jackson/databind/InjectableValues$Std
	at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.<init>(OpenSearchSecuritySSLPlugin.java:195) ~[?:?]

stephen-crawford · 2023-01-12T21:03:36Z

@DarshitChanpura, thanks for the help. I see that too on mine. I am actually running the 2.x and main versions of the install workflow on the backend side to see what the results are. It may be as simple as updating the fasterxml dependency if it fails on the backend too.

Update: 2.x is fine but main fails saying a bunch of jackson stuff does not exist. I am going to look at the changes that impacted jackson recently and see where we will need to add it back in.

Update: Now it seems to be working on main again. I am not sure if it is intermittent build changes on the distribution build that is causing the behavior difference but even when the backend is fixed the dashboards repo is still running into the issue. I am going to look more closely at the specific error mentioned above. For the jackson issue I am unsure what we want to elect as the course of action. It seems like we are running into the issue as a result of the core snapshot having moved jackson around (my best guess without following the path further). We can directly depend on jackson ourselves in order to make sure we have the correct version where we expect but this would add an entire dependency and I know that jackson has been a topic for discussion as of late.

DarshitChanpura · 2023-01-12T22:10:51Z

Update: Now it seems to be working on main again. I am not sure if it is intermittent build changes on the distribution build that is causing the behavior difference but even when the backend is fixed the dashboards repo is still running into the issue. I am going to look more closely at the specific error mentioned above. For the jackson issue I am unsure what we want to elect as the course of action. It seems like we are running into the issue as a result of the core snapshot having moved jackson around (my best guess without following the path further). We can directly depend on jackson ourselves in order to make sure we have the correct version where we expect but this would add an entire dependency and I know that jackson has been a topic for discussion as of late.

The class is present in jackson-databind 2.14.1 which is what we use: https://fasterxml.github.io/jackson-databind/javadoc/2.14/com/fasterxml/jackson/databind/InjectableValues.Std.html
So I'm not sure where exactly the issue lies. I was able to reproduce it locally. Will look into possible solutions now

DarshitChanpura · 2023-01-12T22:14:05Z

The class is present in jackson-databind 2.14.1 which is what we use: https://fasterxml.github.io/jackson-databind/javadoc/2.14/com/fasterxml/jackson/databind/InjectableValues.Std.html So I'm not sure where exactly the issue lies. I was able to reproduce it locally. Will look into possible solutions now

Looks like jackson-databind is not shipped with the packaging distribution, neither in core nor in security which is most likely the root cause. Investigating now

stephen-crawford · 2023-01-13T14:40:54Z

I also saw that core and security seemed to not be building with the jackson dependency. However, it looks like the classes can be found when you just focus on the backend tests--if you run plugin install workflow, everything is successful on main. I suspect this means that dashboards may be getting a different version of either the security plugin or the core SNAPSHOT.

DarshitChanpura · 2023-01-13T20:57:33Z

I also saw that core and security seemed to not be building with the jackson dependency. However, it looks like the classes can be found when you just focus on the backend tests--if you run plugin install workflow, everything is successful on main. I suspect this means that dashboards may be getting a different version of either the security plugin or the core SNAPSHOT.

Interesting. I did confirm that the building the package using main branch of security doesn't include jackson-databind, which makes me think that the way we are building artifacts might be a little skewed.

codecov-commenter · 2023-01-20T20:08:23Z

Codecov Report

Merging #1275 (0c95b14) into main (82c27c7) will not change coverage.
The diff coverage is n/a.

📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

@@           Coverage Diff           @@
##             main    #1275   +/-   ##
=======================================
  Coverage   71.78%   71.78%           
=======================================
  Files          88       88           
  Lines        2027     2027           
  Branches      274      274           
=======================================
  Hits         1455     1455           
  Misses        509      509           
  Partials       63       63

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

RyanL1997 · 2023-01-24T06:07:56Z

Update 01/23/2023

I have successfully fixed the Cypress workflow [1]. However, since the original github action for checkout is broken for windows runner [2]. I have came out a work around: instead of checking-out to functional test repo to run the Cypress test, I clone the FTR into current working directory and directly setup and run Cypress in the same step of OSD setup [3].

Reference

[1] : https://github.com/opensearch-project/security-dashboards-plugin/actions/runs/3993203040/workflow
[2] : actions/checkout#702
[3] : https://github.com/opensearch-project/security-dashboards-plugin/pull/1275/files#diff-1c29b5bd518bbbdff17cb10f614dd430171102ccd8c78497bebe5da820657dbbR91

.github/workflows/integration-test.yml

Signed-off-by: Ryan Liang <jiallian@amazon.com>

RyanL1997 · 2023-01-31T19:49:02Z

Update 01/31/2023

Since the latest distribution build does not have security plugin included, so I'm testing on build version 7028 which includes all the latest changes we made for core and security plugin, and I will change it back to latest once everything is ready.

@cwperks, @peternied, @scrawfor99, and @DarshitChanpura, I have finished some of the refactoring so some of the old reviews of you guys disappeared on the latest change. I also replied some of the comments in the old commits but for better view and understanding I'm manually updating with you guys following:

I have created a action for plugin install and script setup, so that it reduces the redundancy of the commands.
I have deleted all the conditional commands in the action of dashboard installation, since everything is compatible with windows runner.
In the plugin install action I switch to use peternied/download-file@v2 for plugin downloading
For the gecko action, I was checking their repo and it seems like it only has latest as their version control, so that I cannot change it.
For the Firefox action, I have change the latest into v1. However, I cannot change the action for windows, otherwise it will return:

Since it doesn't support for windows, and I can only use my fork for the windows support. (However, due to the IDP server issue, we do not have saml_auth test included in windows runner).

stephen-crawford · 2023-01-31T20:19:48Z

Great work @RyanL1997. I know this has been a major thorn and it is appreciated that you continue to be diligent about it. All of your comments make sense to me. I would also mention that if you were really motivated, you could fork the gecko action and firefox action for windows and then make your own versioned action which you could refer to (this is not necessary but is an option should you want to go that route). Great job overall!

RyanL1997 · 2023-01-31T20:44:13Z

Great work @RyanL1997. I know this has been a major thorn and it is appreciated that you continue to be diligent about it. All of your comments make sense to me. I would also mention that if you were really motivated, you could fork the gecko action and firefox action for windows and then make your own versioned action which you could refer to (this is not necessary but is an option should you want to go that route). Great job overall!

Thanks for the comment @scrawfor99. That makes sense. The reason we were trying to avoid using @latest is that we are trying to avoid to having breaking changes for their update. Using our own fork is actually a good idea for avoiding this.

peternied · 2023-01-31T20:26:25Z

.github/actions/download-plugin/action.yml

+      uses: peternied/download-file@v2
+      if: ${{ runner.os == 'Linux' }}
+      with:
+        url: https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/${{ inputs.opensearch-version }}/7028/linux/x64/tar/builds/opensearch/plugins/${{ inputs.plugin-name }}-${{ inputs.plugin-version }}.zip


Hardcoded build 7028?

Yes, I was checking with infra team this morning and seems like the latest doesnt include security plugin yet and that's also the reason for the failing of pre-requisite test. I will change it back to latest once everything is ready. This is only for now.

Please link the issue where this is being worked on.

I just linked issue in the general comment section.

Since the latest distribution build of 3.0.0 does not have security plugin included, so I'm testing on build version 7028 which includes all the latest changes we made for core and security plugin, and I will change it back to latest once this issue gets fixed opensearch-project/cross-cluster-replication#696

If the security plugin is building on 3.0.0 why don't we have a fresh build of it available? Should we be publishing our own builds to maven for consumption so we aren't stuck because of a separate plugin, or should the build process automatically publish binaries for components that do function?

@RyanL1997 Please create a new issue in opensearch-build and start a thread in our release channel linking that issue, we need to be pushing for a solution to the systemic problem.

@peternied here we go opensearch-project/opensearch-build#3172

Update: They have fixed the build

.github/actions/install-dashboards/action.yml

.github/workflows/cypress-test.yml

.github/workflows/integration-test.yml

peternied · 2023-01-31T22:38:07Z

.github/workflows/integration-test.yml

+      # Browser-action version does not work on Windows
+      - name: Set up Firefox browser for Windows
+        if: ${{ runner.os == 'Windows' }}
+        uses: RyanL1997/setup-browser@main


I don't see any changes in your fork, can you link the commit with the changes you needed? I'm fine using your fork and should use the same fork for both linux/windows workflows

RyanL1997 · 2023-02-01T04:27:47Z

I don't see any changes in your fork, can you link the commit with the changes you needed? I'm fine using your fork and should use the same fork for both linux/windows workflows

I was forking from the other's fork. I think this is the commit from the original author to fix firefox browser on arm64 on windows runner: RyanL1997/setup-browser@4e4eb11#diff-75000eb32e68d8d07d14a37b002c16b1920c849082945fbaf79c4d263a182e87

Signed-off-by: Ryan Liang <jiallian@amazon.com>

RyanL1997 · 2023-02-01T19:05:27Z

Since the latest distribution build of 3.0.0 does not have security plugin included, so I'm testing on build version 7028 which includes all the latest changes we made for core and security plugin, and I will change it back to latest once this issue gets fixed opensearch-project/cross-cluster-replication#696

Signed-off-by: Ryan Liang <jiallian@amazon.com>

cwperks

One qq on yarn add sha.js, but otherwise this looks good to me 🙌

.github/actions/install-dashboards/action.yml

peternied

Lets merge it, thanks for your hardwork @RyanL1997 !

stephen-crawford

Woot woot! Way to go Ryan.

cwperks · 2023-02-03T14:46:10Z

Updating branch protection rules to make integ tests on Ubuntu + Windows a requirement

RyanL1997 changed the title ~~Support Windows runner for CI workflows~~ [WIP] Support Windows runner for CI workflows Dec 20, 2022

RyanL1997 force-pushed the windows-support-for-ci branch from 9999f7e to 0a33ee3 Compare December 21, 2022 19:38

RyanL1997 force-pushed the windows-support-for-ci branch 2 times, most recently from c95647d to 076b2d8 Compare January 6, 2023 22:47

peternied mentioned this pull request Jan 12, 2023

Add Windows Support #845

Closed

3 tasks

RyanL1997 force-pushed the windows-support-for-ci branch 2 times, most recently from d565352 to 549afe5 Compare January 20, 2023 19:57

RyanL1997 force-pushed the windows-support-for-ci branch 4 times, most recently from 4dd8a09 to 833cdff Compare January 24, 2023 05:51

cwperks reviewed Jan 24, 2023

View reviewed changes

.github/workflows/integration-test.yml Outdated Show resolved Hide resolved

RyanL1997 added 2 commits January 26, 2023 15:02

Trim the install dashboards action

c4f2472

Signed-off-by: Ryan Liang <jiallian@amazon.com>

Switch to build id 7028 for security plugin

3dbbe4d

Signed-off-by: Ryan Liang <jiallian@amazon.com>

RyanL1997 force-pushed the windows-support-for-ci branch 4 times, most recently from 916f111 to 429c71e Compare January 31, 2023 19:20

Refactor selenium driver setup

9da6a34

Signed-off-by: Ryan Liang <jiallian@amazon.com>

RyanL1997 force-pushed the windows-support-for-ci branch from 429c71e to 9da6a34 Compare January 31, 2023 19:24

peternied reviewed Jan 31, 2023

View reviewed changes

RyanL1997 added 3 commits January 31, 2023 20:42

Try gecko driver commit id and remove jdk in cypress

0c95b14

Signed-off-by: Ryan Liang <jiallian@amazon.com>

Fix the osd-version naming and revert back

677e6c1

Signed-off-by: Ryan Liang <jiallian@amazon.com>

Fix gecko branch and jdk 11 only

6a6c6e2

Signed-off-by: Ryan Liang <jiallian@amazon.com>

RyanL1997 force-pushed the windows-support-for-ci branch from 1eafb8d to 6a6c6e2 Compare February 1, 2023 17:33

RyanL1997 and others added 2 commits February 2, 2023 13:25

Switch back to 'latest' for distribution build version

e9fcee4

Signed-off-by: Ryan Liang <jiallian@amazon.com>

Merge branch 'main' into windows-support-for-ci

edfd2f0

cwperks reviewed Feb 2, 2023

View reviewed changes

.github/actions/install-dashboards/action.yml Show resolved Hide resolved

peternied approved these changes Feb 2, 2023

View reviewed changes

cwperks approved these changes Feb 3, 2023

View reviewed changes

stephen-crawford approved these changes Feb 3, 2023

View reviewed changes

cwperks merged commit 8e9c4b7 into opensearch-project:main Feb 3, 2023

RyanL1997 mentioned this pull request Feb 8, 2023

[Backport 2.x] Support Windows runner for CI workflows #1320

Merged

3 tasks

This was referenced Aug 24, 2023

When following a link that prompts sign in through SAML or Multi-Auth the destination page is not lost #1557

Merged

Enable SAML Integration tests for windows #1560

Open

derek-ho mentioned this pull request Jan 16, 2024

Run Security dashboards plugin from binary #1726

Merged

3 tasks

Support Windows runner for CI workflows #1275

Support Windows runner for CI workflows #1275

Conversation

RyanL1997 commented Dec 20, 2022

Description

Category

Issues Resolved

Testing

Check List

RyanL1997 commented Dec 20, 2022 • edited Loading

This PR is still a WIP. The major blockers are:

Reference

cwperks commented Dec 20, 2022 • edited Loading

RyanL1997 commented Dec 23, 2022

Updates

RyanL1997 commented Jan 6, 2023

Updates on 01/06/2023

Reference

peternied commented Jan 9, 2023

stephen-crawford commented Jan 9, 2023

RyanL1997 commented Jan 10, 2023

Update 01/09/2023

Reference

RyanL1997 commented Jan 12, 2023 • edited Loading

DarshitChanpura commented Jan 12, 2023 • edited Loading

stephen-crawford commented Jan 12, 2023

DarshitChanpura commented Jan 12, 2023 • edited Loading

stephen-crawford commented Jan 12, 2023 • edited Loading

DarshitChanpura commented Jan 12, 2023

DarshitChanpura commented Jan 12, 2023 • edited Loading

stephen-crawford commented Jan 13, 2023 • edited Loading

DarshitChanpura commented Jan 13, 2023 • edited Loading

codecov-commenter commented Jan 20, 2023 • edited Loading

Codecov Report

RyanL1997 commented Jan 24, 2023

Update 01/23/2023

Reference

RyanL1997 commented Jan 31, 2023 • edited Loading

Update 01/31/2023

stephen-crawford commented Jan 31, 2023

RyanL1997 commented Jan 31, 2023

peternied Jan 31, 2023

Choose a reason for hiding this comment

RyanL1997 Feb 1, 2023 • edited Loading

Choose a reason for hiding this comment

peternied Feb 1, 2023

Choose a reason for hiding this comment

RyanL1997 Feb 1, 2023

Choose a reason for hiding this comment

peternied Feb 2, 2023 • edited Loading

Choose a reason for hiding this comment

RyanL1997 Feb 2, 2023

Choose a reason for hiding this comment

RyanL1997 Feb 2, 2023

Choose a reason for hiding this comment

peternied Jan 31, 2023

Choose a reason for hiding this comment

RyanL1997 commented Feb 1, 2023 • edited Loading

RyanL1997 commented Feb 1, 2023

cwperks left a comment

Choose a reason for hiding this comment

peternied left a comment

Choose a reason for hiding this comment

stephen-crawford left a comment

Choose a reason for hiding this comment

cwperks commented Feb 3, 2023

RyanL1997 commented Dec 20, 2022 •

edited

Loading

cwperks commented Dec 20, 2022 •

edited

Loading

RyanL1997 commented Jan 12, 2023 •

edited

Loading

DarshitChanpura commented Jan 12, 2023 •

edited

Loading

DarshitChanpura commented Jan 12, 2023 •

edited

Loading

stephen-crawford commented Jan 12, 2023 •

edited

Loading

DarshitChanpura commented Jan 12, 2023 •

edited

Loading

stephen-crawford commented Jan 13, 2023 •

edited

Loading

DarshitChanpura commented Jan 13, 2023 •

edited

Loading

codecov-commenter commented Jan 20, 2023 •

edited

Loading

RyanL1997 commented Jan 31, 2023 •

edited

Loading

RyanL1997 Feb 1, 2023 •

edited

Loading

peternied Feb 2, 2023 •

edited

Loading

RyanL1997 commented Feb 1, 2023 •

edited

Loading