Skip to content

Commit

Permalink
Node14 Complete upgrade (#1034)
Browse files Browse the repository at this point in the history
* Fix Node selection when using built in node (#912)

Fixed node selection to use the node executable instead of the user local node.  

Issue resolved:
#908

Signed-off-by: Sven R <admin@hackacad.net>

* [Release] add axios dependency to UI package

This package was missing from release build so have to add it to this
package.

Brought in by custom branding.

Signed-off-by: Kawika Avilla <kavilla414@gmail.com>

* Add support for codeowners to repo (#935)

Signed-off-by: Ryan Bogan <rbogan@amazon.com>

* [LIC] Allows the simplified header for new files (#936)

* [LIC] Allows the simplified header for new files

Signed-off-by: Ashwin Pc <ashwinpc@amazon.com>

* Updates licence variable names

Signed-off-by: Ashwin Pc <ashwinpc@amazon.com>

Co-authored-by: Ashwin Pc <ashwinpc@amazon.com>

* [Git] update PR template (#937)

Update template to remove javadoc check box since we do not have any
javadocs and include how to run the tests we use to verify the build.

Signed-off-by: Kawika Avilla <kavilla414@gmail.com>

* Add release notes for Dashboards 1.2.0 (#944)

This is the backport PR for #944

Signed-off-by: Neumann <neumanns@amazon.com>

* Add versioned document support in OSD

This is PR is to add versioned document support in OSD.
1. Add logic to pick up doc version from package.json and convert it to `latest` if we are on default `main` branch.
2. Refactor doc_link_service to have 3 urls groups: opensearch, opensearchDashboards, and noDocumentation.
3. Update dynamic versioned doc links and clean up unused urls
4. Fix known url bug  #769
5. Add unit tests for doclinks branch name conversion

Signed-off-by: Zuocheng Ding <zding817@gmail.com>

* [Branding] prevent logging when config not set (#941)

Out of the box, the rendering service will check the config
and see the default value and log an info message saying that
the branding config is invalid or not set. Everytime
you refresh the browser you will get those log messages.

This sets it to only log error messages if the user sets
the branding config and it is invalid.

Include using default messages.

Signed-off-by: Kawika Avilla <kavilla414@gmail.com>

* [Version] Increment to 2.0 (#973)

Version bump from 1.2 to 2.0

Signed-off-by: Kawika Avilla <kavilla414@gmail.com>

* Add Lychee Link Checker into OSD (#938)

1. Fix broken links in OSD
2. Generate lycheeexcude list to filter out false negative warnings from test files or external links
3. Add TODO items for internal unavaiable links
4. Integrate with doc link service change.
5. Standardize all opensearch url with `https://opensearch.org/` and add unavilable urls into noDocument list

Signed-off-by: Zuocheng Ding <zding817@gmail.com>

* Fix Lychee Link Checker Error (#1011)

Signed-off-by: Zuocheng Ding <zding817@gmail.com>

* [CI] Add tests to github workflow

Add unit tests to github workflow and also creating a "bad apples"
environment variable. Some unit tests just fail on the CI for
hardware issues. They should be improved but step one will be
calling out the bad apples.

Also due to the flakiness we can cache the previous run results
and only run the tests that failed. It's too random to catch
with the bad apples mechanism. But still added the continue on
error for unit tests because it takes so long to re-run on the
CI. So instead if it does fail we automatically echo there
was a failure and ask them to re-run. However, if we can get
permission for a github action that can add a comment to the PR
then we could automatically add to PR.

Next step will be improving.

Also needed to limit the amount of workers because otherwise the
hardware can't handle well so then it will accidentally create conflicts.
This means we get an accurate test run but it is slower on the CI.

Included integration tests which worked out of the box.

Included e2e tests as well but it the chrome driver for the application
was different from github's chrome so to run it I just upgraded it for
the test run. Not ideal, ideally we should probably set up a
docker env and install the specific versions since we are now
depending on github's virtual env and the dependencies they installed
there. But at least this is a first pace.

Signed-off-by: Kawika Avilla <kavilla414@gmail.com>

* Add bwc tests for osd with bundle (#871)

tests include the following cases:
verify default page work
verify advanced savings work
verify filter and query work

Disable eslint check
Add eslint-disable comment
Revise license content in plugins and support
Simplify filter and query test
modify test name and fix PR comment
update license header and remove env files
fix timestamp issue
update eslint and license

Particailly Resolved:
opensearch-project/opensearch-build#705

Signed-off-by: Anan Zhuang <ananzh@amazon.com>

* Add more bwc tests for osd without bundles (#900)

This PR adds the following bwc tests:
1) verify sample data work properly for bwc
2) verify timeline visualization work properly for bwc
This PR also simplifies check_filter_and_query bwc test.
It first removes Unique Visitors check because even fix
the time interval, the number of unique visitors number
is random. Then it simplifies this bwc test.

add more tests in check_timeline and modify test names
change one query content to make bwc tests more robust
update license header
add missing test and solve timestamp issue
fix eslint and comments

Partially Resolved:
opensearch-project/opensearch-build#705

Signed-off-by: Anan Zhuang <ananzh@amazon.com>

* Add bwc test data for osd without bundle (#927)

This PR contains 13 zipped bwc test data for osd without bundle.
The data has been tested by osd-1.1 and osd-1.2. To use, here are
the steps:
1)unzip the data to opensearch, for example:
tar -xvf odfe-1.13.2.tar.gz
You need to remove data folder first if there is one in opensearch
2)run opensearch: ./bin/opensearch
3)run dashboards: ./bin/opensearch-dashboards
4)run any cypress test

Partically Resolved:
opensearch-project/opensearch-build#705

Signed-off-by: Anan Zhuang <ananzh@amazon.com>

* Add bwc test data for osd with bundle (#940)

This PR contains 13 zipped bwc test data for osd with bundle.
The data has been tested by osd-1.1 and osd-1.2. To use, here are
the steps:
1)unzip the data to opensearch, for example:
tar -xvf odfe-1.13.2.tar.gz
You need to remove data folder first if there is one in opensearch
2)run opensearch: ./bin/opensearch
3)run dashboards: ./bin/opensearch-dashboards
4)run any cypress test

Besides the above manual process, we now offer a script in this PR:
#931

To run bwc test using osd bundle data, use this command:
./cypress/bwctest-osd.sh
-o /path/to/opensearch.tar.gz
-d /path/to/opensearch-dashboards.tar.gz
-b true

Pls see more details in the above PR.

fix data issue for eCommerse data
resubmit data to fix timestamp issue

Partically Resolved:
opensearch-project/opensearch-build#705

Signed-off-by: Anan Zhuang <ananzh@amazon.com>

* Add more bwc tests for osd with bundles (#901)

This PR adds the following bwc tests:
1)verify sample data work properly for bwc
2)verify timeline visualization work properly for bwc

add more commands check in check_timeline and rename sample data check
minimize the login time and make the tests more robust
change query content to make bwc test more robust
update license header
solve timestamp issue
fix comments and eslint

Partically Resolved:

opensearch-project/opensearch-build#705

Signed-off-by: Anan Zhuang <ananzh@amazon.com>

* [Backwards Compatibility] restore URL forwarding from legacy app

Forwarding legacy app to the current format of the application.
This enables the usage of stored URLs and other links that referenced
the format of the application URL that mentioned the application name.

Since we changed the URL forwarding we changed this value and released.
So incase forks were made and depended on this legacy formatted reference
of the application. It will still work. There are also references of the
application.

Issue resolved:
#1013

Signed-off-by: Kawika Avilla <kavilla414@gmail.com>

* [draft] Node.js upgrade from 10 to 14
* precommit checks are failing with an error related to an lmdb-store dependency
* unit tests are failing with an error related to lmdb-store
* bumped react from 16.12 to 16.14 to resolve unmet peer dependencies, but we still need a resolution to remove the old version
* need a replacement for @elastic/good

Signed-off-by: Bishoy Boktor <boktorbb@amazon.com>

* Adds transformIgnorePattern for lmdb-store and dependencies

Signed-off-by: Bishoy Boktor <boktorbb@amazon.com>

* Removes lmdb-store from transformIgnorePatterns and fixes unit test jest issues (#1012)

Signed-off-by: Bishoy Boktor <boktorbb@amazon.com>

* [CI] Add tests to github workflow

Add unit tests to github workflow and also creating a "bad apples"
environment variable. Some unit tests just fail on the CI for
hardware issues. They should be improved but step one will be
calling out the bad apples.

Also due to the flakiness we can cache the previous run results
and only run the tests that failed. It's too random to catch
with the bad apples mechanism. But still added the continue on
error for unit tests because it takes so long to re-run on the
CI. So instead if it does fail we automatically echo there
was a failure and ask them to re-run. However, if we can get
permission for a github action that can add a comment to the PR
then we could automatically add to PR.

Next step will be improving.

Also needed to limit the amount of workers because otherwise the
hardware can't handle well so then it will accidentally create conflicts.
This means we get an accurate test run but it is slower on the CI.

Included integration tests which worked out of the box.

Included e2e tests as well but it the chrome driver for the application
was different from github's chrome so to run it I just upgraded it for
the test run. Not ideal, ideally we should probably set up a
docker env and install the specific versions since we are now
depending on github's virtual env and the dependencies they installed
there. But at least this is a first pace.

Signed-off-by: Kawika Avilla <kavilla414@gmail.com>

* Skips flaky server metrics collector tests

* Adds `--no-deprecation` flag for integration tests caused by `shot`
which is a downstream dependency of `hapi`.
* The ServerMetricsCollector tests are flaky and rely on the existing
v17 hapi library that Dashboards depends on. This will be upgraded
for the 2.0 release along with the Node.js upgrade.

Signed-off-by: Tommy Markley <markleyt@amazon.com>

* Update node version in github workflow to 14

Signed-off-by: Bishoy Boktor <boktorbb@amazon.com>

* Bumps node to 14.18.2 and uses caret range for @types/node

Signed-off-by: Bishoy Boktor <boktorbb@amazon.com>

* Fix local merge issues

Signed-off-by: Bishoy Boktor <boktorbb@amazon.com>

Co-authored-by: Sven R <admin@hackacad.net>
Co-authored-by: Kawika Avilla <kavilla414@gmail.com>
Co-authored-by: Ryan Bogan <10944539+ryanbogan@users.noreply.github.com>
Co-authored-by: Ashwin P Chandran <ashwinpc1993@gmail.com>
Co-authored-by: Ashwin Pc <ashwinpc@amazon.com>
Co-authored-by: Sean Neumann <1413295+seanneumann@users.noreply.github.com>
Co-authored-by: Zuocheng Ding <zding817@gmail.com>
Co-authored-by: Anan <79961084+ananzh@users.noreply.github.com>
Co-authored-by: Tommy Markley <markleyt@amazon.com>
  • Loading branch information
10 people committed Dec 17, 2021
1 parent 51f496b commit 1eafdb9
Show file tree
Hide file tree
Showing 294 changed files with 4,468 additions and 884 deletions.
27 changes: 20 additions & 7 deletions .eslintrc.js
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,20 @@
* GitHub history for details.
*/

const NEW_OSS_HEADER = `
/**
* For new files created by OpenSearch Contributers
*/
const OSD_HEADER = `
/*
* Copyright OpenSearch Contributors
* SPDX-License-Identifier: Apache-2.0
*/
`;

/**
* For files that modify or use code with an exsting OSS header
*/
const OSS_HEADER = `
/*
* SPDX-License-Identifier: Apache-2.0
*
Expand Down Expand Up @@ -151,7 +164,7 @@ module.exports = {
'@osd/eslint/require-license-header': [
'error',
{
license: NEW_OSS_HEADER,
licenses: [OSD_HEADER, OSS_HEADER],
},
],
'@osd/eslint/disallow-license-headers': [
Expand Down Expand Up @@ -187,7 +200,7 @@ module.exports = {
'@osd/eslint/require-license-header': [
'error',
{
license: SAFER_LODASH_SET_LODASH_HEADER,
licenses: [SAFER_LODASH_SET_LODASH_HEADER],
},
],
'@osd/eslint/disallow-license-headers': [
Expand All @@ -209,14 +222,14 @@ module.exports = {
'@osd/eslint/require-license-header': [
'error',
{
license: SAFER_LODASH_SET_HEADER,
licenses: [SAFER_LODASH_SET_HEADER],
},
],
'@osd/eslint/disallow-license-headers': [
'error',
{
licenses: [
NEW_OSS_HEADER,
OSS_HEADER,
ELASTIC_LICENSE_HEADER,
APACHE_2_0_LICENSE_HEADER,
SAFER_LODASH_SET_LODASH_HEADER,
Expand All @@ -233,14 +246,14 @@ module.exports = {
'@osd/eslint/require-license-header': [
'error',
{
license: SAFER_LODASH_SET_DEFINITELYTYPED_HEADER,
licenses: [SAFER_LODASH_SET_DEFINITELYTYPED_HEADER],
},
],
'@osd/eslint/disallow-license-headers': [
'error',
{
licenses: [
NEW_OSS_HEADER,
OSS_HEADER,
ELASTIC_LICENSE_HEADER,
APACHE_2_0_LICENSE_HEADER,
SAFER_LODASH_SET_HEADER,
Expand Down
2 changes: 2 additions & 0 deletions .github/CODEOWNERS
Validating CODEOWNERS rules …
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
# This should match the owning team set up in https://github.com/orgs/opensearch-project/teams
* @opensearch-project/opensearch-dashboards-core
4 changes: 3 additions & 1 deletion .github/pull_request_template.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,8 @@
### Check List
- [ ] New functionality includes testing.
- [ ] All tests pass
- [ ] `yarn test:jest`
- [ ] `yarn test:jest_integration`
- [ ] `yarn test:ftr`
- [ ] New functionality has been documented.
- [ ] New functionality has javadoc added
- [ ] Commits are signed per the DCO using --signoff
31 changes: 31 additions & 0 deletions .github/workflows/links_checker.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
# Copyright OpenSearch Contributors
# SPDX-License-Identifier: Apache-2.0

name: Link Checker

on:
push:
branches: [ main ]
pull_request:
branches: [ main ]

jobs:
linkchecker:

runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v2
- name: Load Excludes
run: |
LYCHEE_EXCLUDE=$(sed -e :a -e 'N;s/\n/ --exclude /;ta' .lycheeexclude)
echo "LYCHEE_EXCLUDE=$LYCHEE_EXCLUDE" >> $GITHUB_ENV
- name: Lychee Link Checker
id: lychee
uses: lycheeverse/lychee-action@v1.0.9
with:
args: --accept=200,403,429 --exclude ${{ env.LYCHEE_EXCLUDE }} --exclude-mail "**/*.html" "**/*.md" "**/*.txt" "**/*.json" "**/*.js" "**/*.ts" "**/*.tsx"
env:
GITHUB_TOKEN: ${{secrets.GITHUB_TOKEN}}
- name: Fail if there were link errors
run: exit ${{ steps.lychee.outputs.exit_code }}
209 changes: 198 additions & 11 deletions .github/workflows/pr_check_workflow.yml
Original file line number Diff line number Diff line change
@@ -1,25 +1,212 @@
# This workflow will do a clean install of node dependencies, build the source code and run tests across different versions of node
# For more information see: https://help.github.com/actions/language-and-framework-guides/using-nodejs-with-github-actions

name: Node.js CI
name: Build and test

on:
push:
branches: [ main ]
pull_request:
branches: [ main ]

jobs:
build:
env:
CACHE_NAME: osd-node-modules
TEST_BROWSER_HEADLESS: 1
CI: 1
GCS_UPLOAD_PREFIX: fake
TEST_OPENSEARCH_DASHBOARDS_HOST: localhost
TEST_OPENSEARCH_DASHBOARDS_PORT: 6610
TEST_OPENSEARCH_TRANSPORT_PORT: 9403
TEST_OPENSEARCH_PORT: 9400

jobs:
build-lint-test:
runs-on: ubuntu-latest
name: Build and Verify
steps:
# Access a cache of set results from a previous run of the job
# This is to prevent re-running steps that were already successful since it is not native to github actions
# Can be used to verify flaky steps with reduced times
- name: Restore the cached run
uses: actions/cache@v2
with:
path: |
job_successful
linter_results
unit_tests_results
integration_tests_results
key: ${{ github.run_id }}-${{ github.job }}-${{ github.sha }}
restore-keys: |
${{ github.run_id }}-${{ github.job }}-${{ github.sha }}
- name: Get if previous job was successful
id: job_successful
run: cat job_successful 2>/dev/null || echo 'false'

- name: Get the previous linter results
id: linter_results
run: cat linter_results 2>/dev/null || echo 'default'

- name: Get the previous unit tests results
id: unit_tests_results
run: cat unit_tests_results 2>/dev/null || echo 'default'

- name: Get the previous integration tests results
id: integration_tests_results
run: cat integration_tests_results 2>/dev/null || echo 'default'

- name: Checkout code
if: steps.job_successful.outputs.job_successful != 'true'
uses: actions/checkout@v2

- name: Setup Node
if: steps.job_successful.outputs.job_successful != 'true'
uses: actions/setup-node@v2
with:
node-version: "14.18.2"
registry-url: 'https://registry.npmjs.org'

- name: Setup Yarn
if: steps.job_successful.outputs.job_successful != 'true'
run: |
npm uninstall -g yarn
npm i -g yarn@1.22.10
- name: Run bootstrap
if: steps.job_successful.outputs.job_successful != 'true'
run: yarn osd bootstrap

- name: Run linter
if: steps.linter_results.outputs.linter_results != 'success'
id: linter
run: yarn lint

# Runs unit tests while limiting workers because github actions will spawn more than it can handle and crash
# Continues on error but will create a comment on the pull request if this step failed.
- name: Run unit tests
if: steps.unit_tests_results.outputs.unit_tests_results != 'success'
id: unit-tests
continue-on-error: true
run: node scripts/jest --ci --colors --maxWorkers=10
env:
SKIP_BAD_APPLES: true

- run: echo Unit tests completed unsuccessfully. However, unit tests are inconsistent on the CI so please verify locally with `yarn test:jest`.
if: steps.unit_tests_results.outputs.unit_tests_results != 'success' && steps.unit-tests.outcome != 'success'

# TODO: This gets rejected, we need approval to add this
# - name: Add comment if unit tests did not succeed
# if: steps.unit_tests_results.outputs.unit_tests_results != 'success' && steps.unit-tests.outcome != 'success'
# uses: actions/github-script@v5
# with:
# github-token: ${{ secrets.GITHUB_TOKEN }}
# script: |
# github.rest.issues.createComment({
# issue_number: context.issue.number,
# owner: context.repo.owner,
# repo: context.repo.repo,
# body: 'Unit tests completed unsuccessfully. However, unit tests are inconsistent on the CI so please verify locally with `yarn test:jest`.'
# })

- name: Run integration tests
if: steps.integration_tests_results.outputs.integration_tests_results != 'success'
id: integration-tests
run: node --no-deprecation scripts/jest_integration --ci --colors --max-old-space-size=5120

# Set cache if linter, unit tests, and integration tests were successful then the job will be marked successful
# Sets individual results to empower re-runs of the same build without re-running successful steps.
- if: |
(steps.linter.outcome == 'success' || steps.linter.outcome == 'skipped') &&
(steps.unit-tests.outcome == 'success' || steps.unit-tests.outcome == 'skipped') &&
(steps.integration-tests.outcome == 'success' || steps.integration-tests.outcome == 'skipped')
run: echo "::set-output name=job_successful::true" > job_successful
- if: steps.linter.outcome == 'success' || steps.linter.outcome == 'skipped'
run: echo "::set-output name=linter_results::success" > linter_results
- if: steps.unit-tests.outcome == 'success' || steps.unit-tests.outcome == 'skipped'
run: echo "::set-output name=unit_tests_results::success" > unit_tests_results
- if: steps.integration-tests.outcome == 'success' || steps.integration-tests.outcome == 'skipped'
run: echo "::set-output name=integration_tests_results::success" > integration_tests_results
functional-tests:
needs: [ build-lint-test ]
runs-on: ubuntu-latest
name: Run functional tests
strategy:
matrix:
group: [ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12 ]
steps:
- uses: actions/checkout@v2
- name: Use Node.js
uses: actions/setup-node@v2
with:
node-version: '14.18.1'
check-latest: false
- run: yarn osd bootstrap
- run: yarn lint
- run: echo Running functional tests for ciGroup${{ matrix.group }}

# Access a cache of set results from a previous run of the job
# This is to prevent re-running a CI group that was already successful since it is not native to github actions
# Can be used to verify flaky steps with reduced times
- name: Restore the cached run
uses: actions/cache@v2
with:
path: |
ftr_tests_results
key: ${{ github.run_id }}-${{ github.job }}-${{ matrix.group }}-${{ github.sha }}
restore-keys: |
${{ github.run_id }}-${{ github.job }}-${{ matrix.group }}-${{ github.sha }}
- name: Get the cached tests results
id: ftr_tests_results
run: cat ftr_tests_results 2>/dev/null || echo 'default'

- name: Checkout code
if: steps.ftr_tests_results.outputs.ftr_tests_results != 'success'
uses: actions/checkout@v2

- name: Setup Node
if: steps.ftr_tests_results.outputs.ftr_tests_results != 'success'
uses: actions/setup-node@v2
with:
node-version: "14.18.2"
registry-url: 'https://registry.npmjs.org'

- name: Setup Yarn
if: steps.ftr_tests_results.outputs.ftr_tests_results != 'success'
run: |
npm uninstall -g yarn
npm i -g yarn@1.22.10
- name: Get cache path
if: steps.ftr_tests_results.outputs.ftr_tests_results != 'success'
id: cache-path
run: echo "::set-output name=CACHE_DIR::$(yarn cache dir)"

- name: Setup cache
if: steps.ftr_tests_results.outputs.ftr_tests_results != 'success'
uses: actions/cache@v2
with:
path: ${{ steps.cache-path.outputs.CACHE_DIR }}
key: ${{ runner.os }}-yarn-${{ env.CACHE_NAME }}-${{ hashFiles('**/yarn.lock') }}
restore-keys: |
${{ runner.os }}-yarn-${{ env.CACHE_NAME }}-
${{ runner.os }}-yarn-
${{ runner.os }}-
# github virtual env is the latest chrome
- name: Setup chromedriver
if: steps.ftr_tests_results.outputs.ftr_tests_results != 'success'
run: yarn add --dev chromedriver@95.0.0

- name: Run bootstrap
if: steps.ftr_tests_results.outputs.ftr_tests_results != 'success'
run: yarn osd bootstrap

- name: Build plugins
if: steps.ftr_tests_results.outputs.ftr_tests_results != 'success'
run: node scripts/build_opensearch_dashboards_platform_plugins --no-examples --workers 10

- if: steps.ftr_tests_results.outputs.ftr_tests_results != 'success'
id: ftr-tests
run: node scripts/functional_tests.js --config test/functional/config.js --include ciGroup${{ matrix.group }}
env:
CI_GROUP: ciGroup${{ matrix.group }}
CI_PARALLEL_PROCESS_NUMBER: ciGroup${{ matrix.group }}
JOB: ci${{ matrix.group }}
CACHE_DIR: ciGroup${{ matrix.group }}

- if: steps.ftr-tests.outcome == 'success' || steps.ftr-tests.outcome == 'skipped'
run: echo "::set-output name=ftr_tests_results::success" > ftr_tests_results

Loading

0 comments on commit 1eafdb9

Please sign in to comment.