Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve Cloud-Pipeline-Distribution GitHub Actions workflow #3370

Merged
merged 37 commits into from
Nov 8, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
815c45c
Use python 3 for Cloud-Pipeline-Distribution GitHub action
tcibinan Aug 31, 2023
733e35e
Use python 3.9 for Cloud-Pipeline-Distribution GitHub action
tcibinan Sep 1, 2023
732b17b
Exclude ntlmaps from pipe macos build
tcibinan Sep 14, 2023
dd6fa24
Use python 3 for pipe macos build
tcibinan Sep 14, 2023
ca748ba
Clean pyinstaller resources in pipe cli module clean task
tcibinan Sep 14, 2023
6f87372
Publish all pipe macos builds
tcibinan Sep 14, 2023
7a0c683
Use prettytable hooks for macos build only
tcibinan Sep 15, 2023
5eb61b4
Publish all cloud pipeline distributions
tcibinan Sep 15, 2023
4cc5ee2
Add temporary debug logs
tcibinan Sep 15, 2023
ce4ee83
Add temporary debug logs
tcibinan Sep 15, 2023
4c8c6af
Add temporary debug logs
tcibinan Sep 15, 2023
4deed8e
Add temporary debug logs
tcibinan Sep 15, 2023
69cc273
Add temporary debug logs
tcibinan Sep 18, 2023
dacfcc0
Add temporary debug logs
tcibinan Sep 18, 2023
48c047d
Build pipe for each platform separately
tcibinan Sep 18, 2023
9ae8e92
Fix pipe build paths
tcibinan Sep 18, 2023
04201bb
Fix pipe build paths
tcibinan Sep 18, 2023
20d0943
Remove temporary debug logs
tcibinan Sep 18, 2023
f9cba96
Revert unnecessary changes
tcibinan Sep 22, 2023
07c50c2
Use separate scripts to build pipe macos
tcibinan Sep 22, 2023
5e7e5a8
Build client separately
tcibinan Sep 22, 2023
5582b9c
Fix typo
tcibinan Sep 22, 2023
eec14bb
Install node
tcibinan Sep 22, 2023
dd17858
Build cloud data separately
tcibinan Oct 25, 2023
c788a5b
Build fsbrowser separately
tcibinan Oct 25, 2023
c849dee
Build data sharing service separately
tcibinan Oct 25, 2023
d351bfa
Add data sharing service to distribution
tcibinan Oct 25, 2023
eff10b6
Build data sharing service beforehand
tcibinan Oct 25, 2023
148d43c
Skip extra builds
tcibinan Nov 6, 2023
bf2373b
Build gpu stat separately
tcibinan Nov 6, 2023
05c46e5
Build necessary modules only once
tcibinan Nov 6, 2023
1844857
Refactor main workflow
tcibinan Nov 6, 2023
887d86f
Remove node install step from build all job
tcibinan Nov 6, 2023
51fa734
Build data transfer service separately
tcibinan Nov 6, 2023
00460e3
Build necessary modules only once
tcibinan Nov 7, 2023
5ad5757
Transfer s3 files more efficiently
tcibinan Nov 7, 2023
8121e62
Upload gha built distributions to a separate directory in s3
tcibinan Nov 7, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
196 changes: 191 additions & 5 deletions .github/workflows/main.yml
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,42 @@ env:
CLOUD_PIPELINE_BUILD_NUMBER_SEED: 30000

jobs:
CLI_Linux:
runs-on: ubuntu-20.04
steps:
- uses: actions/checkout@v2
- uses: actions/setup-java@v2
with:
distribution: 'zulu'
java-version: '8'
- uses: actions/setup-python@v2
with:
python-version: '3.9'
architecture: 'x64'
- uses: aws-actions/configure-aws-credentials@v1
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: ${{ secrets.AWS_DEFAULT_REGION }}
- run: bash deploy/github_actions/gha_build_pipe_linux.sh
CLI_Linux_el6:
runs-on: ubuntu-20.04
steps:
- uses: actions/checkout@v2
- uses: actions/setup-java@v2
with:
distribution: 'zulu'
java-version: '8'
- uses: actions/setup-python@v2
with:
python-version: '3.9'
architecture: 'x64'
- uses: aws-actions/configure-aws-credentials@v1
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: ${{ secrets.AWS_DEFAULT_REGION }}
- run: bash deploy/github_actions/gha_build_pipe_linux_el6.sh
CLI_MacOS:
runs-on: macos-11
steps:
Expand All @@ -26,17 +62,34 @@ jobs:
java-version: '8'
- uses: actions/setup-python@v2
with:
python-version: '2.7'
python-version: '3.9'
architecture: 'x64'
- uses: aws-actions/configure-aws-credentials@v1
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: ${{ secrets.AWS_DEFAULT_REGION }}
- run: bash deploy/github_actions/gha_build_macos.sh
Build_All:
- run: bash deploy/github_actions/gha_build_pipe_macos.sh
CLI_Windows:
runs-on: ubuntu-20.04
steps:
- uses: actions/checkout@v2
- uses: actions/setup-java@v2
with:
distribution: 'zulu'
java-version: '8'
- uses: actions/setup-python@v2
with:
python-version: '3.9'
architecture: 'x64'
- uses: aws-actions/configure-aws-credentials@v1
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: ${{ secrets.AWS_DEFAULT_REGION }}
- run: bash deploy/github_actions/gha_build_pipe_windows.sh
Build_Client:
runs-on: ubuntu-20.04
needs: CLI_MacOS
steps:
- uses: actions/checkout@v2
- uses: actions/setup-java@v2
Expand All @@ -45,7 +98,7 @@ jobs:
java-version: '8'
- uses: actions/setup-python@v2
with:
python-version: '2.7.18'
python-version: '3.9'
architecture: 'x64'
- uses: actions/setup-node@v3
with:
Expand All @@ -55,4 +108,137 @@ jobs:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: ${{ secrets.AWS_DEFAULT_REGION }}
- run: bash deploy/github_actions/gha_build_client.sh
Build_CloudData_Linux:
runs-on: ubuntu-20.04
steps:
- uses: actions/checkout@v2
- uses: actions/setup-java@v2
with:
distribution: 'zulu'
java-version: '8'
- uses: actions/setup-python@v2
with:
python-version: '3.9'
architecture: 'x64'
- uses: aws-actions/configure-aws-credentials@v1
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: ${{ secrets.AWS_DEFAULT_REGION }}
- run: bash deploy/github_actions/gha_build_cloud_data_linux.sh
Build_CloudData_Windows:
runs-on: ubuntu-20.04
steps:
- uses: actions/checkout@v2
- uses: actions/setup-java@v2
with:
distribution: 'zulu'
java-version: '8'
- uses: actions/setup-python@v2
with:
python-version: '3.9'
architecture: 'x64'
- uses: aws-actions/configure-aws-credentials@v1
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: ${{ secrets.AWS_DEFAULT_REGION }}
- run: bash deploy/github_actions/gha_build_cloud_data_windows.sh
Build_DataSharingService:
runs-on: ubuntu-20.04
steps:
- uses: actions/checkout@v2
- uses: actions/setup-java@v2
with:
distribution: 'zulu'
java-version: '8'
- uses: actions/setup-python@v2
with:
python-version: '3.9'
architecture: 'x64'
- uses: actions/setup-node@v3
with:
node-version: 14
- uses: aws-actions/configure-aws-credentials@v1
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: ${{ secrets.AWS_DEFAULT_REGION }}
- run: bash deploy/github_actions/gha_build_data_sharing_service.sh
Build_DataTransferService:
runs-on: ubuntu-20.04
steps:
- uses: actions/checkout@v2
- uses: actions/setup-java@v2
with:
distribution: 'zulu'
java-version: '8'
- uses: aws-actions/configure-aws-credentials@v1
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: ${{ secrets.AWS_DEFAULT_REGION }}
- run: bash deploy/github_actions/gha_build_data_transfer_service.sh
Build_FSBrowser:
runs-on: ubuntu-20.04
steps:
- uses: actions/checkout@v2
- uses: actions/setup-java@v2
with:
distribution: 'zulu'
java-version: '8'
- uses: actions/setup-python@v2
with:
python-version: '3.9'
architecture: 'x64'
- uses: actions/setup-node@v3
with:
node-version: 14
- uses: aws-actions/configure-aws-credentials@v1
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: ${{ secrets.AWS_DEFAULT_REGION }}
- run: bash deploy/github_actions/gha_build_fs_browser.sh
Build_GpuStat:
runs-on: ubuntu-20.04
steps:
- uses: actions/checkout@v2
- uses: actions/setup-java@v2
with:
distribution: 'zulu'
java-version: '8'
- uses: actions/setup-python@v2
with:
python-version: '3.9'
architecture: 'x64'
- uses: aws-actions/configure-aws-credentials@v1
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: ${{ secrets.AWS_DEFAULT_REGION }}
- run: bash deploy/github_actions/gha_build_gpu_stat.sh
Build_All:
runs-on: ubuntu-20.04
needs: [CLI_Linux, CLI_Linux_el6, CLI_MacOS, CLI_Windows,
Build_Client,
Build_CloudData_Linux, Build_CloudData_Windows,
Build_DataSharingService, Build_DataTransferService,
Build_FSBrowser, Build_GpuStat]
steps:
- uses: actions/checkout@v2
- uses: actions/setup-java@v2
with:
distribution: 'zulu'
java-version: '8'
- uses: actions/setup-python@v2
with:
python-version: '3.9'
architecture: 'x64'
- uses: aws-actions/configure-aws-credentials@v1
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: ${{ secrets.AWS_DEFAULT_REGION }}
- run: sudo -E PATH=$PATH bash deploy/github_actions/gha_pack_dist.sh
32 changes: 32 additions & 0 deletions deploy/github_actions/gha_build_client.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
#!/bin/bash

# Copyright 2017-2023 EPAM Systems, Inc. (https://www.epam.com/)
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

set -e

CLOUD_PIPELINE_BUILD_NUMBER=$(($CLOUD_PIPELINE_BUILD_NUMBER_SEED+$GITHUB_RUN_NUMBER))

./gradlew -PbuildNumber=${CLOUD_PIPELINE_BUILD_NUMBER}.${GITHUB_SHA} \
-Pprofile=release \
client:buildUI \
--no-daemon

tar -czf $PWD/client.tar.gz -C client/build .

ls -lh client.tar.gz

if [ "$GITHUB_REPOSITORY" == "epam/cloud-pipeline" ]; then
aws s3 cp --no-progress client.tar.gz s3://cloud-pipeline-oss-builds/temp/$CLOUD_PIPELINE_BUILD_NUMBER/client.tar.gz
fi
30 changes: 30 additions & 0 deletions deploy/github_actions/gha_build_cloud_data_linux.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
#!/bin/bash

# Copyright 2017-2023 EPAM Systems, Inc. (https://www.epam.com/)
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

set -e

CLOUD_PIPELINE_BUILD_NUMBER=$(($CLOUD_PIPELINE_BUILD_NUMBER_SEED+$GITHUB_RUN_NUMBER))

./gradlew -PbuildNumber=${CLOUD_PIPELINE_BUILD_NUMBER}.${GITHUB_SHA} \
-Pprofile=release \
cloud-pipeline-webdav-client:buildLinux \
--no-daemon

ls -lh cloud-pipeline-webdav-client/out/cloud-data-linux.tar.gz

if [ "$GITHUB_REPOSITORY" == "epam/cloud-pipeline" ]; then
aws s3 cp --no-progress cloud-pipeline-webdav-client/out/cloud-data-linux.tar.gz s3://cloud-pipeline-oss-builds/temp/$CLOUD_PIPELINE_BUILD_NUMBER/cloud-data-linux.tar.gz
fi
30 changes: 30 additions & 0 deletions deploy/github_actions/gha_build_cloud_data_windows.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
#!/bin/bash

# Copyright 2017-2023 EPAM Systems, Inc. (https://www.epam.com/)
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

set -e

CLOUD_PIPELINE_BUILD_NUMBER=$(($CLOUD_PIPELINE_BUILD_NUMBER_SEED+$GITHUB_RUN_NUMBER))

./gradlew -PbuildNumber=${CLOUD_PIPELINE_BUILD_NUMBER}.${GITHUB_SHA} \
-Pprofile=release \
cloud-pipeline-webdav-client:buildWin \
--no-daemon

ls -lh cloud-pipeline-webdav-client/out/cloud-data-win64.zip

if [ "$GITHUB_REPOSITORY" == "epam/cloud-pipeline" ]; then
aws s3 cp --no-progress cloud-pipeline-webdav-client/out/cloud-data-win64.zip s3://cloud-pipeline-oss-builds/temp/$CLOUD_PIPELINE_BUILD_NUMBER/cloud-data-win64.zip
fi
30 changes: 30 additions & 0 deletions deploy/github_actions/gha_build_data_sharing_service.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
#!/bin/bash

# Copyright 2017-2023 EPAM Systems, Inc. (https://www.epam.com/)
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

set -e

CLOUD_PIPELINE_BUILD_NUMBER=$(($CLOUD_PIPELINE_BUILD_NUMBER_SEED+$GITHUB_RUN_NUMBER))

./gradlew -PbuildNumber=${CLOUD_PIPELINE_BUILD_NUMBER}.${GITHUB_SHA} \
-Pprofile=release \
data-sharing-service:buildFast \
--no-daemon

ls -lh data-sharing-service/api/build/libs/data-sharing-service.jar

if [ "$GITHUB_REPOSITORY" == "epam/cloud-pipeline" ]; then
aws s3 cp --no-progress data-sharing-service/api/build/libs/data-sharing-service.jar s3://cloud-pipeline-oss-builds/temp/$CLOUD_PIPELINE_BUILD_NUMBER/data-sharing-service.jar
fi
36 changes: 36 additions & 0 deletions deploy/github_actions/gha_build_data_transfer_service.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
#!/bin/bash

# Copyright 2017-2023 EPAM Systems, Inc. (https://www.epam.com/)
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

set -e

CLOUD_PIPELINE_BUILD_NUMBER=$(($CLOUD_PIPELINE_BUILD_NUMBER_SEED+$GITHUB_RUN_NUMBER))

./gradlew -PbuildNumber=${CLOUD_PIPELINE_BUILD_NUMBER}.${GITHUB_SHA} \
-Pprofile=release \
data-transfer-service:build \
data-transfer-service:bundleWindows \
data-transfer-service:bundleLinux \
--no-daemon

ls -lh data-transfer-service/build/libs/data-transfer-service.jar \
data-transfer-service/build/distributions/data-transfer-service-windows.zip \
data-transfer-service/build/distributions/data-transfer-service-linux.zip

if [ "$GITHUB_REPOSITORY" == "epam/cloud-pipeline" ]; then
aws s3 cp --no-progress data-transfer-service/build/libs/data-transfer-service.jar s3://cloud-pipeline-oss-builds/temp/$CLOUD_PIPELINE_BUILD_NUMBER/data-transfer-service.jar
aws s3 cp --no-progress data-transfer-service/build/distributions/data-transfer-service-windows.zip s3://cloud-pipeline-oss-builds/temp/$CLOUD_PIPELINE_BUILD_NUMBER/data-transfer-service-windows.zip
aws s3 cp --no-progress data-transfer-service/build/distributions/data-transfer-service-linux.zip s3://cloud-pipeline-oss-builds/temp/$CLOUD_PIPELINE_BUILD_NUMBER/data-transfer-service-linux.zip
fi
Loading
Loading