Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(actions): updates to gha workflows #11150

Merged
merged 7 commits into from
Aug 14, 2024
Merged

feat(actions): updates to gha workflows #11150

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
50e7545
feat(actions): updates to gha workflows
david-leifker Aug 12, 2024
2bbe5c2
Update action.yml
david-leifker Aug 13, 2024
31f42b9
Update action.yml
david-leifker Aug 13, 2024
acfc361
Update action.yml
david-leifker Aug 13, 2024
f594525
Update docker-unified.yml
david-leifker Aug 13, 2024
c01240b
Update docker-unified.yml
david-leifker Aug 13, 2024
081e625
Merge branch 'master' into github-actions-1
david-leifker Aug 13, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions .github/actions/ci-optimization/action.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
name: 'Identify CI Optimizations'
description: 'Determine if code changes are specific to certain modules.'
name: "Identify CI Optimizations"
description: "Determine if code changes are specific to certain modules."

outputs:
frontend-only:
Expand Down Expand Up @@ -44,9 +44,10 @@ outputs:
runs:
using: "composite"
steps:
- uses: dorny/paths-filter@v2
- uses: dorny/paths-filter@v3
id: filter
with:
token: "" # Empty token forces it to use raw git commands.
filters: |
frontend:
- "datahub-frontend/**"
Expand Down
25 changes: 16 additions & 9 deletions .github/actions/docker-custom-build-and-push/action.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,10 +26,13 @@ inputs:
build-args:
description: "List of build-time variables. Same as docker/build-push-action"
required: false
tags:
# e.g. latest,head,sha12345
description: "List of tags to use for the Docker image"
image_tag:
# e.g. pr12345 OR head OR v0.1.2.3
description: "Main tag to use for the Docker image"
required: true
flavor:
description: 'Image flavor (e.g., slim, full)'
required: false
target:
description: "Sets the target stage to build"
required: false
Expand All @@ -45,13 +48,17 @@ runs:
steps:
- name: Docker meta
id: docker_meta
uses: crazy-max/ghaction-docker-meta@v1
uses: docker/metadata-action@v5
with:
# list of Docker images to use as base name for tags
images: ${{ inputs.images }}
# add git short SHA as Docker tag
tag-custom: ${{ inputs.tags }}
tag-custom-only: true
flavor: |
latest=false
suffix=${{ inputs.flavor && format('-{0}', inputs.flavor) || '' }}
tags: |
type=raw,value=${{ inputs.image_tag }}
type=raw,value=head,enable=${{ github.ref == format('refs/heads/{0}', 'acryl-main') }}
type=ref,event=pr,prefix=pr
type=sha,prefix=,format=short

# Code for testing the build when not pushing to Docker Hub.
- name: Build and Load image for testing (if not publishing)
Expand All @@ -75,7 +82,7 @@ runs:
shell: bash
run: |
TAGS="""
${{ steps.docker_meta.outputs.tags }}
${{ inputs.image_tag }}
"""
echo "SINGLE_TAG=$(echo $TAGS | tr '\n' ' ' | awk -F' ' '{ print $1 }')" >> $GITHUB_OUTPUT
id: single_tag
Expand Down
12 changes: 10 additions & 2 deletions .github/scripts/docker_helpers.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,14 +5,14 @@ export MAIN_BRANCH="master"
export MAIN_BRANCH_TAG="head"

function get_short_sha {
echo $(git rev-parse --short "$GITHUB_SHA")
echo $(git rev-parse --short "$GITHUB_SHA"|head -c7)
}

export SHORT_SHA=$(get_short_sha)
echo "SHORT_SHA: $SHORT_SHA"

function get_tag {
echo $(echo ${GITHUB_REF} | sed -e "s,refs/heads/${MAIN_BRANCH},${MAIN_BRANCH_TAG},g" -e 's,refs/tags/,,g' -e 's,refs/pull/\([0-9]*\).*,pr\1,g'),${SHORT_SHA}
echo $(echo ${GITHUB_REF} | sed -e "s,refs/heads/${MAIN_BRANCH},${MAIN_BRANCH_TAG},g" -e 's,refs/tags/,,g' -e 's,refs/pull/\([0-9]*\).*,pr\1,g')
}

function get_tag_slim {
Expand All @@ -38,3 +38,11 @@ function get_unique_tag_slim {
function get_unique_tag_full {
echo $(echo ${GITHUB_REF} | sed -e "s,refs/heads/${MAIN_BRANCH},${SHORT_SHA}-full,g" -e 's,refs/tags/\(.*\),\1-full,g' -e 's,refs/pull/\([0-9]*\).*,pr\1-full,g')
}

function get_platforms_based_on_branch {
if [ "${{ github.event_name }}" == 'push' && "${{ github.ref }}" == "refs/heads/${MAIN_BRANCH}" ]; then
echo "linux/amd64,linux/arm64"
else
echo "linux/amd64"
fi
}
5 changes: 5 additions & 0 deletions .github/workflows/build-and-test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,11 @@ jobs:
timeout-minutes: 60
needs: setup
steps:
- name: Free up disk space
run: |
sudo apt-get remove 'dotnet-*' azure-cli || true
sudo rm -rf /usr/local/lib/android/ || true
sudo docker image prune -a -f || true
- uses: szenius/set-timezone@v1.2
with:
timezoneLinux: ${{ matrix.timezone }}
Expand Down
49 changes: 24 additions & 25 deletions .github/workflows/docker-unified.yml
View file
Open in desktop
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[actionlint] reported by reviewdog 🐶
shellcheck reported issue in this script: SC2129:style:2:1: Consider using { cmd1; cmd2; } >> file instead of individual redirects [shellcheck]

datahub/.github/workflows/docker-unified.yml

Line 68 in 1f70ba2

run: |

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[actionlint] reported by reviewdog 🐶
shellcheck reported issue in this script: SC2086:info:2:34: Double quote to prevent globbing and word splitting [shellcheck]

datahub/.github/workflows/docker-unified.yml

Line 68 in 1f70ba2

run: |

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[actionlint] reported by reviewdog 🐶
shellcheck reported issue in this script: SC2086:info:3:26: Double quote to prevent globbing and word splitting [shellcheck]

datahub/.github/workflows/docker-unified.yml

Line 68 in 1f70ba2

run: |

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[actionlint] reported by reviewdog 🐶
shellcheck reported issue in this script: SC2086:info:4:36: Double quote to prevent globbing and word splitting [shellcheck]

datahub/.github/workflows/docker-unified.yml

Line 68 in 1f70ba2

run: |

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[actionlint] reported by reviewdog 🐶
shellcheck reported issue in this script: SC2086:info:5:36: Double quote to prevent globbing and word splitting [shellcheck]

datahub/.github/workflows/docker-unified.yml

Line 68 in 1f70ba2

run: |

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[actionlint] reported by reviewdog 🐶
shellcheck reported issue in this script: SC2086:info:6:40: Double quote to prevent globbing and word splitting [shellcheck]

datahub/.github/workflows/docker-unified.yml

Line 68 in 1f70ba2

run: |

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[actionlint] reported by reviewdog 🐶
shellcheck reported issue in this script: SC2086:info:7:50: Double quote to prevent globbing and word splitting [shellcheck]

datahub/.github/workflows/docker-unified.yml

Line 68 in 1f70ba2

run: |

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[actionlint] reported by reviewdog 🐶
shellcheck reported issue in this script: SC2086:info:8:50: Double quote to prevent globbing and word splitting [shellcheck]

datahub/.github/workflows/docker-unified.yml

Line 68 in 1f70ba2

run: |

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[actionlint] reported by reviewdog 🐶
shellcheck reported issue in this script: SC2086:info:9:65: Double quote to prevent globbing and word splitting [shellcheck]

datahub/.github/workflows/docker-unified.yml

Line 68 in 1f70ba2

run: |

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[actionlint] reported by reviewdog 🐶
shellcheck reported issue in this script: SC2086:info:10:69: Double quote to prevent globbing and word splitting [shellcheck]

datahub/.github/workflows/docker-unified.yml

Line 68 in 1f70ba2

run: |

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[actionlint] reported by reviewdog 🐶
shellcheck reported issue in this script: SC2086:info:11:51: Double quote to prevent globbing and word splitting [shellcheck]

datahub/.github/workflows/docker-unified.yml

Line 68 in 50e7545

run: |

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[actionlint] reported by reviewdog 🐶
shellcheck reported issue in this script: SC2086:info:2:55: Double quote to prevent globbing and word splitting [shellcheck]

datahub/.github/workflows/docker-unified.yml

Line 84 in 50e7545

run: |

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[actionlint] reported by reviewdog 🐶
shellcheck reported issue in this script: SC2086:info:2:45: Double quote to prevent globbing and word splitting [shellcheck]

datahub/.github/workflows/docker-unified.yml

Line 95 in 50e7545

run: |

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[actionlint] reported by reviewdog 🐶
shellcheck reported issue in this script: SC2086:info:2:45: Double quote to prevent globbing and word splitting [shellcheck]

datahub/.github/workflows/docker-unified.yml

Line 106 in 50e7545

run: |

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[actionlint] reported by reviewdog 🐶
shellcheck reported issue in this script: SC2086:info:1:118: Double quote to prevent globbing and word splitting [shellcheck]

datahub/.github/workflows/docker-unified.yml

Line 547 in 50e7545

run: echo "tag=${{ needs.setup.outputs.ingestion_base_change == 'true' && needs.setup.outputs.unique_tag || 'head' }}" >> $GITHUB_OUTPUT

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[actionlint] reported by reviewdog 🐶
shellcheck reported issue in this script: SC2086:info:1:128: Double quote to prevent globbing and word splitting [shellcheck]

datahub/.github/workflows/docker-unified.yml

Line 588 in 50e7545

run: echo "tag=${{ needs.setup.outputs.ingestion_base_change == 'true' && needs.setup.outputs.unique_slim_tag || 'head-slim' }}" >> $GITHUB_OUTPUT

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[actionlint] reported by reviewdog 🐶
missing input "image_tag" which is required by action "Custom Docker build and push" defined at "./.github/actions/docker-custom-build-and-push". all required inputs are "image_tag", "images" [action]

datahub/.github/workflows/docker-unified.yml

Line 612 in 50e7545

uses: ./.github/actions/docker-custom-build-and-push

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[actionlint] reported by reviewdog 🐶
input "tags" is not defined in action "Custom Docker build and push" defined at "./.github/actions/docker-custom-build-and-push". available inputs are "build-args", "context", "file", "flavor", "image_tag", "images", "password", "platforms", "publish", "target", "username" [action]

datahub/.github/workflows/docker-unified.yml

Line 617 in 50e7545

tags: ${{ needs.setup.outputs.full_tag }}

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[actionlint] reported by reviewdog 🐶
shellcheck reported issue in this script: SC2086:info:1:123: Double quote to prevent globbing and word splitting [shellcheck]

datahub/.github/workflows/docker-unified.yml

Line 629 in 50e7545

run: echo "tag=${{ needs.setup.outputs.ingestion_base_change == 'true' && needs.setup.outputs.unique_full_tag || 'head' }}" >> $GITHUB_OUTPUT

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[actionlint] reported by reviewdog 🐶
shellcheck reported issue in this script: SC2086:info:1:123: Double quote to prevent globbing and word splitting [shellcheck]

datahub/.github/workflows/docker-unified.yml

Line 683 in 50e7545

run: echo "tag=${{ needs.setup.outputs.ingestion_change == 'true' && needs.setup.outputs.unique_slim_tag || 'head-slim' }}" >> $GITHUB_OUTPUT

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[actionlint] reported by reviewdog 🐶
shellcheck reported issue in this script: SC2086:info:1:113: Double quote to prevent globbing and word splitting [shellcheck]

datahub/.github/workflows/docker-unified.yml

Line 769 in 2bbe5c2

run: echo "tag=${{ needs.setup.outputs.ingestion_change == 'true' && needs.setup.outputs.unique_tag || 'head' }}" >> $GITHUB_OUTPUT

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[actionlint] reported by reviewdog 🐶
shellcheck reported issue in this script: SC2086:info:2:55: Double quote to prevent globbing and word splitting [shellcheck]

datahub/.github/workflows/docker-unified.yml

Line 811 in 2bbe5c2

run: |

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[actionlint] reported by reviewdog 🐶
shellcheck reported issue in this script: SC2086:info:4:63: Double quote to prevent globbing and word splitting [shellcheck]

datahub/.github/workflows/docker-unified.yml

Line 811 in 2bbe5c2

run: |

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[actionlint] reported by reviewdog 🐶
shellcheck reported issue in this script: SC2086:info:6:95: Double quote to prevent globbing and word splitting [shellcheck]

datahub/.github/workflows/docker-unified.yml

Line 811 in 2bbe5c2

run: |

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[actionlint] reported by reviewdog 🐶
shellcheck reported issue in this script: SC2086:info:8:24: Double quote to prevent globbing and word splitting [shellcheck]

datahub/.github/workflows/docker-unified.yml

Line 811 in 2bbe5c2

run: |

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[actionlint] reported by reviewdog 🐶
property "short_sha" is not defined in object type {backend_change: string; backend_only: string; branch_name: string; docker-login: string; elasticsearch_setup_change: string; frontend_change: string; frontend_only: string; full_tag: string; ingestion_base_change: string; ingestion_change: string; ingestion_only: string; kafka_setup_change: string; mysql_setup_change: string; postgres_setup_change: string; pr-publish: string; publish: string; python_release_version: string; repository_name: string; slim_tag: string; tag: string; unique_full_tag: string; unique_slim_tag: string; unique_tag: string} [expression]

datahub/.github/workflows/docker-unified.yml

Line 1061 in 2bbe5c2

message: '{ "command": "git-sync", "args" : {"repoName": "${{ needs.setup.outputs.repository_name }}", "repoOrg": "${{ github.repository_owner }}", "repoBranch": "${{ needs.setup.outputs.branch_name }}", "repoShaShort": "${{ needs.setup.outputs.short_sha }}" }}'

Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,6 @@ jobs:
publish: ${{ steps.publish.outputs.publish }}
pr-publish: ${{ steps.pr-publish.outputs.publish }}
python_release_version: ${{ steps.tag.outputs.python_release_version }}
short_sha: ${{ steps.tag.outputs.short_sha }}
branch_name: ${{ steps.tag.outputs.branch_name }}
repository_name: ${{ steps.tag.outputs.repository_name }}
frontend_change: ${{ steps.ci-optimize.outputs.frontend-change == 'true' }}
Expand Down Expand Up @@ -157,7 +156,7 @@ jobs:
with:
images: |
${{ env.DATAHUB_GMS_IMAGE }}
tags: ${{ needs.setup.outputs.tag }}
image_tag: ${{ needs.setup.outputs.tag }}
username: ${{ secrets.ACRYL_DOCKER_USERNAME }}
password: ${{ secrets.ACRYL_DOCKER_PASSWORD }}
publish: ${{ needs.setup.outputs.publish == 'true' || needs.setup.outputs.pr-publish == 'true' }}
Expand Down Expand Up @@ -221,7 +220,7 @@ jobs:
with:
images: |
${{ env.DATAHUB_MAE_CONSUMER_IMAGE }}
tags: ${{ needs.setup.outputs.tag }}
image_tag: ${{ needs.setup.outputs.tag }}
username: ${{ secrets.ACRYL_DOCKER_USERNAME }}
password: ${{ secrets.ACRYL_DOCKER_PASSWORD }}
publish: ${{ needs.setup.outputs.publish == 'true' || needs.setup.outputs.pr-publish == 'true' }}
Expand Down Expand Up @@ -285,7 +284,7 @@ jobs:
with:
images: |
${{ env.DATAHUB_MCE_CONSUMER_IMAGE }}
tags: ${{ needs.setup.outputs.tag }}
image_tag: ${{ needs.setup.outputs.tag }}
username: ${{ secrets.ACRYL_DOCKER_USERNAME }}
password: ${{ secrets.ACRYL_DOCKER_PASSWORD }}
publish: ${{ needs.setup.outputs.publish == 'true' || needs.setup.outputs.pr-publish == 'true' }}
Expand Down Expand Up @@ -349,7 +348,7 @@ jobs:
with:
images: |
${{ env.DATAHUB_UPGRADE_IMAGE }}
tags: ${{ needs.setup.outputs.tag }}
image_tag: ${{ needs.setup.outputs.tag }}
username: ${{ secrets.ACRYL_DOCKER_USERNAME }}
password: ${{ secrets.ACRYL_DOCKER_PASSWORD }}
publish: ${{ needs.setup.outputs.publish == 'true' || needs.setup.outputs.pr-publish == 'true' }}
Expand Down Expand Up @@ -394,7 +393,7 @@ jobs:
name: Build and Push DataHub Frontend Docker Image
runs-on: ubuntu-latest
needs: setup
if: ${{ needs.setup.outputs.frontend_change == 'true' || needs.setup.outputs.publish == 'true' }}
if: ${{ needs.setup.outputs.frontend_change == 'true' || needs.setup.outputs.publish == 'true' || needs.setup.outputs.pr-publish == 'true'}}
steps:
- name: Set up JDK 17
uses: actions/setup-java@v3
Expand All @@ -415,7 +414,7 @@ jobs:
with:
images: |
${{ env.DATAHUB_FRONTEND_IMAGE }}
tags: ${{ needs.setup.outputs.tag }}
image_tag: ${{ needs.setup.outputs.tag }}
username: ${{ secrets.ACRYL_DOCKER_USERNAME }}
password: ${{ secrets.ACRYL_DOCKER_PASSWORD }}
publish: ${{ needs.setup.outputs.publish == 'true' || needs.setup.outputs.pr-publish == 'true' }}
Expand Down Expand Up @@ -469,7 +468,7 @@ jobs:
with:
images: |
${{ env.DATAHUB_KAFKA_SETUP_IMAGE }}
tags: ${{ needs.setup.outputs.tag }}
image_tag: ${{ needs.setup.outputs.tag }}
username: ${{ secrets.ACRYL_DOCKER_USERNAME }}
password: ${{ secrets.ACRYL_DOCKER_PASSWORD }}
publish: ${{ needs.setup.outputs.publish == 'true' || needs.setup.outputs.pr-publish == 'true' }}
Expand All @@ -490,7 +489,7 @@ jobs:
with:
images: |
${{ env.DATAHUB_MYSQL_SETUP_IMAGE }}
tags: ${{ needs.setup.outputs.tag }}
image_tag: ${{ needs.setup.outputs.tag }}
username: ${{ secrets.ACRYL_DOCKER_USERNAME }}
password: ${{ secrets.ACRYL_DOCKER_PASSWORD }}
publish: ${{ needs.setup.outputs.publish == 'true' || needs.setup.outputs.pr-publish == 'true' }}
Expand All @@ -511,7 +510,7 @@ jobs:
with:
images: |
${{ env.DATAHUB_ELASTIC_SETUP_IMAGE }}
tags: ${{ needs.setup.outputs.tag }}
image_tag: ${{ needs.setup.outputs.tag }}
username: ${{ secrets.ACRYL_DOCKER_USERNAME }}
password: ${{ secrets.ACRYL_DOCKER_PASSWORD }}
publish: ${{ needs.setup.outputs.publish == 'true' || needs.setup.outputs.pr-publish == 'true' }}
Expand All @@ -525,7 +524,7 @@ jobs:
outputs:
tag: ${{ steps.tag.outputs.tag }}
needs: setup
if: ${{ needs.setup.outputs.ingestion_change == 'true' || needs.setup.outputs.publish == 'true' }}
if: ${{ needs.setup.outputs.ingestion_change == 'true' || needs.setup.outputs.publish == 'true' || needs.setup.outputs.pr-publish == 'true' }}
steps:
- name: Check out the repo
uses: acryldata/sane-checkout-action@v3
Expand All @@ -536,7 +535,7 @@ jobs:
target: base
images: |
${{ env.DATAHUB_INGESTION_BASE_IMAGE }}
tags: ${{ needs.setup.outputs.tag }}
image_tag: ${{ needs.setup.outputs.tag }}
username: ${{ secrets.ACRYL_DOCKER_USERNAME }}
password: ${{ secrets.ACRYL_DOCKER_PASSWORD }}
publish: ${{ needs.setup.outputs.publish == 'true' || needs.setup.outputs.pr-publish == 'true' }}
Expand All @@ -552,7 +551,7 @@ jobs:
outputs:
tag: ${{ steps.tag.outputs.tag }}
needs: [setup, datahub_ingestion_base_build]
if: ${{ needs.setup.outputs.ingestion_change == 'true' || needs.setup.outputs.publish == 'true' }}
if: ${{ needs.setup.outputs.ingestion_change == 'true' || needs.setup.outputs.publish == 'true' || needs.setup.outputs.pr-publish == 'true' }}
steps:
- name: Check out the repo
uses: acryldata/sane-checkout-action@v3
Expand All @@ -574,7 +573,7 @@ jobs:
target: slim-install
images: |
${{ env.DATAHUB_INGESTION_BASE_IMAGE }}
tags: ${{ needs.setup.outputs.slim_tag }}
image_tag: ${{ needs.setup.outputs.slim_tag }}
username: ${{ secrets.ACRYL_DOCKER_USERNAME }}
password: ${{ secrets.ACRYL_DOCKER_PASSWORD }}
build-args: |
Expand All @@ -593,7 +592,7 @@ jobs:
outputs:
tag: ${{ steps.tag.outputs.tag }}
needs: [setup, datahub_ingestion_base_build]
if: ${{ needs.setup.outputs.ingestion_change == 'true' || needs.setup.outputs.publish == 'true' }}
if: ${{ needs.setup.outputs.ingestion_change == 'true' || needs.setup.outputs.publish == 'true' || needs.setup.outputs.pr-publish == 'true' }}
steps:
- name: Check out the repo
uses: acryldata/sane-checkout-action@v3
Expand Down Expand Up @@ -636,7 +635,7 @@ jobs:
tag: ${{ steps.tag.outputs.tag }}
needs_artifact_download: ${{ needs.setup.outputs.ingestion_change == 'true' && ( needs.setup.outputs.publish != 'true' && needs.setup.outputs.pr-publish != 'true') }}
needs: [setup, datahub_ingestion_base_slim_build]
if: ${{ needs.setup.outputs.ingestion_change == 'true' || needs.setup.outputs.publish == 'true' }}
if: ${{ needs.setup.outputs.ingestion_change == 'true' || needs.setup.outputs.publish == 'true' || needs.setup.outputs.pr-publish == 'true' }}
steps:
- name: Set up JDK 17
uses: actions/setup-java@v3
Expand All @@ -647,7 +646,7 @@ jobs:
- name: Check out the repo
uses: acryldata/sane-checkout-action@v3
- name: Build codegen
if: ${{ needs.setup.outputs.ingestion_change == 'true' || needs.setup.outputs.publish == 'true' }}
if: ${{ needs.setup.outputs.ingestion_change == 'true' || needs.setup.outputs.publish == 'true' || needs.setup.outputs.pr-publish =='true' }}
run: ./gradlew :metadata-ingestion:codegen
- name: Download Base Image
uses: ishworkh/docker-image-artifact-download@v1
Expand All @@ -661,7 +660,7 @@ jobs:
username: ${{ secrets.ACRYL_DOCKER_USERNAME }}
password: ${{ secrets.ACRYL_DOCKER_PASSWORD }}
- name: Build and push Slim Image
if: ${{ needs.setup.outputs.ingestion_change == 'true' || needs.setup.outputs.publish == 'true' }}
if: ${{ needs.setup.outputs.ingestion_change == 'true' || needs.setup.outputs.publish == 'true' || needs.setup.outputs.pr-publish == 'true' }}
uses: ./.github/actions/docker-custom-build-and-push
with:
target: final
Expand All @@ -672,7 +671,7 @@ jobs:
DOCKER_VERSION=${{ needs.setup.outputs.ingestion_base_change == 'true' && needs.setup.outputs.unique_slim_tag || 'head-slim' }}
RELEASE_VERSION=${{ needs.setup.outputs.python_release_version }}
APP_ENV=slim
tags: ${{ needs.setup.outputs.slim_tag }}
image_tag: ${{ needs.setup.outputs.slim_tag }}
username: ${{ secrets.ACRYL_DOCKER_USERNAME }}
password: ${{ secrets.ACRYL_DOCKER_PASSWORD }}
publish: ${{ needs.setup.outputs.publish == 'true' || needs.setup.outputs.pr-publish == 'true' }}
Expand Down Expand Up @@ -723,7 +722,7 @@ jobs:
tag: ${{ steps.tag.outputs.tag }}
needs_artifact_download: ${{ needs.setup.outputs.ingestion_change == 'true' && ( needs.setup.outputs.publish != 'true' && needs.setup.outputs.pr-publish != 'true' ) }}
needs: [setup, datahub_ingestion_base_full_build]
if: ${{ needs.setup.outputs.ingestion_change == 'true' || needs.setup.outputs.publish == 'true' }}
if: ${{ needs.setup.outputs.ingestion_change == 'true' || needs.setup.outputs.publish == 'true' || needs.setup.outputs.pr-publish == 'true' }}
steps:
- name: Set up JDK 17
uses: actions/setup-java@v3
Expand All @@ -734,7 +733,7 @@ jobs:
- name: Check out the repo
uses: acryldata/sane-checkout-action@v3
- name: Build codegen
if: ${{ needs.setup.outputs.ingestion_change == 'true' || needs.setup.outputs.publish == 'true' }}
if: ${{ needs.setup.outputs.ingestion_change == 'true' || needs.setup.outputs.publish == 'true' || needs.setup.outputs.pr-publish == 'true' }}
run: ./gradlew :metadata-ingestion:codegen
- name: Download Base Image
uses: ishworkh/docker-image-artifact-download@v1
Expand All @@ -748,7 +747,7 @@ jobs:
username: ${{ secrets.ACRYL_DOCKER_USERNAME }}
password: ${{ secrets.ACRYL_DOCKER_PASSWORD }}
- name: Build and push Full Image
if: ${{ needs.setup.outputs.ingestion_change == 'true' || needs.setup.outputs.publish == 'true' }}
if: ${{ needs.setup.outputs.ingestion_change == 'true' || needs.setup.outputs.publish == 'true' || needs.setup.outputs.pr-publish == 'true' }}
uses: ./.github/actions/docker-custom-build-and-push
with:
target: final
Expand All @@ -758,7 +757,7 @@ jobs:
BASE_IMAGE=${{ env.DATAHUB_INGESTION_BASE_IMAGE }}
DOCKER_VERSION=${{ needs.setup.outputs.ingestion_base_change == 'true' && needs.setup.outputs.unique_tag || 'head' }}
RELEASE_VERSION=${{ needs.setup.outputs.python_release_version }}
tags: ${{ needs.setup.outputs.tag }}
image_tag: ${{ needs.setup.outputs.tag }}
username: ${{ secrets.ACRYL_DOCKER_USERNAME }}
password: ${{ secrets.ACRYL_DOCKER_PASSWORD }}
publish: ${{ needs.setup.outputs.publish == 'true' || needs.setup.outputs.pr-publish == 'true' }}
Expand All @@ -776,7 +775,7 @@ jobs:
name: "[Monitoring] Scan Datahub Ingestion images for vulnerabilities"
runs-on: ubuntu-latest
needs: [setup, datahub_ingestion_full_build]
if: ${{ needs.setup.outputs.ingestion_change == 'true' || needs.setup.outputs.publish == 'true' }}
if: ${{ needs.setup.outputs.ingestion_change == 'true' || needs.setup.outputs.publish == 'true' || needs.setup.outputs.pr-publish == 'true' }}
steps:
- name: Checkout # adding checkout step just to make trivy upload happy
uses: acryldata/sane-checkout-action@v3
Expand Down Expand Up @@ -1049,7 +1048,7 @@ jobs:
runs-on: ubuntu-latest
needs: [setup, smoke_test]
steps:
- uses: aws-actions/configure-aws-credentials@v1
- uses: aws-actions/configure-aws-credentials@v4
if: ${{ needs.setup.outputs.publish != 'false' && github.repository_owner == 'datahub-project' && needs.setup.outputs.repository_name == 'datahub' }}
with:
aws-access-key-id: ${{ secrets.AWS_SQS_ACCESS_KEY_ID }}
Expand Down
5 changes: 5 additions & 0 deletions .github/workflows/lint-actions.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,3 +14,8 @@ jobs:
- uses: reviewdog/action-actionlint@v1
with:
reporter: github-pr-review
permissions:
contents: read
checks: write
pull-requests: write
issues: write
5 changes: 5 additions & 0 deletions .github/workflows/metadata-ingestion.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,11 @@ jobs:
- python-version: "3.10"
fail-fast: false
steps:
- name: Free up disk space
run: |
sudo apt-get remove 'dotnet-*' azure-cli || true
sudo rm -rf /usr/local/lib/android/ || true
sudo docker image prune -a -f || true
- name: Set up JDK 17
uses: actions/setup-java@v3
with:
Expand Down
5 changes: 5 additions & 0 deletions .github/workflows/metadata-io.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,11 @@ jobs:
timeout-minutes: 60
needs: setup
steps:
- name: Free up disk space
run: |
sudo apt-get remove 'dotnet-*' azure-cli || true
sudo rm -rf /usr/local/lib/android/ || true
sudo docker image prune -a -f || true
- uses: acryldata/sane-checkout-action@v3
- name: Set up JDK 17
uses: actions/setup-java@v3
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/metadata-model.yml
View file
Open in desktop
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[actionlint] reported by reviewdog 🐶
shellcheck reported issue in this script: SC2086:info:2:51: Double quote to prevent globbing and word splitting [shellcheck]

datahub/.github/workflows/metadata-model.yml

Line 25 in 2bbe5c2

run: |

Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ jobs:
run: ./gradlew :metadata-ingestion:modelDocGen
- name: Configure AWS Credentials
if: ${{ needs.setup.outputs.publish == 'true' }}
uses: aws-actions/configure-aws-credentials@v3
uses: aws-actions/configure-aws-credentials@v4
with:
aws-access-key-id: ${{ secrets.ACRYL_CI_ARTIFACTS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.ACRYL_CI_ARTIFACTS_ACCESS_KEY }}
Expand Down
3 changes: 3 additions & 0 deletions .github/workflows/publish-datahub-jars.yml
View file
Open in desktop
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[actionlint] reported by reviewdog 🐶
shellcheck reported issue in this script: SC2086:info:2:48: Double quote to prevent globbing and word splitting [shellcheck]

datahub/.github/workflows/publish-datahub-jars.yml

Line 29 in 2bbe5c2

run: |

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[actionlint] reported by reviewdog 🐶
shellcheck reported issue in this script: SC2001:style:2:7: See if you can use ${variable//search/replace} instead [shellcheck]

datahub/.github/workflows/publish-datahub-jars.yml

Line 42 in 2bbe5c2

run: |

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[actionlint] reported by reviewdog 🐶
shellcheck reported issue in this script: SC2086:info:2:12: Double quote to prevent globbing and word splitting [shellcheck]

datahub/.github/workflows/publish-datahub-jars.yml

Line 42 in 2bbe5c2

run: |

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[actionlint] reported by reviewdog 🐶
shellcheck reported issue in this script: SC2086:info:3:20: Double quote to prevent globbing and word splitting [shellcheck]

datahub/.github/workflows/publish-datahub-jars.yml

Line 42 in 31f42b9

run: |

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[actionlint] reported by reviewdog 🐶
shellcheck reported issue in this script: SC2086:info:1:17: Double quote to prevent globbing and word splitting [shellcheck]

datahub/.github/workflows/publish-datahub-jars.yml

Line 78 in 31f42b9

run: |

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[actionlint] reported by reviewdog 🐶
shellcheck reported issue in this script: SC2086:info:1:17: Double quote to prevent globbing and word splitting [shellcheck]

datahub/.github/workflows/publish-datahub-jars.yml

Line 91 in 31f42b9

run: |

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[actionlint] reported by reviewdog 🐶
shellcheck reported issue in this script: SC2086:info:1:17: Double quote to prevent globbing and word splitting [shellcheck]

datahub/.github/workflows/publish-datahub-jars.yml

Line 104 in 31f42b9

run: |

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[actionlint] reported by reviewdog 🐶
shellcheck reported issue in this script: SC2086:info:1:17: Double quote to prevent globbing and word splitting [shellcheck]

datahub/.github/workflows/publish-datahub-jars.yml

Line 117 in 31f42b9

run: |

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[actionlint] reported by reviewdog 🐶
shellcheck reported issue in this script: SC2086:info:1:17: Double quote to prevent globbing and word splitting [shellcheck]

datahub/.github/workflows/publish-datahub-jars.yml

Line 130 in 31f42b9

run: |

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[actionlint] reported by reviewdog 🐶
shellcheck reported issue in this script: SC2086:info:1:17: Double quote to prevent globbing and word splitting [shellcheck]

datahub/.github/workflows/publish-datahub-jars.yml

Line 143 in 31f42b9

run: |

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[actionlint] reported by reviewdog 🐶
shellcheck reported issue in this script: SC2086:info:1:17: Double quote to prevent globbing and word splitting [shellcheck]

datahub/.github/workflows/publish-datahub-jars.yml

Line 156 in 31f42b9

run: |

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[actionlint] reported by reviewdog 🐶
shellcheck reported issue in this script: SC2086:info:1:17: Double quote to prevent globbing and word splitting [shellcheck]

datahub/.github/workflows/publish-datahub-jars.yml

Line 169 in 31f42b9

run: |

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[actionlint] reported by reviewdog 🐶
shellcheck reported issue in this script: SC2086:info:1:17: Double quote to prevent globbing and word splitting [shellcheck]

datahub/.github/workflows/publish-datahub-jars.yml

Line 182 in 31f42b9

run: |

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[actionlint] reported by reviewdog 🐶
shellcheck reported issue in this script: SC2086:info:1:17: Double quote to prevent globbing and word splitting [shellcheck]

datahub/.github/workflows/publish-datahub-jars.yml

Line 195 in acfc361

run: |

Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,9 @@ jobs:
echo "tag=$TAG" >> $GITHUB_OUTPUT
publish:
runs-on: ubuntu-latest
permissions:
id-token: write
contents: read
needs: ["check-secret", "setup"]
if: ${{ needs.check-secret.outputs.publish-enabled == 'true' }}
steps:
Expand Down
7 changes: 5 additions & 2 deletions .github/workflows/spark-smoke-test.yml
View file
Open in desktop
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[actionlint] reported by reviewdog 🐶
property "test_strategy" is not defined in object type {} [expression]

datahub/.github/workflows/spark-smoke-test.yml

Line 66 in acfc361

run: |

Original file line number Diff line number Diff line change
Expand Up @@ -44,8 +44,11 @@ jobs:
run: ./metadata-ingestion/scripts/install_deps.sh
- name: Disk Check
run: df -h . && docker images
- name: Remove images
run: docker image prune -a -f || true
- name: Free up disk space
run: |
sudo apt-get remove 'dotnet-*' azure-cli || true
sudo rm -rf /usr/local/lib/android/ || true
sudo docker image prune -a -f || true
- name: Disk Check
run: df -h . && docker images
- name: Smoke test
Expand Down
Loading
Loading