Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Otel v2 #642

Open
wants to merge 10 commits into
base: main
Choose a base branch
from
Open

Otel v2 #642

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
13e6ad4
router
noyoshi Oct 15, 2024
11963f9
try to fix
noyoshi Oct 15, 2024
fb3d59b
fix
noyoshi Oct 15, 2024
4b74c0c
attempt to fix
noyoshi Oct 15, 2024
a894e0d
cargo update
noyoshi Oct 15, 2024
90e2c88
downgrade axum tracing
noyoshi Oct 15, 2024
463b7d8
fix
noyoshi Oct 15, 2024
2ef3798
fix build
noyoshi Oct 15, 2024
7334a54
fix
noyoshi Oct 15, 2024
109ac5a
init tracing otel
noyoshi Oct 15, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
297 changes: 149 additions & 148 deletions .github/workflows/build.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,153 +1,154 @@
name: Build and push docker image to ghcr

on:
workflow_dispatch:
push:
branches:
- 'main'
tags:
- 'v*'
workflow_dispatch:
push:
branches:
- "main"
- "otel-v2"
tags:
- "v*"

jobs:
build-and-push-image:
concurrency:
group: ${{ github.workflow }}-${{ github.job }}-${{ github.head_ref || github.run_id }}
cancel-in-progress: true
runs-on: a100-runner
permissions:
contents: write
packages: write
# This is used to complete the identity challenge
# with sigstore/fulcio when running outside of PRs.
id-token: write
security-events: write

steps:
- name: Log in to GitHub Container Registry
uses: docker/login-action@v1
with:
registry: ghcr.io
username: ${{ github.repository_owner }}
password: ${{ secrets.GHCR_PAT }}
- name: Checkout repository
uses: actions/checkout@v3
with:
submodules: recursive

- name: Free Disk Space (Ubuntu)
uses: jlumbroso/free-disk-space@main
with:
tool-cache: false
android: true
dotnet: true
haskell: true
large-packages: false
swap-storage: true

- name: Install soci
uses: lerentis/soci-installer@v1.0.1
with:
soci-release: 'v0.4.0'

- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v2.10.0

- name: Set up containerd for ubuntu
uses: crazy-max/ghaction-setup-containerd@v2.2.0
with:
config-inline: |
version = 2

# persistent data location
root = "/runner/build/containerd"

- name: Docker meta
id: meta
uses: docker/metadata-action@v5
with:
images: |
ghcr.io/predibase/lorax
tags: |
type=semver,pattern={{version}}
type=semver,pattern={{major}}.{{minor}}
type=sha,prefix=,suffix=,format=short
type=raw,value=main,enable=${{ github.ref == 'refs/heads/main' }}

- name: Create a hash from tags
env:
tags: ${{ steps.meta.outputs.tags }}
id: vars
run: |
tag_hash=$(echo -n "$tags" | md5sum | awk '{print $1}')
echo "tag_hash=$tag_hash" >> $GITHUB_OUTPUT
echo "cache_dir=/runner/build/images/cache" >> $GITHUB_OUTPUT
echo "image_dir=/runner/build/images" >> $GITHUB_OUTPUT
echo "image_path=/runner/build/images/lorax" >> $GITHUB_OUTPUT

- name: Create and update image/cache directory
env:
image_dir: ${{ steps.vars.outputs.image_dir }}
cache_dir: ${{ steps.vars.outputs.cache_dir }}
run: |
sudo mkdir -p $image_dir
sudo chown ubuntu:ubuntu $image_dir

sudo mkdir -p $cache_dir
sudo chown ubuntu:ubuntu $cache_dir

- name: Export Docker image as OCI
uses: docker/build-push-action@v5
with:
context: .
file: ./Dockerfile # Path to your Dockerfile
push: false
tags: ${{ steps.meta.outputs.tags }}
outputs: type=oci,compression=gzip,dest=${{ steps.vars.outputs.image_path }}-${{ steps.vars.outputs.tag_hash }}.tar.gz
cache-from: type=local,src=${{ steps.vars.outputs.cache_dir }}
cache-to: type=local,mode=max,image-manifest=true,oci-mediatypes=true,dest=${{ steps.vars.outputs.cache_dir }}

- name: Import image in containerd
env:
tag_hash: ${{ steps.vars.outputs.tag_hash }}
image_path: ${{ steps.vars.outputs.image_path }}
run: |
echo "Importing $image_path-$tag_hash to Containerd"
sudo ctr i import --no-unpack --all-platforms --digests $image_path-$tag_hash.tar.gz

- name: Push image with containerd
env:
tags: ${{ steps.meta.outputs.tags }}
run: |
for tag in $tags
do
echo "Pushing $tag to GHCR"
sudo ctr i push --user "${{ github.repository_owner }}:${{ secrets.GHCR_PAT }}" $tag
done

- name: Create and push soci index
env:
tags: ${{ steps.meta.outputs.tags }}
run: |
export SOCI_PATH=$HOME/.soci/soci
for tag in $tags
do
echo "Creating soci index for $tag"
sudo $SOCI_PATH create $tag
echo "Pushing soci index for $tag"
sudo $SOCI_PATH push --user ${{ github.repository_owner }}:${{ secrets.GHCR_PAT }} $tag
done

- name: Prune older images
env:
tag_hash: ${{ steps.vars.outputs.tag_hash }}
image_path: ${{ steps.vars.outputs.image_path }}
run: |
# Delete images older than a day from docker store
docker image prune -a -f --filter "until=24h"

# Delete the on disk copy
rm -rf "$image_path-$tag_hash.tar.gz"

# Delete the SHA image(s) from containerd store
sudo ctr i rm $(sudo ctr i ls -q)
build-and-push-image:
concurrency:
group: ${{ github.workflow }}-${{ github.job }}-${{ github.head_ref || github.run_id }}
cancel-in-progress: true
runs-on: a100-runner
permissions:
contents: write
packages: write
# This is used to complete the identity challenge
# with sigstore/fulcio when running outside of PRs.
id-token: write
security-events: write

steps:
- name: Log in to GitHub Container Registry
uses: docker/login-action@v1
with:
registry: ghcr.io
username: ${{ github.repository_owner }}
password: ${{ secrets.GHCR_PAT }}

- name: Checkout repository
uses: actions/checkout@v3
with:
submodules: recursive

- name: Free Disk Space (Ubuntu)
uses: jlumbroso/free-disk-space@main
with:
tool-cache: false
android: true
dotnet: true
haskell: true
large-packages: false
swap-storage: true

- name: Install soci
uses: lerentis/soci-installer@v1.0.1
with:
soci-release: "v0.4.0"

- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v2.10.0

- name: Set up containerd for ubuntu
uses: crazy-max/ghaction-setup-containerd@v2.2.0
with:
config-inline: |
version = 2

# persistent data location
root = "/runner/build/containerd"

- name: Docker meta
id: meta
uses: docker/metadata-action@v5
with:
images: |
ghcr.io/predibase/lorax
tags: |
type=semver,pattern={{version}}
type=semver,pattern={{major}}.{{minor}}
type=sha,prefix=,suffix=,format=short
type=raw,value=main,enable=${{ github.ref == 'refs/heads/main' }}

- name: Create a hash from tags
env:
tags: ${{ steps.meta.outputs.tags }}
id: vars
run: |
tag_hash=$(echo -n "$tags" | md5sum | awk '{print $1}')
echo "tag_hash=$tag_hash" >> $GITHUB_OUTPUT
echo "cache_dir=/runner/build/images/cache" >> $GITHUB_OUTPUT
echo "image_dir=/runner/build/images" >> $GITHUB_OUTPUT
echo "image_path=/runner/build/images/lorax" >> $GITHUB_OUTPUT

- name: Create and update image/cache directory
env:
image_dir: ${{ steps.vars.outputs.image_dir }}
cache_dir: ${{ steps.vars.outputs.cache_dir }}
run: |
sudo mkdir -p $image_dir
sudo chown ubuntu:ubuntu $image_dir

sudo mkdir -p $cache_dir
sudo chown ubuntu:ubuntu $cache_dir

- name: Export Docker image as OCI
uses: docker/build-push-action@v5
with:
context: .
file: ./Dockerfile # Path to your Dockerfile
push: false
tags: ${{ steps.meta.outputs.tags }}
outputs: type=oci,compression=gzip,dest=${{ steps.vars.outputs.image_path }}-${{ steps.vars.outputs.tag_hash }}.tar.gz
cache-from: type=local,src=${{ steps.vars.outputs.cache_dir }}
cache-to: type=local,mode=max,image-manifest=true,oci-mediatypes=true,dest=${{ steps.vars.outputs.cache_dir }}

- name: Import image in containerd
env:
tag_hash: ${{ steps.vars.outputs.tag_hash }}
image_path: ${{ steps.vars.outputs.image_path }}
run: |
echo "Importing $image_path-$tag_hash to Containerd"
sudo ctr i import --no-unpack --all-platforms --digests $image_path-$tag_hash.tar.gz

- name: Push image with containerd
env:
tags: ${{ steps.meta.outputs.tags }}
run: |
for tag in $tags
do
echo "Pushing $tag to GHCR"
sudo ctr i push --user "${{ github.repository_owner }}:${{ secrets.GHCR_PAT }}" $tag
done

- name: Create and push soci index
env:
tags: ${{ steps.meta.outputs.tags }}
run: |
export SOCI_PATH=$HOME/.soci/soci
for tag in $tags
do
echo "Creating soci index for $tag"
sudo $SOCI_PATH create $tag
echo "Pushing soci index for $tag"
sudo $SOCI_PATH push --user ${{ github.repository_owner }}:${{ secrets.GHCR_PAT }} $tag
done

- name: Prune older images
env:
tag_hash: ${{ steps.vars.outputs.tag_hash }}
image_path: ${{ steps.vars.outputs.image_path }}
run: |
# Delete images older than a day from docker store
docker image prune -a -f --filter "until=24h"

# Delete the on disk copy
rm -rf "$image_path-$tag_hash.tar.gz"

# Delete the SHA image(s) from containerd store
sudo ctr i rm $(sudo ctr i ls -q)
Loading
Loading