Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Generate manifest for examples page in viewer #3332

Merged
merged 27 commits into from
Sep 18, 2023
Merged

Generate manifest for examples page in viewer #3332

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
3e07cb5
add `build_args` to example readmes
jprochazk Sep 15, 2023
7480d51
add `thumbnails` script for generating thumbnail_dimensions
jprochazk Sep 15, 2023
e344c1a
add `thumbnail_dimensions` to readmes
jprochazk Sep 15, 2023
b72ed83
add examples build script
jprochazk Sep 15, 2023
19d8f0c
check thumbnails
jprochazk Sep 15, 2023
526c7b9
update `re_viewer` build script to detect and handle CI
jprochazk Sep 15, 2023
26cbc0e
Merge branch 'main' into jan/examples-page-manifest
jprochazk Sep 15, 2023
b8c90c1
release `version/nightly` when pushing on main
jprochazk Sep 15, 2023
92f03b4
fix lints
jprochazk Sep 15, 2023
e3385d5
put `xshell` in workspace deps
jprochazk Sep 15, 2023
de7a86e
fix script path
jprochazk Sep 15, 2023
04e6705
pin serde_yaml to older version
jprochazk Sep 15, 2023
65143c9
fix example manifest links
jprochazk Sep 15, 2023
abe62b4
`main`, not `master`
jprochazk Sep 18, 2023
3896f80
document `re_viewer/build.rs`
jprochazk Sep 18, 2023
9a085b9
update `thumbnails.py` summary
jprochazk Sep 18, 2023
519bf89
only call git once per run on CI
jprochazk Sep 18, 2023
2f4e915
Merge branch 'main' into jan/examples-page-manifest
jprochazk Sep 18, 2023
8604789
explain `nightly` tracking `main`
jprochazk Sep 18, 2023
5c22115
add `demo` field to frontmatter
jprochazk Sep 18, 2023
6036a40
update `build.rs` comment
jprochazk Sep 18, 2023
383f947
Merge branch 'main' into jan/examples-page-manifest
jprochazk Sep 18, 2023
8422e22
Merge branch 'main' into jan/examples-page-manifest
jprochazk Sep 18, 2023
4765fb4
minor cleanup
jprochazk Sep 18, 2023
116ae79
use docker image in rerun lints job
jprochazk Sep 18, 2023
555addd
...remove container from rerun lints job
jprochazk Sep 18, 2023
1ba564f
Merge branch 'main' into jan/examples-page-manifest
jprochazk Sep 18, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 0 additions & 2 deletions .github/workflows/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,6 @@ uploaded anywhere.
- `RELEASE_VERSION` - If producing a release, the version number. This must match the version in `Cargo.toml`.
- [reusable_upload_web.yml](reusable_upload_web.yml) - This job uploads the web assets to google cloud. By default this
only uploads to: `app.rerun.io/commit/<commit>/`
- `MARK_PRERELEASE_FOR_MAINLINE` - If true, then the web assets will go to `app.rerun.io/prelease/
- `MARK_TAGGED_VERSION` - If true, then the web assets will go to `app.rerun.io/version/<RELEASE_VERSION>`
- `RELEASE_VERSION` - If producing a release, the version number.
- `RRD_ARTIFACT_NAME` - Intermediate name of the GitHub rrd artifact. This should match the name passed to
Expand All @@ -54,7 +53,6 @@ uploaded anywhere.
source source code link in the built app always points to the pull request's `HEAD`.
- [reusable_upload_web_demo.yml](reusable_upload_web_demo.yml) - This job uploads the `demo.rerun.io` assets to google cloud. By default this
only uploads to: `demo.rerun.io/commit/<commit>/`
- `MARK_PRERELEASE_FOR_MAINLINE` - If true, then the web assets will go to `demo.rerun.io/prelease/
- `MARK_TAGGED_VERSION` - If true, then the web assets will go to `demo.rerun.io/version/<RELEASE_VERSION>`
- `RELEASE_VERSION` - If producing a release, the version number.
- `UPLOAD_COMMIT_OVERRIDE` - If set, will replace the value of `<commit>`. This is necessary because we want pull
Expand Down
2 changes: 0 additions & 2 deletions .github/workflows/manual_adhoc_web.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,6 @@ jobs:
uses: ./.github/workflows/reusable_upload_web.yml
with:
CONCURRENCY: adhoc-web-${{ github.event.inputs.ADHOC_NAME }}
MARK_PRERELEASE_FOR_MAINLINE: false
MARK_TAGGED_VERSION: false
ADHOC_NAME: ${{ github.event.inputs.ADHOC_NAME }}
RRD_ARTIFACT_NAME: linux-rrd-fast
Expand All @@ -62,7 +61,6 @@ jobs:
uses: ./.github/workflows/reusable_upload_web_demo.yml
with:
CONCURRENCY: adhoc-web-${{ github.event.inputs.ADHOC_NAME }}
MARK_PRERELEASE_FOR_MAINLINE: false
MARK_TAGGED_VERSION: false
ADHOC_NAME: ${{ github.event.inputs.ADHOC_NAME }}
UPLOAD_COMMIT: false
Expand Down
2 changes: 0 additions & 2 deletions .github/workflows/manual_build_release.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -116,7 +116,6 @@ jobs:
uses: ./.github/workflows/reusable_upload_web.yml
with:
CONCURRENCY: manual-dispatch-${{ github.run_id}}
MARK_PRERELEASE_FOR_MAINLINE: false
MARK_TAGGED_VERSION: true
RRD_ARTIFACT_NAME: linux-rrd
RELEASE_VERSION: ${{ github.event.inputs.RELEASE_VERSION }}
Expand All @@ -137,7 +136,6 @@ jobs:
uses: ./.github/workflows/reusable_upload_web_demo.yml
with:
CONCURRENCY: manual-dispatch-${{ github.run_id}}
MARK_PRERELEASE_FOR_MAINLINE: false
MARK_TAGGED_VERSION: true
RELEASE_VERSION: ${{ github.event.inputs.RELEASE_VERSION }}
secrets: inherit
Expand Down
2 changes: 0 additions & 2 deletions .github/workflows/manual_dispatch.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -213,7 +213,6 @@ jobs:
uses: ./.github/workflows/reusable_upload_web.yml
with:
CONCURRENCY: manual-dispatch-${{ github.run_id}}
MARK_PRERELEASE_FOR_MAINLINE: false
MARK_TAGGED_VERSION: false
RRD_ARTIFACT_NAME: linux-rrd-fast
secrets: inherit
Expand All @@ -235,7 +234,6 @@ jobs:
uses: ./.github/workflows/reusable_upload_web_demo.yml
with:
CONCURRENCY: manual-dispatch-${{ github.run_id}}
MARK_PRERELEASE_FOR_MAINLINE: false
MARK_TAGGED_VERSION: false
secrets: inherit

Expand Down
3 changes: 1 addition & 2 deletions .github/workflows/on_push_main.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,6 @@ jobs:
with:
CONCURRENCY: push-${{ github.ref_name }}
RRD_ARTIFACT_NAME: linux-rrd
MARK_PRERELEASE_FOR_MAINLINE: true
secrets: inherit

build-web-demo:
Expand All @@ -73,7 +72,6 @@ jobs:
uses: ./.github/workflows/reusable_upload_web_demo.yml
with:
CONCURRENCY: push-${{ github.ref_name }}
MARK_PRERELEASE_FOR_MAINLINE: true
secrets: inherit

build-rerun_c-and-upload-linux:
Expand Down Expand Up @@ -273,3 +271,4 @@ jobs:
allowUpdates: true
removeArtifacts: true
replacesArtifacts: true

5 changes: 5 additions & 0 deletions .github/workflows/reusable_checks.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -266,6 +266,7 @@ jobs:
- name: Install dependencies
run: |
pip install gitignore_parser python-frontmatter
pip install -r ./scripts/ci/requirements.txt
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Neither of these should be needed if we would run the job on our ci_docker container

    container:
      image: rerunio/ci_docker:0.9

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the docker container is a hack we should use as little as possible, because someone always has to remember to update it any time we add new dependencies. The job here takes 1 minute to run, and it runs in parallel to all the other checks.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I take your point, though the container is not a hack as much as an optimization and a cost saving device (CI minutes cost money). I would also rather not have it, but 🤷

We could get the best of both worlds though, by starting with the ci_docker container and then still calling pip instal here - we won't need to generate a new ci_docker all the time, but when we do these jobs will speed up

Copy link
Member Author

@jprochazk jprochazk Sep 18, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wanted to try it to see how much of a speedup it would be, because I suspect it might actually be a slowdown, but:

python3: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by python3)

Now I don't want to touch it in fear of what other horrors lie beyond. 🙂

though the container is not a hack as much as an optimization and a cost saving device

I understand that, but I think there are better ways to accomplish the same thing. The docker image is basically our own weird cache that we have to manually keep up to date. Doing proper caching in every job would likely have a much bigger impact, but it is a lot more short-term effort than just adding another tool into the image, pushing a new tag, and updating the CI to use it.


- name: Rerun lints
run: |
Expand All @@ -279,6 +280,10 @@ jobs:
run: |
./scripts/ci/check_requirements.py

- name: Check Python example thumbnails
run: |
./scripts/ci/thumbnails.py check

# ---------------------------------------------------------------------------

spell-check:
Expand Down
26 changes: 20 additions & 6 deletions .github/workflows/reusable_upload_web.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,10 +10,6 @@ on:
type: string
required: false
default: ""
MARK_PRERELEASE_FOR_MAINLINE:
required: false
type: boolean
default: false
MARK_TAGGED_VERSION:
required: false
type: boolean
Expand Down Expand Up @@ -100,21 +96,39 @@ jobs:
parent: false

- name: "Upload web-viewer (prerelease)"
if: inputs.MARK_PRERELEASE_FOR_MAINLINE
if: github.ref == 'refs/heads/main'
uses: google-github-actions/upload-cloud-storage@v1
with:
path: "web_viewer"
destination: "rerun-web-viewer/prerelease"
parent: false

- name: "Upload RRD (prerelease)"
if: inputs.MARK_PRERELEASE_FOR_MAINLINE
if: github.ref == 'refs/heads/main'
uses: google-github-actions/upload-cloud-storage@v1
with:
path: "rrd"
destination: "rerun-example-rrd/prerelease"
parent: false

- name: "Upload web-viewer (nightly)"
# TEMP: Tracking `main` until we can do actual nightly builds.
if: github.ref == 'refs/heads/main'
uses: google-github-actions/upload-cloud-storage@v1
with:
path: "web_viewer"
destination: "rerun-web-viewer/version/nightly"
parent: false

- name: "Upload RRD (nightly)"
# TEMP: Tracking `main` until we can do actual nightly builds.
if: github.ref == 'refs/heads/main'
uses: google-github-actions/upload-cloud-storage@v1
with:
path: "rrd"
destination: "rerun-example-rrd/version/nightly"
parent: false

- name: "Upload web-viewer (tagged)"
if: inputs.MARK_TAGGED_VERSION
uses: google-github-actions/upload-cloud-storage@v1
Expand Down
15 changes: 10 additions & 5 deletions .github/workflows/reusable_upload_web_demo.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,10 +10,6 @@ on:
type: string
required: false
default: ""
MARK_PRERELEASE_FOR_MAINLINE:
required: false
type: boolean
default: false
MARK_TAGGED_VERSION:
required: false
type: boolean
Expand Down Expand Up @@ -85,13 +81,22 @@ jobs:
parent: false

- name: "Upload web demo (prerelease)"
if: inputs.MARK_PRERELEASE_FOR_MAINLINE
if: github.ref == 'refs/heads/main'
uses: google-github-actions/upload-cloud-storage@v1
with:
path: "web_demo"
destination: "rerun-demo/prerelease"
parent: false

- name: "Upload web demo (nightly)"
# TEMP: Tracking `main` until we can do actual nightly builds.
if: github.ref == 'refs/heads/main'
uses: google-github-actions/upload-cloud-storage@v1
with:
path: "web_demo"
destination: "rerun-demo/version/nightly"
parent: false

- name: "Upload web demo (tagged)"
if: inputs.MARK_TAGGED_VERSION
uses: google-github-actions/upload-cloud-storage@v1
Expand Down
21 changes: 21 additions & 0 deletions Cargo.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

6 changes: 6 additions & 0 deletions crates/re_viewer/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -113,3 +113,9 @@ web-sys = { version = "0.3.52", features = ["Window"] }

[build-dependencies]
re_build_tools.workspace = true

# External
serde = { version = "1", features = ["derive"] }
serde_json = "1"
serde_yaml = "=0.9.21"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So, afaict this is to parse the markdown frontmatter, correct? Could we be using toml in there instead?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, we'd also have to update everything else that reads the frontmatter. Currently that's at least landing and build_demo_app.py

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please do 🙏 , but can be saved for another PR

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Issue filed: #3355

xshell = "0.2"
Loading
Loading