Generate manifest for examples page in viewer #3332

jprochazk · 2023-09-15T11:04:44Z

What

Part of #3212

Check that examples with a thumbnail have thumbnail_dimensions too
Add scripts/ci/thumbnails.py script for checking/updating thumbnail_dimensions
Add build.rs to re_viewer
- Keeps re_viewer/data/examples_manifest.json up to date
- The file is committed, and is only updated in rerun workspace/CI
- URLs point to:
  - version/nightly on main/local builds
  - version/x.y.z on release branches
  - commit/{short_hash} on PRs

Checklist

I have read and agree to Contributor Guide and the Code of Conduct
I've included a screenshot or gif (if applicable)
I have tested demo.rerun.io (if applicable)

.github/workflows/reusable_upload_web.yml

emilk · 2023-09-17T11:34:30Z

Cargo.lock

@@ -1765,7 +1765,7 @@ checksum = "79386fdcec5e0fde91b1a6a5bcd89677d1f9304f7f986b154a1b9109038854d9"
 dependencies = [
 "az",
 "bytemuck",
- "half 1.8.2",
+ "half 2.3.1",


unrelated change?

I'm not sure why it decided to do this, I only added serde_yaml as a dev dependency. We're already on Rust 1.72, and fixed has its half version set as >=1.8,<3, so it seems harmless.

crates/re_viewer/build.rs

emilk · 2023-09-17T11:43:23Z

crates/re_viewer/build.rs

+        panic!("{err}");
+    }
+}
+
 fn main() {
    re_build_tools::rebuild_if_crate_changed("re_viewer");


This will re-run this build.rs script if anything in re_viewer or any of its dependancies changes. Isn't that a bit too often? I thought only the CI needed to run this at all? And it need to run it always,

The build script has to run locally so that we can commit the examples_manifest.json file when it changes. That way users can build the re_viewer crate without being in the workspace. We need to trigger a run of the build script on any change to the examples directory, but I don't know what the best way to detect if examples were added/changed/removed is. Always running it solves that, I think?

I tried profiling the build script, and maybe it won't turn into an issue. Before this PR, the script takes ~350ms to run. Calling write_examples_manifest added ~500us for me. On CI it will add a bit more, because it also has to run git a few times.

Oh, I missed where I was.

We need to trigger a run of the build script on any change to the examples directory, but I don't know what the best way to detect if examples were added/changed/removed is

For changed/removed, use re_build_tools::rerun_if_changed for each file you read. It does println!("cargo:rerun-if-changed=… internally.

Detecting added files… I don't know how to do.

Always running it solves that, I think?

If you want to force the build.rs file to always run (which you may want to on CI), there is an ugly trick: println!("cargo:rerun-if-changed=path/to/file/that/doesnt/exist");

It should already always run on CI, because all the crates are built from scratch

scripts/ci/thumbnails.py

emilk · 2023-09-18T13:20:35Z

.github/workflows/reusable_checks.yml

@@ -266,6 +266,7 @@ jobs:
      - name: Install dependencies
        run: |
          pip install gitignore_parser python-frontmatter
+          pip install -r ./scripts/ci/requirements.txt


Neither of these should be needed if we would run the job on our ci_docker container

container: image: rerunio/ci_docker:0.9

I think the docker container is a hack we should use as little as possible, because someone always has to remember to update it any time we add new dependencies. The job here takes 1 minute to run, and it runs in parallel to all the other checks.

I take your point, though the container is not a hack as much as an optimization and a cost saving device (CI minutes cost money). I would also rather not have it, but 🤷

We could get the best of both worlds though, by starting with the ci_docker container and then still calling pip instal here - we won't need to generate a new ci_docker all the time, but when we do these jobs will speed up

I wanted to try it to see how much of a speedup it would be, because I suspect it might actually be a slowdown, but:

python3: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by python3)

Now I don't want to touch it in fear of what other horrors lie beyond. 🙂

though the container is not a hack as much as an optimization and a cost saving device

I understand that, but I think there are better ways to accomplish the same thing. The docker image is basically our own weird cache that we have to manually keep up to date. Doing proper caching in every job would likely have a much bigger impact, but it is a lot more short-term effort than just adding another tool into the image, pushing a new tag, and updating the CI to use it.

.github/workflows/reusable_upload_web.yml

emilk · 2023-09-18T13:24:38Z

crates/re_viewer/Cargo.toml

+# External
+serde = { version = "1", features = ["derive"] }
+serde_json = "1"
+serde_yaml = "=0.9.21"


So, afaict this is to parse the markdown frontmatter, correct? Could we be using toml in there instead?

Yes, we'd also have to update everything else that reads the frontmatter. Currently that's at least landing and build_demo_app.py

Please do 🙏 , but can be saved for another PR

Issue filed: #3355

emilk · 2023-09-18T13:30:18Z

crates/re_viewer/build.rs

+        panic!("{err}");
+    }
+}
+
 fn main() {
    re_build_tools::rebuild_if_crate_changed("re_viewer");


Oh, I missed where I was.

We need to trigger a run of the build script on any change to the examples directory, but I don't know what the best way to detect if examples were added/changed/removed is

For changed/removed, use re_build_tools::rerun_if_changed for each file you read. It does println!("cargo:rerun-if-changed=… internally.

Detecting added files… I don't know how to do.

crates/re_viewer/build.rs

emilk · 2023-09-18T13:34:31Z

crates/re_viewer/build.rs

+        panic!("{err}");
+    }
+}
+
 fn main() {
    re_build_tools::rebuild_if_crate_changed("re_viewer");


Always running it solves that, I think?

If you want to force the build.rs file to always run (which you may want to on CI), there is an ugly trick: println!("cargo:rerun-if-changed=path/to/file/that/doesnt/exist");

examples/python/plots/README.md

emilk · 2023-09-18T20:56:12Z

.github/workflows/reusable_checks.yml

@@ -266,6 +266,7 @@ jobs:
      - name: Install dependencies
        run: |
          pip install gitignore_parser python-frontmatter
+          pip install -r ./scripts/ci/requirements.txt


I take your point, though the container is not a hack as much as an optimization and a cost saving device (CI minutes cost money). I would also rather not have it, but 🤷

We could get the best of both worlds though, by starting with the ci_docker container and then still calling pip instal here - we won't need to generate a new ci_docker all the time, but when we do these jobs will speed up

emilk · 2023-09-18T20:58:13Z

crates/re_viewer/Cargo.toml

+# External
+serde = { version = "1", features = ["derive"] }
+serde_json = "1"
+serde_yaml = "=0.9.21"


Issue filed: #3355

emilk · 2023-09-18T21:00:35Z

crates/re_viewer/build.rs

+        let thumbnail_dimensions = get!(example, thumbnail_dimensions);
+
+        Ok(Self {
+            title: get!(example, title),


Why use a macro to unwrap the title: Option<String> instead of using title: String directly?

I think originally I just wanted the parsing from YAML to not fail early, so that I could check for the presence of build_args. I changed it to #[serde(default)] all the fields instead, and removed the macro.

- `Frontmatter` does not need `Option` fields - `parse_release_version` does not need to format the version

jprochazk added 6 commits September 15, 2023 12:13

add build_args to example readmes

3e07cb5

add thumbnails script for generating thumbnail_dimensions

7480d51

add thumbnail_dimensions to readmes

e344c1a

add examples build script

b72ed83

check thumbnails

19d8f0c

update re_viewer build script to detect and handle CI

526c7b9

jprochazk added 🧑‍💻 dev experience developer experience (excluding CI) examples Issues relating to the Rerun examples labels Sep 15, 2023

jprochazk added 2 commits September 15, 2023 13:04

Merge branch 'main' into jan/examples-page-manifest

26cbc0e

release version/nightly when pushing on main

b8c90c1

jprochazk mentioned this pull request Sep 15, 2023

Tracking issue: Revamp of our example building infrastructure #3212

Closed

4 tasks

jprochazk added 5 commits September 15, 2023 13:16

fix lints

92f03b4

put xshell in workspace deps

e3385d5

fix script path

de7a86e

pin serde_yaml to older version

04e6705

fix example manifest links

65143c9

jprochazk marked this pull request as ready for review September 15, 2023 11:51

emilk reviewed Sep 17, 2023

View reviewed changes

jprochazk added 5 commits September 18, 2023 09:38

main, not master

abe62b4

document re_viewer/build.rs

3896f80

update thumbnails.py summary

9a085b9

only call git once per run on CI

519bf89

Merge branch 'main' into jan/examples-page-manifest

2f4e915

jprochazk requested a review from emilk September 18, 2023 09:11

emilk requested changes Sep 18, 2023

View reviewed changes

jprochazk added 4 commits September 18, 2023 15:54

explain nightly tracking main

8604789

add demo field to frontmatter

5c22115

update build.rs comment

6036a40

Merge branch 'main' into jan/examples-page-manifest

383f947

jprochazk requested a review from emilk September 18, 2023 14:31

Merge branch 'main' into jan/examples-page-manifest

8422e22

emilk approved these changes Sep 18, 2023

View reviewed changes

jprochazk added 4 commits September 18, 2023 23:24

minor cleanup

4765fb4

- `Frontmatter` does not need `Option` fields - `parse_release_version` does not need to format the version

use docker image in rerun lints job

116ae79

...remove container from rerun lints job

555addd

Merge branch 'main' into jan/examples-page-manifest

1ba564f

jprochazk merged commit d59a934 into main Sep 18, 2023
26 checks passed

jprochazk deleted the jan/examples-page-manifest branch September 18, 2023 22:02

emilk mentioned this pull request Sep 24, 2023

[REMINDER-0.9] Make sure baked-in example manifest is correct #3213

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Generate manifest for examples page in viewer #3332

Generate manifest for examples page in viewer #3332

jprochazk commented Sep 15, 2023 •

edited by github-actions bot

Loading

emilk Sep 17, 2023

jprochazk Sep 18, 2023

emilk Sep 17, 2023

jprochazk Sep 18, 2023

emilk Sep 18, 2023

emilk Sep 18, 2023

jprochazk Sep 18, 2023

emilk Sep 18, 2023

jprochazk Sep 18, 2023

emilk Sep 18, 2023

jprochazk Sep 18, 2023 •

edited

Loading

emilk Sep 18, 2023

jprochazk Sep 18, 2023

emilk Sep 18, 2023

emilk Sep 18, 2023

emilk Sep 18, 2023

emilk Sep 18, 2023

emilk Sep 18, 2023

emilk Sep 18, 2023

emilk Sep 18, 2023

jprochazk Sep 18, 2023 •

edited

Loading

Generate manifest for examples page in viewer #3332

Generate manifest for examples page in viewer #3332

Conversation

jprochazk commented Sep 15, 2023 • edited by github-actions bot Loading

What

Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jprochazk Sep 18, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jprochazk Sep 18, 2023 • edited Loading

Choose a reason for hiding this comment

jprochazk commented Sep 15, 2023 •

edited by github-actions bot

Loading

jprochazk Sep 18, 2023 •

edited

Loading

jprochazk Sep 18, 2023 •

edited

Loading