DOCS-2930: Add first_run flow for Docker modules #4187

JessamyT · 2025-04-03T20:08:17Z

https://deploy-preview-4187--viam-docs.netlify.app/operate/reference/advanced-modules/#package-and-deploy-using-docker

netlify · 2025-04-03T20:09:08Z

✅ Deploy Preview for viam-docs ready!

Name	Link
🔨 Latest commit	`fb370cf`
🔍 Latest deploy log	https://app.netlify.com/sites/viam-docs/deploys/67f860939aadf8000897087a
😎 Deploy Preview	https://deploy-preview-4187--viam-docs.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.
Lighthouse	1 paths audited Performance: 28 (🔴 down 11 from production) Accessibility: 100 (no change from production) Best Practices: 100 (no change from production) SEO: 92 (no change from production) PWA: 70 (no change from production) View the detailed breakdown and full score reports

To edit notification comments on pull requests, go to your Netlify site configuration.

penguinland

LGTM! All my feedback is either minor or non-actionable.

docs/operate/reference/advanced-modules/_index.md

penguinland · 2025-04-03T20:12:56Z

docs/operate/reference/advanced-modules/_index.md

+     ```sh {id="terminal-prompt" class="command-line" data-prompt="$"}
+     #!/usr/bin/env bash
+
+     if [[ -n "$VIAM_TEST_FAIL_RUN_FIRST" ]]; then


I'm not familiar with this env var. Is it supposed to be in all first_run scripts? What does it do? Maybe I should add it to mine... 😳

Great question--just pulled this in from the setup phase guide on confluence which in turn links to https://github.com/viam-labs/wifi-sensor/blob/7823b6ad3edcbbbf20b06c34b3181453f5f3f078/first_run.sh

I also am unclear on what this is for here. Can you add an explanation in? Does this get set to true if a previous run failed? Or is this jsut for testing?

In the RDK, this env var is mentioned in 2 places: a test and an example first_run.sh. I have cloned about 60 Viam-related repos locally, and it's not mentioned in any of the others. I've searched Github for it, and can only find copies of those 2 locations. I wonder if there was some plan to add this in, but it never happened and these are just the vestiges of a feature that never was.

I think we should remove this paragraph from the example.

Thanks for looking around. Will remove.

penguinland · 2025-04-03T20:14:12Z

docs/operate/reference/advanced-modules/_index.md

+   It will only execute once per module or per version of the module.
+
+1. (Optional) After a first run script runs successfully, Viam adds a marker file with a `.first_run_succeeded` suffix in the module’s data directory on disk.
+   It has the location and form: `/root/.viam/packages/data/module/<MODULE_ID>-<VERSION>-<ARCH>.first_run_succeeded`.


I suspect there should be a slash before .first_run_succeeded, but haven't checked on that personally. Maybe it's correct as-is.

Hmm also from Maxim's guide--not sure if there's a reasonably practical way to check for sure

😳 I went to one of my machines with a module with a first_run script to check, and got surprised that instead of having such a file anywhere, I have a meta.json does not exist, skipping first run message scattered throughout my logs! I can't help, and I've got some modules to go fix.

Thanks for pointing me towards this bug!

Eep! I don't have any Docker modules to test this on... @cheukt since it looks like you've been maintaining the confluence instructions, can you confirm?

it is unpackedModDir + FirstRunSuccessSuffix, an example would be .viam/packages/data/module/e76d1b3b-0468-4efd-bb7f-fb1d2b352fcb-viamrtsp-0_1_0-linux-amd64/bin.first_run_succeeded
https://github.com/viamrobotics/rdk/blob/1af4d812c36e43c2f08b847290e1642489eb1df8/config/module.go#L279

Jessamy had questions about first_run scripts in viamrobotics/docs#4187, I went to check on it for her, and got surprised that instead of running `first_run.sh`, this module instead logs `meta.json does not exist, skipping first run` and continues without running it! I've filed https://viam.atlassian.net/browse/RSDK-10382 so it's harder to get into this position next time. I haven't tried this: local modules don't start up without meta.json, so I think this is only tryable once we deploy to the registry. However, I've run `make -f Makefile.module module.tar.gz` and checked that it successfully puts meta.json in the .tar.gz file.

npentrel

I'm a bit confused at a few points.
I also think it would be really helpful to link to a full example. I think soleng used docker in a few projects, is there none we can link to?

npentrel · 2025-04-04T13:23:39Z

docs/operate/reference/advanced-modules/_index.md

+     #!/usr/bin/env bash
+
+     if [[ -n "$VIAM_TEST_FAIL_RUN_FIRST" ]]; then
+         echo "Sorry, I've failed you."


Does this error indicate an issue with the module? Maybe better to change to something that indicates where the error is

I am not sure where it is. Since this is an edge case documented within a hidden page, I figured it made sense to act quickly and use the existing example provided by Maxim. Sounds like I'll remove the whole paragraph anyway.

npentrel · 2025-04-04T13:24:16Z

docs/operate/reference/advanced-modules/_index.md

+     ```sh {id="terminal-prompt" class="command-line" data-prompt="$"}
+     #!/usr/bin/env bash
+
+     if [[ -n "$VIAM_TEST_FAIL_RUN_FIRST" ]]; then


I also am unclear on what this is for here. Can you add an explanation in? Does this get set to true if a previous run failed? Or is this jsut for testing?

docs/operate/reference/advanced-modules/_index.md

npentrel · 2025-04-04T13:42:56Z

docs/operate/reference/advanced-modules/_index.md

+         exit 1
+     fi
+
+     docker pull mongo:6


So this pulls the image but then how does the container get run? I think we probably want to link to an example of that. Also

And should this run with https://github.com/viam-soleng/viam-docker-manager or is this entirely separate?

I think this example is right: in first_run.sh, you pull the image, and then in your module's main executable, you run the image, and that second part should be obvious for anyone developing with docker. We could add in such an example, but it's trivial:

#!/bin/bash set -euxo pipefail # This example module runs inside the docker image you should have downloaded in first_run.sh SOCKET_DIR=`dirname $1` VIAM_DIR=`realpath ~/.viam` exec docker run \ --rm \ --runtime=nvidia --gpus=all \ -v /etc/passwd:/etc/passwd \ -v /etc/group:/etc/group \ -v $SOCKET_DIR:$SOCKET_DIR \ -v $VIAM_DIR:$VIAM_DIR \ ghcr.io/viam-modules/viam-mlmodelservice-triton/jetpack6:0.9.0 \ LD_PRELOAD=libjemalloc.so.2 VIAM_MODULE_DATA="$VIAM_MODULE_DATA" /opt/viam/bin/viam_mlmodelservice_triton "$@" 2>&1

(that example is from the Triton module, and although there are lots of options specified for the docker line, anyone developing with docker should know which ones they need to set.)

It's really just

#!/bin/bash exec docker run <whatever-the-module-author-intended-to-put-here>

I think it's probably clear to someone who would immediately know that there's an executable that gets called and where to edit it. I don't think it'll hurt us to add this two liner and where to put it here as the entry point script. @JessamyT you can use the https://hub.docker.com/_/hello-world image to test that this works

Okay, added above. Tested using the hello-world image, though it of course doesn't behave exactly like a real module that would continue to run. Did confirm that it ran the first_run and made a marker file.

JessamyT · 2025-04-04T17:59:41Z

RE examples to point to, it looks like https://github.com/abe-winter/python-container-module doesn't actually use a first_run script as such? And https://github.com/viam-labs/wifi-sensor/tree/RSDK-8886-setup-phase is what Maxim pointed to as an example but the main branch of wifi-sensor doesn't use it. Based on the thread it sounds like the sol-eng uses would've been a bit atypical. @abe-winter do you recommend a particular example? Or @penguinland is https://github.com/viam-modules/viam-mlmodelservice-triton/tree/main like you mentioned a good one, and if so, should the examples in this PR be changed to match it?

penguinland · 2025-04-04T18:45:38Z

I've been mentioning the Triton one because that's the only docker module I work with. TBH, it's kind of a pain: the way Drew set it up, in the repo there are merely templates of the module script and first_run script, and you need to generate the actual ones from those templates. My hope is there's a simpler example out there, but I'm not aware of one (because I haven't looked).

Co-authored-by: Naomi Pentrel <5212232+npentrel@users.noreply.github.com>

npentrel

Let's move this into a subpage. It can be referencd from the current page but the rest should be a subpage.

Thank you for testing it. Please do also file a follow up ticket for yourself to test packaging an actual module with docker. Currently the docs really only cover the deploying part.

npentrel · 2025-04-10T12:10:32Z

docs/operate/reference/advanced-modules/_index.md

@@ -46,6 +46,73 @@ Running {{< glossary_tooltip term_id="modular-resource" text="modular resources"

 However, if you are unable to use modular resources because you need to host `viam-server` on a non-Linux system or have an issue with compilation, you may need to [implement a custom component and register it on a server configured as a remote](/operate/reference/advanced-modules/custom-components-remotes/) on your machine.

+## Package and deploy using Docker


For both Custom APIs and components as remote parts, we have subpages. I think we should do the same here.

npentrel · 2025-04-10T12:14:30Z

docs/operate/reference/advanced-modules/_index.md

@@ -46,6 +46,73 @@ Running {{< glossary_tooltip term_id="modular-resource" text="modular resources"

 However, if you are unable to use modular resources because you need to host `viam-server` on a non-Linux system or have an issue with compilation, you may need to [implement a custom component and register it on a server configured as a remote](/operate/reference/advanced-modules/custom-components-remotes/) on your machine.

+## Package and deploy using Docker
+
+In rare cases, you may need to package and deploy a module using Docker.


Be aware that we are not providing complete info here. We're showing how to pull and run a container. We will likely need to look into how to package a module with Docker. So far this really only covers the deploy aspect.

Please file a follow up ticket to cover this. A full example for reference is really what I think we need here.

https://viam.atlassian.net/browse/DOCS-3863

JessamyT · 2025-04-11T00:24:34Z

Please do also file a follow up ticket for yourself to test packaging an actual module with docker. Currently the docs really only cover the deploying part.

@npentrel To be clear, do you mean an additional ticket separate from and before adding the full example how-to, or as part of that? I'd like to do as part of https://viam.atlassian.net/browse/DOCS-3863 whenever that happens.

github-actions · 2025-04-11T16:13:11Z

🔎💬 Inkeep AI search and chat service is syncing content for source 'Viam Docs'

3068firstrundocker

dff049f

viambot added the safe to build This pull request is marked safe to build from a trusted zone label Apr 3, 2025

JessamyT requested review from penguinland and abe-winter April 3, 2025 20:08

penguinland approved these changes Apr 3, 2025

View reviewed changes

Apply suggestions from code review

bb29e83

penguinland mentioned this pull request Apr 3, 2025

include meta.json in the .tar.gz file viam-modules/viam-mlmodelservice-triton#35

Merged

npentrel requested changes Apr 4, 2025

View reviewed changes

Apply suggestions from code review

2ee751b

Address feedback

3776863

JessamyT changed the title ~~DOCS-3068, DOCS-3114: Add first_run flow for Docker modules~~ DOCS-2930: Add first_run flow for Docker modules Apr 4, 2025

JessamyT and others added 2 commits April 9, 2025 12:03

Apply suggestions from code review

a4a501c

Co-authored-by: Naomi Pentrel <5212232+npentrel@users.noreply.github.com>

Changes after testing

1c4fc85

JessamyT requested a review from npentrel April 10, 2025 02:16

npentrel requested changes Apr 10, 2025

View reviewed changes

Move to sub-page

fb370cf

JessamyT requested a review from npentrel April 11, 2025 00:22

npentrel approved these changes Apr 11, 2025

View reviewed changes

JessamyT merged commit 652c828 into viamrobotics:main Apr 11, 2025
12 checks passed

JessamyT deleted the 3068firstrundocker branch April 11, 2025 16:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DOCS-2930: Add first_run flow for Docker modules #4187

DOCS-2930: Add first_run flow for Docker modules #4187

JessamyT commented Apr 3, 2025 •

edited

Loading

netlify bot commented Apr 3, 2025 •

edited

Loading

penguinland left a comment

penguinland Apr 3, 2025

JessamyT Apr 3, 2025

npentrel Apr 4, 2025

penguinland Apr 4, 2025

JessamyT Apr 4, 2025

penguinland Apr 3, 2025

JessamyT Apr 3, 2025

penguinland Apr 3, 2025

JessamyT Apr 3, 2025

cheukt Apr 9, 2025

npentrel left a comment

npentrel Apr 4, 2025

JessamyT Apr 4, 2025

npentrel Apr 4, 2025

npentrel Apr 4, 2025

penguinland Apr 4, 2025

penguinland Apr 4, 2025 •

edited

Loading

npentrel Apr 7, 2025

JessamyT Apr 10, 2025

JessamyT commented Apr 4, 2025

penguinland commented Apr 4, 2025 •

edited

Loading

npentrel left a comment

npentrel Apr 10, 2025

npentrel Apr 10, 2025

JessamyT Apr 11, 2025

JessamyT commented Apr 11, 2025 •

edited

Loading

github-actions bot commented Apr 11, 2025

		@@ -46,6 +46,73 @@ Running {{< glossary_tooltip term_id="modular-resource" text="modular resources"

		However, if you are unable to use modular resources because you need to host `viam-server` on a non-Linux system or have an issue with compilation, you may need to [implement a custom component and register it on a server configured as a remote](/operate/reference/advanced-modules/custom-components-remotes/) on your machine.

		## Package and deploy using Docker

DOCS-2930: Add first_run flow for Docker modules #4187

DOCS-2930: Add first_run flow for Docker modules #4187

Conversation

JessamyT commented Apr 3, 2025 • edited Loading

netlify bot commented Apr 3, 2025 • edited Loading

✅ Deploy Preview for viam-docs ready!

penguinland left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

npentrel left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

penguinland Apr 4, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JessamyT commented Apr 4, 2025

penguinland commented Apr 4, 2025 • edited Loading

npentrel left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JessamyT commented Apr 11, 2025 • edited Loading

github-actions bot commented Apr 11, 2025

JessamyT commented Apr 3, 2025 •

edited

Loading

netlify bot commented Apr 3, 2025 •

edited

Loading

penguinland Apr 4, 2025 •

edited

Loading

penguinland commented Apr 4, 2025 •

edited

Loading

JessamyT commented Apr 11, 2025 •

edited

Loading