feat(docker): /container-init.d for advanced initialization #6577

Caian · 2019-08-16T01:16:42Z

This is a proposal to help solve advanced initialization in container environments being discussed in several issues. It follows the model used by postgres container: https://hub.docker.com/_/postgres

This PR adds support for an optional /container-init.d directory in the container that can be mounted with custom configuration scripts. These scripts are executed if they have +x permission, otherwise they are just sourced.

This model also allows passing environment variables to the custom scripts when launching the container.

This grants flexibility to and delegates the responsability of bulding complex initializations to the users and administrators.

License: MIT
Signed-off-by: Caian Benedicto caianbene@gmail.com

README.md

Caian · 2019-09-02T13:04:53Z

README updated with example per request of @olizilla

lidel

I really like how flexible this approach is. ❤️ 👍

Agree on the main selling point: hook dir removes the need for things like #8326 and allows people to implement custom initialization without burdening the upstream project with additional maintenance surface.

(The mentioned postgresql/docker-entrypoint.sh is very similar in that it executes only during initialization)

Remaining work

Rebase on top of latest master (done)
Add regression test
- as this is something people will leverage in their infra, there should be a test in test/sharness/t0300-docker-image.sh (or a separate file, if thats easier) that confirms that a script mounted into /container-init.d is executed. It could execute ipfs bootstrap add and then we could check if Bootstrap section in the config cot updated.
  - @Caian will you have time to push this over the finish line, or can this be picked up by someone else?

BigLep · 2021-10-01T15:21:45Z

Next step is to add tests.

bin/container_daemon

guseggert · 2021-10-08T17:20:17Z

Adding sharness tests for this will take a bit of time, as the current Docker tests are broken and need to be rewritten.

BigLep · 2022-04-01T16:10:12Z

2022-04-01:

Needs review (@thattommyhall for ergonomics of someone using this, @lidel for sharness)
Is getting hit by docker flakiness (separate work item).

BigLep · 2022-04-01T16:11:52Z

@thattommyhall : it would be great if you can give review of this as as a user of the functionality.

BigLep · 2022-04-01T16:32:50Z

@thattommyhall : did you meean to close this?

License: MIT Signed-off-by: Caian Benedicto <caianbene@gmail.com>

Fixed in init script: - Added some missing quotes around expansions - Fixed INIT_ARGS to not pass any args if IPFS_PROFILE isn't specified - Use printf instead of "echo -e" - Only run scripts in top-level of init dir - Handle filenames correctly when finding init scripts (by using find + xargs)

cleans up containers and images (useful when run on developer machine)

lidel · 2022-04-11T13:37:23Z

README.md

@@ -366,6 +366,48 @@ Basic proof of 'ipfs working' locally:
    # QmT78zSuBmuS4z925WZfrqQ1qHaJ56DQaTfyMUF7F8ff5o
    ipfs cat <that hash>

+To perform a custom initialization and configuration step, one can mount `.sh` scripts to a `/container-init.d` directory, or mount the entire directory. These scripts are executed if they have `+x` permission, otherwise they are just sourced. Executed scripts can exit with error and abort the initialization process. The scripts may also depend on environment variables set by the `docker run` command or in `docker-compose.yml`. The custom initialization step is **only** performed during IPFS initialization, **after** `ipfs init`, **after** after the swarm key is copied to the IPFS data directory, and **before** the daemon is started. This also enables custom containers to layer extra initialization steps to the base image by adding more scripts to `/container-init.d`.


@guseggert @thattommyhall my understanding is that we already run scripts from /container-init.d every time, so we just need to document that + emphasize that scripts should be written in idempotent way.

Would below work?

Suggested change

To perform a custom initialization and configuration step, one can mount `.sh` scripts to a `/container-init.d` directory, or mount the entire directory. These scripts are executed if they have `+x` permission, otherwise they are just sourced. Executed scripts can exit with error and abort the initialization process. The scripts may also depend on environment variables set by the `docker run` command or in `docker-compose.yml`. The custom initialization step is **only** performed during IPFS initialization, **after** `ipfs init`, **after** after the swarm key is copied to the IPFS data directory, and **before** the daemon is started. This also enables custom containers to layer extra initialization steps to the base image by adding more scripts to `/container-init.d`.

To perform a custom initialization or configuration step, one can mount `.sh` scripts to a `/container-init.d` directory, or mount the entire directory. These scripts are executed if they have `+x` permission, otherwise they are just sourced. Executed scripts can exit with error and abort the initialization process. The scripts may also depend on environment variables set by the `docker run` command or in `docker-compose.yml`. **The custom initialization steps from `/container-init.d` are run every time, and should be idempotent.** When run the first time during IPFS initialization, custom scripts run **after** `ipfs init`, **after** the swarm key is copied to the IPFS data directory, and **before** the daemon is started for the very first time. This also enables custom containers to layer extra initialization steps to the base image by adding more scripts to `/container-init.d`.

Yeah it's a good distinction to make, but the prose of this paragraph bothers me a little, let me rewrite it.

I also don't think this advanced stuff is appropriate for the "getting started" section of the readme. And that section is also not docker-specific

I'm going to just remove this and add it to the IPFS Docker documentation , which is already linked to from the readme.

Filled ipfs/ipfs-docs#1114

There is already IPFS Docker documentation where this should live: https://docs.ipfs.io/how-to/run-ipfs-inside-docker/

guseggert · 2022-04-11T15:59:51Z

I've removed the documentation and will open a PR in https://github.com/ipfs/ipfs-docs w/ docs.
Update: filled ipfs/ipfs-docs#1114

* Add initialization directory support to Docker image * Add sharness test, fix bugs in init script Fixed in init script: - Added some missing quotes around expansions - Fixed INIT_ARGS to not pass any args if IPFS_PROFILE isn't specified - Use printf instead of "echo -e" - Only run scripts in top-level of init dir - Handle filenames correctly when finding init scripts (by using find + xargs) * chore: docker cleanup cleans up containers and images (useful when run on developer machine) * remove container init documentation from README There is already IPFS Docker documentation where this should live: https://docs.ipfs.io/how-to/run-ipfs-inside-docker/ Co-authored-by: Caian <caian@ggaunicamp.com> Co-authored-by: Marcin Rataj <lidel@lidel.org> Co-authored-by: Gus Eggert <gus@gus.dev>

thattommyhall · 2022-04-27T23:40:43Z

started a repo of interesting examples of this https://github.com/ipfs-shipyard/go-ipfs-docker-examples

Caian changed the title ~~Feat/container init scripts~~ Docker initscript: Add optional init directory for advanced initialization Aug 16, 2019

olizilla reviewed Sep 2, 2019

View reviewed changes

README.md Outdated Show resolved Hide resolved

Caian requested a review from olizilla October 15, 2019 17:58

olizilla requested a review from lanzafame October 16, 2019 09:01

vk496 mentioned this pull request Aug 12, 2021

Add support for scripts during initialization #8337

Closed

lidel requested review from lidel and guseggert and removed request for lanzafame, guseggert and olizilla September 3, 2021 19:02

lidel changed the title ~~Docker initscript: Add optional init directory for advanced initialization~~ feat(docker): /container-init.d for advanced initialization Sep 3, 2021

lidel force-pushed the feat/container-init-scripts branch 2 times, most recently from 7354767 to 2b3c534 Compare September 3, 2021 19:48

lidel requested changes Sep 4, 2021

View reviewed changes

BigLep assigned lidel Sep 10, 2021

BigLep added this to the go-ipfs 0.11 milestone Sep 10, 2021

lidel added help wanted Seeking public contribution on this issue need_tests labels Sep 24, 2021

lidel removed their assignment Sep 24, 2021

BigLep assigned lidel Oct 5, 2021

BigLep requested a review from guseggert October 7, 2021 15:40

guseggert reviewed Oct 8, 2021

View reviewed changes

bin/container_daemon Outdated Show resolved Hide resolved

bin/container_daemon Outdated Show resolved Hide resolved

guseggert mentioned this pull request Nov 23, 2021

Release v0.11 #8343

Closed

80 tasks

guseggert requested a review from lidel March 30, 2022 16:48

guseggert modified the milestones: Best Effort Track, go-ipfs 0.13 Mar 30, 2022

BigLep requested a review from thattommyhall April 1, 2022 16:11

thattommyhall closed this Apr 1, 2022

BigLep reopened this Apr 1, 2022

lidel self-assigned this Apr 11, 2022

Caian and others added 7 commits April 11, 2022 15:22

Add initialization directory support to container daemon

998a171

License: MIT Signed-off-by: Caian Benedicto <caianbene@gmail.com>

Update README

57e1217

License: MIT Signed-off-by: Caian Benedicto <caianbene@gmail.com>

Add exit on script failure

9422b20

License: MIT Signed-off-by: Caian Benedicto <caianbene@gmail.com>

Create init scripts directory during container build

ea23587

License: MIT Signed-off-by: Caian Benedicto <caianbene@gmail.com>

Update README with an example

0380149

License: MIT Signed-off-by: Caian Benedicto <caianbene@gmail.com>

chore: format code in readme

4ba8d34

lidel force-pushed the feat/container-init-scripts branch from b31c311 to ed202bc Compare April 11, 2022 13:22

chore: docker cleanup

accf6e2

cleans up containers and images (useful when run on developer machine)

lidel requested changes Apr 11, 2022

View reviewed changes

lidel requested a review from guseggert April 11, 2022 15:20

remove container init documentation from README

9935863

There is already IPFS Docker documentation where this should live: https://docs.ipfs.io/how-to/run-ipfs-inside-docker/

guseggert approved these changes Apr 11, 2022

View reviewed changes

guseggert requested a review from lidel April 11, 2022 16:01

lidel mentioned this pull request Apr 12, 2022

Document /container-init.d for Docker ipfs/ipfs-docs#1114

Closed

lidel approved these changes Apr 12, 2022

View reviewed changes

lidel merged commit 63b0025 into ipfs:master Apr 12, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(docker): /container-init.d for advanced initialization #6577

feat(docker): /container-init.d for advanced initialization #6577

Caian commented Aug 16, 2019

Caian commented Sep 2, 2019

lidel left a comment •

edited by guseggert

Loading

BigLep commented Oct 1, 2021

guseggert commented Oct 8, 2021

BigLep commented Apr 1, 2022 •

edited

Loading

BigLep commented Apr 1, 2022

BigLep commented Apr 1, 2022

lidel Apr 11, 2022

guseggert Apr 11, 2022

guseggert Apr 11, 2022 •

edited

Loading

guseggert Apr 11, 2022

lidel Apr 12, 2022

lidel Apr 12, 2022

guseggert commented Apr 11, 2022 •

edited by lidel

Loading

thattommyhall commented Apr 27, 2022

feat(docker): /container-init.d for advanced initialization #6577

feat(docker): /container-init.d for advanced initialization #6577

Conversation

Caian commented Aug 16, 2019

Caian commented Sep 2, 2019

lidel left a comment • edited by guseggert Loading

Choose a reason for hiding this comment

Remaining work

BigLep commented Oct 1, 2021

guseggert commented Oct 8, 2021

BigLep commented Apr 1, 2022 • edited Loading

BigLep commented Apr 1, 2022

BigLep commented Apr 1, 2022

lidel Apr 11, 2022

Choose a reason for hiding this comment

guseggert Apr 11, 2022

Choose a reason for hiding this comment

guseggert Apr 11, 2022 • edited Loading

Choose a reason for hiding this comment

guseggert Apr 11, 2022

Choose a reason for hiding this comment

lidel Apr 12, 2022

Choose a reason for hiding this comment

lidel Apr 12, 2022

Choose a reason for hiding this comment

guseggert commented Apr 11, 2022 • edited by lidel Loading

thattommyhall commented Apr 27, 2022

lidel left a comment •

edited by guseggert

Loading

BigLep commented Apr 1, 2022 •

edited

Loading

guseggert Apr 11, 2022 •

edited

Loading

guseggert commented Apr 11, 2022 •

edited by lidel

Loading