Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Updates to node_modules on image rebuild not reflected in container's node_modules volume #2032

Open
mekarpeles opened this issue Apr 8, 2019 · 3 comments
Labels
Affects: Configuration Related to the configuration of the dev/staging/prod environments, CI, docker, etc. [managed] Lead: @cdrini Issues overseen by Drini (Staff: Team Lead & Solr, Library Explorer, i18n) [managed] Module: Docker Issues related to the configuration or use of Docker. [managed] Priority: 3 Issues that we can consider at our leisure. [managed] Theme: Development Issues related to the developer experience and the dev environment. [managed] Type: Refactor/Clean-up Issues related to reorganization/clean-up of data or code (e.g. for maintainability). [managed]
Milestone

Comments

@mekarpeles
Copy link
Member

mekarpeles commented Apr 8, 2019

Description

Because we volume mount ./ to /openlibrary, /openlibrary/node_modules (which is populated when npm install is run inside the Dockerfile) gets rewritten by what's in the local directory. We've solved this issue by adding another volume mount specifically to /openlibrary/node_modules. This has had the unintended consequence that if the images are rebuilt, even though npm install is run there, when they are launched, they attach to the volume which contains the old version of /openlibrary/node_modules. So npm install has to be run again to update the volume.

What we want:

  1. Rebuilding the image should run npm install
  2. Running docker-compose up should NOT require running npm install
  3. Running docker-compose up should have the same npm versions from the image.

External links:

  • https://jdlm.info/articles/2016/03/06/lessons-building-node-app-docker.html#the-node_modules-volume-trick : a guide on how this problem was handled in 2016 on another project. They use the same kind of volume mount trick, but using non-named / non-persistent volumes. I will investigate whether moving back to non-named volumes will guarantee they are created fresh every docker-compose up with the correct and latest node_modules from the base image.
    The article has a very good description of the node_modules problem in the section "The node_modules Volume Trick" and how mounting the local dev dir overwrites the installed image modules with the empty host node_modules. The blue / red filenames is a great way to illustrate a problem (I have found) difficult to describe clearly.
@mekarpeles mekarpeles added dev-instance Module: Docker Issues related to the configuration or use of Docker. [managed] labels Apr 8, 2019
@mekarpeles mekarpeles assigned hornc and unassigned cdrini Apr 8, 2019
@mekarpeles
Copy link
Member Author

more info here: #2030 (comment)

@cdrini cdrini changed the title Fix updates to node_modules path in docker using anonymous volumes Docker should use image's node_modules instead of volume Apr 21, 2019
@cdrini cdrini changed the title Docker should use image's node_modules instead of volume Fix updates to node_modules path in docker using anonymous volumes Apr 21, 2019
@cdrini cdrini changed the title Fix updates to node_modules path in docker using anonymous volumes Updates to node_modules on image rebuild not reflected in container's node_modules volume Apr 21, 2019
@brad2014 brad2014 added Theme: Development Issues related to the developer experience and the dev environment. [managed] and removed CH: dev-instance labels May 14, 2019
@xayhewalo xayhewalo added Affects: Configuration Related to the configuration of the dev/staging/prod environments, CI, docker, etc. [managed] Priority: 3 Issues that we can consider at our leisure. [managed] State: Backlogged Type: Refactor/Clean-up Issues related to reorganization/clean-up of data or code (e.g. for maintainability). [managed] labels Nov 10, 2019
@mekarpeles mekarpeles added the Lead: @hornc Issues overseen by Charles (Staff: Data Engineering Lead) [managed] label Dec 18, 2019
@hornc hornc removed their assignment Mar 7, 2020
@cclauss cclauss mentioned this issue Jan 7, 2021
14 tasks
@cclauss
Copy link
Collaborator

cclauss commented Mar 26, 2021

@hornc @cdrini @mekarpeles What is the status of this issue?

@cclauss cclauss added this to the Sprint 2023-04 milestone Mar 24, 2023
@mekarpeles mekarpeles modified the milestones: Sprint 2023-04, The Icebox Mar 27, 2023
@mekarpeles mekarpeles added Lead: @cdrini Issues overseen by Drini (Staff: Team Lead & Solr, Library Explorer, i18n) [managed] and removed Lead: @hornc Issues overseen by Charles (Staff: Data Engineering Lead) [managed] labels Mar 27, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Affects: Configuration Related to the configuration of the dev/staging/prod environments, CI, docker, etc. [managed] Lead: @cdrini Issues overseen by Drini (Staff: Team Lead & Solr, Library Explorer, i18n) [managed] Module: Docker Issues related to the configuration or use of Docker. [managed] Priority: 3 Issues that we can consider at our leisure. [managed] Theme: Development Issues related to the developer experience and the dev environment. [managed] Type: Refactor/Clean-up Issues related to reorganization/clean-up of data or code (e.g. for maintainability). [managed]
Projects
None yet
Development

No branches or pull requests

7 participants
@hornc @mekarpeles @cclauss @cdrini @brad2014 @xayhewalo and others