Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docker - Launching stack - service "initialize_app" fails #1803

Open
keebeegee opened this issue Jun 6, 2022 · 6 comments
Open

Docker - Launching stack - service "initialize_app" fails #1803

keebeegee opened this issue Jun 6, 2022 · 6 comments

Comments

@keebeegee
Copy link

keebeegee commented Jun 6, 2022

Problem:
service "initialize_app" didn't completed successfully and therefore prevent other services from starting

Expected behavior:
service "initialize_app" completes successfully so that the following services will be started

Environment:

  • Hyku v4.0.0.
  • Docker Compose version v2.6.0
  • Ubuntu 20.04 LST

Output:

kbg@cx30-580953:~/hyku$ docker-compose up -d web
WARN[0000] The "JAVA_OPTS" variable is not set. Defaulting to a blank string.
[+] Running 19/19
 ⠿ Network hyku_internal            Created                                                                                                                                                                                                                                                                            0.1s
 ⠿ Volume "hyku_redis"              Created                                                                                                                                                                                                                                                                            0.0s
 ⠿ Volume "hyku_solr"               Created                                                                                                                                                                                                                                                                            0.0s
 ⠿ Volume "hyku_zk"                 Created                                                                                                                                                                                                                                                                            0.0s
 ⠿ Volume "hyku_uploads"            Created                                                                                                                                                                                                                                                                            0.0s
 ⠿ Volume "hyku_assets"             Created                                                                                                                                                                                                                                                                            0.0s
 ⠿ Volume "hyku_cache"              Created                                                                                                                                                                                                                                                                            0.0s
 ⠿ Volume "hyku_fcrepo"             Created                                                                                                                                                                                                                                                                            0.0s
 ⠿ Volume "hyku_db"                 Created                                                                                                                                                                                                                                                                            0.0s
 ⠿ Container hyku-redis-1           Started                                                                                                                                                                                                                                                                            5.5s
 ⠿ Container hyku-zoo-1             Healthy                                                                                                                                                                                                                                                                           17.9s
 ⠿ Container hyku-check_volumes-1   Started                                                                                                                                                                                                                                                                            5.5s
 ⠿ Container hyku-db-1              Started                                                                                                                                                                                                                                                                            5.4s
 ⠿ Container hyku-chrome-1          Started                                                                                                                                                                                                                                                                            4.8s
 ⠿ Container hyku-fcrepo-1          Started                                                                                                                                                                                                                                                                            4.7s
 ⠿ Container hyku-solr-1            Healthy                                                                                                                                                                                                                                                                          100.7s
 ⠿ Container hyku-initialize_app-1  Exited                                                                                                                                                                                                                                                                           161.3s
 ⠿ Container hyku-worker-1          Created                                                                                                                                                                                                                                                                            0.2s
 ⠿ Container hyku-web-1             Created  

service "initialize_app" didn't completed successfully: exit 1

See below excerpts from log (initialize_app), see full log here: log.txt

...

hyku-initialize_app-1  | ERROR:  relation "schema_migrations" does not exist
hyku-initialize_app-1  | LINE 1: SELECT version FROM schema_migrations ORDER BY schema_migrat...

....

Migrating to AddThemesToSites (20210121233027)
   (0.3ms)  BEGIN
== 20210121233027 AddThemesToSites: migrating =================================
-- add_column(:sites, :home_theme, :string)
   (0.5ms)  ALTER TABLE "sites" ADD "home_theme" character varying
   -> 0.0009s
rails aborted!
Errno::EACCES: Permission denied @ rb_sysopen - /app/samvera/hyrax-webapp/db/schema.rb
/usr/local/bundle/gems/activerecord-5.2.8/lib/active_record/railties/databases.rake:250:in `initialize'
/usr/local/bundle/gems/activerecord-5.2.8/lib/active_record/railties/databases.rake:250:in `open'
/usr/local/bundle/gems/activerecord-5.2.8/lib/active_record/railties/databases.rake:250:in `block (3 levels) in <top (required)>'
/usr/local/bundle/gems/activerecord-5.2.8/lib/active_record/railties/databases.rake:68:in `block (2 levels) in <top (required)>'
/usr/local/bundle/gems/activerecord-5.2.8/lib/active_record/railties/databases.rake:61:in `block (2 levels) in <top (required)>'
/usr/local/bundle/gems/railties-5.2.8/lib/rails/commands/rake/rake_command.rb:23:in `block in perform'
/usr/local/bundle/gems/railties-5.2.8/lib/rails/commands/rake/rake_command.rb:20:in `perform'
/usr/local/bundle/gems/railties-5.2.8/lib/rails/command.rb:48:in `invoke'
/usr/local/bundle/gems/railties-5.2.8/lib/rails/commands.rb:18:in `<top (required)>'
bin/rails:4:in `require'
bin/rails:4:in `<main>'
Tasks: TOP => db:schema:dump
(See full trace by running task with --trace)
-- add_column(:sites, :show_theme, :string)
   (0.6ms)  ALTER TABLE "sites" ADD "show_theme" character varying
   -> 0.0010s
-- add_column(:sites, :search_theme, :string)
   (0.6ms)  ALTER TABLE "sites" ADD "search_theme" character varying
   -> 0.0008s
== 20210121233027 AddThemesToSites: migrated (0.0030s) ========================
...

Observations:

a) The error

 		rails aborted!
 		Errno::EACCES: Permission denied @ rb_sysopen - /app/samvera/hyrax-webapp/db/schema.rb

seems to be triggered in all cases during execution of the following migration step:

hyku/db/migrate/20210121233027_add_themes_to_sites.rb

b) Migration steps located before and after "20210121233027_add_themes_to_sites.rb" are executed without triggering any errors.

c) Everything seems to work fine in Ubuntu 20.04 - WSL2. Containers start without error:

....  
=> => naming to ghcr.io/samvera/hyku:latest                                                                       0.0s

Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them
[+] Running 19/19
 ⠿ Network hyku_internal            Created                                                                        0.8s
 ⠿ Volume "hyku_cache"              Created                                                                        0.0s
 ⠿ Volume "hyku_fcrepo"             Created                                                                        0.0s
 ⠿ Volume "hyku_solr"               Created                                                                        0.0s
 ⠿ Volume "hyku_zk"                 Created                                                                        0.0s
 ⠿ Volume "hyku_redis"              Created                                                                        0.0s
 ⠿ Volume "hyku_db"                 Created                                                                        0.0s
 ⠿ Volume "hyku_uploads"            Created                                                                        0.0s
 ⠿ Volume "hyku_assets"             Created                                                                        0.0s
 ⠿ Container hyku-fcrepo-1          Started                                                                        7.9s
 ⠿ Container hyku-db-1              Started                                                                        5.9s
 ⠿ Container hyku-check_volumes-1   Started                                                                        8.2s
 ⠿ Container hyku-redis-1           Started                                                                        8.1s
 ⠿ Container hyku-chrome-1          Started                                                                        7.7s
 ⠿ Container hyku-zoo-1             Healthy                                                                       20.6s
 ⠿ Container hyku-solr-1            Healthy                                                                       84.8s
 ⠿ Container hyku-initialize_app-1  Exited                                                                       209.4s
 ⠿ Container hyku-worker-1          Started                                                                      210.1s
 ⠿ Container hyku-web-1             Started
xx@DESKTOP-269UBN3:/mnt/d/xxx/xxx/hyku$
@keebeegee
Copy link
Author

Permission issue most likely caused by ignoring "newgrp docker"

@orangewolf
Copy link
Member

thank you for your report =-)

I think there are fundamentally two issues here. The first one is that we need add a migration to the schema.rb file and commit that in main. When you start up fresh it should load the schema and not need a migration right off the bat, but if one is not checked in then db/schema.rb file will be modified.

Second, and I think this is the bigger issue in regards to running in your original configuration is that inside the container, the db/schema.rb file can not be written too. This means that the docker-compose file mount is not getting the correct permissions. In both MacOS and Windows, bind mounts are handled by a fairly complex mount/sync process. My understanding is that in native Linux this process is simpler. I'm wondering, if you were to run docker-compose run initialize_app sh and look inside the container, are all the file permissions incorrect? Does your code checkout outside the container have the same user / permissions as the files inside it?

I do not have a lot of experience running docker-compose with bind mounts on Linux hosts (we mostly use Linux hosts for deployments and bind mounts are not common there). The first issue is one I'll take care of, but the if you can provide more information about the second it sounds like we at least need to add an optional step to the README.

@keebeegee keebeegee reopened this Jun 14, 2022
@keebeegee
Copy link
Author

Dear @orangewolf thanks for your input. I am re-opening this issue again as a proper start was only achieved once out of ~30 times. All subsequent tries after the single successful one failed as described above. Removing all images and re-creating them would not change anything.

Just to be clear: The err as described above occurred on a Ubuntu 20.04 LTS. I later tried to run the stack on a WIN WSL 2 (Ubuntu 20.04 LTS) and all containers started properly.

Will look into your advise - thanks again for looking into this.

@adamhgriffith-uofu
Copy link

adamhgriffith-uofu commented Sep 14, 2023

Hi @keebeegee and @orangewolf ,

On Linux hosts, the file permissions set on content in the Docker volume are identical from the perspective of the host as well as the container. Since the Dockerfile is rootless and runs as the app user, it doesn't have the permissions to read the mounted volume:

x-app: &app
  ...
  volumes:
   ...
    - .:/app/samvera/hyrax-webapp     <---------------------------------------------

Our group is also exploring this stack on our Linux workstations and found that commenting out the line indicated above resulted in a working environment.

@orangewolf
Copy link
Member

@adamhgriffith-uofu yes, if you remove the bind mount it works ok... but you don't get to edit code outside the container and have it work inside the container. That's ok for a production deployment but makes development very awkward. The good news is that we have a new solution for this using the user id that seems to be successful more uniformly. I'll try to get a PR up about it early next week.

@ryan-lang
Copy link

@orangewolf Have there been any updates on this issue?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants