Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create solr-updater docker service AGAIN #4287

Merged
merged 3 commits into from
Dec 31, 2020
Merged

Create solr-updater docker service AGAIN #4287

merged 3 commits into from
Dec 31, 2020

Conversation

cclauss
Copy link
Collaborator

@cclauss cclauss commented Dec 16, 2020

Closes #4244 (which got in a bad git state and was closed)
Related to #4060

Borrows from https://github.com/internetarchive/olsystem/blob/master/etc/supervisor/conf.d/solr-updater.conf to build a Docker service that can be run from ol-home0 to execute solr-updater.

Technical

Testing

Screenshot

Stakeholders

@cclauss cclauss added Module: Solr Issues related to the configuration or use of the Solr subsystem. [managed] Priority: 1 Do this week, receiving emails, time sensitive, . [managed] Theme: Upgrade to Python 3 labels Dec 16, 2020
@cclauss cclauss requested a review from cdrini December 16, 2020 16:38
@cclauss cclauss marked this pull request as ready for review December 21, 2020 14:11
@cclauss cclauss changed the title Draft: Create solr-updater docker service AGAIN Create solr-updater docker service AGAIN Dec 21, 2020
cclauss and others added 3 commits December 30, 2020 21:29
- Mount /openlibrary so can access local version of code
- Was getting errors about access; needs to ping both web and db
- Ugh, named volumes are created as root by default :/ Internet
recommends pre-creating the directory with the right perms in the
Dockerfile.
- Turns out a named volume can't be a single file :) The more you know!
Copy link
Collaborator

@cdrini cdrini left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, tested, fixed some bugs, and it's working! Fixes:

  • Mount /openlibrary so can access local version of code
  • Was getting errors about access; needs to ping both web and db
  • Ugh, named volumes are created as root by default :/ Internet recommends pre-creating the directory with the right perms in the Dockerfile.
  • Turns out a named volume can't be a single file :) The more you know!

Could you do another test, @cclauss and then merge at your discretion. To test:

cd vendor/infogami
git pull origin master
cd ../..
export COMPOSE_FILE="docker-compose.yml:docker-compose.override.yml:docker-compose.infogami-local.yml"
PYENV_VERSION=3.8.6 docker-compose up -d
docker-compose logs -f solr-updater
# Open the site and edit a book; it should log some stuff

DISCLAIMER: I did something slightly differently, but that should work 👍

@cclauss cclauss merged commit 1830ffc into internetarchive:master Dec 31, 2020
@cclauss cclauss deleted the solrupdater-on-Docker-again branch December 31, 2020 12:37
@cclauss
Copy link
Collaborator Author

cclauss commented Dec 31, 2020

solr-updater_1  | 2020-12-31 14:44:44 [openlibrary.solr-updater] [INFO] saving offset 2020-12-31:6364
solr-updater_1  | Traceback (most recent call last):
solr-updater_1  |   File "scripts/new-solr-updater.py", line 306, in <module>
solr-updater_1  |     main()
solr-updater_1  |   File "scripts/new-solr-updater.py", line 290, in main
solr-updater_1  |     with open(state_file, "w") as f:
solr-updater_1  | PermissionError: [Errno 13] Permission denied: '/solr-updater-data/solr-update.offset'

Sabreen-Parveen pushed a commit to Sabreen-Parveen/openlibrary that referenced this pull request Feb 5, 2021
* Create solr-updater docker service AGAIN

* Fixes to solr-updater volume mounts

- Mount /openlibrary so can access local version of code
- Was getting errors about access; needs to ping both web and db
- Ugh, named volumes are created as root by default :/ Internet
recommends pre-creating the directory with the right perms in the
Dockerfile.
- Turns out a named volume can't be a single file :) The more you know!

* Remove no longer needed solr/solr-updater gitignores

Co-authored-by: Drini Cami <cdrini@gmail.com>
Sabreen-Parveen pushed a commit to Sabreen-Parveen/openlibrary that referenced this pull request Feb 5, 2021
* Create solr-updater docker service AGAIN

* Fixes to solr-updater volume mounts

- Mount /openlibrary so can access local version of code
- Was getting errors about access; needs to ping both web and db
- Ugh, named volumes are created as root by default :/ Internet
recommends pre-creating the directory with the right perms in the
Dockerfile.
- Turns out a named volume can't be a single file :) The more you know!

* Remove no longer needed solr/solr-updater gitignores

Co-authored-by: Drini Cami <cdrini@gmail.com>
Sabreen-Parveen pushed a commit to Sabreen-Parveen/openlibrary that referenced this pull request Feb 5, 2021
* Create solr-updater docker service AGAIN

* Fixes to solr-updater volume mounts

- Mount /openlibrary so can access local version of code
- Was getting errors about access; needs to ping both web and db
- Ugh, named volumes are created as root by default :/ Internet
recommends pre-creating the directory with the right perms in the
Dockerfile.
- Turns out a named volume can't be a single file :) The more you know!

* Remove no longer needed solr/solr-updater gitignores

Co-authored-by: Drini Cami <cdrini@gmail.com>
Sabreen-Parveen pushed a commit to Sabreen-Parveen/openlibrary that referenced this pull request Feb 5, 2021
* Create solr-updater docker service AGAIN

* Fixes to solr-updater volume mounts

- Mount /openlibrary so can access local version of code
- Was getting errors about access; needs to ping both web and db
- Ugh, named volumes are created as root by default :/ Internet
recommends pre-creating the directory with the right perms in the
Dockerfile.
- Turns out a named volume can't be a single file :) The more you know!

* Remove no longer needed solr/solr-updater gitignores

Co-authored-by: Drini Cami <cdrini@gmail.com>
Sabreen-Parveen pushed a commit to Sabreen-Parveen/openlibrary that referenced this pull request Feb 5, 2021
* Create solr-updater docker service AGAIN

* Fixes to solr-updater volume mounts

- Mount /openlibrary so can access local version of code
- Was getting errors about access; needs to ping both web and db
- Ugh, named volumes are created as root by default :/ Internet
recommends pre-creating the directory with the right perms in the
Dockerfile.
- Turns out a named volume can't be a single file :) The more you know!

* Remove no longer needed solr/solr-updater gitignores

Co-authored-by: Drini Cami <cdrini@gmail.com>
Sabreen-Parveen pushed a commit to Sabreen-Parveen/openlibrary that referenced this pull request Feb 15, 2021
* Create solr-updater docker service AGAIN

* Fixes to solr-updater volume mounts

- Mount /openlibrary so can access local version of code
- Was getting errors about access; needs to ping both web and db
- Ugh, named volumes are created as root by default :/ Internet
recommends pre-creating the directory with the right perms in the
Dockerfile.
- Turns out a named volume can't be a single file :) The more you know!

* Remove no longer needed solr/solr-updater gitignores

Co-authored-by: Drini Cami <cdrini@gmail.com>
@mekarpeles mekarpeles added python Pull requests that update Python code and removed Module: Python labels Sep 7, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Module: Solr Issues related to the configuration or use of the Solr subsystem. [managed] Priority: 1 Do this week, receiving emails, time sensitive, . [managed] python Pull requests that update Python code
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants