CI integration tests + do not do a node lookup for multi-value field_member_of

[joecorall]

Link to Github issue or other discussion

#848 and #747

What does this PR do?

• Allows for multi-value field_member_of columns
  • Code change: d9cb22c
  • Test: 7d2f3db
• Adds integration test in CI

What changes were made?

How to test / verify this PR?

Look at the new integration test CI results

Interested Parties

@mjordan

---

Checklist

• Before opening this PR, have you opened an issue explaining what you want to to do?
• Have you included some configuration and/or CSV files useful for testing this PR?
• Have you written unit or integration tests if applicable?
• Does the code added in this PR require a version of Python that is higher than the current minimum version?
• If the changes in this PR require an additional Python library, have you included it in setup.py?
• If the changes in this PR add a new configuration option, have you provided a default for when the option is not present in the .yml file?

[mjordan]

@joecorall islandora_tests_check.py failing is a known issue I started working on a while ago but put aside. Can you remove running that one for now?

[joecorall]

@mjordan - i removed the islandora_tests_check.py test. Also made a few more changes to get the tests working on Mac OS and here in CI. Namely:

• 3c737db - a lot of the test workbench configs have /tmp hardcoded in them. Since that's the case, and mac OS has a different system temp directory than /tmp I changed the value to be hardcoded
• 6d9a948 - some of the islandora_tests.py tests were failing due to the CSV having comments in them that weren't nids. This makes sure only nids get written (and the file is freshly written to instead of appending if exists)

I realize this PR changes the way you were testing locally (replaces isle-dc assumptions with isle-site-template assumptions). Though to get the same setup as CI on your local machine you should be able to run:

git clone git@github.com:islandora-devops/isle-site-template ~/isle-site-template
cd ~/isle-site-template
export ISLANDORA_STARTER_OWNER=Islandora-Devops
export ISLANDORA_STARTER_REF=heads/main
export ISLANDORA_TAG=main
./tests/init-template-starter.sh
docker exec isle-site-template-drupal-dev-1 drush config:set media.settings standalone_url FALSE -y
docker exec isle-site-template-drupal-dev-1 drush cr

And adding the CA to python requests before running the paged content test

export REQUESTS_CA_BUNDLE=~/isle-site-template/certs/rootCA.pem

[mjordan]

@joecorall I'm getting a "no configuration file provided: not found" error when I run ./tests/init-template-starter.sh:

mark@mark-XPS-9320:/tmp/issue-848/isle-site-template$ ./tests/init-template-starter.sh 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100  480k  100  480k    0     0   405k      0  0:00:01  0:00:01 --:--:--  833k
The local CA is already installed in the system trust store! 👍
The local CA is now installed in the Firefox and/or Chrome/Chromium trust store (requires browser restart)! 🦊


Created a new certificate valid for the following names 📜
 - "*.islandora.dev"
 - "islandora.dev"
 - "*.islandora.io"
 - "islandora.io"
 - "*.islandora.info"
 - "islandora.info"
 - "localhost"
 - "127.0.0.1"
 - "::1"

Reminder: X.509 wildcards only go one level deep, so this won't match a.b.islandora.dev ℹ️

The certificate is at "certs/cert.pem" and the key at "certs/privkey.pem" ✅

It will expire on 1 March 2027 🗓

no configuration file provided: not found
no configuration file provided: not found

Any suggestions?

[joecorall]

@mjordan maybe your images are out of date? From inside the isle-site-template directory try running

docker pull islandora/base:main
docker compose --profile dev pull

[joecorall]

@mjordan and what's your output for

docker version && docker compose version

[mjordan]

mark@mark-XPS-9320:~/isle-site-template$ docker version && docker compose version yeilds:

Client:
 Version:           27.2.0
 API version:       1.47
 Go version:        go1.21.13
 Git commit:        3ab4256
 Built:             Fri Sep 20 16:23:22 2024
 OS/Arch:           linux/amd64
 Context:           default

Server:
 Engine:
  Version:          27.2.0
  API version:      1.47 (minimum version 1.24)
  Go version:       go1.21.13
  Git commit:       3ab5c7d
  Built:            Fri Sep 20 16:23:55 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.7.21
  GitCommit:        472731909fa34bd7bc9c087e4c27943f9835f111
 runc:
  Version:          1.1.13
  GitCommit:        
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
Docker Compose version v2.20.3

But when

mark@mark-XPS-9320:~/isle-site-template$ docker compose --profile dev pull
[+] Pulling 16/16
 ✔ houdini-dev Pulled                                                                                                                                                   1.4s 
 ✔ mariadb-dev Pulled                                                                                                                                                   1.4s 
 ✔ homarus-dev Pulled                                                                                                                                                   1.4s 
 ✔ fcrepo-dev Pulled                                                                                                                                                    1.4s 
 ✔ cantaloupe-dev Pulled                                                                                                                                                1.4s 
 ✔ crayfits-dev Pulled                                                                                                                                                  1.4s 
 ✔ alpaca-dev Pulled                                                                                                                                                    1.4s 
 ✔ activemq-dev Pulled                                                                                                                                                  1.4s 
 ✔ traefik-dev Pulled                                                                                                                                                   1.4s 
 ✔ ide Pulled                                                                                                                                                           1.4s 
 ✔ milliner-dev Pulled                                                                                                                                                  1.4s 
 ! drupal-dev Warning                                                                                                                                                   0.5s 
 ✔ solr-dev Pulled                                                                                                                                                      1.4s 
 ✔ hypercube-dev Pulled                                                                                                                                                 1.4s 
 ✔ blazegraph-dev Pulled                                                                                                                                                1.4s 
 ✔ fits-dev Pulled                                                                                                                                                      1.4s 
WARNING: Some service image(s) must be built from source by running:
    docker compose build drupal-dev
1 error occurred:
	* Error response from daemon: unknown: 404 page not found

Taking docker's advice to run docker compose build drupal-dev, I then get:

mark@mark-XPS-9320:~/isle-site-template$ docker compose build drupal-dev
[+] Building 6.5s (9/9) FINISHED                                                                                                                              docker:default
 => [drupal-dev internal] load build definition from Dockerfile                                                                                                         0.1s
 => => transferring dockerfile: 382B                                                                                                                                    0.0s
 => [drupal-dev] resolve image config for docker-image://docker.io/docker/dockerfile:1.5.1                                                                              1.4s
 => [drupal-dev] docker-image://docker.io/docker/dockerfile:1.5.1@sha256:d2d74ff22a0e47b21f4bbde337e2ac4cd0a02a2226ef79264878db3dc7e87df8                               1.5s
 => => resolve docker.io/docker/dockerfile:1.5.1@sha256:d2d74ff22a0e47b21f4bbde337e2ac4cd0a02a2226ef79264878db3dc7e87df8                                                0.0s
 => => sha256:6107562e90a8e27b67c589b63fce32010ddf37f48b701600cfd405e296651a5d 482B / 482B                                                                              0.0s
 => => sha256:89543fde88d7f1fb9104eb8832cfe0e33bbacf255e98e916338940bbd657754f 2.91kB / 2.91kB                                                                          0.0s
 => => sha256:dd092abd7f3683f4e8e7a66e770a1cc279b2132ac7f66d3c11b7d4a0cb529b7d 11.55MB / 11.55MB                                                                        1.3s
 => => sha256:d2d74ff22a0e47b21f4bbde337e2ac4cd0a02a2226ef79264878db3dc7e87df8 7.65kB / 7.65kB                                                                          0.0s
 => => extracting sha256:dd092abd7f3683f4e8e7a66e770a1cc279b2132ac7f66d3c11b7d4a0cb529b7d                                                                               0.2s
 => [drupal-dev internal] load .dockerignore                                                                                                                            0.1s
 => => transferring context: 50B                                                                                                                                        0.1s
 => [drupal-dev internal] load metadata for docker.io/islandora/drupal:main                                                                                             0.0s
 => [drupal-dev internal] load build context                                                                                                                            0.1s
 => => transferring context: 566.88kB                                                                                                                                   0.1s
 => [drupal-dev stage-0 1/3] FROM docker.io/islandora/drupal:main                                                                                                       0.1s
 => [drupal-dev stage-0 2/3] COPY --link rootfs /                                                                                                                       0.0s
 => ERROR [drupal-dev stage-0 3/3] RUN --mount=type=cache,id=custom-drupal-composer-amd64,sharing=locked,target=/root/.composer/cache     composer install -d /var/www  2.0s
------                                                                                                                                                                       
 > [drupal-dev stage-0 3/3] RUN --mount=type=cache,id=custom-drupal-composer-amd64,sharing=locked,target=/root/.composer/cache     composer install -d /var/www/drupal &&     chown -R nginx:nginx /var/www/drupal &&     cleanup.sh:
1.997 Composer could not find a composer.json file in /var/www/drupal
1.997 To initialize a project, please create a composer.json file. See https://getcomposer.org/basic-usage
------
failed to solve: process "/bin/bash -euo pipefail -c composer install -d /var/www/drupal &&     chown -R nginx:nginx /var/www/drupal &&     cleanup.sh" did not complete successfully: exit code: 1