Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

support Dockerized s3 tests #337

Merged
merged 120 commits into from
Dec 7, 2023
Merged

support Dockerized s3 tests #337

merged 120 commits into from
Dec 7, 2023

Commits on Oct 18, 2023

  1. #327 initial jenkins.yml changes for phil's review

    Don Sizemore committed Oct 18, 2023
    Configuration menu
    Copy the full SHA
    4f8d5c9 View commit details
    Browse the repository at this point in the history
  2. #327 only one storage_driver_id, remains file

    Don Sizemore committed Oct 18, 2023
    Configuration menu
    Copy the full SHA
    57cf564 View commit details
    Browse the repository at this point in the history
  3. #327 remove config deletion from dataverse-storage.yml

    Don Sizemore committed Oct 18, 2023
    Configuration menu
    Copy the full SHA
    04c8a47 View commit details
    Browse the repository at this point in the history

Commits on Oct 30, 2023

  1. Merge remote-tracking branch 'origin/develop' into 327_multiple_s3_st…

    …ores
    Don Sizemore committed Oct 30, 2023
    Configuration menu
    Copy the full SHA
    b2eeb54 View commit details
    Browse the repository at this point in the history

Commits on Oct 31, 2023

  1. Merge remote-tracking branch 'origin/develop' into 327_multiple_s3_st…

    …ores
    Don Sizemore committed Oct 31, 2023
    Configuration menu
    Copy the full SHA
    83632c0 View commit details
    Browse the repository at this point in the history
  2. #327 stub out docker/localstack

    Don Sizemore committed Oct 31, 2023
    Configuration menu
    Copy the full SHA
    3e5dda7 View commit details
    Browse the repository at this point in the history
  3. Merge remote-tracking branch 'origin/develop' into 327_multiple_s3_st…

    …ores
    Don Sizemore committed Oct 31, 2023
    Configuration menu
    Copy the full SHA
    547771f View commit details
    Browse the repository at this point in the history
  4. #327 tweak lookup loop

    Don Sizemore committed Oct 31, 2023
    Configuration menu
    Copy the full SHA
    6bfe02d View commit details
    Browse the repository at this point in the history

Commits on Nov 6, 2023

  1. #327 add localstack_create.yml

    Don Sizemore committed Nov 6, 2023
    Configuration menu
    Copy the full SHA
    e9c6adb View commit details
    Browse the repository at this point in the history
  2. #327 add localstack_create_bucket.yml

    Don Sizemore committed Nov 6, 2023
    Configuration menu
    Copy the full SHA
    7137898 View commit details
    Browse the repository at this point in the history
  3. #327 add CORS config

    Don Sizemore committed Nov 6, 2023
    Configuration menu
    Copy the full SHA
    7f783ca View commit details
    Browse the repository at this point in the history

Commits on Nov 7, 2023

  1. #327 add cors.json for S3 testing

    Don Sizemore committed Nov 7, 2023
    Configuration menu
    Copy the full SHA
    8646de4 View commit details
    Browse the repository at this point in the history
  2. #327 remove errant localstack variable name prefix

    Don Sizemore committed Nov 7, 2023
    Configuration menu
    Copy the full SHA
    531cfc6 View commit details
    Browse the repository at this point in the history
  3. #327 correct docker repo installation

    Don Sizemore committed Nov 7, 2023
    Configuration menu
    Copy the full SHA
    d9fca9c View commit details
    Browse the repository at this point in the history
  4. #327 remove second errant localstack variable name prefix

    Don Sizemore committed Nov 7, 2023
    Configuration menu
    Copy the full SHA
    2314dd0 View commit details
    Browse the repository at this point in the history
  5. #327 ansible_user is now ansible_ssh_user

    Don Sizemore committed Nov 7, 2023
    Configuration menu
    Copy the full SHA
    6b53d93 View commit details
    Browse the repository at this point in the history
  6. #327 infer docker user

    Don Sizemore committed Nov 7, 2023
    Configuration menu
    Copy the full SHA
    c4b2918 View commit details
    Browse the repository at this point in the history
  7. #327 do we even need raw/endraw

    Don Sizemore committed Nov 7, 2023
    Configuration menu
    Copy the full SHA
    a520250 View commit details
    Browse the repository at this point in the history
  8. #327 add necessary localstack group_vars

    Don Sizemore committed Nov 7, 2023
    Configuration menu
    Copy the full SHA
    48d99f9 View commit details
    Browse the repository at this point in the history
  9. #327 add missing single ticks

    Don Sizemore committed Nov 7, 2023
    Configuration menu
    Copy the full SHA
    985a22b View commit details
    Browse the repository at this point in the history
  10. #327 container should be container_name

    Don Sizemore committed Nov 7, 2023
    Configuration menu
    Copy the full SHA
    33789ca View commit details
    Browse the repository at this point in the history
  11. #327 switch to docker compose v2

    Don Sizemore committed Nov 7, 2023
    Configuration menu
    Copy the full SHA
    09c8d33 View commit details
    Browse the repository at this point in the history
  12. #327 disable most of S3 configuration in favor of supporting multiple…

    … S3-based datastores
    Don Sizemore committed Nov 7, 2023
    Configuration menu
    Copy the full SHA
    e56b879 View commit details
    Browse the repository at this point in the history
  13. Configuration menu
    Copy the full SHA
    857316f View commit details
    Browse the repository at this point in the history
  14. #327 let's pass AWS keys to localstack

    Don Sizemore committed Nov 7, 2023
    Configuration menu
    Copy the full SHA
    7ebc99c View commit details
    Browse the repository at this point in the history
  15. #327 too many items spoil the broth

    Don Sizemore committed Nov 7, 2023
    Configuration menu
    Copy the full SHA
    6579842 View commit details
    Browse the repository at this point in the history
  16. #327 include_tasks with loop

    Don Sizemore committed Nov 7, 2023
    Configuration menu
    Copy the full SHA
    5f80f14 View commit details
    Browse the repository at this point in the history

Commits on Nov 8, 2023

  1. #327 loop with block

    Don Sizemore committed Nov 8, 2023
    Configuration menu
    Copy the full SHA
    0c595be View commit details
    Browse the repository at this point in the history
  2. #327 attempt include_tasks with_items

    Don Sizemore committed Nov 8, 2023
    Configuration menu
    Copy the full SHA
    5ddd7e6 View commit details
    Browse the repository at this point in the history
  3. #327 now it IS sending item properly

    Don Sizemore committed Nov 8, 2023
    Configuration menu
    Copy the full SHA
    456dd2e View commit details
    Browse the repository at this point in the history
  4. #327 it sees item now

    Don Sizemore committed Nov 8, 2023
    Configuration menu
    Copy the full SHA
    2b5e701 View commit details
    Browse the repository at this point in the history
  5. #327 remove errant item prefix

    Don Sizemore committed Nov 8, 2023
    Configuration menu
    Copy the full SHA
    19ec247 View commit details
    Browse the repository at this point in the history
  6. #327 add AWS binary where needed

    Don Sizemore committed Nov 8, 2023
    Configuration menu
    Copy the full SHA
    8529717 View commit details
    Browse the repository at this point in the history
  7. #327 just use OS-included binary for now

    Don Sizemore committed Nov 8, 2023
    Configuration menu
    Copy the full SHA
    b168fe8 View commit details
    Browse the repository at this point in the history
  8. #327 awscli package name may be specific to RedHat systems

    Don Sizemore committed Nov 8, 2023
    Configuration menu
    Copy the full SHA
    388c6fa View commit details
    Browse the repository at this point in the history
  9. #327 correct Red Hat check

    Don Sizemore committed Nov 8, 2023
    Configuration menu
    Copy the full SHA
    22aa1fd View commit details
    Browse the repository at this point in the history
  10. #327 place AWS config in dataverse user homedir

    Don Sizemore committed Nov 8, 2023
    Configuration menu
    Copy the full SHA
    3d585af View commit details
    Browse the repository at this point in the history
  11. #327 use dataverse homedir for localstack compose

    Don Sizemore committed Nov 8, 2023
    Configuration menu
    Copy the full SHA
    fc7a7c1 View commit details
    Browse the repository at this point in the history

Commits on Nov 9, 2023

  1. #327 don't kill the container

    Don Sizemore committed Nov 9, 2023
    Configuration menu
    Copy the full SHA
    c3f16a0 View commit details
    Browse the repository at this point in the history
  2. #327 perform s3 commands as service account

    Don Sizemore committed Nov 9, 2023
    Configuration menu
    Copy the full SHA
    d043403 View commit details
    Browse the repository at this point in the history

Commits on Nov 10, 2023

  1. #327 add dataverse user to docker group

    Don Sizemore committed Nov 10, 2023
    Configuration menu
    Copy the full SHA
    fd560aa View commit details
    Browse the repository at this point in the history
  2. #327 localstack is trying to nuke its own data_dir

    Don Sizemore committed Nov 10, 2023
    Configuration menu
    Copy the full SHA
    d5b0b66 View commit details
    Browse the repository at this point in the history
  3. #327 Phil's localstack config is much simpler

    Don Sizemore committed Nov 10, 2023
    Configuration menu
    Copy the full SHA
    28c1daf View commit details
    Browse the repository at this point in the history
  4. #327 correct container launch user

    Don Sizemore committed Nov 10, 2023
    Configuration menu
    Copy the full SHA
    d91a83d View commit details
    Browse the repository at this point in the history
  5. #327 localstack claims it doesn't need to be background-ed

    Don Sizemore committed Nov 10, 2023
    Configuration menu
    Copy the full SHA
    26c572d View commit details
    Browse the repository at this point in the history
  6. #327 container stays up when not background-ed

    Don Sizemore committed Nov 10, 2023
    Configuration menu
    Copy the full SHA
    54bda38 View commit details
    Browse the repository at this point in the history
  7. #327 correct conditional evaluation

    Don Sizemore committed Nov 10, 2023
    Configuration menu
    Copy the full SHA
    59f25ed View commit details
    Browse the repository at this point in the history
  8. #327 custom_endpoint_region should just be region

    Don Sizemore committed Nov 10, 2023
    Configuration menu
    Copy the full SHA
    13ac628 View commit details
    Browse the repository at this point in the history
  9. #327 label should be item.label

    Don Sizemore committed Nov 10, 2023
    Configuration menu
    Copy the full SHA
    cfe62fa View commit details
    Browse the repository at this point in the history
  10. #327 loop_control

    Don Sizemore committed Nov 10, 2023
    Configuration menu
    Copy the full SHA
    63c64db View commit details
    Browse the repository at this point in the history
  11. #327 use loop_control in outer loop

    Don Sizemore committed Nov 10, 2023
    Configuration menu
    Copy the full SHA
    8bc009e View commit details
    Browse the repository at this point in the history
  12. #327 item should be outer

    Don Sizemore committed Nov 10, 2023
    Configuration menu
    Copy the full SHA
    59939ba View commit details
    Browse the repository at this point in the history
  13. #327 correct key/value definitions

    Don Sizemore committed Nov 10, 2023
    Configuration menu
    Copy the full SHA
    5de7941 View commit details
    Browse the repository at this point in the history
  14. #327 replace key ticks, whoops

    Don Sizemore committed Nov 10, 2023
    Configuration menu
    Copy the full SHA
    d9cf884 View commit details
    Browse the repository at this point in the history

Commits on Nov 13, 2023

  1. #327 let's define our own dict

    Don Sizemore committed Nov 13, 2023
    Configuration menu
    Copy the full SHA
    cf23d82 View commit details
    Browse the repository at this point in the history

Commits on Nov 14, 2023

  1. #327 debug outer list

    Don Sizemore committed Nov 14, 2023
    Configuration menu
    Copy the full SHA
    2f79dcd View commit details
    Browse the repository at this point in the history
  2. #327 quotes and ticks, quotes and ticks

    Don Sizemore committed Nov 14, 2023
    Configuration menu
    Copy the full SHA
    09e19ca View commit details
    Browse the repository at this point in the history
  3. #327 debug jvm-options

    Don Sizemore committed Nov 14, 2023
    Configuration menu
    Copy the full SHA
    13c5e95 View commit details
    Browse the repository at this point in the history

Commits on Nov 15, 2023

  1. #327 let's set ingestsizelimit to 2000000000

    Don Sizemore committed Nov 15, 2023
    Configuration menu
    Copy the full SHA
    eeb3cb3 View commit details
    Browse the repository at this point in the history
  2. #327 correct key/value syntax

    Don Sizemore committed Nov 15, 2023
    Configuration menu
    Copy the full SHA
    f70f37c View commit details
    Browse the repository at this point in the history
  3. #327 jvm-options should be outer

    Don Sizemore committed Nov 15, 2023
    Configuration menu
    Copy the full SHA
    abd0e34 View commit details
    Browse the repository at this point in the history
  4. #327 jvm-options must be jvm_options

    Don Sizemore committed Nov 15, 2023
    Configuration menu
    Copy the full SHA
    c84b509 View commit details
    Browse the repository at this point in the history
  5. #327 let's debug bucketlist again

    Don Sizemore committed Nov 15, 2023
    Configuration menu
    Copy the full SHA
    8a2334b View commit details
    Browse the repository at this point in the history
  6. #327 Ansible handles bucket and CORS

    Don Sizemore committed Nov 15, 2023
    Configuration menu
    Copy the full SHA
    df0b84b View commit details
    Browse the repository at this point in the history
  7. #327 swap inner and outer loop_vars

    Don Sizemore committed Nov 15, 2023
    Configuration menu
    Copy the full SHA
    9ae290a View commit details
    Browse the repository at this point in the history
  8. #327 do we need to escape custom_endpoint_url?

    Don Sizemore committed Nov 15, 2023
    Configuration menu
    Copy the full SHA
    45c585d View commit details
    Browse the repository at this point in the history

Commits on Nov 17, 2023

  1. #327 correct unescaped custom endpoint url

    Don Sizemore committed Nov 17, 2023
    Configuration menu
    Copy the full SHA
    e0401c0 View commit details
    Browse the repository at this point in the history
  2. #327 I was using ticks instead of quotes

    Don Sizemore committed Nov 17, 2023
    Configuration menu
    Copy the full SHA
    1a4c641 View commit details
    Browse the repository at this point in the history
  3. #327 we bothered to escape the var, let's use it

    Don Sizemore committed Nov 17, 2023
    Configuration menu
    Copy the full SHA
    73d9607 View commit details
    Browse the repository at this point in the history

Commits on Nov 20, 2023

  1. #327 forgot to escape a dash

    Don Sizemore committed Nov 20, 2023
    Configuration menu
    Copy the full SHA
    4441add View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    ff12dc2 View commit details
    Browse the repository at this point in the history
  3. #327 correct cors.json syntax

    Don Sizemore committed Nov 20, 2023
    Configuration menu
    Copy the full SHA
    1e5db65 View commit details
    Browse the repository at this point in the history
  4. #327 whoops, cors settings requires custom_endpoint_escaped

    Don Sizemore committed Nov 20, 2023
    Configuration menu
    Copy the full SHA
    6f00946 View commit details
    Browse the repository at this point in the history
  5. #327 let's do MinIO as well, all in

    Don Sizemore committed Nov 20, 2023
    Configuration menu
    Copy the full SHA
    01b0b7a View commit details
    Browse the repository at this point in the history
  6. #327 stub out MinIO settings

    Don Sizemore committed Nov 20, 2023
    Configuration menu
    Copy the full SHA
    a7a4b86 View commit details
    Browse the repository at this point in the history

Commits on Nov 21, 2023

  1. Configuration menu
    Copy the full SHA
    f9973bf View commit details
    Browse the repository at this point in the history
  2. #327 untested stab at minio task

    Don Sizemore committed Nov 21, 2023
    Configuration menu
    Copy the full SHA
    5ffba8b View commit details
    Browse the repository at this point in the history
  3. #327 correct spacing, add some DV-specific jvm-options

    Don Sizemore committed Nov 21, 2023
    Configuration menu
    Copy the full SHA
    8cd0dea View commit details
    Browse the repository at this point in the history
  4. #327 correct group_vars syntax

    Don Sizemore committed Nov 21, 2023
    Configuration menu
    Copy the full SHA
    498075c View commit details
    Browse the repository at this point in the history
  5. #327 correct minio root setting tests

    Don Sizemore committed Nov 21, 2023
    Configuration menu
    Copy the full SHA
    dbf37bc View commit details
    Browse the repository at this point in the history
  6. #327 docker-compose.yml should be minio_compose.yml

    Don Sizemore committed Nov 21, 2023
    Configuration menu
    Copy the full SHA
    bcccf09 View commit details
    Browse the repository at this point in the history

Commits on Nov 27, 2023

  1. #327 correct minio template src filename

    Don Sizemore committed Nov 27, 2023
    Configuration menu
    Copy the full SHA
    4034d0b View commit details
    Browse the repository at this point in the history
  2. #327 correct variable typo in minio_compose.j2

    Don Sizemore committed Nov 27, 2023
    Configuration menu
    Copy the full SHA
    2912023 View commit details
    Browse the repository at this point in the history
  3. #327 correct variable typos in minio template

    Don Sizemore committed Nov 27, 2023
    Configuration menu
    Copy the full SHA
    4031296 View commit details
    Browse the repository at this point in the history

Commits on Nov 28, 2023

  1. #327 one final typo in MinIO jinja template

    Don Sizemore committed Nov 28, 2023
    Configuration menu
    Copy the full SHA
    7a199c6 View commit details
    Browse the repository at this point in the history
  2. #327 Ansible Docker module requires python-docker

    Don Sizemore committed Nov 28, 2023
    Configuration menu
    Copy the full SHA
    2f38d4c View commit details
    Browse the repository at this point in the history
  3. #327 make that python3-docker

    Don Sizemore committed Nov 28, 2023
    Configuration menu
    Copy the full SHA
    eafe3b9 View commit details
    Browse the repository at this point in the history
  4. #327 Ansible needs docker-compose python package

    Don Sizemore committed Nov 28, 2023
    Configuration menu
    Copy the full SHA
    3c6289d View commit details
    Browse the repository at this point in the history
  5. #327 what a difference a colon makes

    Don Sizemore committed Nov 28, 2023
    Configuration menu
    Copy the full SHA
    3589b92 View commit details
    Browse the repository at this point in the history
  6. #327 now pip errors. sigh.

    Don Sizemore committed Nov 28, 2023
    Configuration menu
    Copy the full SHA
    1edfef3 View commit details
    Browse the repository at this point in the history
  7. #327 specify minio_compose.yml

    Don Sizemore committed Nov 28, 2023
    Configuration menu
    Copy the full SHA
    073ff0f View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    bba73ad View commit details
    Browse the repository at this point in the history
  9. #327 don't use special docker network unless we have to

    Don Sizemore committed Nov 28, 2023
    Configuration menu
    Copy the full SHA
    15011d9 View commit details
    Browse the repository at this point in the history

Commits on Nov 29, 2023

  1. #327 launch minio as dataverse service account

    Don Sizemore committed Nov 29, 2023
    Configuration menu
    Copy the full SHA
    4156091 View commit details
    Browse the repository at this point in the history
  2. #327 do we need minio docker user/group?

    Don Sizemore committed Nov 29, 2023
    Configuration menu
    Copy the full SHA
    f82e48a View commit details
    Browse the repository at this point in the history
  3. #327 variables must exist, length 0

    Don Sizemore committed Nov 29, 2023
    Configuration menu
    Copy the full SHA
    5d9dff3 View commit details
    Browse the repository at this point in the history
  4. #327 keep variables, comment out in case

    Don Sizemore committed Nov 29, 2023
    Configuration menu
    Copy the full SHA
    62c4a22 View commit details
    Browse the repository at this point in the history
  5. #327 adjust commented spacing

    Don Sizemore committed Nov 29, 2023
    Configuration menu
    Copy the full SHA
    f04faaf View commit details
    Browse the repository at this point in the history
  6. #327 adjust template indentation

    Don Sizemore committed Nov 29, 2023
    Configuration menu
    Copy the full SHA
    db5f630 View commit details
    Browse the repository at this point in the history
  7. #327 if blocks causing trouble

    Don Sizemore committed Nov 29, 2023
    Configuration menu
    Copy the full SHA
    5e74320 View commit details
    Browse the repository at this point in the history

Commits on Nov 30, 2023

  1. #327 set MinIO jvm-options

    Don Sizemore committed Nov 30, 2023
    Configuration menu
    Copy the full SHA
    34cd897 View commit details
    Browse the repository at this point in the history
  2. #327 quote MinIO CEU value

    Don Sizemore committed Nov 30, 2023
    Configuration menu
    Copy the full SHA
    a497741 View commit details
    Browse the repository at this point in the history
  3. #327 escape custom-endpoint-url colons

    Don Sizemore committed Nov 30, 2023
    Configuration menu
    Copy the full SHA
    0fa758a View commit details
    Browse the repository at this point in the history
  4. #327 underscores only

    Don Sizemore committed Nov 30, 2023
    Configuration menu
    Copy the full SHA
    18aa4b2 View commit details
    Browse the repository at this point in the history

Commits on Dec 1, 2023

  1. #327 update minio variables

    Don Sizemore committed Dec 1, 2023
    Configuration menu
    Copy the full SHA
    e00dae3 View commit details
    Browse the repository at this point in the history
  2. #327 localstack1, mybucket

    Don Sizemore committed Dec 1, 2023
    Configuration menu
    Copy the full SHA
    e9a271a View commit details
    Browse the repository at this point in the history
  3. #327 localstack label, caps count

    Don Sizemore committed Dec 1, 2023
    Configuration menu
    Copy the full SHA
    01dc08f View commit details
    Browse the repository at this point in the history
  4. #327 LocalStack: match Phil's options exactly

    Don Sizemore committed Dec 1, 2023
    Configuration menu
    Copy the full SHA
    fd34213 View commit details
    Browse the repository at this point in the history
  5. #327 touch up variables post-Zoom

    Don Sizemore committed Dec 1, 2023
    Configuration menu
    Copy the full SHA
    e00d8d5 View commit details
    Browse the repository at this point in the history

Commits on Dec 4, 2023

  1. #327 run tests and reports as Dataverse service account

    Don Sizemore committed Dec 4, 2023
    Configuration menu
    Copy the full SHA
    801c6ee View commit details
    Browse the repository at this point in the history
  2. #327 disable global s3 creds/config as a test

    Don Sizemore committed Dec 4, 2023
    Configuration menu
    Copy the full SHA
    463763a View commit details
    Browse the repository at this point in the history
  3. #327 correct localstack variables

    Don Sizemore committed Dec 4, 2023
    Configuration menu
    Copy the full SHA
    76a21e6 View commit details
    Browse the repository at this point in the history
  4. #327 correct identifier group_var syntax

    Don Sizemore committed Dec 4, 2023
    Configuration menu
    Copy the full SHA
    4a3731f View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    7c1844e View commit details
    Browse the repository at this point in the history
  6. #327 correct minio secret-access-key

    Don Sizemore committed Dec 4, 2023
    Configuration menu
    Copy the full SHA
    d2ec1d8 View commit details
    Browse the repository at this point in the history

Commits on Dec 5, 2023

  1. Configuration menu
    Copy the full SHA
    48594cf View commit details
    Browse the repository at this point in the history
  2. #327 duplicate Docker CIDR to default/vagrant

    Don Sizemore committed Dec 5, 2023
    Configuration menu
    Copy the full SHA
    5b458dc View commit details
    Browse the repository at this point in the history

Commits on Dec 6, 2023

  1. #327 hopefully final touch-ups

    Don Sizemore committed Dec 6, 2023
    Configuration menu
    Copy the full SHA
    d66f79f View commit details
    Browse the repository at this point in the history
  2. #327 keep aws_config.yml for now

    Don Sizemore committed Dec 6, 2023
    Configuration menu
    Copy the full SHA
    2efaafa View commit details
    Browse the repository at this point in the history
  3. #327 correct minio secret_key

    Don Sizemore committed Dec 6, 2023
    Configuration menu
    Copy the full SHA
    37bae1f View commit details
    Browse the repository at this point in the history