Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Re-sync of a collection fails when a newer version is available #1767

Closed
markafarrell opened this issue Feb 23, 2024 · 6 comments
Closed

Re-sync of a collection fails when a newer version is available #1767

markafarrell opened this issue Feb 23, 2024 · 6 comments
Labels
Issue Triage-Needed

Comments

@markafarrell
Copy link

Version

pulp-ansible==0.21.1
pulp-certguard==1.8.0
pulp-container==2.18.0
pulp-deb==3.1.1
pulp-file==1.16.0
pulp-gem==0.5.0
pulp-glue==0.23.2
pulp-glue-gem==0.3.0
pulp-maven==0.8.0
pulp-ostree==2.3.0
pulp-python==3.11.0
pulp-rpm==3.25.1
pulpcore==3.46.0

Describe the bug
Re-sync of a collection fails when a newer version is available

To Reproduce

sudo rm -rf container-data

docker run --detach \
    --rm \
    --publish 8443:443 \
    --name pulp \
    --volume "$(pwd)/container-data/settings":/etc/pulp \
    --volume "$(pwd)/container-data/pulp_storage":/var/lib/pulp \
    --volume "$(pwd)/container-data/pgsql":/var/lib/pgsql \
    --volume "$(pwd)/container-data/containers":/var/lib/containers \
    -e PULP_SECRET_KEY='Vf59GLiOInlH8lBcQ09YdW2JQ1OvWryU' \
    -e PULP_CONTENT_ORIGIN='https://localhost:8443' \
    -e PULP_ANSIBLE_CONTENT_HOSTNAME='https://localhost:8443/pulp/content' \
    -e PULP_ANSIBLE_API_HOSTNAME='https://localhost:8443' \
    -e https_proxy="http://172.17.0.1:3128" \
    -e http_proxy="http://172.17.0.1:3128" \
    -e PULP_API_WORKERS_MAX_REQUESTS=10 \
    -e PULP_HTTPS=true \
    --health-cmd='curl --noproxy 127.0.0.1 --fail --insecure https://127.0.0.1/pulp/api/v3/status/ || exit 1' \
    --health-interval=1s \
    --health-retries=0 \
    --health-timeout=1s \
    --health-start-period=5s \
    pulp/pulp:3.46.0

while [ $(docker inspect -f '{{ .State.Health.Status }}' pulp) != 'healthy' ]
do
    sleep 1
    echo -n '.'
done
echo

docker exec -it pulp pip3 freeze | grep pulp

docker exec -it pulp bash -c 'pulpcore-manager reset-admin-password -p 3aEqMY23ckanMhEo'

virtualenv .venv
source .venv/bin/activate
pip3 install pulp-cli[pygments]

pulp config create --username admin --base-url https://localhost:8443 --password 3aEqMY23ckanMhEo --no-verify-ssl --overwrite

pulp user create --username readonly
pulp user role-assignment add --username=readonly --role "ansible.ansiblerepository_viewer" --domain ""
pulp user update --username=readonly --password="readonly"


pulp ansible repository create --name "galaxy_mirror"

pulp ansible distribution create --name "galaxy_mirror" --base-path "galaxy_mirror" --repository "galaxy_mirror"

pulp ansible remote -t "collection" create --name "galaxy_collections" --proxy-url="http://172.17.0.1:3128" --url "https://galaxy.ansible.com/" --requirements $'{"collections": [ {"name":"openstack.cloud", "version": "2.1.0"}]}'
pulp ansible remote -t "collection" show --name "galaxy_collections"
pulp ansible repository sync --name "galaxy_mirror" --remote "galaxy_collections"

pulp ansible remote -t "collection" update --name "galaxy_collections" --proxy-url="http://172.17.0.1:3128" --url "https://galaxy.ansible.com/" --requirements $'{"collections": [ {"name":"openstack.cloud" }]}'
pulp ansible remote -t "collection" show --name "galaxy_collections"
pulp ansible repository sync --name "galaxy_mirror" --remote "galaxy_collections"

docker stop pulp

deactivate

sudo rm -rf container-data

Output

Created config file at '/home/d384492/.config/pulp/cli.toml'.
{
  "pulp_href": "/pulp/api/v3/users/2/",
  "id": 2,
  "username": "readonly",
  "first_name": "",
  "last_name": "",
  "email": "",
  "is_staff": false,
  "is_active": true,
  "date_joined": "2024-02-23T03:47:36.839112Z",
  "groups": [],
  "hidden_fields": [
    {
      "name": "password",
      "is_set": true
    }
  ]
}

{
  "pulp_href": "/pulp/api/v3/users/2/roles/018dd413-64de-75b8-a367-70c00e8e72a0/",
  "pulp_created": "2024-02-23T03:47:37.312135Z",
  "role": "ansible.ansiblerepository_viewer",
  "content_object": null,
  "description": null,
  "permissions": [
    "ansible.view_ansiblerepository"
  ],
  "domain": null
}

{
  "pulp_href": "/pulp/api/v3/repositories/ansible/ansible/018dd413-68d3-732e-b138-d09502d9e252/",
  "pulp_created": "2024-02-23T03:47:38.324446Z",
  "versions_href": "/pulp/api/v3/repositories/ansible/ansible/018dd413-68d3-732e-b138-d09502d9e252/versions/",
  "pulp_labels": {},
  "latest_version_href": "/pulp/api/v3/repositories/ansible/ansible/018dd413-68d3-732e-b138-d09502d9e252/versions/0/",
  "name": "galaxy_mirror",
  "description": null,
  "retain_repo_versions": null,
  "remote": null,
  "last_synced_metadata_time": null,
  "gpgkey": null,
  "last_sync_task": null,
  "private": false
}

Started background task /pulp/api/v3/tasks/018dd413-6b2f-78b6-9ae8-ef0f034b5de7/
.Done.
{
  "pulp_href": "/pulp/api/v3/distributions/ansible/ansible/018dd413-6c39-73c3-a842-9a26a032ddb9/",
  "pulp_created": "2024-02-23T03:47:39.194285Z",
  "base_path": "galaxy_mirror",
  "content_guard": null,
  "name": "galaxy_mirror",
  "repository": "/pulp/api/v3/repositories/ansible/ansible/018dd413-68d3-732e-b138-d09502d9e252/",
  "repository_version": null,
  "client_url": "https://localhost:8443/pulp_ansible/galaxy/galaxy_mirror/",
  "pulp_labels": {}
}
{
  "pulp_href": "/pulp/api/v3/remotes/ansible/collection/018dd413-803e-712f-950b-c9954d865c26/",
  "pulp_created": "2024-02-23T03:47:44.319527Z",
  "name": "galaxy_collections",
  "url": "https://galaxy.ansible.com/",
  "ca_cert": null,
  "client_cert": null,
  "tls_validation": true,
  "proxy_url": "http://172.17.0.1:3128",
  "pulp_labels": {},
  "pulp_last_updated": "2024-02-23T03:47:44.319539Z",
  "download_concurrency": null,
  "max_retries": null,
  "policy": "immediate",
  "total_timeout": null,
  "connect_timeout": null,
  "sock_connect_timeout": null,
  "sock_read_timeout": null,
  "headers": null,
  "rate_limit": null,
  "hidden_fields": [
    {
      "name": "client_key",
      "is_set": false
    },
    {
      "name": "proxy_username",
      "is_set": false
    },
    {
      "name": "proxy_password",
      "is_set": false
    },
    {
      "name": "username",
      "is_set": false
    },
    {
      "name": "password",
      "is_set": false
    },
    {
      "name": "token",
      "is_set": false
    }
  ],
  "requirements_file": "{'collections': [{'name': 'openstack.cloud', 'version': '2.1.0'}]}",
  "auth_url": null,
  "sync_dependencies": true,
  "signed_only": false,
  "last_sync_task": null
}

{
  "pulp_href": "/pulp/api/v3/remotes/ansible/collection/018dd413-803e-712f-950b-c9954d865c26/",
  "pulp_created": "2024-02-23T03:47:44.319527Z",
  "name": "galaxy_collections",
  "url": "https://galaxy.ansible.com/",
  "ca_cert": null,
  "client_cert": null,
  "tls_validation": true,
  "proxy_url": "http://172.17.0.1:3128",
  "pulp_labels": {},
  "pulp_last_updated": "2024-02-23T03:47:44.319539Z",
  "download_concurrency": null,
  "max_retries": null,
  "policy": "immediate",
  "total_timeout": null,
  "connect_timeout": null,
  "sock_connect_timeout": null,
  "sock_read_timeout": null,
  "headers": null,
  "rate_limit": null,
  "hidden_fields": [
    {
      "name": "client_key",
      "is_set": false
    },
    {
      "name": "proxy_username",
      "is_set": false
    },
    {
      "name": "proxy_password",
      "is_set": false
    },
    {
      "name": "username",
      "is_set": false
    },
    {
      "name": "password",
      "is_set": false
    },
    {
      "name": "token",
      "is_set": false
    }
  ],
  "requirements_file": "{'collections': [{'name': 'openstack.cloud', 'version': '2.1.0'}]}",
  "auth_url": null,
  "sync_dependencies": true,
  "signed_only": false,
  "last_sync_task": null
}

Started background task /pulp/api/v3/tasks/018dd413-8419-74df-80d4-4e54db293ee8/
...........Done.

Started background task /pulp/api/v3/tasks/018dd414-39a2-7434-9c5f-a7f45514fd8a/
Done.
{
  "pulp_href": "/pulp/api/v3/remotes/ansible/collection/018dd413-803e-712f-950b-c9954d865c26/",
  "pulp_created": "2024-02-23T03:47:44.319527Z",
  "name": "galaxy_collections",
  "url": "https://galaxy.ansible.com/",
  "ca_cert": null,
  "client_cert": null,
  "tls_validation": true,
  "proxy_url": "http://172.17.0.1:3128",
  "pulp_labels": {},
  "pulp_last_updated": "2024-02-23T03:48:31.802040Z",
  "download_concurrency": null,
  "max_retries": null,
  "policy": "immediate",
  "total_timeout": null,
  "connect_timeout": null,
  "sock_connect_timeout": null,
  "sock_read_timeout": null,
  "headers": null,
  "rate_limit": null,
  "hidden_fields": [
    {
      "name": "client_key",
      "is_set": false
    },
    {
      "name": "proxy_username",
      "is_set": false
    },
    {
      "name": "proxy_password",
      "is_set": false
    },
    {
      "name": "username",
      "is_set": false
    },
    {
      "name": "password",
      "is_set": false
    },
    {
      "name": "token",
      "is_set": false
    }
  ],
  "requirements_file": "{'collections': [{'name': 'openstack.cloud'}]}",
  "auth_url": null,
  "sync_dependencies": true,
  "signed_only": false,
  "last_sync_task": {
    "pk": "018dd413-8419-74df-80d4-4e54db293ee8",
    "error": null,
    "state": "completed",
    "finished_at": "2024-02-23T03:47:57.736981+00:00",
    "pulp_created": "2024-02-23T03:47:45.30679+00:00"
  }
}

Started background task /pulp/api/v3/tasks/018dd414-3eda-7af1-bc7a-0c3737d26054/
....................Error: Task /pulp/api/v3/tasks/018dd414-3eda-7af1-bc7a-0c3737d26054/ failed: 'duplicate key value violates unique constraint "unique_is_highest"
DETAIL:  Key (collection_id, is_highest)=(018dd413-b3a6-7350-9b15-fb047f732089, t) already exists.'

Expected behavior
We are able to re-sync an ansible collection when a new version is available

Additional context
I believe that this is potentially fixed by #1710

Is there a plan to release a bugfix release (0.21.2?) including this change?
@mdellweg
Copy link
Member

Let's try.

@mdellweg
Copy link
Member

Please try again.

@mdellweg
Copy link
Member

Assuming that releasing that backport fixed the issue, I'm closing it. Please reopen if needed.

@pulpbot pulpbot moved this to Done in RH Pulp Kanban board Feb 26, 2024
@markafarrell
Copy link
Author

@mdellweg Looks like its fixed the issue. Any chance we can build and push a version of the pulp/pulp (https://github.com/pulp/pulp-oci-images) docker image that includes this version?

@mdellweg
Copy link
Member

They are created nightly, I think.

@markafarrell
Copy link
Author

It looks like the nightly ci job is failing

https://github.com/pulp/pulp-oci-images/actions/runs/8042330428

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Issue Triage-Needed
Projects
No open projects
RH Pulp Kanban board
Archived in project
Milestone
No milestone
Development

No branches or pull requests
2 participants
@markafarrell @mdellweg