Error response from daemon: unknown: artifact stencilapi:4.3.0.1 not found

[wei-hsin-story]

upgraded from v2.2.1 to v2.6.0 .

I found some conditions when pull partial image occurred "Error response from daemon: unknown: artifact stencilapi:4.3.0.1 not found"
But from UI , it exactly existed the image.

Then I deleted the image stencilapi and re-push ,
I feel confused from UI.

It have two the same image repository
Now it is work for me.
But how to remove redundant repositories ?
I have no idea to do , because delete it from UI seems to not work.

Is there more methods for correct operation ?

[tm-chang]

Hello, We upgrade harbor from 2.2. We also encountered the similar issue. We found different from Harbor UI to API request.
For example:
Use API:
https://xxx.xxx.com/api/v2.0/projects//repositories//artifacts/4.3.0.12/tags?page=1&page_size=10&with_signature=false&with_immutable_status=false
The result:
{
"errors": [
{
"code": "NOT_FOUND",
"message": "/<repo_name>:4.3.0.12 not found"
}
]
}

UI:

Pls help find root cause.
Thanks,
TM

[willislin0206]

Today we also had the same issues.
When upgrade to latest version , we found partial images CAN NOT pull.
Total image repository 2047 , about 60 counts occurred "Not Found"

any solutions or else advices for me?
Thanks!

[chlins]

Hello, We upgrade harbor from 2.2. We also encountered the similar issue. We found different from Harbor UI to API request. For example: Use API: https://xxx.xxx.com/api/v2.0/projects//repositories//artifacts/4.3.0.12/tags?page=1&page_size=10&with_signature=false&with_immutable_status=false The result: { "errors": [ { "code": "NOT_FOUND", "message": "/<repo_name>:4.3.0.12 not found" } ] }

UI:

Pls help find root cause. Thanks, TM

@tm-chang It seems your API path is not correct, please refer to harbor swagger docs.

[chlins]

@wei-hsin-story @willy2653 Please check the artifact table to look for the missing artifacts.

[stonezdj]

You could try this workaround:

1. Backup your database data and expose the database port
2. Check the duplicate repository name

// change the localhost to the hostname of your database host, and also change the username/password if required
docker run --rm firstfloor/before_reindex:latest --dburl=postgres://postgres:root123@localhost:5432/registry --command=list

3. Fix the duplicated repository name

// change the localhost to the hostname of your database host,and also change the username/password if required
docker run --rm firstfloor/before_reindex:latest --dburl=postgres://postgres:root123@localhost:5432/registry --command=fix

4. Reindex the repository table

REINDEX TABLE repository;

[wei-hsin-story]

@wei-hsin-story @willy2653 Please check the artifact table to look for the missing artifacts.
Hi @chlins
some imags were impacted can't pull by Tag Name.
It's very tricky.

If using the sha256 to pull, it's actually work !

[wei-hsin-story]

@stonezdj
I will try the method to remove duplicate repository name.
thanks for your advice and response !

[wy65701436]

dup with: #15620

[stonezdj]

@wei-hsin-story Does it work?

[wei-hsin-story]

@stonezdj
sorry for later.
I prepare the new environment to validate.

[wy65701436]

@stonezdj we still need to investigate why the index on repository name is corrupted.

[thechristschn]

I currently have the same problem after updating from 2.1.1 to 2.6.2 with 10 of 1272 repositories.

@stonezdj I would really like to try your proposed fix. But is the source of firstfloor/before_reindex:latest public? I would like to know what the image does, before I execute it in our environment.

[stonezdj]

The source of firstfloor/before_reindex:latest: https://github.com/stonezdj/before_reindex

[stonezdj]

Because this issue is uneasy to reproduce in our env, and we have no clear conclusion for it, handle it case by case.

[github-actions]

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.

[wy65701436]

Hey @stonezdj, what do you think about closing this issue and using the workaround mentioned in #17544 (comment)? Also, I didn't see any reports of this issue in v2.8.

[github-actions]

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.

[github-actions]

This issue was closed because it has been stalled for 30 days with no activity. If this issue is still relevant, please re-open a new issue.

[stonezdj]

The issue might related to the glibc2.28 change. the glibc 2.28 changed some character sort order esp -_$ with a-z A-Z
see
https://wiki.postgresql.org/wiki/Locale_data_changes
https://postgresql.verite.pro/blog/2018/08/27/glibc-upgrade.html

It caused the database index corruption.

Harbor-db v2.2.x --> glibc 2.26
Harbor-db v2.3.0 --> glibc 2.30
The only way to fix is to remove the duplicate item and reindex the table.