Server misbehaving when trying to push/pull/login to docker registry

[jessielw]

Description

Issue with package registry. I've not changed anything in my configuration and all of a sudden I can't push/pull images from the package registry. I am on the latest nightly build.

I tried docker logout/login and still I get errors like this.

Error response from daemon: Get "https://URL/v2/": Get "http://gitea:3000/gitea/v2/token?account=username&client_id=docker&offline_token=true&service=container_registry": dial tcp: lookup gitea on 127.0.0.53:53: server misbehaving

I get this error in the logs of gitea

2024/05/16 23:58:47 ...eb/routing/logger.go:102:func1() [I] router: completed GET /v2/ for 172.18.0.17:51504, 401 Unauthorized in 0.1ms @ container/container.go:124(container.ReqContainerAccess)

Gitea Version

Current nightly build

Can you reproduce the bug on the Gitea demo site?

No

Log Gist

No response

Screenshots

No response

Git Version

nightly

Operating System

Docker/UnRaid

How are you running Gitea?

Docker

Database

MySQL/MariaDB

---

Edit

Rolling back to https://hub.docker.com/layers/gitea/gitea/1.22.0-rc1/images/sha256-031ba2851bbc155db8f676154730257aa803a7f46058e9eb409a89f6527fe09d?context=explore resolved the issue. I don't know the go language completely but I assume the failure comes somewhere in this commit 67c1a07#diff-36426f1aaff58f9b2084f7fc13593b237474e1d75a4e2883e0a44351e367ff76

maybe in here

Let me know if there is something I need to do on my end, thanks!

[wxiaoguang]

Related to #30885

• How do you deploy the servers? Is there a reverse proxy, and are the "Host" / "X-Forwarded-Proto" headers correctly set?
• What does the "Self Check" page say in the admin panel (nightly)?

[jessielw]

Related to #30885

• How do you deploy the servers? Is there a reverse proxy, and are the "Host" / "X-Forwarded-Proto" headers correctly set?
• What does the "Self Check" page say in the admin panel (nightly)?

I deploy with docker/nginx (swag). I have the nginx config setup properly (as advised in the docs) and haven't modified any of that. I did double check all of that though just to be sure.

I did have some database warnings in the health page that I used the doctor to resolve.

For MySQL/MariaDB users, you could use the "gitea doctor convert" command to fix the collation problems, or you could also fix the problem by "ALTER ... COLLATE ..." SQLs manually.

This was present on nightly before I rolled back. Fixing it didn't resolve the issue.

[wxiaoguang]

Could you share your nginx config?

---

I did a quick test:

upstream gitea {
  server 127.0.0.1:3000;
}

server {
  server_name this-host.local;
  listen 80;
  location / {
    proxy_pass http://gitea;
    ## proxy_set_header Host $host;
    ## proxy_set_header X-Forwarded-Proto $scheme;
  }
}

Without these proxy_set_header:

$ curl -v http://this-host.local/v2
< Www-Authenticate: Bearer realm="http://gitea/v2/token",service="container_registry",scope="*"

With these proxy_set_header (uncomment):

$ curl -v http://this-host.local/v2
< Www-Authenticate: Bearer realm="http://this-host.local/v2/token",service="container_registry",scope="*"

I think these are expected behaviors.

[wxiaoguang]

And one more thing, I can see that you are using Gitea in a sub-path.

So have you checked/changed your /v2 config section in your nginx to make sure the headers are correctly set?

[wxiaoguang]

And one more thing, I can see that you are using Gitea in a sub-path.

So have you checked/changed your /v2 config section in your nginx to make sure the headers are correctly set?

Hmm, I can see one new problem here.

By design (required by the "container" standard ....), the container /v2 path should always be in the root. But AppURL (and the new GuessCurrentAppURL) always have the "sub-path". I will make a new PR to remove the sub-path from "realm".

[jessielw]

This is the configuration I've been running for a while. I followed the documentation/an issue on the tracker here to set it up like so. It's been working really well until that commit as far as I am aware. Here is my nginx.

location /gitea {
    return 301 $scheme://$host/gitea/;
}

location /gitea/ {
    client_max_body_size 512M;

    # make nginx use unescaped URI, keep "%2F" as is
    rewrite ^ $request_uri;
    rewrite ^/gitea(/.*) $1 break;
    proxy_pass http://gitea:3000$uri;

    proxy_set_header Connection $http_connection;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
}

# This forwards docker traffic to gitea
location /v2/ {
    client_max_body_size 10G;
    proxy_pass http://gitea:3000/v2/;
}

Let me know if anything should be changed.

Yes I am using gitea in a sub-path. Thanks for the help so far!

[wxiaoguang]

# This forwards docker traffic to gitea
location /v2/ {
    client_max_body_size 10G;
    proxy_pass http://gitea:3000/v2/;
}

So, please add proxy_set_header in this section too. 🎉

(just copy these from the location /gitea/ section)

[jessielw]

I will add that as well when I can get back on my computer in the morning. Thanks for the advice and looking into the bug. I didn't realize that was supposed to go in that block as well.

I'll be back on in the AM. Thanks again!

[KN4CK3R]

Wasn't there a section in https://docs.gitea.com/administration/reverse-proxies which described the additional /v2/ route? I'm missing that part in the current docs.

[wxiaoguang]

Wasn't there a section in https://docs.gitea.com/administration/reverse-proxies which described the additional /v2/ route? I'm missing that part in the current docs.

That's why I proposed: Improve reverse proxy documents #31003

[KN4CK3R]

Found the docs. It's only available in the "next" version, so with 1.22 docs. https://docs.gitea.com/next/administration/reverse-proxies#docker--container-registry

[wxiaoguang]

Found the docs. It's only available in the "next" version, so with 1.22 docs. https://docs.gitea.com/next/administration/reverse-proxies#docker--container-registry

The problem is that the section is too simple, it doesn't mention that the header & URI decoding requirements.

[jessielw]

@wxiaoguang so should I try nightly again while adding the headers to the docker v2 block or wait until a patch?

[wxiaoguang]

@wxiaoguang so should I try nightly again while adding the headers to the docker v2 block or wait until a patch?

No patch for logic change at the moment.

"Improve reverse proxy documents #31003" is a document improvement:

4. Make sure `Host` and `X-Fowarded-Proto` headers are correctly passed to Gitea to make sure Gitea sees the real URL being visited.

### Use a sub-path

Usually it's **not recommended** to put Gitea in a sub-path, it's not widely used and may have some issues in rare cases.

If you really need to do so, to make Gitea work with sub-path (eg: `https://common.example.com/gitea/`),
here are the extra requirements besides the general configuration above:

1. Use `[server] ROOT_URL = https://common.example.com/gitea/` in your `app.ini` file.
2. Make the reverse-proxy pass `https://common.example.com/gitea/foo` to `http://gitea:3000/foo`.
3. If you'd like to use container registry, the container registry uses a fixed sub-path `/v2` in the root, which is unchangeable and required by container registry standard.
   - Make reverse-proxy pass `https://common.example.com/v2` to `http://gitea:3000/v2`.
   - Make sure the URI and headers are also correctly passed (see the general configuration above).

So you could use nightly and add these headers to the /v2 nginx config section.

[jessielw]

I tested it, adding the headers did the trick. I agree the documentation wasn't quite clear on this particular use case when it comes to nginx.

I'm not sure if you want me to close this or if you're wanting to keep it open until the documentation patch comes through?

For anyone that is curious for a complete example nginx config with a sub folder (this is used with swag)

location /gitea {
    return 301 $scheme://$host/gitea/;
}

location /gitea/ {
    client_max_body_size 512M;

    # make nginx use unescaped URI, keep "%2F" as is
    rewrite ^ $request_uri;
    rewrite ^/gitea(/.*) $1 break;
    proxy_pass http://gitea:3000$uri;

    # common http headers
    proxy_set_header Connection $http_connection;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    
}

# This forwards docker traffic to gitea
location /v2/ {
    client_max_body_size 10G;
    proxy_pass http://gitea:3000/v2/;
    
    # common http headers
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;   
}

[wxiaoguang]

I added this example to the doc PR #31003 :

server {
    ...
    location ~ ^/(gitea|v2)($|/) {
        client_max_body_size 512M;

        # make nginx use unescaped URI, keep "%2F" as-is, remove the "/gitea" sub-path prefix, pass "/v2" as-is.
        rewrite ^ $request_uri;
        rewrite ^(/gitea)?(/.*) $2 break;
        proxy_pass http://127.0.0.1:3000$uri;

        # other common HTTP headers, see the "Nginx" config section above
        proxy_set_header Connection $http_connection;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}