Markdown Preview can't be viewed - 403 Forbidden

[MPeti1]

Gitea Version

v1.15.5

Git Version

No response

Operating System

Raspbian Buster

How are you running Gitea?

I build Gitea myself on an x86 machine in an Ubuntu VM, for armv7 architecture, with docker buildx. I do this by cloning this repo, and without making changes running the following command:

sudo docker buildx build --platform linux/arm/v7 . --output=type=docker,dest=./gitea_${GITEA_VERSION}_img.tar --tag local/gitea/gitea:$GITEA_VERSION --build-arg GITEA_VERSION=v$GITEA_VERSION

$GITEA_VERSION here is the git tag of the version which I want to build. The container I currently run was built with this being set to 1.15.5.

When this completes, I get a tar file, which I transfer to my RPi 4, and there I can sudo docker image load -i [image_path].
Then I adjust my docker-compose configuration to use this new image.

Database

MySQL

Can you reproduce the bug on the Gitea demo site?

No

Log Gist

https://gist.github.com/MPeti1/e508529515b3cbfae94b09f40087b536

Description

The problem appears in the text editor I use when creating a new issue, in the text editor of issue comments, in the text editor of wiki pages, and in the text editor of MD files of the repository too.

In the web UI, this ends in an endless "Loading..." in the place of the preview:

While debugging this problem, with my browser's developer console I've found that when I open the "Preview" tab of the editor, a request is made to https://[domain_name]/api/v1/repos/[username]/[reponame]/markdown, which ends up with the status "403 - Forbidden".
At the same time, these 2 lines are logged in the Gitea container:

2021/10/22 22:32:22 Started POST /api/v1/repos/Home-infrastructure/Node804/markdown for [client_IP]:0
2021/10/22 22:32:22 Completed POST /api/v1/repos/Home-infrastructure/Node804/markdown 403 Forbidden in 1.112074ms

Please note, that for the log gist I tried to set the log options described here, but it did not seem to work, even though I've restarted the container.
Here is how /etc/gitea/conf/app.ini looks like inside the Gitea container:

bash-5.1# cat /etc/gitea/conf/app.ini
[ui]
DEFAULT_THME = arc-green

[log]
MODE = console
LEVEL = debug ; please set the level to debug when we are debugging a problem
ROUTER = console
COLORIZE = false ; this can be true if you can strip out the ansi coloring

Screenshots

No response

[noerw]

Are you sure your session is still valid / you're still logged in after refreshing?
I can't reproduce this on 1.15.5 instance..
The only 403 response source I can see in the entire code path for this route comes from the sudo() middleware.
Can you verify that you don't have a Sudo http header set for what ever reason by inspecting requests sent by your browser?

[MPeti1]

Are you sure your session is still valid / you're still logged in after refreshing?

Yes, after a page reload I'm still logged in, and I can do anything that a logged in user can.
Sometimes I'm logged out and need to log in again, thought, but that is not very frequent, I think it might happen around every 2 weeks or something like that.

Can you verify that you don't have a Sudo http header set for what ever reason by inspecting requests sent by your browser?

There is no Sudo header set in my requests:

POST /api/v1/repos/Home-infrastructure/Node804/markdown undefined
Host: gitea.rpi.lan
User-Agent: Mozilla/5.0 (Windows NT 10.0; rv:78.0) Gecko/20100101 Firefox/78.0
Accept: */*
Accept-Encoding: gzip, deflate
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
Content-Length: 391
Origin: https://gitea.[domain]
DNT: 1
Connection: keep-alive

[MPeti1]

Oh, I got the header above from my browsers's developer console. Did you mean to inspect it at the server?

[MPeti1]

Found out how to capture the requests in the Gitea container, so here is what I found.
I installed tcpdump and grep, then used this command (source) to inspect my requests:

tcpdump -i eth1 -A -s 10240 'tcp port 3000 and (((ip[2:2] - ((ip[0]&0xf)<<2)) - ((tcp[12]&0xf0)>>2)) != 0)' | egrep --line-buffered "^........(GET |HTTP\/|POST |HEAD )|^[A-Za-z0-9-]+: " | sed -r 's/^........(GET |HTTP\/|POST |HEAD )/\n\1/g'

Output:

...
POST /api/v1/repos/Home-infrastructure/Node804/markdown HTTP/1.1
Host: gitea.rpi.lan
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Firefox/78.0
Content-Length: 130
Accept: */*
Accept-Encoding: gzip, deflate, br
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
Dnt: 1
Origin: https://gitea.[domain]
Te: trailers
X-Forwarded-For: [client ip]
X-Forwarded-Host: gitea.[domain]
X-Forwarded-Port: 443
X-Forwarded-Proto: https
X-Forwarded-Server: [middleman reverse proxy's hostname]
X-Real-Ip: [client ip]

HTTP/1.1 403 Forbidden
Content-Type: application/json;charset=utf-8
Set-Cookie: i_like_gitea=[random string]; Path=/; HttpOnly; SameSite=Lax
Set-Cookie: _csrf=[long random string]; Path=/; Expires=Sun, 24 Oct 2021 18:11:24 GMT; HttpOnly; SameSite=Lax
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
Date: Sat, 23 Oct 2021 18:11:24 GMT
Content-Length: 59
...

I cut off requests that seemed to not be relevant.

[MPeti1]

Also, there's a detail that I forgot to tell but it might be interesting to you.

I use uMatrix in my browser, and unless I turn off "Forbid mixed content" for my gitea's domain, it simply blocks the request.
If I turn that option off, the request will go through, and Gitea will respond with 403.
I think this detail is interesting, because I haven't experienced other occasions where uMatrix would have been wrong with this check, but if I turn that option off, the Firefox devtools says that the request that resulted in the 403 was made through a secure connection.

uMatrix is a content blocker like uBlock, and the "Forbid mixed content" option blocks plain HTTP requests on HTTPS sites. My Gitea instance is accesed through HTTPS from the clients to my reverse proxy, and it's HTTP from the reverse proxy to Gitea, but that HTTP connection happens inside a Docker bridge network so it shouldn't be a big problem.

And before I send this, there's an other one:
Not sure what changed, but I think earlier the /markdown requests did not return a response body, just the 403 error code.
Now, if I switch to the preview tab of an editor, I see that the request returns a response body:

{"message":"Only signed in user is allowed to call APIs."}

And while this happens, I'm still logged in, as a test I just changed my biography in my profile and it persisted.

[MPeti1]

Can it be the problem that no cookies are sent in requests to /markdown?
As a test I just made a comment to an issue, and it's requests included a "Cookie" header, but POST requests to /markdown do not have a "Cookie" header, as you can see in the above snippet, but I also see it in my browser devtools so it's not a problem with the reverse proxy.

You said that you wasn't able to reproduce the problem, and others haven't complained either, so it must be something on my end. Is there a simple way I could debug the Gitea client? I see that the JS has probably been minified.

[noerw]

Yeah it looks like a problem with your setup. You mentioned you can't repro on try.gitea.io, so I'd also assume it's not something your browser extensions mess up.
I suspect this may related to the domain you access gitea with, due to some cross site restrictions or similar.
If your config is really as minimal as posted above that may be the culprit, you should probably set at least ROOT_URL and/or DOMAIN.·

Interestingly when I modify the request removing all cookies, I get a 404 back, not a 403 (as expected; I'm not aware of more code paths that return a 403..), so this is odd.

For further support please visit the gitea discord :)

[MPeti1]

If your config is really as minimal as posted above that may be the culprit, you should probably set at least ROOT_URL and/or DOMAIN.·

I have both set in the main app.ini. But now that you word it this way.. isn't both of the app.ini files read by Gitea? The "main" app.ini (at /data/gitea/conf/app.ini) is much longer, but I assumed I only need to put customizations in the /etc/gitea/conf/app.ini, and the default configuration and required settings can stay in the main app.ini.

For further support please visit the gitea discord :)

Sorry, but I don't have access to discord, because they require phone and email verification on registerning a new account (even if it's an anonymous one).

[noerw]

Gitea reads only a single file, which one should be displayed in the startup log messages.
Of course there are default values for most variables, but those defaults are in the binary.
I never deployed gitea via docker, so I'm not sure what the standard config layout is there, but look at https://docs.gitea.io/en-us/install-with-docker/#customization.

Sorry, but I don't have access to discord, because they require phone and email verification on registering a new account (even if it's an anonymous one).

They require both now?! That's something I wouldn't get on board with either..

[MPeti1]

Gitea reads only a single file, which one should be displayed in the startup log messages.

I'm kind of surprised then that my Gitea was working at all. I'll move my settings over there and report back if the issue was fixed by this.

They require both now?! That's something I wouldn't get on board with either..

It's a bit complicated.

I think not everyone is required both or any of these, but it seems that if your browser is configured for higher privacy (privacy.resistfingerprinting and such), they flag your IP (at least for some time) so any registrations (even from a brand new VM with a brand new browser profile) will end up in requiring you to provide these. In some circumstances they will disable your account right after the verification, but that might only happen if you use temporary mail and sms services for that.
Tl;Dr: it seems they implemented a kind of a KYC system

[MPeti1]

Well, looked up the Docker installation guide again here, and it seems above I have inserted the wrong config file. So in the end the log options did not work because I put them in the wrong place (🤦), but the other news is that at the correct place (/data/gitea/conf/app.ini) I have a complete looking configuration.
Here DOMAIN and SSH_DOMAIN are set to gitea.[domain], and ROOT_URL is set to http://gitea.[domain]/.

Aaand found the problem. It seems the value of ROOT_URL should always have https as protocol if your browser will access Gitea though HTTPS, probably because it is sometimes inserted into the HTML sent to the client.
After changing the protocol to https, preview now works!

Thank you for making me find this config value!
And also for making me realize that in the Docker image there are no such paths as /etc/gitea (confusion from 2nd paragrapgh here) and /data/custom (I don't even know now where did I find this one)