Preview of Markdown not working, getting 403

[fhuberts]

Gitea Version

1.15.6+27-gd29a0fc3b

Git Version

2.33.1

Operating System

Fedora 35 x64

How are you running Gitea?

Self-made RPM package, see https://gitlab.com/fhuberts/rpmsUpstream/-/tree/master/fedora/gitea

Database

PostgreSQL

Can you reproduce the bug on the Gitea demo site?

No

Log Gist

No response

Description

Nov 28 14:57:07 git.somewhere gitea[3372]: 2021/11/28 14:57:07 Started POST /api/v1/repos/servers/server-installs/markdown for 192.168.180.11:0
Nov 28 14:57:07 git.somewhere gitea[3372]: 2021/11/28 14:57:07 Completed POST /api/v1/repos/servers/server-installs/markdown 403 Forbidden in 821.289µs

Screenshots

No response

[fhuberts]

The same 403 is present in the apache webserver log.
I run gitea behind apache, on a unix domain socket.

[28/Nov/2021:15:18:04 +0100] "POST /git/api/v1/repos/servers/server-installs/markdown HTTP/1.1" 403 59 "-" "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:94.0) Gecko/20100101 Firefox/94.0"

[wxiaoguang]

I just remembered another similar issue:

• Markdown Preview can't be viewed - 403 Forbidden #17408

[fhuberts]

This was also present on 1.14.
Just upgraded my server to 1.15 in the hopes that it would work now...

[fhuberts]

I've tried changing the ROOT_URL as mentioned in the referenced issue, but that doesn't fix the problem.

[fhuberts]

Setting the *_VISIBILITY variables to public also doesn't work.

Don't know what else to check now.

[wxiaoguang]

Can you check (inspect) the requests sent by your browser? What's the difference between a correct request and an incorrect request?

[fhuberts]

From the Firefox console.

Headers:

{
	"POST": {
		"scheme": "https",
		"host": "<hostname>",
		"filename": "/git/api/v1/repos/servers/server-installs/markdown",
		"remote": {
			"Address": "<host ip>:443"
		}
	}
}

Request:

_csrf=<token>&mode=gfm&context=%2Fgit%2Fservers%2Fserver-installs&text=something&wiki=true

Must note that a different token appears to be sent than is stored in my cookies.

Also, Firefox reports 'CORS Missing Allow Origin' in the transferred column
for the request.

[wxiaoguang]

Must note that a different token appears to be sent than is stored in my cookies.

Also, Firefox reports 'CORS Missing Allow Origin' in the transferred column for the request.

Maybe that's the problem. On my side, the CSRF token in POST and in COOKIE are the same.

Can you check the source code of the page to see which CSRF token is correct? which is incorrect? The correct CSRF token should also exist in the page source code.

ps: You can safely paste the starting sub-string of your CSRF tokens here, it won't cause any security problem.

[fhuberts]

The csrf is changed by the 403 response.
I checked after the response so that is why the token was different

[zeripath]

What are your cookie settings in app.ini? It might just be helpful to paste that up entirely - skipping the secrets and the confidential stuff but explaining what the values are. (For example if your root URL is something like https://foo.example.com/bar put this in instead.)

Could you check if the problem still occurs in an incognito window?

Have you tried deleting all of your Gitea cookies?

[fhuberts]

I have changed nothing in app.ini wrt cookies, everything is at defaults.
I've not set ROOT_URL.

Problem is present is private window as well.

Removing all cookies doesn't help.

[fhuberts]

Even disabling selinux doesn't help, so it appears to be somewhere inside the application code

[fhuberts]

This can easily be reproduced by setting up a Fedora 35 machine/VM and installing the RPM I created (see https://gitlab.com/fhuberts/rpmsUpstream/-/tree/master/fedora/gitea). I'll attach a built RPM package in a minute.

This should make it a lot easier to debug.

[fhuberts]

Github refuses to attach the rpms/gz/zip. bummer.

It's quite easy to build the RPM yourself, or just ping me

[fhuberts]

Ha, easy workaround. Put them on gitlab: https://gitlab.com/fhuberts/gitea-rpms

[fhuberts]

Ok, so I'm only having this problem on wiki pages. Preview works fine on issues.
So the issue does seem to be located within Gitea itself.

[zeripath]

Ah that helps!

[fhuberts]

Didn't know that before, hit that by accident.

[zeripath]

OK on both 1.15.7 and 1.16-dev I'm getting the right thing on the wiki.

What permissions does the user you're trying to write with have in that repository?

[fhuberts]

the logged in user is the owner of the repo.

[fhuberts]

and the filesystem owner of the wiki repo is the user under which gitea runs (e.g. user 'git')

[zeripath]

I still cannot reproduce this.

I think it would be useful to see your app.ini.

[fhuberts]

I put the RPM packages on gitlab (#17843 (comment)), so you can completely reproduce my setup by installing a Fedora VM with those RPMs.
If you want more up-to-date RPMs then please let me know, the issue is still present in gitea v1.15.9 (and today both wiki and issues don't show a preview).

[fhuberts]

And now it works again for a very small example in an issue.
Might it be that there is a timeout somewhere that is a bit tight?

[zeripath]

Thanks for the rpms, but it's a bit much to expect us to install vagrant, download a fedora cloud-base, create a virtual machine install 288 packages into that virtual machine, then install your homemade packages into that virtual machine. (Which ones?! All of them?) Even after doing all that we would then still have to guess what the eventual configuration would be.

So instead:

• What is your app.ini?

• In particular what session provider are you using?

• I see from your rpms that you're using apache as a reverse proxy. What is that configuration?
• In your local browser inspector it would be useful to check and compare what the session id cookies are being sent to the api as compared to your non api endroutes.

It may be that 1.15.10 will solve this issue as it might be that you were using a memory provider and this an issue to do with the multiple sessioners in 1.15.0-1.15.9.

[fhuberts]

As I've tried to explain, the RPMs exactly reproduce my setup. There is no need for vagrant, just a plain Fedora 35 server with those RPMs installed. Installing the RPMs will drag in all dependencies and configure the server, exactly reproducing my setup.

I will update those RPMs, but most importantly I will upload the configuration files you've asked for in that repo as well.

What do you mean with session provider? It's a daemon, there is no session. Or do you mean gitea user session? --> local login.

I'm running 1.15.9 and it still has the issue.

I'll add a comment after uploading all the files in that repo.

[fhuberts]

I've updated the RPMs and added all config files.
It's all on https://gitlab.com/fhuberts/gitea-rpms

[zeripath]

Are you sure that the ProxyPassReverse is correct?

https://docs.gitea.io/en-us/reverse-proxies/#apache-httpd Does not recommend a ProxyPassReverse

[fhuberts]

Removed the ProxyPassReverse:

• makes no difference, same problem as before: preview works on issues, but not on wiki pages

While playing around I did see something very strange:

• when editing a wiki page and saving it unchanged the backing file is changed and suddenly has cr/lf line endings (which is very unexpected and unwanted !)
• the preview also doesn't work on that wiki page with cr/lf

[wxiaoguang]

1. ProxyPass /git unix:/run/gitea/gitea.sock|http://localhost:3000 nocanon , the http://localhost:3000 seems incorrect either. Please make sure all your hosts and ports are correct. It's highly recommended that you use a non-unix-socket to test everything and make sure things work first, then try to migrate it to a unix-socket setup, then you can find where the problem occurs.

2. the CRLF: it is the browser's behavior, Gitea now justs saves content as it is (well, you can call it a flaw, or a feature , whatever)

3. There are 4 RPMs, 3 apache config files, we have no idea how to test or reproduce your problem. As (1), it's recommended to make a simple setup first, and you could only upload the gitea binary with it's config and apache config in a tar.gz.

[fhuberts]

I'm sorry but I have to react to point 3 here.
I've done exactly as you asked and uploaded all configurtion files. I've uploaded all RPMs I use, and that will exactly reproduce my setup. Still you persist in claiming that you can't reproduce my setup or that I haven't uploaded the configuration files.
Which is all very disappointing to me. I've tried very hard to give you all the info you need, and to follow all your suggestions...

I will investigate point 1 you mention but I'm not holding out hope that that will solve the problem.

Would it be so bad to consider that there might actually be a bug in gitea? You seem to be only pushing back on me instead of trying to understand the problem and investigating the gitea side.

Thanks for the clarification on point 2.

I'll report back when I've tried your point 1.

[fhuberts]

And here are the configuration files, so that you don't have to go out to the git repo I put together for this issue.

configuration.zip

[fhuberts]

So I've tried without the unix domain socket, without SSL, just HTTP proxy: exact same problem.
It really does appear that the proxy setup doesn't function correctly, on the gitea side.
The issue previews work fine, the wiki previews do not work at all.

[zeripath]

In your browser console have you watched the network to check how your browser is communicating with gitea when you click the preview page?

Are you certain that the request is going to the correct place?

[wolflu05]

I saw that problem by another user using my gitea instance. First i wondered, why it doesn't render the wiki edit preview, because on my machine (using chrome) everything works fine. Maybe its due to the browser, because the other user was using edge.

[zeripath]

OK so I've finally managed to start a fedora docker with systemd using https://hub.docker.com/r/jrei/systemd-fedora/

docker pull jrei/systemd-fedora
docker run -d --name systemd-fedora --tmpfs /tmp --tmpfs /run --tmpfs /run/lock -v /sys/fs/cgroup:/sys/fs/cgroup:ro jrei/systemd-fedora
docker exec -it systemd-fedora bash

Then in that container:

yum install https://gitlab.com/fhuberts/gitea-rpms/-/raw/master/gitea-1.15.9-1.fc35.x86_64.rpm

Now for whatever reason that postgres didn't come up for me straight away, but I think the db is irrelevant so I switched to sqlite by editing the app.ini and restarted gitea:

vi /etc/gitea/app.ini # to edit the app.ini as above
systemctl restart gitea

Then on the host I navigate to:

https://172.17.0.2/git/

created a user and then a test repository.

Then gone to wiki markdown and yes I can reproduce the issue, but looking in the network console I see:

POST https://8eba9d421607/git/api/v1/repos/administrator/simple-test/markdown

---

Now I remember I've not set a ROOT_URL in app.ini ... (Gitea requires that you set the ROOT_URL to match the site.)

---

So now I go back to command line and change the app.ini using vi /etc/gitea/app.ini:

DOMAIN=172.17.0.2
ROOT_URL=https://172.17.0.2/git

then I run

systemctl restart gitea

And refresh my edit page.

et voila ça marche.

[fhuberts]

I saw that problem by another user using my gitea instance. First i wondered, why it doesn't render the wiki edit preview, because on my machine (using chrome) everything works fine. Maybe its due to the browser, because the other user was using edge.

I tried both Firefox and Chrome. Both show the same result.

[fhuberts]

@zeripath
If you use the RPM then the ROOT_URL will have been setup already, on my server it shows like:
DOMAIN = hostname.domain.name
ROOT_URL = https://%(DOMAIN)s/git/

Thanks for looking into this, it helped me narrow down the problem...

I made it work on my server, and the problem is that I need to access the server with its FQDN to make it work!

This seems very weird and actually a bug.
More so because for the preview of an issue comment there is no such requirement, that works fine for both the FQDN and its simple hostname.

Literally the only thing different is using the 'hostname.domain.name' or 'hostname'.

And the POST goes out to the same (apart from the FQDN) URL.
The POST are the same.

[fhuberts]

Setting domain to 'hostname' has the following effect:

• when the server is accessed with 'hostname' both issue comment and wiki page previews work.
• when the server is accessed with its FQDN neither issue comment nor wiki page previews work.

Same/reverse holds when setting domain to 'hostname.domain.name'.

(I did see it work for issue comment previews a few times but can't reproduce that now)

Seems like a hostname check is too strict.

[fhuberts]

For now I've implemented a workaround for this bug (see https://gitlab.com/fhuberts/rpmsUpstream/-/commit/d403b45ede235a1852e57be9546b169b44fdcbe3?w=1).

[zeripath]

It would be useful if you could check the cookies that are set. You'll probably find that the cookies are being set as per the FQDN or hostname as appropriate.

https://stackoverflow.com/questions/1062963/how-do-browser-cookie-domains-work

[fhuberts]

Without the workaround cookies are stored under 'hostname', with the workaround cookies are stored under 'domain.name'.

[fhuberts]

I also just discovered why issue comment previews sometimes seem to work: in a list with issues, the links to the issues (on the issue titles) are to the FQDN of the issue, even though the page with the list is opened on the 'hostname'!

[fhuberts]

Summary:
Previews only work when the server is accessed through its full hostname (FQDN).

[wxiaoguang]

Currently, Gitea uses an absolute full URL <a class="preview item" data-url="{{$.Repository.APIURL}}/markdown" data-context="{{$.RepoLink}}">{{$.i18n.Tr "preview"}}</a> to do preview. The absolute full URL is generated by ROOT_URL, it must match the FQDN you are visiting.

That's why we asked many times to check whether the CSRF token and COOKIE were working correctly.

[wxiaoguang]

After many refactoring to the URL and CSRF modules, this issue can be closed

• Show messages for users if the ROOT_URL is wrong, show JavaScript errors #18971
• Never use /api/v1 from Gitea UI Pages #19318
• Remove dependent on session auth for api/v1 routers #19321

And a new issue for the root problem:

• Multi-domain support, ROOT_URL problem, use relative URLs in templates instead of absolute URLs #19345