docker https fails with web_graceful [E] Failed to load https cert file 0.0.0.0:3000 : open : no such file or directory #4196

r-pufky · 2018-06-09T13:36:24Z

Gitea version (or commit ref): 8786c15 (docker gitea/gitea:latest)
Git version: 2.15.2
Operating system: docker-ce on ubuntu 16.04, not using compose.
Database (use [x]):
- PostgreSQL
- MySQL
- MSSQL
- SQLite
Can you reproduce the bug at https://try.gitea.io:
- Yes (provide example URL)
- No
- Not relevant
Log gist:

2018/06/09 12:46:58 [.../cmd/web_graceful.go:37 runHTTPS()] [E] Failed to load https cert file 0.0.0.0:3000: open : no such file or directory

Description

It looks like CERT_FILE and CERT_KEY are not being resolved / loaded properly.

When specifying CERT_FILE/CERT_KEY with absolute container paths, the cert files are not found and no cert file is specified in the error:

2018/06/09 12:46:58 [.../cmd/web_graceful.go:37 runHTTPS()] [E] Failed to load https cert file 0.0.0.0:3000: open : no such file or directory

cert creation done inside docker

docker exec -it gitea /bin/bash
cd /app/gitea
./gitea cert --ca --host 10.10.10.10
mv *.pem /data/gitea/conf

app.ini

[server]
PROTOCOL = https
DOMAIN = 10.10.10.10
ROOT_URL = https://10.10.10.10:10000
CERT_FILE = /data/gitea/conf/cert.pem
CERT_KEY = /data/gitea/conf/key.pem

cert.pem is 0644
key.pem is 0600
both are owned by the docker container and can be read inside the docker container fine:

docker exec -it gitea /bin/bash
ls -l /data/gitea/conf/*.pem
-rw-r--r-- 1 git git 1159 jun 9 12:43 /data/gitea/conf/cert.pem
-rw------ 1 git git 1679 jun 9 12:43 /data/gitea/conf/key.pem

Relative paths expose cert file in error

However, if I use relative paths for the certificate, the log message changes, exposing the actual relative path for the requested cert:

2018/06/09 13:08:18 [.../cmd/web_graceful.go:37 runHTTPS()] [E] Failed to load https cert file 0.0.0.0:3000: open gitea/conf/cert.pem: not a directory

cert creation done inside docker

docker exec -it gitea /bin/bash
cd /app/gitea
./gitea cert --ca --host 10.10.10.10
mv *.pem /data/gitea/conf

app.ini

[server]
PROTOCOL = https
DOMAIN = 10.10.10.10
ROOT_URL = https://10.10.10.10:10000
CERT_FILE = gitea/conf/cert.pem
CERT_KEY = gitea/conf/key.pem

cert.pem is 0644
key.pem is 0600
both are owned by the docker container and can be read inside the docker container fine:

docker exec -it gitea /bin/bash
ls -l /data/gitea/conf/*.pem
-rw-r--r-- 1 git git 1159 jun 9 12:43 /data/gitea/conf/cert.pem
-rw------ 1 git git 1679 jun 9 12:43 /data/gitea/conf/key.pem

Disabling https, gitea works fine.

Disabling https results in gitea work properly.
Using the same setup from gogs:latest works fine in gogs.
Setting unrealistic permissions on certs (0644 for all) does not fix the issue.

Something has changed within gitea which is resulting in these certs not being loaded properly.

The text was updated successfully, but these errors were encountered:

techknowlogick · 2018-06-09T21:24:47Z

Can you try

CERT_FILE = conf/cert.pem
CERT_KEY = conf/key.pem

in your app.ini?

r-pufky · 2018-06-10T21:13:49Z

It's the same relative path error, as in case #2 above. I reconfirmed state from original submission as well.

conf/cert.pem case:

CERT_FILE = conf/cert.pem
CERT_KEY = conf/key.pem

2018/06/10 21:06:28 [.../cmd/web_graceful.go:37 runHTTPS()] [E] Failed to load https cert file 0.0.0.0:3000: open conf/cert.pem: no such file or directory

Additionally, following the current instructions: https://docs.gitea.io/en-us/https-setup/
I get the same file error:

CERT_FILE = cert.pem
CERT_KEY = key.pem

2018/06/10 21:08:53 [.../cmd/web_graceful.go:37 runHTTPS()] [E] Failed to load https cert file 0.0.0.0:3000: open cert.pem: no such file or directory

This seems like a regression?

I'd additionally recommend updating the documentation to explicitly state:

where the certs should be stored
whether relative or absolute paths should be used

I'm happy to submit a CL to update the documentation once this is figured out.

lafriks · 2018-06-19T11:34:11Z

Seams to be problem resolving relative paths

canit00 · 2018-10-10T16:37:15Z

@lafriks we are using :1 build-date: 2018-10-09 container image having the exact problem. Any chance this will be resolved anytime soon?

It doesn't matter how one configures/loads the certs this is brorken.

stale · 2019-01-07T07:37:25Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs during the next 2 weeks. Thank you for your contributions.

felix-exon · 2019-01-07T09:30:45Z

I'm having the same problem !

2019/01/07 10:27:33 [I] Log Mode: File(Info)
2019/01/07 10:27:33 [I] XORM Log Mode: File(Info)
2019/01/07 10:27:33 [I] Cache Service Enabled
2019/01/07 10:27:33 [I] Session Service Enabled
2019/01/07 10:27:33 [I] Git Version: 2.1.4
2019/01/07 10:27:33 [I] SQLite3 Supported
2019/01/07 10:27:33 [I] Run Mode: Production
2019/01/07 10:27:33 [I] Listen: https://0.0.0.0:3000
2019/01/07 10:27:33 [I] LFS server enabled
2019/01/07 10:27:33 [.../cmd/web_graceful.go:37 runHTTPS()] [E] Failed to load https cert file 0.0.0.0:3000: open cert.pem: no such file or directory

techknowlogick · 2019-01-07T09:57:55Z

ping @0x5c as they had problems with relative paths for HTTPS certs, perhaps they could post their solution.

0x5c · 2019-01-07T17:17:08Z

Aaah, yes

The docker image is one hell of a picky boi, and refuses anything else than /data/gitea/key.pem (or cert), the path itself seemingly needing to exactly match the path in the GITEA_CUSTOM environment variable.

stale · 2019-03-08T17:52:06Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs during the next 2 weeks. Thank you for your contributions.

r-pufky · 2019-03-08T18:06:26Z

FYI,

Before 0x5c's post, I ended up using nginx as a reverse proxy to serve gitea over https.

0x5c · 2019-03-08T20:15:03Z

It would make sense to keep the issue open, though, since the bug remain no-less present.

zeripath · 2019-06-22T20:22:33Z

OK the problem is here:

gitea/cmd/web.go

Line 176 in baefea3

    
           err = runHTTPS(listenAddr, setting.CertFile, setting.KeyFile, context2.ClearHandler(m))

Which calls:

gitea/cmd/web_graceful.go

Line 35 in baefea3

config.Certificates[0], err = tls.LoadX509KeyPair(certFile, keyFile)

The path of the cert and key file will be resolved against the current working directory - not against the GITEA_CUSTOM directory.

Now,

https://github.com/go-gitea/gitea/blame/master/docs/content/doc/advanced/config-cheat-sheet.en-us.md#L138-L139

Although it doesn't make this clear, those default paths are consistent with being relative to the working directory.

So I guess the question is:

To where should these files be relative?

Clearly it appears that the working directory isn't quite what people expect. Should it be GITEA_CUSTOM?

djagoo · 2020-01-18T10:03:45Z

Same problem here. I already tried any location and config variable possible.

The pem files are copied to:
/ , /app/gitea/ , /data/, /data/gitea/ , /data/gitea/conf

I tried any of these values in app.ini:
cert.pem , /cert.pem , conf/cert.pem , /conf/cert.pem, /data/gitea/cert.pem , /data/gitea/conf/cert.pem

Can anyone please supply a workaround till this is fixed?

zeripath · 2020-01-19T09:15:55Z

Just put an absolute file name in.

I'll put a breaking PR up today to change this to be relative to the custom path.

djagoo · 2020-01-19T09:54:30Z

Thats what I already tried... Right now I have in app.ini:

CERT_FILE = /data/gitea/cert.pem
CERT_KEY = /data/gitea/key.pem

bash-5.0# ls -asl /data/gitea/
total 8
     0 drwxr-xr-x    4 git      git             60 Jan 19 10:48 .
     0 drwxr-xr-x    5 root     root            41 Jan 18 08:26 ..
     4 -rw-r--r--    1 git      git           2311 Jan 18 09:24 cert.pem
     0 drwxr-xr-x    2 git      git             21 Jan 19 10:48 conf
     4 -rw-r-----    1 git      git           1675 Jan 18 09:24 key.pem
     0 drwxr-xr-x    2 git      git              6 Jan 18 08:26 log

which results in:

2020/01/19 10:50:40 cmd/web.go:107:runWeb() [I] Starting Gitea on PID: 426
2020/01/19 10:50:40 ...dules/setting/git.go:91:newGit() [I] Git Version: 2.24.1, Wire Protocol Version 2 Enabled
2020/01/19 10:50:40 routers/init.go:87:GlobalInit() [T] AppPath: /app/gitea/gitea
2020/01/19 10:50:40 routers/init.go:88:GlobalInit() [T] AppWorkPath: /app/gitea
2020/01/19 10:50:40 routers/init.go:89:GlobalInit() [T] Custom path: /data/gitea
2020/01/19 10:50:40 routers/init.go:90:GlobalInit() [T] Log path: /data/gitea/log
2020/01/19 10:50:40 ...dules/setting/log.go:233:newLogService() [I] Gitea v1.12.0+dev-113-gaac8250b4 built with GNU Make 4.2.1, go1.13.6 : bindata, sqlite, sqlite_unlock_notify
2020/01/19 10:50:40 ...dules/setting/log.go:276:newLogService() [I] Gitea Log Mode: Console(Console:info)
2020/01/19 10:50:40 ...les/setting/cache.go:45:newCacheService() [I] Cache Service Enabled
2020/01/19 10:50:40 ...s/setting/session.go:44:newSessionService() [I] Session Service Enabled
2020/01/19 10:50:40 routers/init.go:122:GlobalInit() [I] SQLite3 Supported
2020/01/19 10:50:40 routers/init.go:46:checkRunMode() [I] Run Mode: Development
2020/01/19 10:50:40 cmd/web.go:161:runWeb() [I] Listen: https://0.0.0.0:443
2020/01/19 10:50:40 ...s/graceful/server.go:53:NewServer() [I] Starting new server: tcp:0.0.0.0:443 on PID: 426
2020/01/19 10:50:40 ...s/graceful/server.go:105:ListenAndServeTLS() [E] Failed to load https cert file /data/gitea/cert.pem for tcp:0.0.0.0:443: open : no such file or directory
2020/01/19 10:50:40 cmd/web.go:204:runWeb() [C] Failed to start server: open : no such file or directory
2020/01/19 10:50:40 cmd/web.go:206:runWeb() [I] HTTP Listener: 0.0.0.0:443 Closed

It's a fresh container, created just yesterday from gitea/gitea:latest

The current code will absolute CertFile and KeyFile against the current working directory. This is quite unexpected for users. This code makes relative paths absolute against the CustomPath. Fix go-gitea#4196

zeripath · 2020-01-19T11:13:26Z

Hmm now that is odd...

zeripath · 2020-01-19T11:16:17Z

So the error report is unhelpful in the extreme here. I actually suspect the issue is the key.pem is not being loaded.

What user is running gitea?

djagoo · 2020-01-19T11:24:27Z

It's running as user git. But even if I tried to run it as root the error is the same.

djagoo · 2020-01-19T11:49:28Z

Just tried again on another machine.

copy/pasted the docker-compose.yml file from docker hub
started the container to create app.ini and file structure
added https, domain, root_url and the two file paths to app.ini
entered the container to create the files with docker cert --host git.lan.djagoo.io (as user git)
checked, that /data/gitea/cert.pem and key.pem are created and are readable by user git

The result is the same as on my main docker host.

Don't know if it helps but I ran strace to look if the path is correct:

140   openat(AT_FDCWD, "/data/gitea/cert.pem", O_RDONLY|O_CLOEXEC <unfinished ...>
143   <... futex resumed>)              = 1
138   <... futex resumed>)              = 0
140   <... openat resumed>)             = 3</data/gitea/cert.pem>
138   futex(0xc00261e148, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
143   write(1</dev/pts/0>, "\33[36m2020/01/19 14:40:10 \33[0m\33[3"..., 134 <unfinished ...>
140   epoll_ctl(4<anon_inode:[eventpoll]>, EPOLL_CTL_ADD, 3</data/gitea/cert.pem>, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=3384057512, u64=140375800192680}} <unfinished ...>
143   <... write resumed>)              = 134
140   <... epoll_ctl resumed>)          = -1 EPERM (Operation not permitted)
143   futex(0xc00261e148, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
140   epoll_ctl(4<anon_inode:[eventpoll]>, EPOLL_CTL_DEL, 3</data/gitea/cert.pem>, 0xc004a153e4 <unfinished ...>
143   <... futex resumed>)              = 1
138   <... futex resumed>)              = 0
140   <... epoll_ctl resumed>)          = -1 EPERM (Operation not permitted)
143   write(1</dev/pts/0>, "\33[36m2020/01/19 14:40:10 \33[0m\33[3"..., 170 <unfinished ...>
138   nanosleep({tv_sec=0, tv_nsec=3000},  <unfinished ...>
143   <... write resumed>)              = 170
140   fstat(3</data/gitea/cert.pem>,  <unfinished ...>
143   futex(0xc004a8e148, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
140   <... fstat resumed>{st_dev=makedev(0, 0x39), st_ino=2827748, st_mode=S_IFREG|0644, st_nlink=1, st_uid=1000, st_gid=1000, st_blksize=4096, st_blocks=8, st_size=2311, st_atime=1579433848 /* 2020-01-19T12:37:28.258175189+0100 */, st_atime_nsec=258175189, st_mtime=1579433753 /* 2020-01-19T12:35:53.467844618+0100 */, st_mtime_nsec=467844618, st_ctime=1579433848 /* 2020-01-19T12:37:28.071507931+0100 */, st_ctime_nsec=71507931}) = 0
138   <... nanosleep resumed>NULL)      = 0
140   read(3</data/gitea/cert.pem>,  <unfinished ...>
138   futex(0xc00261e148, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
140   <... read resumed>"-----BEGIN CERTIFICATE-----\nMIID"..., 2823) = 2311
140   read(3</data/gitea/cert.pem>, "", 512) = 0
140   close(3</data/gitea/cert.pem>)    = 0
140   openat(AT_FDCWD, "", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)

It reads cert.pem and then tries to read "" so the path of the keyfile is empty.

djagoo · 2020-01-19T14:13:48Z

Just found the error. The variable is named KEY_FILE and not CERT_KEY. I copied it every time wrong from the posts above. It's correct in the docs and the cheat sheet.
Sorry for me being that blind.

zeripath · 2020-01-19T19:07:10Z

Nah it's not your fault. The error report was just totally unhelpful. I've put a PR up to fix this for you.

* Make CertFile and KeyFile relative to CustomPath The current code will absolute CertFile and KeyFile against the current working directory. This is quite unexpected for users. This code makes relative paths absolute against the CustomPath. Fix #4196 * Improve error reporting when reading certificates * Apply suggestions from code review Co-Authored-By: guillep2k <18600385+guillep2k@users.noreply.github.com> Co-authored-by: guillep2k <18600385+guillep2k@users.noreply.github.com>

* Make CertFile and KeyFile relative to CustomPath The current code will absolute CertFile and KeyFile against the current working directory. This is quite unexpected for users. This code makes relative paths absolute against the CustomPath. Fix go-gitea#4196 * Improve error reporting when reading certificates * Apply suggestions from code review Co-Authored-By: guillep2k <18600385+guillep2k@users.noreply.github.com> Co-authored-by: guillep2k <18600385+guillep2k@users.noreply.github.com>

* Make CertFile and KeyFile relative to CustomPath The current code will absolute CertFile and KeyFile against the current working directory. This is quite unexpected for users. This code makes relative paths absolute against the CustomPath. Fix #4196 * Improve error reporting when reading certificates * Apply suggestions from code review Co-Authored-By: guillep2k <18600385+guillep2k@users.noreply.github.com> Co-authored-by: guillep2k <18600385+guillep2k@users.noreply.github.com> Co-authored-by: guillep2k <18600385+guillep2k@users.noreply.github.com>

lafriks added the type/bug label Jun 19, 2018

stale bot added the issue/stale label Jan 7, 2019

stale bot removed the issue/stale label Jan 7, 2019

stale bot added the issue/stale label Mar 8, 2019

stale bot removed the issue/stale label Mar 8, 2019

lafriks added the issue/confirmed Issue has been reviewed and confirmed to be present or accepted to be implemented label Mar 10, 2019

zeripath mentioned this issue Jan 19, 2020

Make CertFile and KeyFile relative to CustomPath #9867

Closed

zeripath mentioned this issue Jan 19, 2020

Make CertFile and KeyFile relative to CustomPath #9868

Merged

zeripath closed this as completed in #9868 Jan 19, 2020

zeripath mentioned this issue Jan 19, 2020

Make CertFile and KeyFile relative to CustomPath (#9868) #9874

Merged

go-gitea locked and limited conversation to collaborators Nov 24, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

docker https fails with web_graceful [E] Failed to load https cert file 0.0.0.0:3000 : open : no such file or directory #4196

docker https fails with web_graceful [E] Failed to load https cert file 0.0.0.0:3000 : open : no such file or directory #4196

r-pufky commented Jun 9, 2018

techknowlogick commented Jun 9, 2018

r-pufky commented Jun 10, 2018 •

edited

Loading

lafriks commented Jun 19, 2018

canit00 commented Oct 10, 2018 •

edited

Loading

stale bot commented Jan 7, 2019

felix-exon commented Jan 7, 2019 •

edited

Loading

techknowlogick commented Jan 7, 2019

0x5c commented Jan 7, 2019 •

edited

Loading

stale bot commented Mar 8, 2019

r-pufky commented Mar 8, 2019

0x5c commented Mar 8, 2019

zeripath commented Jun 22, 2019

djagoo commented Jan 18, 2020 •

edited

Loading

zeripath commented Jan 19, 2020

djagoo commented Jan 19, 2020

zeripath commented Jan 19, 2020

zeripath commented Jan 19, 2020

djagoo commented Jan 19, 2020

djagoo commented Jan 19, 2020 •

edited

Loading

djagoo commented Jan 19, 2020 •

edited

Loading

zeripath commented Jan 19, 2020

docker https fails with web_graceful [E] Failed to load https cert file 0.0.0.0:3000 : open : no such file or directory #4196

docker https fails with web_graceful [E] Failed to load https cert file 0.0.0.0:3000 : open : no such file or directory #4196

Comments

r-pufky commented Jun 9, 2018

Description

Relative paths expose cert file in error

Disabling https, gitea works fine.

techknowlogick commented Jun 9, 2018

r-pufky commented Jun 10, 2018 • edited Loading

lafriks commented Jun 19, 2018

canit00 commented Oct 10, 2018 • edited Loading

stale bot commented Jan 7, 2019

felix-exon commented Jan 7, 2019 • edited Loading

techknowlogick commented Jan 7, 2019

0x5c commented Jan 7, 2019 • edited Loading

stale bot commented Mar 8, 2019

r-pufky commented Mar 8, 2019

0x5c commented Mar 8, 2019

zeripath commented Jun 22, 2019

djagoo commented Jan 18, 2020 • edited Loading

zeripath commented Jan 19, 2020

djagoo commented Jan 19, 2020

zeripath commented Jan 19, 2020

zeripath commented Jan 19, 2020

djagoo commented Jan 19, 2020

djagoo commented Jan 19, 2020 • edited Loading

djagoo commented Jan 19, 2020 • edited Loading

zeripath commented Jan 19, 2020

r-pufky commented Jun 10, 2018 •

edited

Loading

canit00 commented Oct 10, 2018 •

edited

Loading

felix-exon commented Jan 7, 2019 •

edited

Loading

0x5c commented Jan 7, 2019 •

edited

Loading

djagoo commented Jan 18, 2020 •

edited

Loading

djagoo commented Jan 19, 2020 •

edited

Loading

djagoo commented Jan 19, 2020 •

edited

Loading