[Gitea] Stuck on deploying (2/3) version .34 onwards

[Boostflow]

As of version 1.16.0_5.0.35 Gitea is stuck on deploying 2/3 pods. Rollback to version 1.16.0_5.0.34 is confirmed to work. Here are all the requested artifacts:

Scale version: TrueNAS-SCALE-22.02-RC.2
App version: 1.16.0_5.0.39

Logs:

[stavros-k]

Note for whoever picks it up:

Fail appeared right after the common change to add Upgrade and Install initContainers
Commits:
04a3aa9
0496669

Seems that init Container 2-configure-gitea fails

Events:
  Type     Reason            Age                 From               Message
  ----     ------            ----                ----               -------
  Warning  FailedScheduling  16m                 default-scheduler  0/1 nodes are available: 1 pod has unbound immediate PersistentVolumeClaims.
  Warning  FailedScheduling  16m                 default-scheduler  0/1 nodes are available: 1 pod has unbound immediate PersistentVolumeClaims.
  Normal   Scheduled         16m                 default-scheduler  Successfully assigned ix-gitea/gitea-6c7ffdc8f5-575sf to ix-truenas
  Normal   AddedInterface    16m                 multus             Add eth0 [172.16.0.65/16] from ix-net
  Normal   Pulled            16m                 kubelet            Container image "ghcr.io/truecharts/alpine:v3.14.2@sha256:4095394abbae907e94b1f2fd2e2de6c4f201a5b9704573243ca8eb16db8cdb7c" already present on machine
  Normal   Created           16m                 kubelet            Created container autopermissions
  Normal   Started           16m                 kubelet            Started container autopermissions
  Normal   Pulled            16m                 kubelet            Container image "ghcr.io/truecharts/alpine:v3.14.2@sha256:4095394abbae907e94b1f2fd2e2de6c4f201a5b9704573243ca8eb16db8cdb7c" already present on machine
  Normal   Created           16m                 kubelet            Created container inotify
  Normal   Started           16m                 kubelet            Started container inotify
  Normal   Pulled            16m                 kubelet            Container image "ghcr.io/truecharts/postgresql:v14.1.0@sha256:f729127ca5517840f75e3ab107e2b60efe8257b27f2c679e070dffcaf534e88a" already present on machine
  Normal   Created           16m                 kubelet            Created container postgresql-init
  Normal   Started           16m                 kubelet            Started container postgresql-init
  Normal   Pulling           15m                 kubelet            Pulling image "tccr.io/truecharts/gitea:v1.16.0@sha256:d248e0dcc76de23704e75c447711f8b747f870aa799ba3e0f3c62759a1a11c5b"
  Normal   Pulled            15m                 kubelet            Successfully pulled image "tccr.io/truecharts/gitea:v1.16.0@sha256:d248e0dcc76de23704e75c447711f8b747f870aa799ba3e0f3c62759a1a11c5b" in 23.688872187s
  Normal   Created           15m                 kubelet            Created container 1-init-directories
  Normal   Started           15m                 kubelet            Started container 1-init-directories
  Normal   Pulled            14m (x3 over 15m)   kubelet            Container image "tccr.io/truecharts/gitea:v1.16.0@sha256:d248e0dcc76de23704e75c447711f8b747f870aa799ba3e0f3c62759a1a11c5b" already present on machine
  Normal   Created           14m (x3 over 15m)   kubelet            Created container 2-configure-gitea
  Normal   Started           14m (x3 over 15m)   kubelet            Started container 2-configure-gitea
  Warning  BackOff           86s (x65 over 15m)  kubelet            Back-off restarting failed container
scale# k logs -n ix-gitea pod/gitea-6c7ffdc8f5-575sf -c 2-configure-gitea
Wait for database to become avialable...
gitea-postgresql (172.17.220.120:5432) open
==== BEGIN GITEA MIGRATION ====
2022/02/07 19:24:37 ...s/setting/setting.go:543:SetCustomPathAndConf() [W] Using 'custom' directory as relative origin for configuration file: '/data/gitea/"/etc/gitea/app.ini"'
2022/02/07 19:24:37 ...s/setting/setting.go:543:SetCustomPathAndConf() [W] Using 'custom' directory as relative origin for configuration file: '/data/gitea/"/etc/gitea/app.ini"'
2022/02/07 19:24:37 ...s/setting/setting.go:583:loadFromConf() [F] Unable to find configuration file: "/data/gitea/\"/etc/gitea/app.ini\"".
        Ensure you are running in the correct environment or set the correct configuration file with -c.

[progmboy]

I think this is a bug of Gitea-1.16.0
looks like go-gitea/gitea#18524 fix this bug.
Just update gitea >=1.16.1 will resolve this.

[PrivatePuffin]

Thanks for the memo, i'll push an extra round of container updates right away! :)

[Boostflow]

I'm afraid with the latest update the issue still persists. I'm on 1.16.1_5.0.41

[PrivatePuffin]

This. Sucks.

[stavros-k]

Okay, this seems wrong Unable to find configuration file: "/data/gitea/\"/etc/gitea/app.ini\"".
Opened a PR, with couple of things and let CI test it. I have to go for now, will come back later to check how it did go.

[stavros-k]

Well The problem seems to be upstream on gitea.
I narrowed it down to this commit PR.
https://github.com/go-gitea/gitea/pull/18535/files
As the error seems to match
Unable to find configuration file: "/data/gitea/\"/data/gitea/conf/app.ini\"".

Also the date merged and the days we update our images matches also.

If you agree also, we can report it upstream

[stavros-k]

I was right! :) Now we wait the new image to be released ;)

go-gitea/gitea#18690

[PrivatePuffin]

To be fair:
This is precisely one of the reasons I want stricter testing on the enterprise train :)

[Boostflow]

I just deployed the latest update 1.16.1_5.0.44 which I see included version 1.16.1 Gitea. Reading here: https://github.com/go-gitea/gitea/releases/tag/v1.16.1 I assume to bug fix should have been shipped with this version (correct me if I'm wrong). However running this latest version on Truenas still poses an issue with running Gitea:

Is there something I'm missing here on anything I can do to add to the solution?

[stavros-k]

This release is 15days ago, while fix is merged 12 days ago.. So..

[PrivatePuffin]

@stavros-k No there is another fix that also needs to be merged which they did not release yet.

go-gitea/gitea#18699

[PrivatePuffin]

Cleaned up the "When updateszzz"...
Use discord please people, unless you have code to report on.

[PrivatePuffin]

@Boostflow There where some issues with 1.6.2 getting released, should be released within 24 hours

[PrivatePuffin]

@all-contributors please add @Boostflow for bugs

[allcontributors]

@Ornias1993

I've put up a pull request to add @Boostflow! 🎉

[truecharts-admin]

This issue is locked to prevent necro-posting on closed issues. Please create a new issue or contact staff on discord of the problem persists