The storage.repo-archive and storage.packages configs do not accept custom STORAGE_TYPE #20703

mmeier86 · 2022-08-07T19:29:47Z

Description

Hello,

I am using S3 storage provided by my Ceph cluster for Gitea's LFS, Avatar, attachment, repo-archive and packages storage. My storage configuration currently looks like this:

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
[storage]
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
STORAGE_TYPE = my_minio

[lfs]
STORAGE_TYPE = my_minio

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
[storage.repo-archive]
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
STORAGE_TYPE            = minio
MINIO_ENDPOINT          = s3.my-domain.net
MINIO_BUCKET            = gitea
MINIO_LOCATION          = homenet
MINIO_USE_SSL           = true
MINIO_ACCESS_KEY_ID     = correct_key
MINIO_SECRET_ACCESS_KEY = correct_key

[storage.packages]
STORAGE_TYPE            = minio
MINIO_ENDPOINT          = s3.mei-home.net:7481
MINIO_BUCKET            = gitea
MINIO_LOCATION          = homenet
MINIO_USE_SSL           = true
MINIO_ACCESS_KEY_ID     = correct_key
MINIO_SECRET_ACCESS_KEY = correct_key

[storage.my_minio]
STORAGE_TYPE            = minio
MINIO_ENDPOINT          = s3.my-domain.net
MINIO_BUCKET            = gitea
MINIO_LOCATION          = homenet
MINIO_USE_SSL           = true
MINIO_ACCESS_KEY_ID     = correct_key
MINIO_SECRET_ACCESS_KEY = correct_key

This configuration works without problems. But I need to repeat my S3 configuration three times. Once for the default storage config used by attachments and so on, and then once each for the package and repo-archive storage.

Now to the bug itself. I had originally hoped that the two new storage items, packages and repo-archive would work similar to the lfs section above, by only setting the STORAGE_TYPE option to my_minio, avoiding the multiple repeats of the S3 configs.
But with the following config, I'm getting errors during Gitea startup:

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
[storage]
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
; storage type
STORAGE_TYPE = my_minio

[lfs]
STORAGE_TYPE = my_minio

; customize storage
[storage.my_minio]
STORAGE_TYPE            = minio
MINIO_ENDPOINT          = s3.my-domain.net
MINIO_BUCKET            = gitea
MINIO_LOCATION          = homenet
MINIO_USE_SSL           = true
MINIO_ACCESS_KEY_ID     = correct_key
MINIO_SECRET_ACCESS_KEY = correct_key

This seems to working in so far as the package and repo-archive storage now try to use my_minio, but trying to use that then fails with this log output:

2022/08/07 19:10:45 cmd/web.go:106:runWeb() [I] Starting Gitea on PID: 20
2022/08/07 19:10:45 ...s/setting/setting.go:1256:CreateOrAppendToCustomConf() [I] Settings saved to: "/data/gitea/conf/app.ini"
2022/08/07 19:10:45 ...s/setting/setting.go:989:loadFromConf() [I] Default UI Location is Europe/Berlin
2022/08/07 19:10:45 cmd/web.go:157:runWeb() [I] Global init
2022/08/07 19:10:45 ...s/setting/setting.go:989:loadFromConf() [I] Default UI Location is Europe/Berlin
2022/08/07 19:10:45 routers/init.go:104:GlobalInitInstalled() [I] Git Version: 2.36.2, Wire Protocol Version 2 Enabled (home: /data/gitea_data/home)
2022/08/07 19:10:45 routers/init.go:107:GlobalInitInstalled() [I] AppPath: /usr/local/bin/gitea
2022/08/07 19:10:45 routers/init.go:108:GlobalInitInstalled() [I] AppWorkPath: /app/gitea
2022/08/07 19:10:45 routers/init.go:109:GlobalInitInstalled() [I] Custom path: /data/gitea
2022/08/07 19:10:45 routers/init.go:110:GlobalInitInstalled() [I] Log path: /data/gitea_data/logs
2022/08/07 19:10:45 routers/init.go:111:GlobalInitInstalled() [I] Configuration file: /data/gitea/conf/app.ini
2022/08/07 19:10:45 routers/init.go:112:GlobalInitInstalled() [I] Run Mode: Prod
2022/08/07 19:10:45 ...dules/setting/log.go:288:newLogService() [I] Gitea v1.17.0 built with GNU Make 4.3, go1.18.4 : bindata, timetzdata, sqlite, sqlite_unlock_notify
2022/08/07 19:10:45 ...dules/setting/log.go:335:newLogService() [I] Gitea Log Mode: Console(Console:info)
2022/08/07 19:10:45 ...dules/setting/log.go:249:generateNamedLogger() [I] Router Log: Console(console:info)
2022/08/07 19:10:45 ...les/setting/cache.go:76:newCacheService() [I] Cache Service Enabled
2022/08/07 19:10:45 ...les/setting/cache.go:91:newCacheService() [I] Last Commit Cache Service Enabled
2022/08/07 19:10:45 ...s/setting/session.go:73:newSessionService() [I] Session Service Enabled
2022/08/07 19:10:45 ...es/setting/mailer.go:127:newMailService() [I] Mail Service Enabled
2022/08/07 19:10:45 ...s/storage/storage.go:176:initAttachments() [I] Initialising Attachment storage with type: minio
2022/08/07 19:10:45 ...les/storage/minio.go:91:NewMinioStorage() [I] Creating Minio storage at s3.my-domain.net:gitea with base path attachments/
2022/08/07 19:10:45 ...s/storage/storage.go:170:initAvatars() [I] Initialising Avatar storage with type: minio
2022/08/07 19:10:45 ...les/storage/minio.go:91:NewMinioStorage() [I] Creating Minio storage at s3.my-domain.net:gitea with base path avatars/
2022/08/07 19:10:45 ...s/storage/storage.go:188:initRepoAvatars() [I] Initialising Repository Avatar storage with type: minio
2022/08/07 19:10:45 ...les/storage/minio.go:91:NewMinioStorage() [I] Creating Minio storage at s3.my-domain.net:gitea with base path repo-avatars/
2022/08/07 19:10:45 ...s/storage/storage.go:182:initLFS() [I] Initialising LFS storage with type: minio
2022/08/07 19:10:45 ...les/storage/minio.go:91:NewMinioStorage() [I] Creating Minio storage at s3.my-domain.net:gitea with base path lfs/
2022/08/07 19:10:45 ...s/storage/storage.go:194:initRepoArchives() [I] Initialising Repository Archive storage with type: my_minio
2022/08/07 19:10:45 routers/init.go:57:mustInit() [F] code.gitea.io/gitea/modules/storage.Init failed: Unsupported storage type: my_minio

What I would expect to happen: Like for the lfs config, the default storage config, in turn using my customized my_minio storage is used, instead of having to configure the same options three times.

Gitea Version

1.17.0

Can you reproduce the bug on the Gitea demo site?

No

Log Gist

No response

Screenshots

No response

Git Version

N/A

Operating System

Run via Docker

How are you running Gitea?

I'm using the official Gitea Docker container, with one slight adaption, namely completely disabling the external SSH daemon by not even starting it.

The container is deployed on a Nomad cluster.

The most important difference perhaps: Instead of MinIO, I'm using S3 provided by Ceph.

Database

PostgreSQL

The text was updated successfully, but these errors were encountered:

lunny · 2022-08-08T07:34:21Z

As a workround, you can remove all the related configuration and only left below:

[storage]
STORAGE_TYPE            = minio
MINIO_ENDPOINT          = s3.my-domain.net
MINIO_BUCKET            = gitea
MINIO_LOCATION          = homenet
MINIO_USE_SSL           = true
MINIO_ACCESS_KEY_ID     = correct_key
MINIO_SECRET_ACCESS_KEY = correct_key

mmeier86 · 2022-08-08T19:07:04Z

And that did the trick. Gitea config considerably shortened. Thanks @lunny!

For me this is an even better solution than solving the actual bug would be. But should we still leave the issue open for a future fix enabling package and repo archive storage to use a custom storage config?

viceice · 2022-09-22T07:54:13Z

[storage]
STORAGE_TYPE            = minio
MINIO_ENDPOINT          = s3.my-domain.net
MINIO_BUCKET            = gitea
MINIO_LOCATION          = homenet
MINIO_USE_SSL           = true
MINIO_ACCESS_KEY_ID     = correct_key
MINIO_SECRET_ACCESS_KEY = correct_key

Does this work for all storage types? so i don't need to configure attachments / lfs / picture ... manually?

lunny · 2022-09-22T08:45:40Z

[storage]
STORAGE_TYPE            = minio
MINIO_ENDPOINT          = s3.my-domain.net
MINIO_BUCKET            = gitea
MINIO_LOCATION          = homenet
MINIO_USE_SSL           = true
MINIO_ACCESS_KEY_ID     = correct_key
MINIO_SECRET_ACCESS_KEY = correct_key

Does this work for all storage types? so i don't need to configure attachments / lfs / picture ... manually?

Yes, except you have special configuration which will override it.

navanshu · 2023-01-02T06:28:36Z

As a workround, you can remove all the related configuration and only left below:

[storage]
STORAGE_TYPE            = minio
MINIO_ENDPOINT          = s3.my-domain.net
MINIO_BUCKET            = gitea
MINIO_LOCATION          = homenet
MINIO_USE_SSL           = true
MINIO_ACCESS_KEY_ID     = correct_key
MINIO_SECRET_ACCESS_KEY = correct_key

Does this also work with [attachment] configuration.

lunny · 2023-01-14T10:03:44Z

I will add some test and possible fix after #22405 merge.

…23911) Follow up #22405 Fix #20703 This PR rewrites storage configuration read sequences with some breaks and tests. It becomes more strict than before and also fixed some inherit problems. - Move storage's MinioConfig struct into setting, so after the configuration loading, the values will be stored into the struct but not still on some section. - All storages configurations should be stored on one section, configuration items cannot be overrided by multiple sections. The prioioty of configuration is `[attachment]` > `[storage.attachments]` | `[storage.customized]` > `[storage]` > `default` - For extra override configuration items, currently are `SERVE_DIRECT`, `MINIO_BASE_PATH`, `MINIO_BUCKET`, which could be configured in another section. The prioioty of the override configuration is `[attachment]` > `[storage.attachments]` > `default`. - Add more tests for storages configurations. - Update the storage documentations. --------- Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>

…o-gitea#23911) Follow up go-gitea#22405 Fix go-gitea#20703 This PR rewrites storage configuration read sequences with some breaks and tests. It becomes more strict than before and also fixed some inherit problems. - Move storage's MinioConfig struct into setting, so after the configuration loading, the values will be stored into the struct but not still on some section. - All storages configurations should be stored on one section, configuration items cannot be overrided by multiple sections. The prioioty of configuration is `[attachment]` > `[storage.attachments]` | `[storage.customized]` > `[storage]` > `default` - For extra override configuration items, currently are `SERVE_DIRECT`, `MINIO_BASE_PATH`, `MINIO_BUCKET`, which could be configured in another section. The prioioty of the override configuration is `[attachment]` > `[storage.attachments]` > `default`. - Add more tests for storages configurations. - Update the storage documentations. --------- Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>

@lunny

…23911) (#25244) Backport #23911 by @lunny Follow up #22405 Fix #20703 This PR rewrites storage configuration read sequences with some breaks and tests. It becomes more strict than before and also fixed some inherit problems. - Move storage's MinioConfig struct into setting, so after the configuration loading, the values will be stored into the struct but not still on some section. - All storages configurations should be stored on one section, configuration items cannot be overrided by multiple sections. The prioioty of configuration is `[attachment]` > `[storage.attachments]` | `[storage.customized]` > `[storage]` > `default` - For extra override configuration items, currently are `SERVE_DIRECT`, `MINIO_BASE_PATH`, `MINIO_BUCKET`, which could be configured in another section. The prioioty of the override configuration is `[attachment]` > `[storage.attachments]` > `default`. - Add more tests for storages configurations. - Update the storage documentations. Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com> Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>

mmeier86 added the type/bug label Aug 7, 2022

lunny mentioned this issue Jan 14, 2023

Repository archive settings does not support custom storage #21899

Closed

lunny added this to the 1.19.1 milestone Apr 4, 2023

lunny self-assigned this Apr 4, 2023

lunny mentioned this issue Apr 4, 2023

Fix all possible setting error related storages and added some tests #23911

Merged

6543 modified the milestones: 1.19.1, 1.19.2 Apr 12, 2023

lunny modified the milestones: 1.19.2, 1.19.3, 1.19.4 Apr 28, 2023

lunny mentioned this issue May 26, 2023

Add test to confirm storage could be inherit correctly #24940

Closed

delvh removed this from the 1.19.4 milestone Jun 8, 2023

lunny closed this as completed in #23911 Jun 14, 2023

GiteaBot mentioned this issue Jun 14, 2023

Fix all possible setting error related storages and added some tests (#23911) #25244

Merged

github-actions bot locked as resolved and limited conversation to collaborators Jul 30, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

The storage.repo-archive and storage.packages configs do not accept custom STORAGE_TYPE #20703

The storage.repo-archive and storage.packages configs do not accept custom STORAGE_TYPE #20703

mmeier86 commented Aug 7, 2022 •

edited

Loading

lunny commented Aug 8, 2022

mmeier86 commented Aug 8, 2022

viceice commented Sep 22, 2022

lunny commented Sep 22, 2022

navanshu commented Jan 2, 2023

lunny commented Jan 14, 2023

The storage.repo-archive and storage.packages configs do not accept custom STORAGE_TYPE #20703

The storage.repo-archive and storage.packages configs do not accept custom STORAGE_TYPE #20703

Comments

mmeier86 commented Aug 7, 2022 • edited Loading

Description

Gitea Version

Can you reproduce the bug on the Gitea demo site?

Log Gist

Screenshots

Git Version

Operating System

How are you running Gitea?

Database

lunny commented Aug 8, 2022

mmeier86 commented Aug 8, 2022

viceice commented Sep 22, 2022

lunny commented Sep 22, 2022

navanshu commented Jan 2, 2023

lunny commented Jan 14, 2023

mmeier86 commented Aug 7, 2022 •

edited

Loading