Minio S3 storage not working for LFS or attachments

[adamb70]

• Gitea version (or commit ref): 1.14.0+dev-69-gd2ad4dec6
• Git version: 2.26.2
• Operating system: Ubuntu 20

Using latest Docker image.

• Database (use [x]):
  • PostgreSQL
  • MySQL
  • MSSQL
  • SQLite
• Log gist: https://gist.github.com/adamb70/0f58c68486d826cd4810792e1dad9b66

Description

I am trying to use S3 to host files in attachments and LFS using the minio storage type, but the files are instead being stored locally. I think this may be a problem of improper configuration, as there are no errors logged so it seems that Gitea doesn't even try to save the files to S3. The relevant parts of the ini file are below, any help is appreciated.

[server]
APP_DATA_PATH    = /data/gitea
DOMAIN           = git.mydomain.com
SSH_DOMAIN       = git.mydomain.com
HTTP_PORT        = 3000
ROOT_URL         = https://git.mydomain.com/
DISABLE_SSH      = false
SSH_PORT         = 22
SSH_LISTEN_PORT  = 22
LFS_START_SERVER = true
LFS_CONTENT_PATH = /data/git/lfs
LFS_JWT_SECRET   = {secret}
OFFLINE_MODE     = false

[lfs]
STORAGE_TYPE = minio
SERVE_DIRECT = false
MINIO_ENDPOINT = s3.eu-west-1.amazonaws.com
MINIO_ACCESS_KEY_ID = {secret}
MINIO_SECRET_ACCESS_KEY = {secret}
MINIO_BUCKET = my-gitea-bucket
MINIO_LOCATION = eu-west-1
MINIO_BASE_PATH = /lfs
MINIO_USE_SSL = true

[attachment]
STORAGE_TYPE = minio
SERVE_DIRECT = false
MINIO_ENDPOINT = s3.eu-west-1.amazonaws.com
MINIO_ACCESS_KEY_ID = {secret}
MINIO_SECRET_ACCESS_KEY = {secret}
MINIO_BUCKET = my-gitea-bucket
MINIO_LOCATION = eu-west-1
MINIO_BASE_PATH = /attachment
MINIO_USE_SSL = true

[lunny]

Could you find storages logs when gitea start, like below

2020/10/24 09:16:20 ...s/storage/storage.go:151:initAttachments() [I] Initialising Attachment storage with type:
2020/10/24 09:16:20 ...les/storage/local.go:44:NewLocalStorage() [I] Creating new Local Storage at /data/gitea/data/attachments
2020/10/24 09:16:20 ...s/storage/storage.go:145:initAvatars() [I] Initialising Avatar storage with type:
2020/10/24 09:16:20 ...les/storage/local.go:44:NewLocalStorage() [I] Creating new Local Storage at /data/gitea/data/avatars
2020/10/24 09:16:20 ...s/storage/storage.go:163:initRepoAvatars() [I] Initialising Repository Avatar storage with type:
2020/10/24 09:16:20 ...les/storage/local.go:44:NewLocalStorage() [I] Creating new Local Storage at /data/gitea/data/repo-avatars
2020/10/24 09:16:20 ...s/storage/storage.go:157:initLFS() [I] Initialising LFS storage with type:
2020/10/24 09:16:20 ...les/storage/local.go:44:NewLocalStorage() [I] Creating new Local Storage at /data/gitea/data/lfs

[adamb70]

I don't see anything about storage in the logs. Here is the startup log: https://gist.github.com/adamb70/5415b31be481db6fc325773c0c956c12

And here is the logging config in my ini file:

[log]
MODE                 = console
LEVEL                = trace
REDIRECT_MACARON_LOG = true
MACARON              = console
ROUTER               = console
ROOT_PATH            = /data/gitea/log

If it helps at all, here is my full gitea-app.ini file: https://gist.github.com/adamb70/963accf8c1e8b28f43c5a336f6262259

[lunny]

It seems there are something lost in your log?

[adamb70]

I am starting Gitea up through docker-compose, so maybe some log info doesn't get sent through to the console?

I have tried again with setting log MODE=file, and the content in gitea.log is still the same as the console output so it doesn't seem like it's Docker related.

[zeripath]

@adamb70 please update to at least g25b7766673. We added logging in from that point to help us diagnose what was going on with storage.

(Assuming you're not running gitea with --port - which you shouldn't anyway - just jump to the latest)

[adamb70]

Thanks for the new release @zeripath. The full log is as follows: https://gist.github.com/adamb70/5e315aa4bc8afde849e64033703e7ddb

The relevant storage part is here:

gitea      | 2020/10/24 21:37:42 ...s/storage/storage.go:151:initAttachments() [I] Initialising Attachment storage with type:
gitea      | 2020/10/24 21:37:42 ...les/storage/local.go:44:NewLocalStorage() [I] Creating new Local Storage at /data/gitea/attachments
gitea      | 2020/10/24 21:37:42 ...s/storage/storage.go:145:initAvatars() [I] Initialising Avatar storage with type:
gitea      | 2020/10/24 21:37:42 ...les/storage/local.go:44:NewLocalStorage() [I] Creating new Local Storage at /data/gitea/avatars
gitea      | 2020/10/24 21:37:42 ...s/storage/storage.go:163:initRepoAvatars() [I] Initialising Repository Avatar storage with type:
gitea      | 2020/10/24 21:37:42 ...les/storage/local.go:44:NewLocalStorage() [I] Creating new Local Storage at /data/gitea/repo-avatars
gitea      | 2020/10/24 21:37:42 ...s/storage/storage.go:157:initLFS() [I] Initialising LFS storage with type:
gitea      | 2020/10/24 21:37:42 ...les/storage/local.go:44:NewLocalStorage() [I] Creating new Local Storage at /data/git/lfs

It seems like it's still trying to use local storage, despite the app.ini file.

I am binding the ini file in the Docker container to a local file outside so it can be managed from outside the container. This seems to work fine to change any other settings in the ini file, it's just the storage I'm seeing the issue with.

docker-compose.yml

services:
  gitea:
    image: gitea/gitea:latest
    container_name: gitea
	...
    volumes:
      - gitea:/data
      - ./gitea-app.ini:/data/gitea/conf/app.ini  # <-----
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro

[zeripath]

Looks like the storage type isn't being set...

[zeripath]

OK it looks like there's a bug here:

gitea/modules/setting/storage.go

Line 49 in c5020cf

storage.Type = sec.Key("STORAGE_TYPE").MustString("")

I bet that this should actually default to typ instead of "".

in the meantime I think if you create a new ini section:

[storage.minio]
STORAGE_TYPE=minio

That would make the minio work

[adamb70]

Strangely I'm now getting the following error:

gitea      | 2020/10/24 22:34:07 ...s/storage/storage.go:151:initAttachments() [I] Initialising Attachment storage with type: minio
gitea      | 2020/10/24 22:34:07 ...les/storage/minio.go:91:NewMinioStorage() [I] Creating Minio storage at : with base path
gitea      | 2020/10/24 22:34:07 routers/init.go:60:NewServices() [F] storage init failed: Endpoint:  does not follow ip address or domain name standards.

It seems like it is interpreting the endpoint URL as an empty string for some reason. Here's the line that I think is throwing the error https://github.com/minio/minio-go/blob/master/utils.go#L91 .

app.ini

[lfs]
STORAGE_TYPE = minio
MINIO_BASE_PATH = /lfs

[attachment]
STORAGE_TYPE = minio
MINIO_BASE_PATH = /attachment

[storage.minio]
STORAGE_TYPE=minio
SERVE_DIRECT = false
MINIO_ENDPOINT = s3.eu-west-1.amazonaws.com
MINIO_ACCESS_KEY_ID = {secret}
MINIO_SECRET_ACCESS_KEY = {secret}
MINIO_BUCKET = my-gitea-bucket
MINIO_LOCATION = eu-west-1
MINIO_BASE_PATH = /attachment
MINIO_USE_SSL = true

[adamb70]

It seems like the config in NewMinioStorage() is set to null values. I've never used golang before, but is this line correct?

gitea/modules/storage/minio.go

Line 89 in 25b7766

config := configInterface.(MinioStorageConfig)

Some Googling suggests it should be config, ok := configInterface.(MinioStorageConfig).

[zeripath]

nope that's not it.

The problem will be in modules/setting/storage.go

[zeripath]

Actually it wasn't!

The issue is that the toConfig string wasn't matching the Mappable interface because the config wasn't being passed as a pointer.

There was a further bug in the MapTo.

Sorry about this - I think I have it finally fixed now.