Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Storage in S3 buckets using minio not working in Gitea 1.20.0 with working config from Gitea 1.19.4 #25984

Closed
tobiasbp opened this issue Jul 19, 2023 · 8 comments · Fixed by #26420
Labels
issue/critical This issue should be fixed ASAP. If it is a PR, the PR should be merged ASAP type/bug

Comments

@tobiasbp
Copy link
Contributor

Description

Storage in s3 buckets (using minio) no longer works when using Gitea 1.20.0 with chart v9.0.0. Changing the Gitea version to 1.19.4 with chart v9.0.0 (same configuration) makes storage in s3 buckets work again as described in the issue here:
https://gitea.com/gitea/helm-chart/issues/469

The storage configuration is:

    storage:
      MINIO_ACCESS_KEY_ID: ***
      MINIO_BUCKET: my-existing-bucket
      MINIO_ENDPOINT: storage.googleapis.com
      MINIO_INSECURE_SKIP_VERIFY: false
      MINIO_SECRET_ACCESS_KEY: ***
      MINIO_USE_SSL: true
      SERVE_DIRECT: false
      STORAGE_TYPE: minio

With the above storage config, Gitea 1.20.0 says:

2023/07/18 16:33:54 ...s/setting/session.go:74:loadSessionFrom() [I] Session Service Enabled
2023/07/18 16:33:54 ...s/storage/storage.go:176:initAttachments() [I] Initialising Attachment storage with type: minio
2023/07/18 16:33:54 ...les/storage/minio.go:81:NewMinioStorage() [I] Creating Minio storage at storage.googleapis.com:my-existing-bucket with base path attachments/
2023/07/18 16:33:55 routers/init.go:60:mustInit() [F] code.gitea.io/gitea/modules/storage.Init failed: The requested bucket name is not available. The bucket namespace is shared by all users of the system. Please select a different name and try again

Is Gitea/minio trying to create the existing bucket?

Gitea Version

1.20.0

Can you reproduce the bug on the Gitea demo site?

No

Log Gist

No response

Screenshots

No response

Git Version

No response

Operating System

No response

How are you running Gitea?

In Kubernetes with chart v9.0.0.

Database

PostgreSQL

@tobiasbp tobiasbp changed the title Storage in S3 buckets using minio not working in 1.2.0 Storage in S3 buckets using minio not working in 1.20.0 Jul 19, 2023
@tobiasbp
Copy link
Contributor Author

I have updated to Gitea 1.20.1 (Now using Gitea chart 9.0.4). If I configure MINIO_LOCATION to the location of my existong bucket, it sounds like minio tries to create the bucket??

Defaulted container "gitea" out of: gitea, init-directories (init), init-app-ini (init), configure-gitea (init)
2023/07/25 15:51:00 cmd/web.go:223:runWeb() [I] Starting Gitea on PID: 7
2023/07/25 15:51:00 cmd/web.go:148:serveInstalled() [I] Gitea version: 1.20.1 built with GNU Make 4.4.1, go1.20.6 : bindata, timetzdata, sqlite, sqlite_unlock_notify
2023/07/25 15:51:00 cmd/web.go:149:serveInstalled() [I] App path: /usr/local/bin/gitea
2023/07/25 15:51:00 cmd/web.go:150:serveInstalled() [I] Work path: /data
2023/07/25 15:51:00 cmd/web.go:151:serveInstalled() [I] Custom path: /data/gitea
2023/07/25 15:51:00 cmd/web.go:152:serveInstalled() [I] Config file: /data/gitea/conf/app.ini
2023/07/25 15:51:00 cmd/web.go:153:serveInstalled() [I] Run mode: dev
2023/07/25 15:51:00 cmd/web.go:154:serveInstalled() [I] Prepare to run web server
2023/07/25 15:51:01 routers/init.go:112:InitWebInstalled() [I] Git version: 2.40.1, Wire Protocol Version 2 Enabled (home: /data/home)
2023/07/25 15:51:02 ...les/setting/cache.go:75:loadCacheFrom() [I] Cache Service Enabled
2023/07/25 15:51:02 ...les/setting/cache.go:90:loadCacheFrom() [I] Last Commit Cache Service Enabled
2023/07/25 15:51:02 ...s/setting/session.go:74:loadSessionFrom() [I] Session Service Enabled
2023/07/25 15:51:02 ...s/storage/storage.go:176:initAttachments() [I] Initialising Attachment storage with type: minio
2023/07/25 15:51:02 ...les/storage/minio.go:81:NewMinioStorage() [I] Creating Minio storage at storage.googleapis.com:gitea-stg-aqrk with base path attachments/
2023/07/25 15:51:02 routers/init.go:60:mustInit() [F] code.gitea.io/gitea/modules/storage.Init failed: permission denied

@tobiasbp tobiasbp changed the title Storage in S3 buckets using minio not working in 1.20.0 Storage in S3 buckets using minio not working in Gitea 1.20.0 with working config from Gitea 1.19.4 Jul 25, 2023
@lunny
Copy link
Member

lunny commented Jul 25, 2023

I have updated to Gitea 1.20.1 (Now using Gitea chart 9.0.4). If I configure MINIO_LOCATION to the location of my existong bucket, it sounds like minio tries to create the bucket??

Defaulted container "gitea" out of: gitea, init-directories (init), init-app-ini (init), configure-gitea (init)
2023/07/25 15:51:00 cmd/web.go:223:runWeb() [I] Starting Gitea on PID: 7
2023/07/25 15:51:00 cmd/web.go:148:serveInstalled() [I] Gitea version: 1.20.1 built with GNU Make 4.4.1, go1.20.6 : bindata, timetzdata, sqlite, sqlite_unlock_notify
2023/07/25 15:51:00 cmd/web.go:149:serveInstalled() [I] App path: /usr/local/bin/gitea
2023/07/25 15:51:00 cmd/web.go:150:serveInstalled() [I] Work path: /data
2023/07/25 15:51:00 cmd/web.go:151:serveInstalled() [I] Custom path: /data/gitea
2023/07/25 15:51:00 cmd/web.go:152:serveInstalled() [I] Config file: /data/gitea/conf/app.ini
2023/07/25 15:51:00 cmd/web.go:153:serveInstalled() [I] Run mode: dev
2023/07/25 15:51:00 cmd/web.go:154:serveInstalled() [I] Prepare to run web server
2023/07/25 15:51:01 routers/init.go:112:InitWebInstalled() [I] Git version: 2.40.1, Wire Protocol Version 2 Enabled (home: /data/home)
2023/07/25 15:51:02 ...les/setting/cache.go:75:loadCacheFrom() [I] Cache Service Enabled
2023/07/25 15:51:02 ...les/setting/cache.go:90:loadCacheFrom() [I] Last Commit Cache Service Enabled
2023/07/25 15:51:02 ...s/setting/session.go:74:loadSessionFrom() [I] Session Service Enabled
2023/07/25 15:51:02 ...s/storage/storage.go:176:initAttachments() [I] Initialising Attachment storage with type: minio
2023/07/25 15:51:02 ...les/storage/minio.go:81:NewMinioStorage() [I] Creating Minio storage at storage.googleapis.com:gitea-stg-aqrk with base path attachments/
2023/07/25 15:51:02 routers/init.go:60:mustInit() [F] code.gitea.io/gitea/modules/storage.Init failed: permission denied

What's your workround? Maybe I can get some clues to fix it.

@tobiasbp
Copy link
Contributor Author

tobiasbp commented Jul 25, 2023

What's your workround? Maybe I can get some clues to fix it.

I have no workaround? I can no longer use my S3 storage. It's just that the error i the log changes if I set MINIO_LOCATION.

I assume my problem is related to Rework storage settings -> #23911

But since I only have one section with storage config, I think my config should still work. I have a hard time following the changes being described in #23911

@lunny
Copy link
Member

lunny commented Jul 25, 2023

What's your workround? Maybe I can get some clues to fix it.

I have no workaround? I can no longer use my S3 storage.

I assume my problem is related to Rework storage settings -> #23911

Yes, I think so. Can you post your generated storage configuration in app.ini and remove the seurity informations. And I will figure it out tomorrow.

@tobiasbp
Copy link
Contributor Author

Here is my app.ini. I have had to set storage.STORAGE_TYPE = local to get the pod to start up. The rest of the config in [storage] is untouched. I don't know if I should configure storage.MINIO_LOCATION or not? The error changes when I do (As described in earlier posts):

/var/lib/gitea $ cat /data/gitea/conf/app.ini


APP_NAME = Gitea (stg)
RUN_MODE = dev
WORK_PATH = /data

[storage]
MINIO_LOCATION = europe-north1
MINIO_USE_SSL = true
MINIO_ENDPOINT = storage.googleapis.com
STORAGE_TYPE = local
SERVE_DIRECT = false
MINIO_BUCKET = gitea-stg-aqrk
MINIO_SECRET_ACCESS_KEY = **REDACTED**
MINIO_ACCESS_KEY_ID = **REDACTED**
MINIO_INSECURE_SKIP_VERIFY = false

[indexer]
ISSUE_INDEXER_TYPE = db

[server]
LFS_JWT_SECRET = **REDACTED**
ROOT_URL = **REDACTED**
LFS_START_SERVER = true
ENABLE_PPROF = false
LANDING_PAGE = login
SSH_PORT = 22
APP_DATA_PATH = /data
SSH_LISTEN_PORT = 2222
PROTOCOL = http
HTTP_PORT = 3000
START_SSH_SERVER = true
DISABLE_SSH = true
SSH_DOMAIN = **REDACTED**
DOMAIN = **REDACTED**

[database]
NAME = gitea-db
SSL_MODE = disable
DB_TYPE = postgres
HOST = sql-proxy-gcloud-sqlproxy:5432
USER = gitea-user
PASSWD = **REDACTED**

[service]
SHOW_MILESTONES_DASHBOARD_PAGE = false
REQUIRE_SIGNIN_VIEW = true
DISABLE_REGISTRATION = true
EXPLORE = map[DISABLE_USERS_PAGE:true REQUIRE_SIGNIN_VIEW:true]
DEFAULT_ORG_VISIBILITY = private
DEFAULT_USER_VISIBILITY = private
DEFAULT_ALLOW_CREATE_ORGANIZATION = false

[repository]
DISABLE_STARS = true
DISABLE_MIGRATIONS = true
FORCE_PRIVATE = true
ROOT = /data/git/gitea-repositories
DISABLED_REPO_UNITS = repo.wiki, repo.ext_wiki, repo.packages, repo.projects, repo.issues, repo.ext_issues, repo.pulls
DEFAULT_PRIVATE = true

[oauth2_client]
REGISTER_EMAIL_CONFIRM = false
USERNAME = email
UPDATE_AVATAR = true
ENABLE_AUTO_REGISTRATION = true
OPENID_CONNECT_SCOPES = openid profile email

[security]
DISABLE_WEBHOOKS = true
INTERNAL_TOKEN = **REDACTED**
SECRET_KEY = **REDACTED**
INSTALL_LOCK = true

[session]
PROVIDER_CONFIG = redis://redis-1.source-access.private:6379/0?pool_size=100&idle_timeout=180s
PROVIDER = redis

[cache]
HOST = redis://redis-1.source-access.private:6379/0?pool_size=100&idle_timeout=180s
ENABLED = true
ADAPTER = redis

[admin]
DISABLE_REGULAR_ORG_CREATION = true
DEFAULT_EMAIL_NOTIFICATIONS = false

[metrics]
ENABLED = true

[oauth2]
JWT_SECRET = **REDACTED**

@lunny lunny added the issue/critical This issue should be fixed ASAP. If it is a PR, the PR should be merged ASAP label Jul 27, 2023
@evanreichard
Copy link

evanreichard commented Jul 27, 2023

I'm assuming this is related, but since the update my [storage.packages] override no longer works. I don't think I violate the incompatibility detailed in #23911

For the time being I just set up a symbolic link.

My redacted configuration:

APP_NAME = Gitea: Git with a cup of tea
RUN_MODE = prod
RUN_USER = git
WORK_PATH = /data/gitea

[repository]
ROOT = /data/git/repositories

[repository.local]
LOCAL_COPY_PATH = /data/gitea/tmp/local-repo

[repository.upload]
TEMP_PATH = /data/gitea/uploads

[server]
APP_DATA_PATH = /data/gitea
SSH_DOMAIN = <REDACTED>
HTTP_PORT = 3000
ROOT_URL = <REDACTED>
DISABLE_SSH = false
SSH_PORT = 22
SSH_LISTEN_PORT = 22
LFS_START_SERVER = true
DOMAIN = <REDACTED>
LFS_JWT_SECRET = <REDACTED>
OFFLINE_MODE = false

[database]
PATH = /data/gitea/gitea.db
DB_TYPE = sqlite3
HOST = localhost:3306
NAME = gitea
USER = <REDACTED>
PASSWD = <REDACTED>
SCHEMA = 
SSL_MODE = disable
CHARSET = utf8

[indexer]
ISSUE_INDEXER_PATH = /data/gitea/indexers/issues.bleve

[session]
PROVIDER_CONFIG = /data/gitea/sessions
PROVIDER = file

[picture]
AVATAR_UPLOAD_PATH = /data/gitea/avatars
REPOSITORY_AVATAR_UPLOAD_PATH = /data/gitea/repo-avatars
DISABLE_GRAVATAR = false
ENABLE_FEDERATED_AVATAR = true

[log]
ROOT_PATH = /data/gitea/log
MODE = file
LEVEL = info

[security]
INSTALL_LOCK = true
SECRET_KEY = <REDACTED>
INTERNAL_TOKEN = <REDACTED>

[service]
DISABLE_REGISTRATION = true
REQUIRE_SIGNIN_VIEW = false
REGISTER_EMAIL_CONFIRM = false
ENABLE_NOTIFY_MAIL = false
ALLOW_ONLY_EXTERNAL_REGISTRATION = false
ENABLE_CAPTCHA = false
DEFAULT_KEEP_EMAIL_PRIVATE = false
DEFAULT_ALLOW_CREATE_ORGANIZATION = true
DEFAULT_ENABLE_TIMETRACKING = true
NO_REPLY_ADDRESS = noreply.localhost
SHOW_REGISTRATION_BUTTON = false

[oauth2]
JWT_SECRET = <REDACTED>

[mailer]
ENABLED = false

[openid]
ENABLE_OPENID_SIGNIN = false
ENABLE_OPENID_SIGNUP = false

[ui]
DEFAULT_THEME = arc-green
THEMES = gitea,arc-green

[lfs]
PATH = /data/git/lfs

[attachment]
PATH = /data/gitea/attachments
MAX_SIZE = 50

[webhook]
ALLOWED_HOST_LIST = <REDACTED>

[storage.packages]
PATH = /packages

@tobiasbp
Copy link
Contributor Author

tobiasbp commented Aug 1, 2023

Any idea as to what could be causing this issue? Is it a problem in Minio?

@Lapotor
Copy link

Lapotor commented Aug 9, 2023

On v1.20.2 the error still appears.
I think that is a problem on the implementation because it tries to create the bucket on every storage.

wxiaoguang pushed a commit that referenced this issue Aug 12, 2023
…st and then create one if it doesn't exist (#26420)

For some reason, the permission of the client_id and secret may cannot
create bucket, so now we will check whether bucket does exist first and
then try to create a bucket if it doesn't exist.

Try to fix #25984

Co-authored-by: silverwind <me@silverwind.io>
GiteaBot pushed a commit to GiteaBot/gitea that referenced this issue Aug 12, 2023
…st and then create one if it doesn't exist (go-gitea#26420)

For some reason, the permission of the client_id and secret may cannot
create bucket, so now we will check whether bucket does exist first and
then try to create a bucket if it doesn't exist.

Try to fix go-gitea#25984

Co-authored-by: silverwind <me@silverwind.io>
silverwind added a commit that referenced this issue Aug 12, 2023
#26465)

Backport #26420 by @lunny

For some reason, the permission of the client_id and secret may cannot
create bucket, so now we will check whether bucket does exist first and
then try to create a bucket if it doesn't exist.

Try to fix #25984

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: silverwind <me@silverwind.io>
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Sep 27, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
issue/critical This issue should be fixed ASAP. If it is a PR, the PR should be merged ASAP type/bug
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants