Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update fails with '$dcr snuba-api snuba bootstrap --no-migrate --force' exited with status 1 #2744

Closed
adminZebralog opened this issue Jan 29, 2024 · 4 comments

Comments

@adminZebralog
Copy link

Self-Hosted Version

23.11.0.dev0

CPU Architecture

x86_64

Docker Version

25.0.1

Docker Compose Version

2.24.2

Steps to Reproduce

  1. git fetch
  2. git checkout 24.1.1 - but every version I tried gave the same error
  3. ./install.sh --report-self-hosted-issues

OS is Debian 12 Bookworm headless

Expected Result

Installer is upgrading Sentry

Actual Result

Installer fails with '$dcr snuba-api snuba bootstrap --no-migrate --force' exited with status 1 error message.

[...]
Docker images built.

▶ Downloading and installing wal2json ...

▶ Bootstrapping and migrating Snuba ...
 Network sentry-self-hosted_default  Creating
 Network sentry-self-hosted_default  Created
 Volume "sentry-self-hosted_sentry-zookeeper-log"  Creating
 Volume "sentry-self-hosted_sentry-zookeeper-log"  Created
 Container sentry-self-hosted-redis-1  Creating
 Container sentry-self-hosted-zookeeper-1  Creating
 Container sentry-self-hosted-clickhouse-1  Creating
 Container sentry-self-hosted-clickhouse-1  Created
 Container sentry-self-hosted-redis-1  Created
 Container sentry-self-hosted-zookeeper-1  Created
 Container sentry-self-hosted-kafka-1  Creating
 Container sentry-self-hosted-kafka-1  Created
 Container sentry-self-hosted-clickhouse-1  Starting
 Container sentry-self-hosted-zookeeper-1  Starting
 Container sentry-self-hosted-redis-1  Starting
 Container sentry-self-hosted-redis-1  Started
 Container sentry-self-hosted-zookeeper-1  Started
 Container sentry-self-hosted-zookeeper-1  Waiting
 Container sentry-self-hosted-clickhouse-1  Started
 Container sentry-self-hosted-zookeeper-1  Healthy
 Container sentry-self-hosted-kafka-1  Starting
 Container sentry-self-hosted-kafka-1  Started
dependency failed to start: container sentry-self-hosted-kafka-1 is unhealthy
Error in install/bootstrap-snuba.sh:3.
'$dcr snuba-api bootstrap --no-migrate --force' exited with status 1
-> ./install.sh:main:31
--> install/bootstrap-snuba.sh:source:3

install/error-handling.sh: line 81: /usr/bin/docker: Argument list too long

Event ID

No response

@aldy505
Copy link
Collaborator

aldy505 commented Jan 30, 2024

Your Kafka is unhealthy, can we see the logs with sudo docker compose logs kafka -t --since 24h?

@getsantry getsantry bot moved this to Waiting for: Community in GitHub Issues with 👀 2 Jan 30, 2024
@getsantry getsantry bot moved this from Waiting for: Community to Waiting for: Product Owner in GitHub Issues with 👀 2 Jan 30, 2024
@adminZebralog
Copy link
Author

Ok we have the next error:

'$dcr web upgrade' exited with status 1
-> ./install.sh:main:34
--> install/set-up-and-migrate-database.sh:source:12

install/error-handling.sh: line 81: /usr/bin/docker: Argument list too long

What I did:
I restored a working snapshot after failed upgrade yesterday to get the sentry logs I tried to do the update again but now I get this new error.

[...]
  Applying sentry.0628_better_menv_latest_index... OK
  Applying sentry.0629_eventattachment_index... OK
  Applying sentry.0630_better_monitor_latest_index... OK
  Applying sentry.0631_add_priority_columns_to_groupedmessage... OK
Api Tokens: N/A% |                                             | ETA:  --:--:--Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/django/utils/module_loading.py", line 30, in import_string
    return cached_import(module_path, class_name)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/utils/module_loading.py", line 16, in cached_import
    return getattr(module, class_name)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: module 'django.core.cache.backends.memcached' has no attribute 'MemcachedCache'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/django/core/cache/__init__.py", line 47, in create_connection
    backend_cls = import_string(backend)
                  ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/utils/module_loading.py", line 32, in import_string
    raise ImportError(
ImportError: Module "django.core.cache.backends.memcached" does not define a "MemcachedCache" attribute/class

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/sentry/options/store.py", line 119, in get_cache
    value = self.cache.get(cache_key)
            ^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/utils/connection.py", line 15, in __getattr__
    return getattr(self._connections[self._alias], item)
                   ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/utils/connection.py", line 62, in __getitem__
    conn = self.create_connection(alias)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sentry_sdk/integrations/django/caching.py", line 108, in sentry_create_connection
    cache = original_create_connection(self, alias)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/core/cache/__init__.py", line 49, in create_connection
    raise InvalidCacheBackendError(
django.core.cache.backends.base.InvalidCacheBackendError: Could not find backend 'django.core.cache.backends.memcached.MemcachedCache': Module "django.core.cache.backends.memcached" does not define a "MemcachedCache" attribute/class
07:41:35 [WARNING] sentry: Unable to fetch option cache for apitoken.auto-add-last-chars (key='apitoken.auto-add-last-chars')
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/django/utils/module_loading.py", line 30, in import_string
    return cached_import(module_path, class_name)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/utils/module_loading.py", line 16, in cached_import
    return getattr(module, class_name)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: module 'django.core.cache.backends.memcached' has no attribute 'MemcachedCache'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/django/core/cache/__init__.py", line 47, in create_connection
    backend_cls = import_string(backend)
                  ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/utils/module_loading.py", line 32, in import_string
    raise ImportError(
ImportError: Module "django.core.cache.backends.memcached" does not define a "MemcachedCache" attribute/class

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/sentry/options/store.py", line 262, in set_cache
    self.cache.set(cache_key, value, self.ttl)
    ^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/utils/connection.py", line 15, in __getattr__
    return getattr(self._connections[self._alias], item)
                   ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/utils/connection.py", line 62, in __getitem__
    conn = self.create_connection(alias)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sentry_sdk/integrations/django/caching.py", line 108, in sentry_create_connection
    cache = original_create_connection(self, alias)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/core/cache/__init__.py", line 49, in create_connection
    raise InvalidCacheBackendError(
django.core.cache.backends.base.InvalidCacheBackendError: Could not find backend 'django.core.cache.backends.memcached.MemcachedCache': Module "django.core.cache.backends.memcached" does not define a "MemcachedCache" attribute/class
07:41:35 [WARNING] sentry: Unable to update option cache for apitoken.auto-add-last-chars (key='apitoken.auto-add-last-chars')
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/django/utils/module_loading.py", line 30, in import_string
    return cached_import(module_path, class_name)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/utils/module_loading.py", line 16, in cached_import
    return getattr(module, class_name)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: module 'django.core.cache.backends.memcached' has no attribute 'MemcachedCache'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/django/core/cache/__init__.py", line 47, in create_connection
    backend_cls = import_string(backend)
                  ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/utils/module_loading.py", line 32, in import_string
    raise ImportError(
ImportError: Module "django.core.cache.backends.memcached" does not define a "MemcachedCache" attribute/class

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/sentry/options/store.py", line 119, in get_cache
    value = self.cache.get(cache_key)
            ^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/utils/connection.py", line 15, in __getattr__
    return getattr(self._connections[self._alias], item)
                   ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/utils/connection.py", line 62, in __getitem__
    conn = self.create_connection(alias)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sentry_sdk/integrations/django/caching.py", line 108, in sentry_create_connection
    cache = original_create_connection(self, alias)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/core/cache/__init__.py", line 49, in create_connection
    raise InvalidCacheBackendError(
django.core.cache.backends.base.InvalidCacheBackendError: Could not find backend 'django.core.cache.backends.memcached.MemcachedCache': Module "django.core.cache.backends.memcached" does not define a "MemcachedCache" attribute/class
07:41:35 [WARNING] sentry: Unable to fetch option cache for transaction-events.force-disable-internal-project (key='transaction-events.force-disable-internal-project')
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/django/utils/module_loading.py", line 30, in import_string
    return cached_import(module_path, class_name)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/utils/module_loading.py", line 16, in cached_import
    return getattr(module, class_name)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: module 'django.core.cache.backends.memcached' has no attribute 'MemcachedCache'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/django/core/cache/__init__.py", line 47, in create_connection
    backend_cls = import_string(backend)
                  ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/utils/module_loading.py", line 32, in import_string
    raise ImportError(
ImportError: Module "django.core.cache.backends.memcached" does not define a "MemcachedCache" attribute/class

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/sentry/options/store.py", line 262, in set_cache
    self.cache.set(cache_key, value, self.ttl)
    ^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/utils/connection.py", line 15, in __getattr__
    return getattr(self._connections[self._alias], item)
                   ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/utils/connection.py", line 62, in __getitem__
    conn = self.create_connection(alias)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sentry_sdk/integrations/django/caching.py", line 108, in sentry_create_connection
    cache = original_create_connection(self, alias)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/core/cache/__init__.py", line 49, in create_connection
    raise InvalidCacheBackendError(
django.core.cache.backends.base.InvalidCacheBackendError: Could not find backend 'django.core.cache.backends.memcached.MemcachedCache': Module "django.core.cache.backends.memcached" does not define a "MemcachedCache" attribute/class
07:41:35 [WARNING] sentry: Unable to update option cache for transaction-events.force-disable-internal-project (key='transaction-events.force-disable-internal-project')
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/django/utils/module_loading.py", line 30, in import_string
    return cached_import(module_path, class_name)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/utils/module_loading.py", line 16, in cached_import
    return getattr(module, class_name)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: module 'django.core.cache.backends.memcached' has no attribute 'MemcachedCache'. Did you mean: 'PyMemcacheCache'?

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/django/core/cache/__init__.py", line 47, in create_connection
    backend_cls = import_string(backend)
                  ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/utils/module_loading.py", line 32, in import_string
    raise ImportError(
ImportError: Module "django.core.cache.backends.memcached" does not define a "MemcachedCache" attribute/class

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/bin/sentry", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sentry/runner/__init__.py", line 190, in main
    func(**kwargs)
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sentry/runner/decorators.py", line 29, in inner
    return ctx.invoke(f, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sentry/runner/commands/upgrade.py", line 117, in upgrade
    _upgrade(
  File "/usr/local/lib/python3.11/site-packages/sentry/runner/commands/upgrade.py", line 52, in _upgrade
    dj_call_command(
  File "/usr/local/lib/python3.11/site-packages/django/core/management/__init__.py", line 194, in call_command
    return command.execute(*args, **defaults)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/core/management/base.py", line 458, in execute
    output = self.handle(*args, **options)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/core/management/base.py", line 106, in wrapper
    res = handle_func(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/core/management/commands/migrate.py", line 356, in handle
    post_migrate_state = executor.migrate(
                         ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/migrations/executor.py", line 135, in migrate
    state = self._migrate_all_forwards(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/migrations/executor.py", line 167, in _migrate_all_forwards
    state = self.apply_migration(
            ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sentry/new_migrations/monkey/executor.py", line 140, in apply_migration
    return super().apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/migrations/executor.py", line 252, in apply_migration
    state = migration.apply(state, schema_editor)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sentry/new_migrations/migrations.py", line 22, in apply
    return super().apply(project_state, schema_editor, collect_sql)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/migrations/migration.py", line 132, in apply
    operation.database_forwards(
  File "/usr/local/lib/python3.11/site-packages/django/db/migrations/operations/special.py", line 193, in database_forwards
    self.code(from_state.apps, schema_editor)
  File "/usr/local/lib/python3.11/site-packages/sentry/migrations/0632_apitoken_backfill_last_chars.py", line 16, in backfill_last_token_characters
    api_token.save()
  File "/usr/local/lib/python3.11/site-packages/sentry/silo/base.py", line 145, in override
    return original_method(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sentry/models/apitoken.py", line 66, in save
    return super().save(**kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sentry/db/models/outboxes.py", line 258, in save
    super().save(*args, **kwds)
  File "/usr/local/lib/python3.11/site-packages/django/db/models/base.py", line 814, in save
    self.save_base(
  File "/usr/local/lib/python3.11/site-packages/sentry/silo/base.py", line 145, in override
    return original_method(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/models/base.py", line 892, in save_base
    post_save.send(
  File "/usr/local/lib/python3.11/site-packages/django/dispatch/dispatcher.py", line 176, in send
    return [
           ^
  File "/usr/local/lib/python3.11/site-packages/django/dispatch/dispatcher.py", line 177, in <listcomp>
    (receiver, receiver(signal=self, sender=sender, **named))
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sentry/db/models/manager/base.py", line 181, in __post_save
    cache.set(
    ^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/utils/connection.py", line 15, in __getattr__
    return getattr(self._connections[self._alias], item)
                   ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/utils/connection.py", line 62, in __getitem__
    conn = self.create_connection(alias)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sentry_sdk/integrations/django/caching.py", line 108, in sentry_create_connection
    cache = original_create_connection(self, alias)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/core/cache/__init__.py", line 49, in create_connection
    raise InvalidCacheBackendError(
django.core.cache.backends.base.InvalidCacheBackendError: Could not find backend 'django.core.cache.backends.memcached.MemcachedCache': Module "django.core.cache.backends.memcached" does not define a "MemcachedCache" attribute/class
Error in install/set-up-and-migrate-database.sh:12.
'$dcr web upgrade' exited with status 1
-> ./install.sh:main:34
--> install/set-up-and-migrate-database.sh:source:12

install/error-handling.sh: line 81: /usr/bin/docker: Argument list too long

@aldy505
Copy link
Collaborator

aldy505 commented Jan 30, 2024

At this point, you must update your sentry/sentry.conf.py file. There's an update to the Memcached problem.

@adminZebralog
Copy link
Author

How could I miss that note...
https://github.com/getsentry/self-hosted/pull/2722/files
helped here

Thanks

@aldy505 aldy505 mentioned this issue Jan 30, 2024
18 tasks
@github-actions github-actions bot locked and limited conversation to collaborators Feb 15, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
Archived in project
Archived in project
Development

No branches or pull requests

2 participants