Migrations sentry.0640_index_together fails

[pajowu]

Environment

self-hosted (https://develop.sentry.dev/self-hosted/)

Steps to Reproduce

1. Checkout the nightly self-hosted repo into an existing project
2. Run install.sh

Expected Result

Upgrade runs successfully

Actual Result

Upgrade fails in migration sentry.0640_index_together with the error below. This is caused by this following migration step which calls RenameIndex, which fails because two indices exist on these columns: one created by index_together as well as one created by unique_together.

This is a django bug that is fixed with this django commit. After applying this commit into our sentry installation, the migration ran successfully.

Traceback (most recent call last):
  File "/usr/local/bin/sentry", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sentry/runner/__init__.py", line 190, in main
    func(**kwargs)
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sentry/runner/decorators.py", line 29, in inner
    return ctx.invoke(f, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sentry/runner/commands/django.py", line 14, in django
    execute_from_command_line(argv=[ctx.command_path] + list(management_args))
  File "/usr/local/lib/python3.11/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.11/site-packages/django/core/management/__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.11/site-packages/django/core/management/base.py", line 412, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.11/site-packages/django/core/management/base.py", line 458, in execute
    output = self.handle(*args, **options)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/core/management/base.py", line 106, in wrapper
    res = handle_func(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/core/management/commands/migrate.py", line 356, in handle
    post_migrate_state = executor.migrate(
                         ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/migrations/executor.py", line 135, in migrate
    state = self._migrate_all_forwards(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/migrations/executor.py", line 167, in _migrate_all_forwards
    state = self.apply_migration(
            ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sentry/new_migrations/monkey/executor.py", line 140, in apply_migration
    return super().apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/migrations/executor.py", line 252, in apply_migration
    state = migration.apply(state, schema_editor)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sentry/new_migrations/migrations.py", line 22, in apply
    return super().apply(project_state, schema_editor, collect_sql)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/migrations/migration.py", line 132, in apply
    operation.database_forwards(
  File "/usr/local/lib/python3.11/site-packages/django/db/migrations/operations/models.py", line 1042, in database_forwards
    raise ValueError(
ValueError: Found wrong number (2) of indexes for sentry_groupedmessage(project_id, id).

Product Area

Unknown

Link

No response

DSN

No response

Version

nightly

[getsantry]

Assigning to @getsentry/support for routing ⏲️

[Christophe31]

I got this issue with this event id
ee4e9e869ab6a19f443f9b338159211c

[SherinBloemendaal]

+1 got the same error. My sentry envelope id is: ee4e9e869ab6a19f443f9b338159211c

[codedmonkey]

Is it even possible to install Sentry at the moment?

Is there any workaround?

[Christophe31]

This is an index migration, it may impact performances but it looks like you can launch sentry without this migration as long as there are no structural migrations after.

Or even better, instead of running on master (nightly) branch, you may go to a tag and use a stable release: https://github.com/getsentry/self-hosted/tags

[hubertdeng123]

This seems to be due to this PR here:
getsentry/sentry#64158

In SaaS, it looks like this was due to some dangerous migrations not being performed. These can be performed manually and you can get these migrations with
git log -G thetablename -- src/sentry/migrations

[hubertdeng123]

Or, you can opt for a more hacky fix that I tried by going into your postgres database.

SELECT indexname, indexdef FROM pg_indexes WHERE tablename = 'sentry_groupedmessage';
DROP INDEX sentry_groupedmessage_project_id_id_515aaa7e_idx;
CREATE INDEX sentry_grou_project_41a5ce_idx ON public.sentry_groupedmessage USING btree (project_id, id);

[myonlylonely]

Or, you can opt for a more hacky fix that I tried by going into your postgres database.

SELECT indexname, indexdef FROM pg_indexes WHERE tablename = 'sentry_groupedmessage';
DROP INDEX sentry_groupedmessage_project_id_id_515aaa7e_idx;
CREATE INDEX sentry_grou_project_41a5ce_idx ON public.sentry_groupedmessage USING btree (project_id, id);

This hack does not work for me.

[codedmonkey]

@Christophe31 I feel so stupid, I did try to downgrade to a release tag but I forgot to update my .env.custom. Thank you!

[hubertdeng123]

Hey folks, we've recently upgraded our nightly image to django 5. Could you do us a favor and try upgrading again to see if you get the same error?

We think that this may be resolved as django 5 should pick up django/django@7045661

[tankerkiller125]

@hubertdeng123 Pulled the latest master branch details and ran the install.sh command, no luck, same error message.

[hubertdeng123]

Yeah, I would assume that sentry_groupedmessage_project_id_id_515aaa7e_uniq is the one to drop. @ZVilusinsky I apologize I may have misremembered the fix we included in the install script, perhaps you should not have dropped sentry_groupedmessage_project_id_515aaa7e_idx.

[JustDoItSascha]

Will there be a automatic fix in an upcoming version? I tried to upgrade and are now facing the same issue...

[black3r]

Faced the same issue here upgrading from 23.11.2 to 24.2.0 (also an archaic instance, was setup in August 2016 and slowly upgraded throughout since then).

The install script tried to fix the issue by deleting the sentry_groupedmessage_project_id_id_515aaa7e_uniq which was mentioned here, but it looked in the wrong database (postgres instead of sentry) which confused me as well so I had to figure out an alternate solution.

My alternate solution was faking the migration (INSERT INTO "django_migrations" (app, name, applied) VALUES ('sentry', '0640_index_together', current_timestamp);), running ./install.sh again and setting up the indices manually after the update successfully finished to match the table in #2758 (comment) (I assumed that's the current reference). Only at that point I realized what was wrong.

[mmorejon]

I have the same error migrating from the helm chart 20.12.2 -> 21.5.0. (24.1.2)

ValueError: Found wrong number (2) of indexes for sentry_groupedmessage(project_id, id).
  Applying sentry.0640_index_together...% 

Do you know how I can fix it?

[xvilo]

Running drop index sentry_groupedmessage_project_id_id_515aaa7e_uniq in the postgres DB did indeed resolve this for us

[dedronek]

The same error while migrating from the chart version 20.6.0 -> 21.6.1 (sentry 23.9.1 -> 24.2.0).

It's fixed by running drop index sentry_groupedmessage_project_id_id_515aaa7e_uniq on the postgres DB.

[hubertdeng123]

A general fix for this issue has been merged and now released, so I'm going to close this. For people who are still experiencing this issue, please attempt to try dropping this index in postgres sentry_groupedmessage_project_id_id_515aaa7e_uniq