Post Process Forwarder - KafkaError "Offset Out of Range"

[DandyDeveloper]

After a while of running, my post process forwarder appear to be having issues with communication to Kafka:

Traceback (most recent call last):
  File "/usr/local/bin/sentry", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/site-packages/sentry/runner/__init__.py", line 164, in main
    cli(prog_name=get_prog(), obj={}, max_content_width=100)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/sentry/runner/decorators.py", line 68, in inner
    return ctx.invoke(f, *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/sentry/runner/decorators.py", line 30, in inner
    return ctx.invoke(f, *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/sentry/runner/commands/run.py", line 318, in post_process_forwarder
    initial_offset_reset=options["initial_offset_reset"],
  File "/usr/local/lib/python2.7/site-packages/sentry/utils/services.py", line 104, in <lambda>
    context[key] = (lambda f: lambda *a, **k: getattr(self, f)(*a, **k))(key)
  File "/usr/local/lib/python2.7/site-packages/sentry/eventstream/kafka/backend.py", line 184, in run_post_process_forwarder
    raise Exception(error)
Exception: KafkaError{code=OFFSET_OUT_OF_RANGE,val=1,str="Broker: Offset out of range"}

Oddly, my Snuba consumer is working fine and Sentry appears to be running as expected and I see the Snuba consumer running in the Kafka logs.

Can someone give me some suggestions on what to look for to figure out what the post-processor is looking for?

[BYK]

Can you try modifying this line to:

command: run post-process-forwarder --commit-batch-size 1 --initial-offset-reset earliest

and see if it helps?

[DandyDeveloper]

@BYK Thank you, I'll have a try on the manifest and update it.

Just an FYI I did fix this by effectively destroying the volume for the Kafka backend and recreate it.

I also noticed my volume size is pretty low for the amount of tasks we have coming in. Does Kafka have an inherent GC that cleans things up? It looks as though the disk was filling upwards of 8GB.

[DandyDeveloper]

@BYK It came back, looks like with both earliest and latest offsets I get the same error. This is POST fixing things, so it's unexpectedly broken itself again

Trying to think of what I can do to fix it.

[ProxiBlue]

Hello,
I am getting same issue, and it eventually kills sentry, it simply stops logging any issues.
Been going on for some time
I can give you access to a Digital Ocean Droplet where this is happening, if that will help.
I upped the instance to 8gb memory, to test if resources is issue.

https://sentry.uptactics.com/share/issue/2a83806a230e4c4196f86c83890ad62e/

[ProxiBlue]

I upped the instance to 8gb memory, to test if resources is issue.

I think that helped. Today (since 7h ago) the error stopped.

The readme/docs note at least 2GB memory. Seems I needed 8GB to stop the error. Possible resource related?

[joscha-alisch]

Thanks @DandyDeveloper for outlining your solution path. We had the same issue and were also able to resolve it for now by destroying the persistent volume of Kafka. Until now, we didn't have the issue occurring again like it did in your case.

[rmisyurev]

I have a solution:

1. Log on to Kafka container
   docker exec -it onpremise_kafka_1 bash
2. Receive consumers list
   kafka-consumer-groups --bootstrap-server 127.0.0.1:9092 --list
3. Get group info
   kafka-consumer-groups --bootstrap-server 127.0.0.1:9092 --group snuba-post-processor -describe
4. Watching what gonna happend with offset by using dry-run (optional)
   kafka-consumer-groups --bootstrap-server 127.0.0.1:9092 --group snuba-post-processor --topic events --reset-offsets --to-latest
5. Set offset to latest and execute
   kafka-consumer-groups --bootstrap-server 127.0.0.1:9092 --group snuba-post-processor --topic events --reset-offsets --to-latest --execute
6. Profit

This solution work the same way for other Snuba components (just replace group and\or topic)

[flokoe]

@rmisyurev This actually worked for me, thank you very much! But to be honest I don't know why.

The first time I got an error message like this:

Error: Assignments can only be reset if the group 'snuba-post-processor' is inactive, but the current state is Stable.

I just stopped all containers, then only started zookeeper and kafka containers, executed the statement above and stopped both containers again.
After that I started all containers and everything was fine.

I am sure it is not the right solution, so be careful, but for me it worked.

[BYK]

@rmisyurev - thanks for sharing the steps that helped you! I'll make sure to add these to some future documentation.