Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Regression: Consume stream doesn't return after restart #1814

Closed
sehz opened this issue Oct 27, 2021 · 1 comment
Closed

Regression: Consume stream doesn't return after restart #1814

sehz opened this issue Oct 27, 2021 · 1 comment
Assignees
Labels
bug Something isn't working SPU SPU related Stability/Scalability
Milestone

Comments

@sehz
Copy link
Contributor

sehz commented Oct 27, 2021

Steps to reproduce:

  1. Create a new local cluster
  2. Create topic and produce test data
  3. Kill SPU: pkill -f "fluvio-run spu -i 5001"
  4. Restart SPU: ``fluvio run spu -i 5001 -p 0.0.0.0:9010 -v 0.0.0.0:9011 --log-base-dir /home/ubuntu/.fluvio/data`

fluvio consume <topic> -B doesn't return data.

Seems to be cause by: #1726

@sehz sehz self-assigned this Oct 27, 2021
@sehz sehz added bug Something isn't working Priority/blocker SPU SPU related Stability/Scalability labels Oct 27, 2021
@sehz sehz added this to the 0.9.12 milestone Oct 27, 2021
@sehz
Copy link
Contributor Author

sehz commented Oct 27, 2021

With working SPU, retrieving partition looks like this:

Oct 27 04:07:37.523 DEBUG fetch{stream_id=2 replica=test-0 sink=828}:send_back_records{starting_offset=0 stream_id=2}:read_records:read_partition_slice:read_records{start_offset=0 max_offset=None max_len=1000000}: fluvio_storage::replica: starting read records hw=1000 leo=1000
Oct 27 04:07:37.523 DEBUG fetch{stream_id=2 replica=test-0 sink=828}:send_back_records{starting_offset=0 stream_id=2}:read_records:read_partition_slice:read_records{start_offset=0 max_offset=None max_len=1000000}:find_segment{offset=0}: fluvio_storage::replica: is in active segment active_base_offset=0
Oct 27 04:07:37.523 DEBUG fetch{stream_id=2 replica=test-0 sink=828}:send_back_records{starting_offset=0 stream_id=2}:read_records:read_partition_slice:read_records{start_offset=0 max_offset=None max_len=1000000}: fluvio_storage::replica: active segment with base offset: 0 found for offset: 0
Oct 27 04:07:37.524 DEBUG fetch{stream_id=2 replica=test-0 sink=828}:send_back_records{starting_offset=0 stream_id=2}:read_records:read_partition_slice:read_records{start_offset=0 max_offset=None max_len=1000000}: fluvio_storage::replica: retrieved record slice fd: 825, position: 0, len: 191890

but with restarted SPU, retrieved slice returns 0

Oct 27 03:53:49.064 DEBUG fetch{stream_id=1 replica=test-0 sink=810}: fluvio_spu::services::public::stream_fetch: stream fetch max_bytes=1000000 max_fetch_bytes=1000000 isolation=ReadUncommitted stream_id=1 sink=810 starting_offset=0
Oct 27 03:53:49.064 DEBUG fetch{stream_id=1 replica=test-0 sink=810}:send_back_records{starting_offset=0 stream_id=1}:read_records:read_partition_slice:read_records{start_offset=0 max_offset=None max_len=1000000}: fluvio_storage::replica: starting read records hw=1000 leo=1000
Oct 27 03:53:49.064 DEBUG fetch{stream_id=1 replica=test-0 sink=810}:send_back_records{starting_offset=0 stream_id=1}:read_records:read_partition_slice:read_records{start_offset=0 max_offset=None max_len=1000000}:find_segment{offset=0}: fluvio_storage::replica: is in active segment active_base_offset=0
Oct 27 03:53:49.064 DEBUG fetch{stream_id=1 replica=test-0 sink=810}:send_back_records{starting_offset=0 stream_id=1}:read_records:read_partition_slice:read_records{start_offset=0 max_offset=None max_len=1000000}: fluvio_storage::replica: active segment with base offset: 0 found for offset: 0
Oct 27 03:53:49.064 DEBUG fetch{stream_id=1 replica=test-0 sink=810}:send_back_records{starting_offset=0 stream_id=1}:read_records:read_partition_slice:read_records{start_offset=0 max_offset=None max_len=1000000}: fluvio_storage::replica: retrieved record slice fd: 807, position: 0, len: 0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working SPU SPU related Stability/Scalability
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant