Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[libbeat] Disk queue implementation #21176

Merged
merged 97 commits into from
Sep 28, 2020
Merged

[libbeat] Disk queue implementation #21176

merged 97 commits into from
Sep 28, 2020

Commits on Apr 8, 2020

  1. Initial disk queue skeleton

    @faec
    faec committed Apr 8, 2020
    Configuration menu
    Copy the full SHA
    e76a41b View commit details
    Browse the repository at this point in the history

Commits on Apr 9, 2020

  1. Merge branch 'feature-disk-queue' into disk-queue

    @faec
    faec committed Apr 9, 2020
    Configuration menu
    Copy the full SHA
    1d8bf65 View commit details
    Browse the repository at this point in the history

Commits on Apr 10, 2020

  1. Sketching out some top-level disk queue data structures

    @faec
    faec committed Apr 10, 2020
    Configuration menu
    Copy the full SHA
    e8c8128 View commit details
    Browse the repository at this point in the history

Commits on Apr 16, 2020

  1. add queue type registration

    @faec
    faec committed Apr 16, 2020
    Configuration menu
    Copy the full SHA
    97a7ed5 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'feature-disk-queue' into disk-queue

    @faec
    faec committed Apr 16, 2020
    Configuration menu
    Copy the full SHA
    f5ad9a2 View commit details
    Browse the repository at this point in the history

  3. use new registry helper

    @faec
    faec committed Apr 16, 2020
    Configuration menu
    Copy the full SHA
    a78b85c View commit details
    Browse the repository at this point in the history

  4. connect external user config to the queue's Settings struct

    @faec
    faec committed Apr 16, 2020
    Configuration menu
    Copy the full SHA
    b12020c View commit details
    Browse the repository at this point in the history

  5. Fill out more default settings

    @faec
    faec committed Apr 16, 2020
    Configuration menu
    Copy the full SHA
    67540d6 View commit details
    Browse the repository at this point in the history

Commits on Apr 23, 2020

  1. review comments

    @faec
    faec committed Apr 23, 2020
    Configuration menu
    Copy the full SHA
    94f125c View commit details
    Browse the repository at this point in the history

  2. review comments (add panic to unimplemented functions)

    @faec
    faec committed Apr 23, 2020
    Configuration menu
    Copy the full SHA
    f29a96f View commit details
    Browse the repository at this point in the history

Commits on Apr 24, 2020

  1. Merge branch 'feature-disk-queue' into disk-queue

    @faec
    faec committed Apr 24, 2020
    Configuration menu
    Copy the full SHA
    7ce01f9 View commit details
    Browse the repository at this point in the history

Commits on May 27, 2020

  1. some state file stuff

    @faec
    faec committed May 27, 2020
    Configuration menu
    Copy the full SHA
    a04980e View commit details
    Browse the repository at this point in the history

Commits on Jun 1, 2020

  1. revising code to match new design

    @faec
    faec committed Jun 1, 2020
    Configuration menu
    Copy the full SHA
    26b4248 View commit details
    Browse the repository at this point in the history

  2. more state file handling

    @faec
    faec committed Jun 1, 2020
    Configuration menu
    Copy the full SHA
    f30f30b View commit details
    Browse the repository at this point in the history

Commits on Jun 2, 2020

  1. reading data frames from segments

    @faec
    faec committed Jun 2, 2020
    Configuration menu
    Copy the full SHA
    c312d69 View commit details
    Browse the repository at this point in the history

Commits on Jun 3, 2020

  1. fleshing out segment logic

    @faec
    faec committed Jun 3, 2020
    Configuration menu
    Copy the full SHA
    1a40b06 View commit details
    Browse the repository at this point in the history

Commits on Jun 9, 2020

  1. lots of partial work on reader and writer

    @faec
    faec committed Jun 9, 2020
    Configuration menu
    Copy the full SHA
    ce65718 View commit details
    Browse the repository at this point in the history

Commits on Jun 16, 2020

  1. reworking segments

    @faec
    faec committed Jun 16, 2020
    Configuration menu
    Copy the full SHA
    22ae148 View commit details
    Browse the repository at this point in the history

Commits on Jun 18, 2020

  1. reworking reader code

    @faec
    faec committed Jun 18, 2020
    Configuration menu
    Copy the full SHA
    3f5f8fe View commit details
    Browse the repository at this point in the history

Commits on Jul 17, 2020

  1. working on writer loop

    @faec
    faec committed Jul 17, 2020
    Configuration menu
    Copy the full SHA
    61fa5d7 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' into disk-queue-0

    @faec
    faec committed Jul 17, 2020
    Configuration menu
    Copy the full SHA
    0191dc6 View commit details
    Browse the repository at this point in the history

  3. fix most build errors

    @faec
    faec committed Jul 17, 2020
    Configuration menu
    Copy the full SHA
    3bf35ff View commit details
    Browse the repository at this point in the history

Commits on Jul 21, 2020

  1. checksumType -> ChecksumType

    @faec
    faec committed Jul 21, 2020
    Configuration menu
    Copy the full SHA
    50bd450 View commit details
    Browse the repository at this point in the history

Commits on Jul 30, 2020

  1. working on read / write loops

    @faec
    faec committed Jul 30, 2020
    Configuration menu
    Copy the full SHA
    04c9b60 View commit details
    Browse the repository at this point in the history

  2. replace filebeat with a queue wrapper for testing

    @faec
    faec committed Jul 30, 2020
    Configuration menu
    Copy the full SHA
    7a2e09a View commit details
    Browse the repository at this point in the history

Commits on Aug 12, 2020

  1. adapting encoder stuff from the disk spool

    @faec
    faec committed Aug 12, 2020
    Configuration menu
    Copy the full SHA
    132ba8e View commit details
    Browse the repository at this point in the history

  2. add most of the api logic for the reader / writer loops

    @faec
    faec committed Aug 12, 2020
    Configuration menu
    Copy the full SHA
    e73f55f View commit details
    Browse the repository at this point in the history

Commits on Aug 13, 2020

  1. filling in segment-deletion api

    @faec
    faec committed Aug 13, 2020
    Configuration menu
    Copy the full SHA
    6d2ca31 View commit details
    Browse the repository at this point in the history

  2. connect consumer ack endpoints

    @faec
    faec committed Aug 13, 2020
    Configuration menu
    Copy the full SHA
    988cef6 View commit details
    Browse the repository at this point in the history

  3. organize, delete dead code

    @faec
    faec committed Aug 13, 2020
    Configuration menu
    Copy the full SHA
    7774dc4 View commit details
    Browse the repository at this point in the history

  4. comment / delete more old code

    @faec
    faec committed Aug 13, 2020
    Configuration menu
    Copy the full SHA
    7146525 View commit details
    Browse the repository at this point in the history

  5. cleanup, plug in consumer acks

    @faec
    faec committed Aug 13, 2020
    Configuration menu
    Copy the full SHA
    1001565 View commit details
    Browse the repository at this point in the history

Commits on Aug 21, 2020

  1. finish reader loop api / logic

    @faec
    faec committed Aug 21, 2020
    Configuration menu
    Copy the full SHA
    bb56b8f View commit details
    Browse the repository at this point in the history

Commits on Sep 3, 2020

  1. make things build

    @faec
    faec committed Sep 3, 2020
    Configuration menu
    Copy the full SHA
    a99e869 View commit details
    Browse the repository at this point in the history

  2. clean up naming, plug in queue initialization, start fixing writer loop

    @faec
    faec committed Sep 3, 2020
    Configuration menu
    Copy the full SHA
    89da2b2 View commit details
    Browse the repository at this point in the history

Commits on Sep 4, 2020

  1. finish writer loop rework

    @faec
    faec committed Sep 4, 2020
    Configuration menu
    Copy the full SHA
    6fa9d33 View commit details
    Browse the repository at this point in the history

  2. plug in writing to actual files

    @faec
    faec committed Sep 4, 2020
    Configuration menu
    Copy the full SHA
    e77dfa6 View commit details
    Browse the repository at this point in the history

  3. make diskQueue.logger a top-level field

    @faec
    faec committed Sep 4, 2020
    Configuration menu
    Copy the full SHA
    d2a65dc View commit details
    Browse the repository at this point in the history

  4. move segment allocation to the producer handler, clean up naming

    @faec
    faec committed Sep 4, 2020
    Configuration menu
    Copy the full SHA
    b93fe83 View commit details
    Browse the repository at this point in the history

  5. properly handle writer loop responses

    @faec
    faec committed Sep 4, 2020
    Configuration menu
    Copy the full SHA
    3a3bfcd View commit details
    Browse the repository at this point in the history

  6. implement readSegmentHeader, more cleanup / naming changes

    @faec
    faec committed Sep 4, 2020
    Configuration menu
    Copy the full SHA
    cb1107d View commit details
    Browse the repository at this point in the history

Commits on Sep 8, 2020

  1. move queueSettings out of queueSegment, fix bugs

    @faec
    faec committed Sep 8, 2020
    Configuration menu
    Copy the full SHA
    3020962 View commit details
    Browse the repository at this point in the history

  2. delete old code, reorganize some structures

    @faec
    faec committed Sep 8, 2020
    Configuration menu
    Copy the full SHA
    df31c0e View commit details
    Browse the repository at this point in the history

  3. move frame-wrapping responsibility to the writer loop

    @faec
    faec committed Sep 8, 2020
    Configuration menu
    Copy the full SHA
    4a22ccd View commit details
    Browse the repository at this point in the history

  4. add event deserialization

    @faec
    faec committed Sep 8, 2020
    Configuration menu
    Copy the full SHA
    d4e1dcd View commit details
    Browse the repository at this point in the history

  5. properly assign frame ids after reading

    @faec
    faec committed Sep 8, 2020
    Configuration menu
    Copy the full SHA
    0e12495 View commit details
    Browse the repository at this point in the history

Commits on Sep 9, 2020

  1. remove obsolete code

    @faec
    faec committed Sep 9, 2020
    Configuration menu
    Copy the full SHA
    56fe5ba View commit details
    Browse the repository at this point in the history

  2. compute / verify real checksums

    @faec
    faec committed Sep 9, 2020
    Configuration menu
    Copy the full SHA
    eaf6e2b View commit details
    Browse the repository at this point in the history

  3. don't read everything twice

    @faec
    faec committed Sep 9, 2020
    Configuration menu
    Copy the full SHA
    5f0376d View commit details
    Browse the repository at this point in the history

  4. Fix ACKing / deletion

    @faec
    faec committed Sep 9, 2020
    Configuration menu
    Copy the full SHA
    d63cbf0 View commit details
    Browse the repository at this point in the history

  5. revert testing wrapper around filebeat

    @faec
    faec committed Sep 9, 2020
    Configuration menu
    Copy the full SHA
    98e61f0 View commit details
    Browse the repository at this point in the history

Commits on Sep 10, 2020

  1. filebeat ingestion is working!

    @faec
    faec committed Sep 10, 2020
    Configuration menu
    Copy the full SHA
    e48c815 View commit details
    Browse the repository at this point in the history

  2. clean up capacity checking / add maybeUnblockProducers

    @faec
    faec committed Sep 10, 2020
    Configuration menu
    Copy the full SHA
    1575640 View commit details
    Browse the repository at this point in the history

  3. move all dq.maybe... calls out of the handle... helpers

    @faec
    faec committed Sep 10, 2020
    Configuration menu
    Copy the full SHA
    d495aa8 View commit details
    Browse the repository at this point in the history

  4. simplify / clarify the writer loop response logic

    @faec
    faec committed Sep 10, 2020
    Configuration menu
    Copy the full SHA
    dd08e2c View commit details
    Browse the repository at this point in the history

  5. add retrying to file creation / writing

    @faec
    faec committed Sep 10, 2020
    Configuration menu
    Copy the full SHA
    8db0acf View commit details
    Browse the repository at this point in the history

  6. correct / clarify comments

    @faec
    faec committed Sep 10, 2020
    Configuration menu
    Copy the full SHA
    26d226d View commit details
    Browse the repository at this point in the history

  7. move segment read / write positions to the segments struct

    @faec
    faec committed Sep 10, 2020
    Configuration menu
    Copy the full SHA
    442a513 View commit details
    Browse the repository at this point in the history

  8. remove the separate 'core loop' struct and coalesce its helpers in di… 

    …skQueue
    @faec
    faec committed Sep 10, 2020
    Configuration menu
    Copy the full SHA
    1a1742e View commit details
    Browse the repository at this point in the history

Commits on Sep 14, 2020

  1. set output channel buffer to a more reasonable size

    @faec
    faec committed Sep 14, 2020
    Configuration menu
    Copy the full SHA
    5ca039d View commit details
    Browse the repository at this point in the history

  2. Expose some user configuration settings

    @faec
    faec committed Sep 14, 2020
    Configuration menu
    Copy the full SHA
    fdd4be6 View commit details
    Browse the repository at this point in the history

Commits on Sep 16, 2020

  1. send producer acks

    @faec
    faec committed Sep 16, 2020
    Configuration menu
    Copy the full SHA
    f7a446d View commit details
    Browse the repository at this point in the history

  2. remove checksum type as a configurable field

    @faec
    faec committed Sep 16, 2020
    Configuration menu
    Copy the full SHA
    79a81de View commit details
    Browse the repository at this point in the history

Commits on Sep 17, 2020

  1. Batch producer ACKs / fsync after writes

    @faec
    faec committed Sep 17, 2020
    Configuration menu
    Copy the full SHA
    f88481c View commit details
    Browse the repository at this point in the history

  2. remove queueSegment.header which is no longer needed

    @faec
    faec committed Sep 17, 2020
    Configuration menu
    Copy the full SHA
    8b61f5b View commit details
    Browse the repository at this point in the history

  3. reorganizing files a bit, starting the final pass of consumer acks

    @faec
    faec committed Sep 17, 2020
    Configuration menu
    Copy the full SHA
    8c32477 View commit details
    Browse the repository at this point in the history

  4. initialize the read position properly, continue consumer ack revision

    @faec
    faec committed Sep 17, 2020
    Configuration menu
    Copy the full SHA
    0a6f9f4 View commit details
    Browse the repository at this point in the history

Commits on Sep 18, 2020

  1. still rewriting consumer acks

    @faec
    faec committed Sep 18, 2020
    Configuration menu
    Copy the full SHA
    91bc3b2 View commit details
    Browse the repository at this point in the history

  2. plug consumer acks back in

    @faec
    faec committed Sep 18, 2020
    Configuration menu
    Copy the full SHA
    aea4cad View commit details
    Browse the repository at this point in the history

  3. ...increment the loop variable -.-

    @faec
    faec committed Sep 18, 2020
    Configuration menu
    Copy the full SHA
    a8ca56a View commit details
    Browse the repository at this point in the history

  4. cleanups

    @faec
    faec committed Sep 18, 2020
    Configuration menu
    Copy the full SHA
    6d91ab5 View commit details
    Browse the repository at this point in the history

Commits on Sep 21, 2020

  1. remove unused variables

    @faec
    faec committed Sep 21, 2020
    Configuration menu
    Copy the full SHA
    767aabb View commit details
    Browse the repository at this point in the history

  2. shutdown cleanly

    @faec
    faec committed Sep 21, 2020
    Configuration menu
    Copy the full SHA
    0bf668e View commit details
    Browse the repository at this point in the history

  3. delete the final segment on shutdown if everything is ACKed

    @faec
    faec committed Sep 21, 2020
    Configuration menu
    Copy the full SHA
    857077c View commit details
    Browse the repository at this point in the history

  4. turn on basic producer canceling

    @faec
    faec committed Sep 21, 2020
    Configuration menu
    Copy the full SHA
    d4cd0bb View commit details
    Browse the repository at this point in the history

Commits on Sep 22, 2020

  1. propagate event flags thru the queue

    @faec
    faec committed Sep 22, 2020
    Configuration menu
    Copy the full SHA
    91ce56f View commit details
    Browse the repository at this point in the history

  2. clean up state_file / remove unused code

    @faec
    faec committed Sep 22, 2020
    Configuration menu
    Copy the full SHA
    f9dbfb1 View commit details
    Browse the repository at this point in the history

  3. save queue position as events are ACKed

    @faec
    faec committed Sep 22, 2020
    Configuration menu
    Copy the full SHA
    93d3fff View commit details
    Browse the repository at this point in the history

  4. remove producer cancel boilerplate (not needed in v1)

    @faec
    faec committed Sep 22, 2020
    Configuration menu
    Copy the full SHA
    0ac493c View commit details
    Browse the repository at this point in the history

  5. various review comments

    @faec
    faec committed Sep 22, 2020
    Configuration menu
    Copy the full SHA
    3af1421 View commit details
    Browse the repository at this point in the history

  6. ...save everything before commit

    @faec
    faec committed Sep 22, 2020
    Configuration menu
    Copy the full SHA
    333cb76 View commit details
    Browse the repository at this point in the history

  7. clarify the errors that "can't" fail

    @faec
    faec committed Sep 22, 2020
    Configuration menu
    Copy the full SHA
    0dfffe1 View commit details
    Browse the repository at this point in the history

  8. be more consistent about unsignedness

    @faec
    faec committed Sep 22, 2020
    Configuration menu
    Copy the full SHA
    43c4aaa View commit details
    Browse the repository at this point in the history

  9. change the format of the deleter loop response

    @faec
    faec committed Sep 22, 2020
    Configuration menu
    Copy the full SHA
    0b33c4b View commit details
    Browse the repository at this point in the history

  10. make helper functions to initialize the helper loop structures

    @faec
    faec committed Sep 22, 2020
    Configuration menu
    Copy the full SHA
    7b2f4ba View commit details
    Browse the repository at this point in the history

  11. clarify comments

    @faec
    faec committed Sep 22, 2020
    Configuration menu
    Copy the full SHA
    a626cec View commit details
    Browse the repository at this point in the history

  12. minor tweaks

    @faec
    faec committed Sep 22, 2020
    Configuration menu
    Copy the full SHA
    53da09a View commit details
    Browse the repository at this point in the history

  13. remove unused event field in writeFrame

    @faec
    faec committed Sep 22, 2020
    Configuration menu
    Copy the full SHA
    565a99f View commit details
    Browse the repository at this point in the history

  14. ...save all files before commit...

    @faec
    faec committed Sep 22, 2020
    Configuration menu
    Copy the full SHA
    008fbe7 View commit details
    Browse the repository at this point in the history

  15. don't initialize waitgroup to a nil pointer

    @faec
    faec committed Sep 22, 2020
    Configuration menu
    Copy the full SHA
    b5ec926 View commit details
    Browse the repository at this point in the history

Commits on Sep 23, 2020

  1. error message cleanups

    @faec
    faec committed Sep 23, 2020
    Configuration menu
    Copy the full SHA
    11e55a8 View commit details
    Browse the repository at this point in the history

  2. simplify position write-to-disk

    @faec
    faec committed Sep 23, 2020
    Configuration menu
    Copy the full SHA
    ff04bb5 View commit details
    Browse the repository at this point in the history

Commits on Sep 24, 2020

  1. Merge branch 'master' into disk-queue-0

    @faec
    faec committed Sep 24, 2020
    Configuration menu
    Copy the full SHA
    cc49879 View commit details
    Browse the repository at this point in the history

  2. add some unit tests

    @faec
    faec committed Sep 24, 2020
    Configuration menu
    Copy the full SHA
    2baf1d4 View commit details
    Browse the repository at this point in the history

  3. more verbose config errors

    @faec
    faec committed Sep 24, 2020
    Configuration menu
    Copy the full SHA
    30cf56c View commit details
    Browse the repository at this point in the history

  4. Merge branch 'master' into disk-queue-0

    @faec
    faec committed Sep 24, 2020
    Configuration menu
    Copy the full SHA
    c619d6a View commit details
    Browse the repository at this point in the history

  5. change queue acks back to a pointer since it can't be used w/o initia… 

    …lization
    @faec
    faec committed Sep 24, 2020
    Configuration menu
    Copy the full SHA
    5b41351 View commit details
    Browse the repository at this point in the history