Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

SinceDB should support file offset and not only the keys #54

Open
ph opened this issue Sep 16, 2015 · 0 comments
Open

SinceDB should support file offset and not only the keys #54

ph opened this issue Sep 16, 2015 · 0 comments
Labels
enhancement

Comments

@ph
Copy link
Contributor

ph commented Sep 16, 2015

The current sincedb implementation of this plugin only relies on the object key and doesn't use the file offset at all, so when we stop logstash in a middle of reading a file we don't have the choice to read the file back at the beginning causing duplicates in the log stream.

We should investigate if we could use the filewatch plugins to actually do the file reading and the S3 input could act as a downloading agent.
@ph ph mentioned this issue Sep 16, 2015
Closed
ph added a commit that referenced this issue Sep 22, 2015
@ph @jordansissel
Refactor the shutdown sequence of this plugin to follow the new semantic
221bb96 
This plugin is now interruptable, this plugin require a bit more care to
shutdown it down for a few reasons:

1. It can be stuck streaming file to disk from a S3 bucket.
2. It can be stuck reading a large file

For case 1, we cancel the streaming and delete the incomplete temp file
and not update the sincedb.

For case 2, we will cancel reading the file and not update the sincedb,
this will result in duplicates in the logging stream which is fine for now since the S3 plugin
doesn't support saving offset yet. see #54

This PR also add a fix to non deterministic test run in a local
environment by specifying a new sincedb for each run.

Fixes: #53 #42

Fixes #55
@eherot eherot mentioned this issue Nov 2, 2023
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@ph @kaisecheng