Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Filebeat] Add option for S3 input to work without SQS notification #27332

Merged
merged 21 commits into from
Aug 17, 2021
Merged

[Filebeat] Add option for S3 input to work without SQS notification #27332

merged 21 commits into from
Aug 17, 2021

Conversation

aspacca
Copy link

@aspacca aspacca commented Aug 12, 2021

What does this PR do?

Introduces an alternative polling method to ingest logs in S3 buckets through buckets listing instead of SQS notifications

Why is it important?

Some users are prevented to attach SQS notifications to S3 buckets and this enhancement will allow them to use Filebeat to ingest logs from the buckets anyway

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

  • [ ]

How to test this PR locally

  • Create an S3 bucket and uploads logs file for a fileset of your choice, mixing uncompressed format and tar.gz format
  • In the fileset configuration use the following config:
    enabled: true
    input.type: aws-s3
    input.bucket_arn: 'aws:s3:::created-s3-bucket'
    input.number_of_workers: 2
    input.bucket_list_interval: 5s
  • Run Filebeat with full debug messages (-d *)
  • Assure that events from the logs file are published on the target elasticsearch instance
  • Keep Filebeat running and check in the debug logs an entry like:
2021-08-18T10:16:37.654+0200    DEBUG   [input.aws-s3.s3_poller]        awss3/s3.go:177 skipping state. {"id": "4F67B762988BFE04", "bucket_arn": "aws:s3:::created-s3-bucket", "state": {"id":"created-s3-bucketfilename.ext\"etag\"2021-08-04 11:06:25 +0000 UTC","bucket":"created-s3-bucket","key":"filename.ext","etag":"\"etag\"","last_modified":"2021-08-04T11:06:25Z","stored":false,"error":false}}
  • Add a new log file in the bucket and check that events from it are published on the target elasticsearch instance
  • Stop Filebeat and restart
  • Assure that the events from the logs are not sent again
  • Check for debug entries like:
2021-08-18T10:16:37.654+0200    DEBUG   [input.aws-s3.s3_poller]        awss3/s3.go:177 skipping state. {"id": "4F67B762988BFE04", "bucket_arn": "aws:s3:::created-s3-bucket", "state": {"id":"created-s3-bucketfilename.ext\"etag\"2021-08-04 11:06:25 +0000 UTC","bucket":"created-s3-bucket","key":"filename.ext","etag":"\"etag\"","last_modified":"2021-08-04T11:06:25Z","stored":false,"error":false}}

Related issues

Closes #18205

Use cases

Screenshots

Logs

@aspacca aspacca added enhancement Team:Integrations Label for the Integrations team labels Aug 12, 2021
@elasticmachine
Copy link
Collaborator

Pinging @elastic/integrations (Team:Integrations)

@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Aug 12, 2021
@aspacca aspacca changed the title changelog [Filebeat] Add option for S3 input to work without SQS notification Aug 12, 2021
@elasticmachine
Copy link
Collaborator

elasticmachine commented Aug 12, 2021

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2021-08-16T18:59:28.072+0000

  • Duration: 158 min 55 sec

  • Commit: 4087528

Test stats 🧪

Test Results
Failed 0
Passed 53349
Skipped 5320
Total 58669

Trends 🧪

Image of Build Times

Image of Tests

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test Results
Failed 0
Passed 53349
Skipped 5320
Total 58669

@aspacca
Copy link
Author

aspacca commented Aug 13, 2021

=== RUN   TestBenchmarkInputS3
+-------------------+------------------------+---------------------------+-----------------------+--------------------+------------------+--------------+------+
| NUMBER OF WORKERS | OBJECTS LISTED PER SEC | OBJECTS PROCESSED PER SEC | OBJECTS ACKED PER SEC |   EVENTS PER SEC   | S3 BYTES PER SEC |  TIME (SEC)  | CPUS |
+-------------------+------------------------+---------------------------+-----------------------+--------------------+------------------+--------------+------+
|                 1 |     1480.0946260194296 |         740.0473130097148 |     740.0473130097148 |  46622.98071961203 | 1.6 MB           | 13.512649562 |   16 |
|                 2 |     2542.4276952352016 |        1271.2138476176008 |    1271.2138476176008 |  80086.47239990885 | 2.7 MB           |  7.866497064 |   16 |
|                 4 |     3830.9806879633325 |        1915.4903439816662 |    1915.4903439816662 | 120675.89167084497 | 4.1 MB           |  5.220595359 |   16 |
|                 8 |      5156.841457565499 |        2578.4207287827494 |    2578.4207287827494 | 162440.50591331322 | 5.6 MB           |  3.878343006 |   16 |
|                16 |      5740.807503567915 |        2870.4037517839574 |    2870.4037517839574 | 180835.43636238933 | 6.2 MB           |  3.483830452 |   16 |
|                32 |      5795.253020568674 |         2897.626510284337 |     2897.626510284337 | 182550.47014791323 | 6.3 MB           |  3.451100397 |   16 |
|                64 |       6076.29982837242 |          3038.14991418621 |      3038.14991418621 | 191403.44459373123 | 6.6 MB           |  3.291476814 |   16 |
|               128 |      6329.421405740139 |        3164.7107028700693 |    3164.7107028700693 | 199376.77428081437 | 6.8 MB           |  3.159846488 |   16 |
|               256 |      6622.546088022572 |         3311.273044011286 |     3311.273044011286 |   208610.201772711 | 7.2 MB           |  3.019986533 |   16 |
|               512 |      6632.511804289157 |        3316.2559021445786 |    3316.2559021445786 | 208924.12183510847 | 7.2 MB           |  3.015448836 |   16 |
|              1024 |     6876.8124812793185 |        3438.4062406396592 |    3438.4062406396592 |  216619.5931602985 | 7.4 MB           |  2.908324177 |   16 |
+-------------------+------------------------+---------------------------+-----------------------+--------------------+------------------+--------------+------+

@elasticmachine
Copy link
Collaborator

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

Copy link
Member

@andrewkroh andrewkroh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. My comments are all minor things and a few questions.

For other reviewers - please look over states.go. I didn't spend much time on that.

filebeat/docs/modules/aws.asciidoc Outdated Show resolved Hide resolved
x-pack/filebeat/input/awss3/_meta/terraform/README.md Outdated Show resolved Hide resolved
@@ -28,8 +28,8 @@ func newEventACKTracker(ctx context.Context) *eventACKTracker {
return &eventACKTracker{ctx: ctx, cancel: cancel}
}

// Add increments the number of pending ACKs by the specified amount.
func (a *eventACKTracker) Add(messageCount int64) {
// Add increments the number of pending ACKs
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// Add increments the number of pending ACKs
// Add increments the number of pending ACKs.

QueueURL string `config:"queue_url"`
S3Bucket string `config:"s3_bucket"`
S3BucketPollInterval time.Duration `config:"s3_bucket_poll_interval"`
S3BucketNumberOfWorkers int `config:"s3_bucket_number_of_workers"`
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm thinking about the config names. How about s3.bucket and s3.bucket_list_interval (I think "list" might be more descriptive than "poll")?

On the s3_bucket_number_of_workers, thinking about the future, perhaps we should call this number_of_workers. Then if we do make the changes you mentioned to SQS we could use this option for both.

x-pack/filebeat/input/awss3/config.go Outdated Show resolved Hide resolved
Bucket string `json:"bucket" struct:"bucket"`
Key string `json:"key" struct:"key"`
Etag string `json:"etag" struct:"etag"`
LastModified time.Time `json:"last_modified" struct:"last_modifed"`
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
LastModified time.Time `json:"last_modified" struct:"last_modifed"`
LastModified time.Time `json:"last_modified" struct:"last_modified"`


previousState := s.FindPrevious(state)

// status is forget. if there is no previous state and
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// status is forget. if there is no previous state and
// status is forgotten. if there is no previous state and

// No existing state found, add new one
s.idx[id] = len(s.states)
s.states = append(s.states, newState)
logp.Debug("input", "New state added for %s", newState.Id)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This function is deprecated. Prefer creating a logger per instance, but if you must then use lopg.L().Debugf.

return true, nil
}

// try to decode. Ingore faulty/incompatible values.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// try to decode. Ingore faulty/incompatible values.
// try to decode. Ignore faulty/incompatible values.

x-pack/filebeat/input/awss3/state.go Show resolved Hide resolved
@aspacca
Copy link
Author

aspacca commented Aug 16, 2021

@andrewkroh
I've found what seems to be a caveat with the eventACKTracker:

func TestEventACKHandlerWait(t *testing.T) {
	ctx, cancel := context.WithCancel(context.Background())
	t.Cleanup(cancel)

	// Create acker. Add one pending ACK.
	acker := newEventACKTracker(ctx)
	acker.Add()
	acker.ACK()
	acker.Wait()
	acker.Add()

	assert.EqualValues(t, 1, acker.pendingACKs)
	assert.ErrorIs(t, acker.ctx.Err(), context.Canceled)
}

As soon there's an even number of Add and ACK the pendingACKs property goes to zero and the context is canceled, therefore Wait will return.

Similarly to what I reported I the other PR: we should clarify that Add/Wait should be called sequentially, with the latter only when every expected Add are already called

That's what we do for both input method so it's not a problem.

@andrewkroh
Copy link
Member

The godocs should be clarified that the eventACKTracker cannot be reused to wait for multiple independent sets of ACKs.

But we could also change the implementation to allow for reuse if this is what you need. Do you need that?

@aspacca
Copy link
Author

aspacca commented Aug 16, 2021

The godocs should be clarified that the eventACKTracker cannot be reused to wait for multiple independent sets of ACKs.

https://github.com/elastic/beats/pull/27332/files#diff-86ce06343e1fd48ccaa25dad04bb1e799646e4d7326d48e2a27058e38915bd89R54-R57

But we could also change the implementation to allow for reuse if this is what you need. Do you need that?

I don't need it, just wanted to be sure I didn't overlook the limitation

@aspacca
Copy link
Author

aspacca commented Aug 16, 2021

/test

@aspacca aspacca added the backport-v7.15.0 Automated backport with mergify label Aug 16, 2021
@aspacca
Copy link
Author

aspacca commented Aug 16, 2021

@andrewkroh I should have addressed all the comments, please check, thanks

Copy link
Member

@andrewkroh andrewkroh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM


Interval between list requests to the S3 bucket. Default to be 120 seconds.
Wait interval between completion of a list request to the S3 bucket and beginning of the nest one. Default to be 120 seconds.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Wait interval between completion of a list request to the S3 bucket and beginning of the nest one. Default to be 120 seconds.
Wait interval between completion of a list request to the S3 bucket and beginning of the next one. Default to be 120 seconds.

s3ObjectsListedTotal *monitoring.Uint // Number of S3 objects listed.
s3ObjectsProcessedTotal *monitoring.Uint // Number of S3 objects processed.
// s3ObjectsListedTotal is the number of S3 objects processed that were fully ACKed.
s3ObjectsAckedTotal *monitoring.Uint // Number of S3 objects fully ACKed.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This one has two godocs.

Suggested change
s3ObjectsAckedTotal *monitoring.Uint // Number of S3 objects fully ACKed.
s3ObjectsAckedTotal *monitoring.Uint

@@ -44,6 +49,9 @@ Example config:
cloudtrail:
enabled: false
#var.queue_url: https://sqs.myregion.amazonaws.com/123456/myqueue
#var.bucket: 'arn:aws:s3:::mybucket
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
#var.bucket: 'arn:aws:s3:::mybucket
#var.bucket: 'arn:aws:s3:::mybucket'

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and same for the other sections below.

Copy link
Contributor

@kaiyan-sheng kaiyan-sheng left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Besides the missing quote, everything else looks great. Thanks!

@aspacca
Copy link
Author

aspacca commented Aug 16, 2021

/test

@aspacca aspacca merged commit 928f9c5 into elastic:master Aug 17, 2021
mergify bot pushed a commit that referenced this pull request Aug 17, 2021
…27332)

* [Filebeat] Add option for S3 input to work without SQS notification

(cherry picked from commit 928f9c5)
aspacca pushed a commit that referenced this pull request Aug 17, 2021
…27332) (#27414)

* [Filebeat] Add option for S3 input to work without SQS notification

(cherry picked from commit 928f9c5)

Co-authored-by: Andrea Spacca <andrea.spacca@elastic.co>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport-v7.15.0 Automated backport with mergify enhancement Filebeat Filebeat review Team:Integrations Label for the Integrations team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Filebeat] Add option for S3 input to work without SQS notification
4 participants