[Filebeat] Add option for S3 input to work without SQS notification #27332

aspacca · 2021-08-12T08:34:55Z

What does this PR do?

Introduces an alternative polling method to ingest logs in S3 buckets through buckets listing instead of SQS notifications

Why is it important?

Some users are prevented to attach SQS notifications to S3 buckets and this enhancement will allow them to use Filebeat to ingest logs from the buckets anyway

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

[ ]

How to test this PR locally

Create an S3 bucket and uploads logs file for a fileset of your choice, mixing uncompressed format and tar.gz format
In the fileset configuration use the following config:

    enabled: true
    input.type: aws-s3
    input.bucket_arn: 'aws:s3:::created-s3-bucket'
    input.number_of_workers: 2
    input.bucket_list_interval: 5s

Run Filebeat with full debug messages (-d *)
Assure that events from the logs file are published on the target elasticsearch instance
Keep Filebeat running and check in the debug logs an entry like:

2021-08-18T10:16:37.654+0200    DEBUG   [input.aws-s3.s3_poller]        awss3/s3.go:177 skipping state. {"id": "4F67B762988BFE04", "bucket_arn": "aws:s3:::created-s3-bucket", "state": {"id":"created-s3-bucketfilename.ext\"etag\"2021-08-04 11:06:25 +0000 UTC","bucket":"created-s3-bucket","key":"filename.ext","etag":"\"etag\"","last_modified":"2021-08-04T11:06:25Z","stored":false,"error":false}}

Add a new log file in the bucket and check that events from it are published on the target elasticsearch instance
Stop Filebeat and restart
Assure that the events from the logs are not sent again
Check for debug entries like:

2021-08-18T10:16:37.654+0200    DEBUG   [input.aws-s3.s3_poller]        awss3/s3.go:177 skipping state. {"id": "4F67B762988BFE04", "bucket_arn": "aws:s3:::created-s3-bucket", "state": {"id":"created-s3-bucketfilename.ext\"etag\"2021-08-04 11:06:25 +0000 UTC","bucket":"created-s3-bucket","key":"filename.ext","etag":"\"etag\"","last_modified":"2021-08-04T11:06:25Z","stored":false,"error":false}}

Related issues

Closes #18205

Use cases

Screenshots

Logs

elasticmachine · 2021-08-12T08:34:57Z

Pinging @elastic/integrations (Team:Integrations)

elasticmachine · 2021-08-12T08:57:05Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2021-08-16T18:59:28.072+0000
Duration: 158 min 55 sec
Commit: 4087528

Test stats 🧪

Test	Results
Failed	0
Passed	53349
Skipped	5320
Total	58669

Trends 🧪

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	53349
Skipped	5320
Total	58669

aspacca · 2021-08-13T09:01:27Z

=== RUN   TestBenchmarkInputS3
+-------------------+------------------------+---------------------------+-----------------------+--------------------+------------------+--------------+------+
| NUMBER OF WORKERS | OBJECTS LISTED PER SEC | OBJECTS PROCESSED PER SEC | OBJECTS ACKED PER SEC |   EVENTS PER SEC   | S3 BYTES PER SEC |  TIME (SEC)  | CPUS |
+-------------------+------------------------+---------------------------+-----------------------+--------------------+------------------+--------------+------+
|                 1 |     1480.0946260194296 |         740.0473130097148 |     740.0473130097148 |  46622.98071961203 | 1.6 MB           | 13.512649562 |   16 |
|                 2 |     2542.4276952352016 |        1271.2138476176008 |    1271.2138476176008 |  80086.47239990885 | 2.7 MB           |  7.866497064 |   16 |
|                 4 |     3830.9806879633325 |        1915.4903439816662 |    1915.4903439816662 | 120675.89167084497 | 4.1 MB           |  5.220595359 |   16 |
|                 8 |      5156.841457565499 |        2578.4207287827494 |    2578.4207287827494 | 162440.50591331322 | 5.6 MB           |  3.878343006 |   16 |
|                16 |      5740.807503567915 |        2870.4037517839574 |    2870.4037517839574 | 180835.43636238933 | 6.2 MB           |  3.483830452 |   16 |
|                32 |      5795.253020568674 |         2897.626510284337 |     2897.626510284337 | 182550.47014791323 | 6.3 MB           |  3.451100397 |   16 |
|                64 |       6076.29982837242 |          3038.14991418621 |      3038.14991418621 | 191403.44459373123 | 6.6 MB           |  3.291476814 |   16 |
|               128 |      6329.421405740139 |        3164.7107028700693 |    3164.7107028700693 | 199376.77428081437 | 6.8 MB           |  3.159846488 |   16 |
|               256 |      6622.546088022572 |         3311.273044011286 |     3311.273044011286 |   208610.201772711 | 7.2 MB           |  3.019986533 |   16 |
|               512 |      6632.511804289157 |        3316.2559021445786 |    3316.2559021445786 | 208924.12183510847 | 7.2 MB           |  3.015448836 |   16 |
|              1024 |     6876.8124812793185 |        3438.4062406396592 |    3438.4062406396592 |  216619.5931602985 | 7.4 MB           |  2.908324177 |   16 |
+-------------------+------------------------+---------------------------+-----------------------+--------------------+------------------+--------------+------+

elasticmachine · 2021-08-13T20:06:32Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

andrewkroh

Looks good. My comments are all minor things and a few questions.

For other reviewers - please look over states.go. I didn't spend much time on that.

filebeat/docs/modules/aws.asciidoc

x-pack/filebeat/input/awss3/_meta/terraform/README.md

andrewkroh · 2021-08-13T19:42:42Z

x-pack/filebeat/input/awss3/acker.go

@@ -28,8 +28,8 @@ func newEventACKTracker(ctx context.Context) *eventACKTracker {
 	return &eventACKTracker{ctx: ctx, cancel: cancel}
 }

-// Add increments the number of pending ACKs by the specified amount.
-func (a *eventACKTracker) Add(messageCount int64) {
+// Add increments the number of pending ACKs


Suggested change

// Add increments the number of pending ACKs

// Add increments the number of pending ACKs.

andrewkroh · 2021-08-13T19:48:57Z

x-pack/filebeat/input/awss3/config.go

+	QueueURL                string               `config:"queue_url"`
+	S3Bucket                string               `config:"s3_bucket"`
+	S3BucketPollInterval    time.Duration        `config:"s3_bucket_poll_interval"`
+	S3BucketNumberOfWorkers int                  `config:"s3_bucket_number_of_workers"`


I'm thinking about the config names. How about s3.bucket and s3.bucket_list_interval (I think "list" might be more descriptive than "poll")?

On the s3_bucket_number_of_workers, thinking about the future, perhaps we should call this number_of_workers. Then if we do make the changes you mentioned to SQS we could use this option for both.

x-pack/filebeat/input/awss3/config.go

andrewkroh · 2021-08-15T20:56:16Z

x-pack/filebeat/input/awss3/state.go

+	Bucket       string    `json:"bucket" struct:"bucket"`
+	Key          string    `json:"key" struct:"key"`
+	Etag         string    `json:"etag" struct:"etag"`
+	LastModified time.Time `json:"last_modified" struct:"last_modifed"`


Suggested change

LastModified time.Time `json:"last_modified" struct:"last_modifed"`

LastModified time.Time `json:"last_modified" struct:"last_modified"`

andrewkroh · 2021-08-15T20:57:42Z

x-pack/filebeat/input/awss3/states.go

+
+	previousState := s.FindPrevious(state)
+
+	// status is forget. if there is no previous state and


Suggested change

// status is forget. if there is no previous state and

// status is forgotten. if there is no previous state and

andrewkroh · 2021-08-15T20:58:55Z

x-pack/filebeat/input/awss3/states.go

+		// No existing state found, add new one
+		s.idx[id] = len(s.states)
+		s.states = append(s.states, newState)
+		logp.Debug("input", "New state added for %s", newState.Id)


This function is deprecated. Prefer creating a logger per instance, but if you must then use lopg.L().Debugf.

andrewkroh · 2021-08-15T21:00:07Z

x-pack/filebeat/input/awss3/states.go

+			return true, nil
+		}
+
+		// try to decode. Ingore faulty/incompatible values.


Suggested change

// try to decode. Ingore faulty/incompatible values.

// try to decode. Ignore faulty/incompatible values.

x-pack/filebeat/input/awss3/state.go

aspacca · 2021-08-16T12:34:08Z

@andrewkroh
I've found what seems to be a caveat with the eventACKTracker:

func TestEventACKHandlerWait(t *testing.T) {
	ctx, cancel := context.WithCancel(context.Background())
	t.Cleanup(cancel)

	// Create acker. Add one pending ACK.
	acker := newEventACKTracker(ctx)
	acker.Add()
	acker.ACK()
	acker.Wait()
	acker.Add()

	assert.EqualValues(t, 1, acker.pendingACKs)
	assert.ErrorIs(t, acker.ctx.Err(), context.Canceled)
}

As soon there's an even number of Add and ACK the pendingACKs property goes to zero and the context is canceled, therefore Wait will return.

Similarly to what I reported I the other PR: we should clarify that Add/Wait should be called sequentially, with the latter only when every expected Add are already called

That's what we do for both input method so it's not a problem.

andrewkroh · 2021-08-16T12:50:06Z

The godocs should be clarified that the eventACKTracker cannot be reused to wait for multiple independent sets of ACKs.

But we could also change the implementation to allow for reuse if this is what you need. Do you need that?

aspacca · 2021-08-16T13:05:05Z

The godocs should be clarified that the eventACKTracker cannot be reused to wait for multiple independent sets of ACKs.

https://github.com/elastic/beats/pull/27332/files#diff-86ce06343e1fd48ccaa25dad04bb1e799646e4d7326d48e2a27058e38915bd89R54-R57

But we could also change the implementation to allow for reuse if this is what you need. Do you need that?

I don't need it, just wanted to be sure I didn't overlook the limitation

aspacca · 2021-08-16T13:05:51Z

/test

…ithout-SQS-notification-refactored

aspacca · 2021-08-16T16:07:53Z

@andrewkroh I should have addressed all the comments, please check, thanks

andrewkroh

LGTM

andrewkroh · 2021-08-16T16:11:36Z

filebeat/docs/modules/aws.asciidoc


-Interval between list requests to the S3 bucket. Default to be 120 seconds.
+Wait interval between completion of a list request to the S3 bucket and beginning of the nest one. Default to be 120 seconds.


Suggested change

Wait interval between completion of a list request to the S3 bucket and beginning of the nest one. Default to be 120 seconds.

Wait interval between completion of a list request to the S3 bucket and beginning of the next one. Default to be 120 seconds.

andrewkroh · 2021-08-16T16:14:51Z

x-pack/filebeat/input/awss3/metrics.go

-	s3ObjectsListedTotal    *monitoring.Uint // Number of S3 objects listed.
-	s3ObjectsProcessedTotal *monitoring.Uint // Number of S3 objects processed.
+	// s3ObjectsListedTotal is the number of S3 objects processed that were fully ACKed.
+	s3ObjectsAckedTotal *monitoring.Uint // Number of S3 objects fully ACKed.


This one has two godocs.

Suggested change

s3ObjectsAckedTotal *monitoring.Uint // Number of S3 objects fully ACKed.

s3ObjectsAckedTotal *monitoring.Uint

kaiyan-sheng · 2021-08-16T16:43:41Z

filebeat/docs/modules/aws.asciidoc

@@ -44,6 +49,9 @@ Example config:
  cloudtrail:
    enabled: false
    #var.queue_url: https://sqs.myregion.amazonaws.com/123456/myqueue
+    #var.bucket: 'arn:aws:s3:::mybucket


Suggested change

#var.bucket: 'arn:aws:s3:::mybucket

#var.bucket: 'arn:aws:s3:::mybucket'

and same for the other sections below.

kaiyan-sheng

Besides the missing quote, everything else looks great. Thanks!

aspacca · 2021-08-16T18:58:56Z

/test

…27332) * [Filebeat] Add option for S3 input to work without SQS notification (cherry picked from commit 928f9c5)

…27332) (#27414) * [Filebeat] Add option for S3 input to work without SQS notification (cherry picked from commit 928f9c5) Co-authored-by: Andrea Spacca <andrea.spacca@elastic.co>

aspacca added enhancement Team:Integrations Label for the Integrations team labels Aug 12, 2021

botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Aug 12, 2021

Andrea Spacca added 2 commits August 12, 2021 10:35

[Filebeat] Add option for S3 input to work without SQS notification

af149f8

changelog

4c43eab

aspacca changed the title ~~changelog~~ [Filebeat] Add option for S3 input to work without SQS notification Aug 12, 2021

Andrea Spacca added 7 commits August 12, 2021 11:18

docs and configs

23d8488

make update

4b91953

aws credentials and config

84677ab

state and states

5ef9d12

s3 bucket polling refactored

e4e5950

make update

4ed3e5a

benchmark, bugfix and metrics

a1f9e53

aspacca requested a review from andrewkroh August 13, 2021 09:02

Andrea Spacca added 4 commits August 13, 2021 13:16

fix bug

d7e6d74

cr fixes

a997b91

integration test

0895cbc

revert missing build tags

b3b19bc

andrewkroh added Filebeat Filebeat Team:Security-External Integrations labels Aug 13, 2021

andrewkroh added the review label Aug 13, 2021

andrewkroh reviewed Aug 15, 2021

View reviewed changes

aspacca requested review from exekias and kaiyan-sheng August 16, 2021 08:03

cr fixes

193664b

Andrea Spacca added 4 commits August 16, 2021 11:55

make update

42284d0

fix config test

c99c412

cr fix

e9049e6

eventACKTracke Add/Wait must be sequential

cf6c277

aspacca added the backport-v7.15.0 Automated backport with mergify label Aug 16, 2021

Merge branch 'master' into Filebeat-Add-option-for-S3-input-to-work-w…

839770a

…ithout-SQS-notification-refactored

andrewkroh approved these changes Aug 16, 2021

View reviewed changes

cr fixes

5e8594e

kaiyan-sheng reviewed Aug 16, 2021

View reviewed changes

kaiyan-sheng approved these changes Aug 16, 2021

View reviewed changes

fix quotes in docs

4087528

aspacca merged commit 928f9c5 into elastic:master Aug 17, 2021

mergify bot pushed a commit that referenced this pull request Aug 17, 2021

[Filebeat] Add option for S3 input to work without SQS notification (#…

29cdd57

…27332) * [Filebeat] Add option for S3 input to work without SQS notification (cherry picked from commit 928f9c5)

mergify bot mentioned this pull request Aug 17, 2021

[7.x](backport #27332) [Filebeat] Add option for S3 input to work without SQS notification #27414

Merged

aspacca mentioned this pull request Aug 18, 2021

Add vars in modules.d/aws.yml.disabled #27454

Merged

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Filebeat] Add option for S3 input to work without SQS notification #27332

[Filebeat] Add option for S3 input to work without SQS notification #27332

aspacca commented Aug 12, 2021 •

edited

Loading

elasticmachine commented Aug 12, 2021

elasticmachine commented Aug 12, 2021 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

Trends 🧪

Test stats 🧪

aspacca commented Aug 13, 2021 •

edited

Loading

elasticmachine commented Aug 13, 2021

andrewkroh left a comment

andrewkroh Aug 13, 2021

andrewkroh Aug 13, 2021

andrewkroh Aug 15, 2021

andrewkroh Aug 15, 2021

andrewkroh Aug 15, 2021

andrewkroh Aug 15, 2021

aspacca commented Aug 16, 2021

andrewkroh commented Aug 16, 2021

aspacca commented Aug 16, 2021

aspacca commented Aug 16, 2021

aspacca commented Aug 16, 2021

andrewkroh left a comment

andrewkroh Aug 16, 2021

andrewkroh Aug 16, 2021

kaiyan-sheng Aug 16, 2021

kaiyan-sheng Aug 16, 2021

kaiyan-sheng left a comment

aspacca commented Aug 16, 2021

	// Add increments the number of pending ACKs
	// Add increments the number of pending ACKs.

	LastModified time.Time `json:"last_modified" struct:"last_modifed"`
	LastModified time.Time `json:"last_modified" struct:"last_modified"`


		previousState := s.FindPrevious(state)

		// status is forget. if there is no previous state and

	// status is forget. if there is no previous state and
	// status is forgotten. if there is no previous state and

	// try to decode. Ingore faulty/incompatible values.
	// try to decode. Ignore faulty/incompatible values.


		Interval between list requests to the S3 bucket. Default to be 120 seconds.
		Wait interval between completion of a list request to the S3 bucket and beginning of the nest one. Default to be 120 seconds.

	s3ObjectsAckedTotal *monitoring.Uint // Number of S3 objects fully ACKed.
	s3ObjectsAckedTotal *monitoring.Uint

	#var.bucket: 'arn:aws:s3:::mybucket
	#var.bucket: 'arn:aws:s3:::mybucket'

[Filebeat] Add option for S3 input to work without SQS notification #27332

[Filebeat] Add option for S3 input to work without SQS notification #27332

Conversation

aspacca commented Aug 12, 2021 • edited Loading

What does this PR do?

Why is it important?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

elasticmachine commented Aug 12, 2021

elasticmachine commented Aug 12, 2021 • edited by jenkins-beats-ci bot Loading

💚 Build Succeeded

Build stats

Test stats 🧪

Trends 🧪

💚 Flaky test report

Test stats 🧪

aspacca commented Aug 13, 2021 • edited Loading

elasticmachine commented Aug 13, 2021

andrewkroh left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aspacca commented Aug 16, 2021

andrewkroh commented Aug 16, 2021

aspacca commented Aug 16, 2021

aspacca commented Aug 16, 2021

aspacca commented Aug 16, 2021

andrewkroh left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kaiyan-sheng left a comment

Choose a reason for hiding this comment

aspacca commented Aug 16, 2021

aspacca commented Aug 12, 2021 •

edited

Loading

elasticmachine commented Aug 12, 2021 •

edited by jenkins-beats-ci bot

Loading

aspacca commented Aug 13, 2021 •

edited

Loading