Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

S3 input: Add fallback configuration for content type so that events can be decoded #25697

Closed
ppf2 opened this issue May 12, 2021 · 1 comment · Fixed by #25772
Closed

S3 input: Add fallback configuration for content type so that events can be decoded #25697

ppf2 opened this issue May 12, 2021 · 1 comment · Fixed by #25772
Assignees
@leehinman
Labels
enhancement Filebeat Filebeat

Comments

@ppf2
Copy link
Member

ppf2 commented May 12, 2021

If the S3 object has a Content-Type of application/json, we will decode it as JSON in the input. Sometimes, the uploaded S3 objects do not have content type set. For these cases, it will be helpful to have an optional configuration to tell the S3 input to fallback to a specified content type of choice when no Content-Type is specified so that we will still attempt to decode them.

@andrewkroh
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label May 12, 2021
@ppf2 ppf2 changed the title S3 input: Add fallback to application/json so that events can be decoded S3 input: Add fallback configuration for content type so that events can be decoded May 12, 2021
@elasticmachine
Copy link
Collaborator

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label May 12, 2021
@andrewkroh andrewkroh added the Filebeat Filebeat label May 12, 2021
@leehinman leehinman self-assigned this May 14, 2021
@leehinman leehinman mentioned this issue May 18, 2021
Merged
4 tasks
leehinman added a commit to leehinman/beats that referenced this issue May 18, 2021
@leehinman
aws-s3 input default content-type
87f4f26 
- new option `content_type`
- can be set at input or file selector level
- overrides Content-Type that was given to the S3 object when it was
  uploaded.

Closes elastic#25697
leehinman added a commit that referenced this issue May 19, 2021
@leehinman @kaiyan-sheng
aws-s3 input default content-type (#25772)
d62b1be 
* aws-s3 input default content-type

- new option `content_type`
- can be set at input or file selector level
- overrides Content-Type that was given to the S3 object when it was
  uploaded.

Closes #25697

Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co>
mergify bot pushed a commit that referenced this issue May 19, 2021
@leehinman
aws-s3 input default content-type (#25772)
fba4db2 
* aws-s3 input default content-type

- new option `content_type`
- can be set at input or file selector level
- overrides Content-Type that was given to the S3 object when it was
  uploaded.

Closes #25697

Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co>
(cherry picked from commit d62b1be)
leehinman added a commit that referenced this issue May 19, 2021
@mergify @leehinman
aws-s3 input default content-type (#25772) (#25786)
c1acdf5 
* aws-s3 input default content-type

- new option `content_type`
- can be set at input or file selector level
- overrides Content-Type that was given to the S3 object when it was
  uploaded.

Closes #25697

Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co>
(cherry picked from commit d62b1be)

Co-authored-by: Lee Hinman <57081003+leehinman@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@leehinman leehinman

Labels
enhancement Filebeat Filebeat
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

aws-s3 input default content-type leehinman/beats
4 participants
@andrewkroh @ppf2 @elasticmachine @leehinman