Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cursor input and manager implementation #19571

Merged
merged 2 commits into from
Jul 2, 2020
Merged

Cursor input and manager implementation #19571

merged 2 commits into from
Jul 2, 2020

Conversation

urso
Copy link

@urso urso commented Jul 1, 2020 •
edited by andresrc
Loading

  • Enhancement

What does this PR do?

This change finally implements the input manager and actual input execution for stateful
inputs.

The full list of changes will include:

Why is it important?

Support for statefull inputs

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
    - [ ] I have made corresponding changes to the documentation
    - [ ] I have made corresponding change to the default configuration files
    - [ ] I have added tests that prove my fix is effective or that my feature works
    - [ ] I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

  • [ ]

How to test this PR locally

Related issues

@urso urso added review Filebeat Filebeat needs_backport PR is waiting to be backported to other branches. Project:Filebeat-Input-v2 Team:Services (Deprecated) Label for the former Integrations-Services team v7.9.0 labels Jul 1, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/integrations-services (Team:Services)

@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Jul 1, 2020
@urso urso mentioned this pull request Jul 1, 2020
Merged
3 tasks
@elasticmachine
Copy link
Collaborator

elasticmachine commented Jul 1, 2020 •
edited
Loading

💚 Build Succeeded

Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: [Pull request #19571 updated]

  • Start Time: 2020-07-02T13:35:17.212+0000

  • Duration: 32 min 38 sec

Test stats 🧪

Test Results
Failed 0
Passed 555
Skipped 128
Total 683

Steps errors

Expand to view the steps failures

  • Name: Report to Codecov

    • Description: curl -sSLo codecov https://codecov.io/bash for i in auditbeat filebeat heartbeat libbeat metricbeat packetbeat winlogbeat journalbeat do FILE="${i}/build/coverage/full.cov" if [ -f "${FILE}" ]; then bash codecov -f "${FILE}" fi done

    • Duration: 2 min 22 sec

    • Start Time: 2020-07-02T13:54:33.411+0000

    • log

@urso urso assigned kvch Jul 2, 2020
kvch
kvch reviewed Jul 2, 2020
View reviewed changes
filebeat/input/v2/input-cursor/input.go Outdated
source Source,
pipeline beat.PipelineConnector,
) (err error) {
// Setup error recovery/reporting
Copy link
Contributor

@kvch kvch Jul 2, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: this comment is unnecessary.

EDIT: IMHO all comments in this function are unnecessary. The name of the functions you are calling are telling the story.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will remove. They are leftovers from prototyping and having a panic implementation

urso added 2 commits July 2, 2020 14:36
Cursor input and manager implementation
760fb81 
This change finally implements the input manager and actual input execution for stateful
inputs.

The full list of changes will include:
- Introduce v2 API interfaces
- Introduce [compatibility layer](https://github.com/urso/beats/tree/fb-input-v2-combined/filebeat/input/v2/compat) to integrate API with existing functionality
- Introduce helpers for writing [stateless](https://github.com/urso/beats/blob/fb-input-v2-combined/filebeat/input/v2/input-stateless/stateless.go) inputs.
- Introduce helpers for writing [inputs that store a state](https://github.com/urso/beats/tree/fb-input-v2-combined/filebeat/input/v2/input-cursor) between restarts.
- Integrate new API with [existing inputs and modules](https://github.com/urso/beats/blob/fb-input-v2-combined/filebeat/beater/filebeat.go#L301) in filebeat.
review
8b9fa3a
@urso urso force-pushed the input-v2-cursor-manager-impl branch from 651f5d8 to 8b9fa3a Compare July 2, 2020 12:37
@urso urso merged commit 9a98bee into elastic:master Jul 2, 2020
@urso urso deleted the input-v2-cursor-manager-impl branch July 2, 2020 15:17
v1v added a commit to v1v/beats that referenced this pull request Jul 3, 2020
@v1v
Merge remote-tracking branch 'upstream/master' into feature/windows-o…
0ba6f3b 
…ne-beats

* upstream/master: (35 commits)
  [ci] fix env variable name for xpack filebeats (elastic#19617)
  Cache error responses for cloudfoundry apps metadata (elastic#19181)
  ci: user fixed type of agent (elastic#19625)
  Input v2 cursor testing (elastic#19573)
  Update Jenkinsfile to not inspect removed vendor (elastic#19610)
  Fix ordering and duplicate configs on autodiscover (elastic#19317)
  Prepare input/file for changes in the registrar (elastic#19516)
  Cursor input and manager implementation (elastic#19571)
  [Filebeat] Fix tls mapping in suricata module (elastic#19494)
  [Ingest Manager] Make Agent beta and Constraints experimental (elastic#19586)
  Accept prefix as metric_types for stackdriver metricset in GCP (elastic#19345)
  Implement memlog store operations (elastic#19533)
  introduce journalbeat/pkg in order to provide reusable shared code (elastic#19581)
  Add descriptions to HAProxy fields in Metricbeat (elastic#19561)
  ci: apm-server-update trigered only on upstream, comments, and manual triggered (elastic#19590)
  ci: enable upstream triggering on the packaging job (elastic#19589)
  ci: some jjbb improvements (elastic#19588)
  [MetricBeat] set tags correctly if the dimension value is ARN (elastic#19433)
  [Filebeat] Add default_fields: false to fields.yml in aws module (elastic#19568)
  Add publisher implementation for stateful inputs (elastic#19530)
  ...
urso pushed a commit to urso/beats that referenced this pull request Jul 8, 2020
Cursor input and manager implementation (elastic#19571)
e62dde2 
This change finally implements the input manager and actual input execution for stateful
inputs.
@urso urso mentioned this pull request Jul 8, 2020
Merged
3 tasks
@urso urso removed the needs_backport PR is waiting to be backported to other branches. label Jul 8, 2020
urso pushed a commit that referenced this pull request Jul 8, 2020
Cherry-pick #19518, #19528, #19529, #19530, #19571, #19573 to 7.x: Im…
56d4dd0 
…plement storage handling for cursor inputs (#19732)
melchiormoulin pushed a commit to melchiormoulin/beats that referenced this pull request Oct 14, 2020
@melchiormoulin
Cursor input and manager implementation (elastic#19571)
17b6fa9 
This change finally implements the input manager and actual input execution for stateful
inputs.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kvch kvch kvch approved these changes

Assignees

@kvch kvch

Labels
Filebeat Filebeat Project:Filebeat-Input-v2 review skip-test-plan Team:Services (Deprecated) Label for the former Integrations-Services team v7.9.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@urso @elasticmachine @kvch @andresrc