[Filebeat] keep track of bytes read when max_bytes exceeded

[leehinman]

What does this PR do?

Keeps track of bytes read when max_bytes is exceeded.

Why is it important?

Without this offset is wrong in the events and the registry. This
leads to re-reading events from the point where max_bytes is
exceeded.

Checklist

• My code follows the style guidelines of this project
  - [ ] I have commented my code, particularly in hard-to-understand areas
  - [ ] I have made corresponding changes to the documentation
  - [ ] I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

See #28317

Related issues

• Closes [Filebeat] offset not correctly recorded when max_bytes is exceeded in linereader #28317

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b 28317_offset_and_max_bytes upstream/28317_offset_and_max_bytes
git merge upstream/master
git push upstream 28317_offset_and_max_bytes

[mergify]

This pull request does not have a backport label. Could you fix it @leehinman? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-v./d./d./d is the label to automatically backport to the 7./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2021-10-12T19:36:38.656+0000

• Duration: 151 min 19 sec

• Commit: 403e576

Test stats 🧪

Test	Results
Failed	0
Passed	53759
Skipped	5346
Total	59105

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate the packages and run the E2E tests.

• /beats-tester : Run the installation tests with beats-tester.

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b 28317_offset_and_max_bytes upstream/28317_offset_and_max_bytes
git merge upstream/master
git push upstream 28317_offset_and_max_bytes

[andrewkroh]

@leehinman Could you try to write some godocs for the Next() method. Without a specification for its behavior I worry that someone might either change the implementation unknowingly or use it incorrectly.

IIUC the int that is returned by LineReader#Next() accumulates the sizes of skipped lines and has little relation to the length of the returned []byte?

[leehinman]

@leehinman Could you try to write some godocs for the Next() method. Without a specification for its behavior I worry that someone might either change the implementation unknowingly or use it incorrectly.

IIUC the int that is returned by LineReader#Next() accumulates the sizes of skipped lines and has little relation to the length of the returned []byte?

Good idea. Updated.

[andrewkroh]

LGTM