Extract and store Cowrie file download metadata#861
Closed
remo-lab wants to merge 1 commit intointelowlproject:mainfrom
Closed
Extract and store Cowrie file download metadata#861remo-lab wants to merge 1 commit intointelowlproject:mainfrom
remo-lab wants to merge 1 commit intointelowlproject:mainfrom
Conversation
Signed-off-by: remo-lab <remopanda7@gmail.com>
Contributor
Author
|
@regulartim Plelase take a look at this! |
Collaborator
|
Hey @remo-lab ! Your PR is for |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR implements support for extracting and storing Cowrie file download/upload metadata, specifically the SHA-256 hashes that were previously not being captured.
While going through the Cowrie extraction pipeline, I noticed that even though
cowrie.session.file_downloadandcowrie.session.file_uploadevents include fields likeshasum,url, anddestfile, we weren’t actually fetching or storing this data.In this PR, I:
Downloadmodel linked toCowrieSessionto storeshasum, URL, destination filename, and timestamp.shasumanddestfiletoREQUIRED_FIELDSso Elasticsearch returns them._process_session_hit()to handle file download and upload events and collect download metadata per session.save_download()repository method with deduplication using a unique constraint on(shasum, session)._get_url_downloads()when available for better observability.The changes follow the existing extraction patterns and keep everything scoped to the Cowrie pipeline.
Related issues
Closes #848
Type of change
Checklist
develop.Ruff) gave 0 errors.