Redline CSV Import

[jaegeral]

I plan to write something to ingest redline csv format that is used by a lot of different teams globally.
As redline does not have the same fields as timesketch expects, there is some parsing for dates and re-arranging needed.

My plan is to write a python script that ingests redline format and exports timesketchable csv.

If there is a good reason to not do it that way, please speak up.

Importing it directly might easier for users, but atm I do not feel I want to go that deep.

However, the code that might be in the helper script could be re-used at a later stage.

[berggren]

This is a great idea! If it is just reordering the fields and add a header line I think the easiest way is to hook this up in Timesketch directly. The code would live in timesketch/lib/utils.py and as a function like "read_and_validate_redline_csv()".

Take a look at the read_and_validate_csv(path, delimiter) in the same file and reuse the logic there. Key is to return a generator in order for us to be memory efficient when reading large files.

[jaegeral]

So here is the first shot, that at least works:
https://github.com/deralexxx/redline2timesketch

It is currently a standalone script that does cleaning re-ordering and such.
I tested it with exports around 100 MB, works.

Next step is then to do as you suggested to make it a built in module.
During testing I had to play with the util file anyway, so should be no problem.

[berggren]

Awesome! This should be simple to add to utils.py and hook up to the core upload feature. Looking forward to the PR

[kiddinn]

isn't this already done/completed? Can we close this issue?

[berggren]

It is for tsctl, but not enabled for imports using the new analyzers etc. Will add that asap.

[garanews]

It would be useful if script will be able to read mans directly and not only csv (because opening mans and exporting in csv is requiring resources and time at client side)

[garanews]

@deralexxx I see you already wrote https://github.com/deralexxx/RedLineTimelineDB_extractor , so just need to integrate directly in timesketch so people can upload directly mans, will look if possible to use multiprocessing in order to speed up the process.

[joachimmetz]

Please add setup.py, register this on PyPI.

Add a release to https://github.com/deralexxx/RedLineTimelineDB_extractor/releases and PyPI.

[jaegeral]

@deralexxx I see you already wrote https://github.com/deralexxx/RedLineTimelineDB_extractor , so just need to integrate directly in timesketch so people can upload directly mans, will look if possible to use multiprocessing in order to speed up the process.

#937

[jaegeral]

I think that is still needed, as besides mans file a csv can be exported from redline which was already proposed in #642.

[kiddinn]

can we not just use the timesketch importer directly now, since this is a CSV file?

I already added a formatter.yaml file to the importer client that can be used to identify different CSV files and apply correct formatting for them to import into TS... and the first example of such formatter is for redline: https://github.com/google/timesketch/blob/master/importer_client/python/timesketch_import_client/data/formatter.yaml