Added import stream object for more flexible data upload #1004

kiddinn · 2019-10-23T23:10:18Z

Adding an import stream object that has the ability to:

To upload pandas data frames, python dict, JSON and Excel sheets.
Allowing streaming results from external parsers to TS.

Other features:

Changed the API to allow appending data to an index
Changed the AIP to allow for only indexing newly added data, and no analyzer runs
Added the ability to kick start auto analyzers on an index to the API.
Minor bugfixes in the API and the API client

kiddinn · 2019-10-23T23:11:16Z

This is still WIP, just uploading to demonstrate a bit where I was going with this.

Still need to test it (haven't tested it once), which will most likely results in some unit tests being added as well as major or minor changes to the actual code.

berggren

First round of comments.

berggren · 2019-10-29T13:46:36Z

api_client/python/timesketch_api_client/client.py

+                string_items.append('{0:s} = {{0!s}}'.format(column))
+            format_message_string = ' '.join(string_items)
+
+        response = None


maybe rename to streamer_response ?

berggren · 2019-10-29T13:47:07Z

api_client/python/timesketch_api_client/client.py

+            return 'No return value.'
+
+        return_lines = []
+        for timesketch_object in response.get('objects', []):


sketch_object?

berggren · 2019-10-29T13:50:30Z

timesketch/api/v1/resources.py

@@ -1368,6 +1369,50 @@ def post(self, sketch_id):
                annotations, status_code=HTTP_STATUS_CODE_CREATED)


+class AnalyzerPipelineResource(ResourceMixin, Resource):


Maybe we want to extend this to also accept an analyzer name? So we can run a specific analyzer instead of all.

that is already supported in the AnalyzerRunResource

Decided to do this differently, removed this entire resource, merged it with the run resource, and made it so that we can run a single or multiple analyzers.

also deleted the form so that we now just use request.json instead of the form

timesketch/api/v1/resources.py

berggren · 2019-10-29T13:52:28Z

timesketch/api/v1/resources.py

-        # Create the search index in the Timesketch database
-        searchindex = SearchIndex.get_or_create(
+        # Check if search index already exists.
+        searchindex = SearchIndex.query.filter_by(


hm, why not use get_or_create() here?

I don't want to create one here if it doesn't exist, only querying whether it exists or not.

timesketch/api/v1/routes.py

berggren · 2019-10-29T13:56:02Z

docs/UploadDataViaAPI.md

@@ -0,0 +1,201 @@
+# Create Timeline From Other Sources


Kudos for great documentation :)

;)

there are some TODOs there though

api_client/python/timesketch_api_client/client.py

…into stream_upload

kiddinn

done with round

api_client/python/timesketch_api_client/client.py

kiddinn · 2019-10-29T14:02:20Z

api_client/python/timesketch_api_client/client.py

+                string_items.append('{0:s} = {{0!s}}'.format(column))
+            format_message_string = ' '.join(string_items)
+
+        response = None


kiddinn · 2019-10-29T14:02:49Z

api_client/python/timesketch_api_client/client.py

+            return 'No return value.'
+
+        return_lines = []
+        for timesketch_object in response.get('objects', []):


kiddinn · 2019-10-29T14:03:07Z

docs/UploadDataViaAPI.md

@@ -0,0 +1,201 @@
+# Create Timeline From Other Sources


;)

there are some TODOs there though

kiddinn · 2019-10-29T14:31:45Z

timesketch/api/v1/resources.py

@@ -1368,6 +1369,50 @@ def post(self, sketch_id):
                annotations, status_code=HTTP_STATUS_CODE_CREATED)


+class AnalyzerPipelineResource(ResourceMixin, Resource):


that is already supported in the AnalyzerRunResource

Decided to do this differently, removed this entire resource, merged it with the run resource, and made it so that we can run a single or multiple analyzers.

kiddinn · 2019-10-29T14:32:01Z

timesketch/api/v1/resources.py

@@ -1368,6 +1369,50 @@ def post(self, sketch_id):
                annotations, status_code=HTTP_STATUS_CODE_CREATED)


+class AnalyzerPipelineResource(ResourceMixin, Resource):


also deleted the form so that we now just use request.json instead of the form

kiddinn · 2019-10-29T15:03:48Z

timesketch/api/v1/resources.py

-        # Create the search index in the Timesketch database
-        searchindex = SearchIndex.get_or_create(
+        # Check if search index already exists.
+        searchindex = SearchIndex.query.filter_by(


I don't want to create one here if it doesn't exist, only querying whether it exists or not.

timesketch/api/v1/resources.py

timesketch/api/v1/routes.py

kiddinn · 2019-10-29T15:06:16Z

@berggren PTAL

berggren

Just one small thing, otherwise LG

timesketch/api/v1/resources.py

kiddinn

done

timesketch/api/v1/resources.py

berggren

LGTM

Initial stream uploader.

b85b5da

kiddinn added 3 commits October 24, 2019 09:27

Making some changes to the streamer.

a94406e

Some updates.

896342c

minor change

745e4a9

kiddinn mentioned this pull request Oct 24, 2019

[Idea] Import arbitrary data into Timesketch #967

Closed

kiddinn added 21 commits October 24, 2019 14:03

Minor minor change

f9a98b0

Merge branch 'master' into stream_upload

479c5dc

First version of documentation and minor changes.

04a1db2

Adding an API to trigger analyzer pipeline run.

c28382b

minor bug fix

6f0de80

minor change to aPI

7faf4b5

Minor changes.

1782630

Moving the uploader to a separate file.

86fcd85

Making few changes.

477912d

Updating documentation as well as adding a JSON importer.

71dd2e7

More fixes.

cc08bf1

Improving the file import

84e0c0a

Adding an excel read method.

24aad81

Minor changes

bb4bf62

Merge branch 'master' into stream_upload

9ee7b39

forgot an equal sign

941c673

Merge branch 'master' into stream_upload

63cd2cb

Adding tests for the importer.

8a2864f

Merge branch 'master' into stream_upload

a41f92a

Adding the ability to list available analyzers in the client.

fbb629b

Minor bug in the list aggregation.

e6f2eb5

kiddinn changed the title ~~Added more upload options to API client~~ Added import stream object to the client API for more flexible data upload Oct 28, 2019

kiddinn requested a review from berggren October 28, 2019 16:32

kiddinn self-assigned this Oct 28, 2019

kiddinn added the New feature label Oct 28, 2019

kiddinn marked this pull request as ready for review October 28, 2019 16:32

Upgrading client version.

42cc941

kiddinn mentioned this pull request Oct 28, 2019

List stored aggregations no longer works. #1014

Closed

Expanding excel imports.

793b918

kiddinn changed the title ~~Added import stream object to the client API for more flexible data upload~~ Added import stream object for more flexible data upload Oct 28, 2019

kiddinn added 2 commits October 29, 2019 08:08

Merge branch 'master' into stream_upload

8f35e9f

Merge branch 'master' into stream_upload

dbe1671

berggren requested changes Oct 29, 2019

View reviewed changes

kiddinn added 4 commits October 29, 2019 14:42

Responding to comments, as well as simplifying API calls.

e3b9393

Merge branch 'stream_upload' of https://github.com/kiddinn/timesketch …

c701017

…into stream_upload

Making minor changes to the analyzer form resource.

ae371fa

Minor adjustments.

a5eb78d

kiddinn commented Oct 29, 2019

View reviewed changes

kiddinn requested a review from berggren October 29, 2019 15:06

Merge branch 'master' into stream_upload

1b7571e

berggren requested changes Oct 29, 2019

View reviewed changes

timesketch/api/v1/resources.py Outdated Show resolved Hide resolved

Changing analyzer_name to analyzer_names

59251d3

kiddinn commented Oct 29, 2019

View reviewed changes

timesketch/api/v1/resources.py Outdated Show resolved Hide resolved

kiddinn requested a review from berggren October 29, 2019 15:24

berggren approved these changes Oct 29, 2019

View reviewed changes

kiddinn merged commit b5fff65 into google:master Oct 29, 2019

kiddinn deleted the stream_upload branch October 29, 2019 15:35

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added import stream object for more flexible data upload #1004

Added import stream object for more flexible data upload #1004

kiddinn commented Oct 23, 2019 •

edited

Loading

kiddinn commented Oct 23, 2019

berggren left a comment

berggren Oct 29, 2019

kiddinn Oct 29, 2019

berggren Oct 29, 2019

kiddinn Oct 29, 2019

berggren Oct 29, 2019

kiddinn Oct 29, 2019

kiddinn Oct 29, 2019

berggren Oct 29, 2019

berggren Oct 29, 2019

kiddinn Oct 29, 2019

berggren Oct 29, 2019

berggren Oct 29, 2019

kiddinn Oct 29, 2019

kiddinn left a comment

kiddinn Oct 29, 2019

kiddinn Oct 29, 2019

kiddinn Oct 29, 2019

kiddinn Oct 29, 2019

kiddinn Oct 29, 2019

kiddinn Oct 29, 2019

kiddinn commented Oct 29, 2019

berggren left a comment

kiddinn left a comment

berggren left a comment

		@@ -1368,6 +1369,50 @@ def post(self, sketch_id):
		annotations, status_code=HTTP_STATUS_CODE_CREATED)


		class AnalyzerPipelineResource(ResourceMixin, Resource):

Added import stream object for more flexible data upload #1004

Added import stream object for more flexible data upload #1004

Conversation

kiddinn commented Oct 23, 2019 • edited Loading

kiddinn commented Oct 23, 2019

berggren left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kiddinn left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kiddinn commented Oct 29, 2019

berggren left a comment

Choose a reason for hiding this comment

kiddinn left a comment

Choose a reason for hiding this comment

berggren left a comment

Choose a reason for hiding this comment

kiddinn commented Oct 23, 2019 •

edited

Loading