Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Elastic-ecs mapping improvements for network traffic attributes #1410

Merged
merged 61 commits into from
Apr 27, 2023
Merged
Changes from 1 commit
Commits
Show all changes
61 commits
Select commit Hold shift + click to select a range
4e52ee3
consolidate network-traffic and x-ecs-network attributes
Harmedox Feb 20, 2023
65772c4
consolidate network-traffic and x-ecs-network attributes
Harmedox Feb 20, 2023
ca9ac4d
consolidate network-traffic and x-ecs-network attributes
Harmedox Feb 20, 2023
75fc74e
add missing network.vlan attribute in elastic_ecs mapping
Harmedox Feb 20, 2023
3982504
add missing network.vlan attribute in elastic_ecs mapping
Harmedox Feb 20, 2023
a841279
add missing network.vlan attribute in elastic_ecs mapping
Harmedox Feb 20, 2023
9eb8031
combine inner vlan and vlan into a single custom object
Harmedox Feb 28, 2023
3264667
combine inner vlan and vlan into a single custom object
Harmedox Feb 28, 2023
9a20947
combine inner vlan and vlan into a single custom object
Harmedox Feb 28, 2023
338ab75
consolidate the object reference for the network-traffic custom attri…
Harmedox Feb 28, 2023
177fec1
transfer network attribute changes to STIX 2.1
Harmedox Feb 28, 2023
011fcba
transfer network attribute changes to STIX 2.1
Harmedox Feb 28, 2023
4b14606
transfer network attribute changes to STIX 2.1
Harmedox Feb 28, 2023
6a8903e
consolidate x-ecs-user attributes into user-account
Harmedox Mar 3, 2023
c05e679
consolidate x-ecs-user attributes into user-account
Harmedox Mar 3, 2023
1162f48
consolidate x-ecs-user attributes into user-account
Harmedox Mar 3, 2023
902749a
consolidate x-ecs-user attributes into user-account
Harmedox Mar 3, 2023
9880a9c
consolidate x-ecs-user attributes into user-account
Harmedox Mar 3, 2023
3316347
consolidate x-ecs-user attributes into user-account
Harmedox Mar 3, 2023
1a9aad8
Merge branch 'opencybersecurityalliance:develop' into develop
Harmedox Mar 7, 2023
b825580
update network object unit test
Harmedox Mar 8, 2023
03e9b50
WIP: represent the pe attributes in file SCO using software SCO
Harmedox Mar 8, 2023
1a94ee4
refer to pe attributes in file object using the software SCO
Harmedox Mar 9, 2023
498426c
refer to pe attributes in file object using the software SCO
Harmedox Mar 9, 2023
cc81195
refer to pe attributes in file object using the software SCO
Harmedox Mar 9, 2023
ee05033
refer to pe attributes in file object using the software SCO in v2.1
Harmedox Mar 9, 2023
f1b2151
refer to pe attributes in file object using the software SCO in v2.1
Harmedox Mar 9, 2023
3b0c997
refer to pe attributes in file object using the software SCO in v2.1
Harmedox Mar 9, 2023
6cb1cde
remove tests for x_pe from the elastic_ecs_connector mapping
Harmedox Mar 9, 2023
b1073ab
add software_ref test to replace the x_pe test in test_x_ecs_event
Harmedox Mar 11, 2023
0ec548b
add product and description as custom atributes of the Software SCO
Harmedox Mar 11, 2023
675fd07
add product and description as custom atributes of the Software SCO
Harmedox Mar 11, 2023
9f2537c
add product and description as custom atributes of the Software SCO v2.1
Harmedox Mar 11, 2023
4341a8d
add test for the product and description attributes for the software …
Harmedox Mar 11, 2023
4c1ea5a
Merge branch 'develop' into develop
mdazam1942 Mar 14, 2023
62ed1c5
Merge branch 'develop' into develop
Mar 15, 2023
81d2a7a
elastic-ecs: make os and host attributes reference the software SCO
Harmedox Mar 20, 2023
5c3b506
Merge branch 'opencybersecurityalliance:develop' into develop
Harmedox Mar 20, 2023
946b616
Merge branch 'develop' of https://github.com/Harmedox/stix-shifter in…
Harmedox Mar 20, 2023
1ab3f87
Merge branch 'opencybersecurityalliance:develop' into develop
Harmedox Mar 20, 2023
8c5669f
WIP:consolidate x-ecs-source, x-ecs-destination, x-ecs-client, x-ecs-…
Harmedox Mar 30, 2023
4ac842c
WIP:consolidate x-ecs-source, x-ecs-destination, x-ecs-client, x-ecs-…
Harmedox Mar 30, 2023
04e37e3
map nat ip and port to the ipv4-addr and ipv6-addr SCO objects
Harmedox Mar 30, 2023
0557432
represent registered_domain and top_level_domain ecs objects as domai…
Harmedox Mar 30, 2023
f3ed547
fix geo reference to x-ecs-geo custom SCO object
Harmedox Mar 30, 2023
cd793c4
Merge branch 'opencybersecurityalliance:develop' into develop
Harmedox Mar 30, 2023
d25ce68
fix port references in ipv4 and ipv6 objects
Harmedox Mar 31, 2023
bf6c94d
fix geo references
Harmedox Mar 31, 2023
137a5b3
remove x-ecs-[source, destination,client, server] objects in from_sti…
Harmedox Mar 31, 2023
c61cfd4
consolidate dll attributes into the file SCO
Harmedox Mar 31, 2023
8508f75
remove x-ecs-dll object
Harmedox Mar 31, 2023
dfb82d7
Merge branch 'opencybersecurityalliance:develop' into develop
Harmedox Apr 10, 2023
9742c18
Merge branch 'opencybersecurityalliance:develop' into develop
Harmedox Apr 25, 2023
c4953ea
consolidate observer object into x-oca-asset
Harmedox Apr 25, 2023
1176d50
represent geo attributes with a generic object x-oca-geo
Harmedox Apr 25, 2023
b8eba06
Merge branch 'develop' of https://github.com/Harmedox/stix-shifter in…
Harmedox Apr 26, 2023
d809b60
Merge branch 'develop' into develop
mdazam1942 Apr 26, 2023
f841e49
update stix_to_query unit test to reflect elastic_ecs mapping changes
Harmedox Apr 26, 2023
3dbc456
Merge branch 'develop' of https://github.com/Harmedox/stix-shifter in…
Harmedox Apr 26, 2023
fb84135
Merge branch 'develop' into develop
Harmedox Apr 26, 2023
ff8cb3f
Merge branch 'develop' into develop
Apr 27, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
update stix_to_query unit test to reflect elastic_ecs mapping changes
Harmedox committed Apr 26, 2023

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
commit f841e4904cb7b780c9e1df7aa3a37eaf94b7d90c
Original file line number Diff line number Diff line change
@@ -40,14 +40,14 @@ def test_ipv4_query(self):
stix_pattern = "[ipv4-addr:value = '192.168.122.83' OR ipv4-addr:value = '192.168.122.84']"
translated_query = translation.translate('elastic_ecs', 'query', '{}', stix_pattern)
translated_query['queries'] = _remove_timestamp_from_query(translated_query['queries'])
test_query = ['((source.ip : "192.168.122.84" OR destination.ip : "192.168.122.84" OR client.ip : "192.168.122.84" OR server.ip : "192.168.122.84" OR host.ip : "192.168.122.84" OR dns.resolved_ip : "192.168.122.84") OR (source.ip : "192.168.122.83" OR destination.ip : "192.168.122.83" OR client.ip : "192.168.122.83" OR server.ip : "192.168.122.83" OR host.ip : "192.168.122.83" OR dns.resolved_ip : "192.168.122.83"))']
test_query = ['((source.ip : "192.168.122.84" OR destination.ip : "192.168.122.84" OR client.ip : "192.168.122.84" OR server.ip : "192.168.122.84" OR host.ip : "192.168.122.84" OR dns.resolved_ip : "192.168.122.84" OR source.nat.ip : "192.168.122.84" OR destination.nat.ip : "192.168.122.84" OR client.nat.ip : "192.168.122.84" OR server.nat.ip : "192.168.122.84") OR (source.ip : "192.168.122.83" OR destination.ip : "192.168.122.83" OR client.ip : "192.168.122.83" OR server.ip : "192.168.122.83" OR host.ip : "192.168.122.83" OR dns.resolved_ip : "192.168.122.83" OR source.nat.ip : "192.168.122.83" OR destination.nat.ip : "192.168.122.83" OR client.nat.ip : "192.168.122.83" OR server.nat.ip : "192.168.122.83"))']
_test_query_assertions(translated_query, test_query)

def test_ipv6_query(self):
stix_pattern = "[ipv6-addr:value = '3001:0:0:0:0:0:0:2']"
translated_query = translation.translate('elastic_ecs', 'query', '{}', stix_pattern)
translated_query['queries'] = _remove_timestamp_from_query(translated_query['queries'])
test_query = ['(source.ip : "3001:0:0:0:0:0:0:2" OR destination.ip : "3001:0:0:0:0:0:0:2" OR client.ip : "3001:0:0:0:0:0:0:2" OR server.ip : "3001:0:0:0:0:0:0:2" OR host.ip : "3001:0:0:0:0:0:0:2" OR dns.resolved_ip : "3001:0:0:0:0:0:0:2")']
test_query = ['(source.ip : "3001:0:0:0:0:0:0:2" OR destination.ip : "3001:0:0:0:0:0:0:2" OR client.ip : "3001:0:0:0:0:0:0:2" OR server.ip : "3001:0:0:0:0:0:0:2" OR host.ip : "3001:0:0:0:0:0:0:2" OR dns.resolved_ip : "3001:0:0:0:0:0:0:2" OR source.nat.ip : "3001:0:0:0:0:0:0:2" OR destination.nat.ip : "3001:0:0:0:0:0:0:2" OR client.nat.ip : "3001:0:0:0:0:0:0:2" OR server.nat.ip : "3001:0:0:0:0:0:0:2")']
_test_query_assertions(translated_query, test_query)

def test_url_query(self):
@@ -82,36 +82,37 @@ def test_file_query(self):
stix_pattern = "[file:name = 'some_file.exe']"
translated_query = translation.translate('elastic_ecs', 'query', '{}', stix_pattern)
translated_query['queries'] = _remove_timestamp_from_query(translated_query['queries'])
test_query = ['(file.name : "some_file.exe" OR file.path : "some_file.exe" OR process.name : "some_file.exe" OR process.executable : "some_file.exe" OR process.parent.name : "some_file.exe" OR process.parent.executable : "some_file.exe")']
test_query = ['(file.name : "some_file.exe" OR dll.name : "some_file.exe" OR file.path : "some_file.exe" OR process.name : "some_file.exe" OR process.executable : "some_file.exe" OR process.parent.name : "some_file.exe" OR process.parent.executable : "some_file.exe")']
_test_query_assertions(translated_query, test_query)

def test_complex_query(self):
stix_pattern = "[network-traffic:protocols[*] LIKE 'ipv_' AND network-traffic:src_port>443] START t'2019-04-11T08:42:39.297Z' STOP t'2019-04-11T08:43:39.297Z' OR [user-account:user_id = '_' AND artifact:payload_bin LIKE '%'] START t'2019-04-11T14:35:44.011Z' STOP t'2019-04-21T16:35:44.011Z' AND [process:pid<700 OR url:value LIKE '%' AND process:creator_user_ref.user_id IN ('root','admin')] START t'2019-04-11T14:35:44.011Z' STOP t'2019-04-17T14:35:44.011Z'"
translated_query = translation.translate('elastic_ecs', 'query', '{}', stix_pattern)
print(str(translated_query))
test_query = ['((source.port:>443 OR client.port:>443) AND (network.transport : ipv? OR network.type : ipv? OR network.protocol : ipv?)) AND (@timestamp:["2019-04-11T08:42:39.297Z" TO "2019-04-11T08:43:39.297Z"])',
'(event.original : * AND (user.name : "_" OR user.id : "_")) AND (@timestamp:["2019-04-11T14:35:44.011Z" TO "2019-04-21T16:35:44.011Z"])', '(((user.name : ("root" OR "admin") AND url.original : *)) OR (process.pid:<700 OR process.ppid:<700 OR process.parent.pid:<700 OR process.parent.ppid:<700)) AND (@timestamp:["2019-04-11T14:35:44.011Z" TO "2019-04-17T14:35:44.011Z"])']
test_query = ['((source.port:>443 OR client.port:>443 OR source.nat.port:>443 OR client.nat.port:>443) AND (network.transport : ipv? OR network.type : ipv? OR network.protocol : ipv?)) AND (@timestamp:["2019-04-11T08:42:39.297Z" TO "2019-04-11T08:43:39.297Z"])',
'(event.original : * AND (user.name : "_" OR user.id : "_")) AND (@timestamp:["2019-04-11T14:35:44.011Z" TO "2019-04-21T16:35:44.011Z"])',
'(((user.name : ("root" OR "admin") AND url.original : *)) OR (process.pid:<700 OR process.ppid:<700 OR process.parent.pid:<700 OR process.parent.ppid:<700)) AND (@timestamp:["2019-04-11T14:35:44.011Z" TO "2019-04-17T14:35:44.011Z"])']
assert translated_query['queries'] == test_query

def test_file_not_equal_query(self):
stix_pattern = "[file:name != 'some_file.exe']"
translated_query = translation.translate('elastic_ecs', 'query', '{}', stix_pattern)
translated_query['queries'] = _remove_timestamp_from_query(translated_query['queries'])
test_query = ['((NOT file.name : "some_file.exe" AND file.name:*) OR (NOT file.path : "some_file.exe" AND file.path:*) OR (NOT process.name : "some_file.exe" AND process.name:*) OR (NOT process.executable : "some_file.exe" AND process.executable:*) OR (NOT process.parent.name : "some_file.exe" AND process.parent.name:*) OR (NOT process.parent.executable : "some_file.exe" AND process.parent.executable:*))']
test_query = ['((NOT file.name : "some_file.exe" AND file.name:*) OR (NOT dll.name : "some_file.exe" AND dll.name:*) OR (NOT file.path : "some_file.exe" AND file.path:*) OR (NOT process.name : "some_file.exe" AND process.name:*) OR (NOT process.executable : "some_file.exe" AND process.executable:*) OR (NOT process.parent.name : "some_file.exe" AND process.parent.name:*) OR (NOT process.parent.executable : "some_file.exe" AND process.parent.executable:*))']
_test_query_assertions(translated_query, test_query)

def test_port_queries(self):
stix_pattern = "[network-traffic:src_port = 12345 OR network-traffic:dst_port = 23456]"
translated_query = translation.translate('elastic_ecs', 'query', '{}', stix_pattern)
translated_query['queries'] = _remove_timestamp_from_query(translated_query['queries'])
test_query = ['((destination.port : "23456" OR server.port : "23456") OR (source.port : "12345" OR client.port : "12345"))']
test_query = ['((destination.port : "23456" OR server.port : "23456" OR destination.nat.port : "23456" OR server.nat.port : "23456") OR (source.port : "12345" OR client.port : "12345" OR source.nat.port : "12345" OR client.nat.port : "12345"))']
_test_query_assertions(translated_query, test_query)

def test_port_queries_lessthan_greaterthan(self):
stix_pattern = "[network-traffic:src_port > 12345 AND network-traffic:dst_port < 23456]"
translated_query = translation.translate('elastic_ecs', 'query', '{}', stix_pattern)
translated_query['queries'] = _remove_timestamp_from_query(translated_query['queries'])
test_query = ['((destination.port:<23456 OR server.port:<23456) AND (source.port:>12345 OR client.port:>12345))']
test_query = ['((destination.port:<23456 OR server.port:<23456 OR destination.nat.port:<23456 OR server.nat.port:<23456) AND (source.port:>12345 OR client.port:>12345 OR source.nat.port:>12345 OR client.nat.port:>12345))']
_test_query_assertions(translated_query, test_query)

def test_port_queries_src_ref_equal(self):
@@ -132,14 +133,14 @@ def test_port_queries_greaterthanorequal(self):
stix_pattern = "[network-traffic:src_port >=443]"
translated_query = translation.translate('elastic_ecs', 'query', '{}', stix_pattern)
translated_query['queries'] = _remove_timestamp_from_query(translated_query['queries'])
test_query = ['(source.port:>=443 OR client.port:>=443)']
test_query = ['(source.port:>=443 OR client.port:>=443 OR source.nat.port:>=443 OR client.nat.port:>=443)']
_test_query_assertions(translated_query, test_query)

def test_port_queries_lessthanorequal(self):
stix_pattern = "[network-traffic:src_port <=443]"
translated_query = translation.translate('elastic_ecs', 'query', '{}', stix_pattern)
translated_query['queries'] = _remove_timestamp_from_query(translated_query['queries'])
test_query = ['(source.port:<=443 OR client.port:<=443)']
test_query = ['(source.port:<=443 OR client.port:<=443 OR source.nat.port:<=443 OR client.nat.port:<=443)']
_test_query_assertions(translated_query, test_query)

def test_network_traffic_in_operator(self):
@@ -153,7 +154,7 @@ def test_network_traffic_queries(self):
stix_pattern = "[network-traffic:src_port = 12345 OR network-traffic:protocols[*] LIKE '_n%']"
translated_query = translation.translate('elastic_ecs', 'query', '{}', stix_pattern)
translated_query['queries'] = _remove_timestamp_from_query(translated_query['queries'])
test_query = ['((network.transport : ?n* OR network.type : ?n* OR network.protocol : ?n*) OR (source.port : "12345" OR client.port : "12345"))']
test_query = ['((network.transport : ?n* OR network.type : ?n* OR network.protocol : ?n*) OR (source.port : "12345" OR client.port : "12345" OR source.nat.port : "12345" OR client.nat.port : "12345"))']
_test_query_assertions(translated_query, test_query)

def test_unmapped_attribute_handling_with_OR(self):
@@ -225,8 +226,8 @@ def test_combined_observations_with_one_qualifier(self):
stix_pattern = "([network-traffic:src_port = 37020 AND user-account:user_id = 'root'] OR [ipv4-addr:value = '192.168.122.83']) START {} STOP {}".format(start_time, stop_time)
translated_query = translation.translate('elastic_ecs', 'query', '{}', stix_pattern)
translated_query['queries'][-1] = _remove_timestamp_from_query(translated_query['queries'][-1])
test_query = ['(source.ip : "192.168.122.83" OR destination.ip : "192.168.122.83" OR client.ip : "192.168.122.83" OR server.ip : "192.168.122.83" OR host.ip : "192.168.122.83" OR dns.resolved_ip : "192.168.122.83") AND (@timestamp:["2019-04-01T01:30:00.123Z" TO "2019-04-01T02:20:00.123Z"])',
'((user.name : "root" OR user.id : "root") AND (source.port : "37020" OR client.port : "37020"))']
test_query = ['(source.ip : "192.168.122.83" OR destination.ip : "192.168.122.83" OR client.ip : "192.168.122.83" OR server.ip : "192.168.122.83" OR host.ip : "192.168.122.83" OR dns.resolved_ip : "192.168.122.83" OR source.nat.ip : "192.168.122.83" OR destination.nat.ip : "192.168.122.83" OR client.nat.ip : "192.168.122.83" OR server.nat.ip : "192.168.122.83") AND (@timestamp:["2019-04-01T01:30:00.123Z" TO "2019-04-01T02:20:00.123Z"])',
'((user.name : "root" OR user.id : "root") AND (source.port : "37020" OR client.port : "37020" OR source.nat.port : "37020" OR client.nat.port : "37020"))']
assert len(translated_query['queries']) == 2
_test_query_assertions(translated_query, test_query)

@@ -237,8 +238,8 @@ def test_start_stop_qualifiers_with_two_observations(self):
stop_time_02 = "t'2019-04-01T04:30:24.743Z'"
stix_pattern = "[network-traffic:src_port = 37020 AND user-account:user_id = 'root'] START {} STOP {} OR [ipv4-addr:value = '192.168.122.83'] START {} STOP {}".format(start_time_01, stop_time_01, start_time_02, stop_time_02)
translated_query = translation.translate('elastic_ecs', 'query', '{}', stix_pattern)
test_query = ['((user.name : "root" OR user.id : "root") AND (source.port : "37020" OR client.port : "37020")) AND (@timestamp:["2019-04-01T01:30:00.123Z" TO "2019-04-01T02:20:00.123Z"])',
'(source.ip : "192.168.122.83" OR destination.ip : "192.168.122.83" OR client.ip : "192.168.122.83" OR server.ip : "192.168.122.83" OR host.ip : "192.168.122.83" OR dns.resolved_ip : "192.168.122.83") AND (@timestamp:["2019-04-01T03:55:00.123Z" TO "2019-04-01T04:30:24.743Z"])']
test_query = ['((user.name : "root" OR user.id : "root") AND (source.port : "37020" OR client.port : "37020" OR source.nat.port : "37020" OR client.nat.port : "37020")) AND (@timestamp:["2019-04-01T01:30:00.123Z" TO "2019-04-01T02:20:00.123Z"])',
'(source.ip : "192.168.122.83" OR destination.ip : "192.168.122.83" OR client.ip : "192.168.122.83" OR server.ip : "192.168.122.83" OR host.ip : "192.168.122.83" OR dns.resolved_ip : "192.168.122.83" OR source.nat.ip : "192.168.122.83" OR destination.nat.ip : "192.168.122.83" OR client.nat.ip : "192.168.122.83" OR server.nat.ip : "192.168.122.83") AND (@timestamp:["2019-04-01T03:55:00.123Z" TO "2019-04-01T04:30:24.743Z"])']
assert len(translated_query['queries']) == 2
_test_query_assertions(translated_query, test_query)

@@ -251,8 +252,9 @@ def test_start_stop_qualifiers_with_three_observations(self):
start_time_01, stop_time_01, start_time_02, stop_time_02)
translated_query = translation.translate('elastic_ecs', 'query', '{}', stix_pattern)
translated_query['queries'][-1] = _remove_timestamp_from_query(translated_query['queries'][-1])
test_query = ['((destination.port : "635" OR server.port : "635") AND (source.port : "37020" OR client.port : "37020")) AND (@timestamp:["2019-04-01T00:00:00.123Z" TO "2019-04-01T01:11:11.456Z"])',
'(source.ip : "333.333.333.0" OR destination.ip : "333.333.333.0" OR client.ip : "333.333.333.0" OR server.ip : "333.333.333.0" OR host.ip : "333.333.333.0" OR dns.resolved_ip : "333.333.333.0") AND (@timestamp:["2019-04-07T02:22:22.789Z" TO "2019-04-07T03:33:33.012Z"])', 'url.original : "www.example.com"']
test_query = ['((destination.port : "635" OR server.port : "635" OR destination.nat.port : "635" OR server.nat.port : "635") AND (source.port : "37020" OR client.port : "37020" OR source.nat.port : "37020" OR client.nat.port : "37020")) AND (@timestamp:["2019-04-01T00:00:00.123Z" TO "2019-04-01T01:11:11.456Z"])',
'(source.ip : "333.333.333.0" OR destination.ip : "333.333.333.0" OR client.ip : "333.333.333.0" OR server.ip : "333.333.333.0" OR host.ip : "333.333.333.0" OR dns.resolved_ip : "333.333.333.0" OR source.nat.ip : "333.333.333.0" OR destination.nat.ip : "333.333.333.0" OR client.nat.ip : "333.333.333.0" OR server.nat.ip : "333.333.333.0") AND (@timestamp:["2019-04-07T02:22:22.789Z" TO "2019-04-07T03:33:33.012Z"])',
'url.original : "www.example.com"']
assert len(translated_query['queries']) == 3
_test_query_assertions(translated_query, test_query)