Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

LIKE operator only added for events queries during translation involving dnsdomainname #1521

Conversation

KaneBrennan132
Copy link
Contributor

@KaneBrennan132 KaneBrennan132 commented Jun 27, 2023

The like operator was being added to flow searches after the domain name property name updates, added to the if statement in _parse_mapped_fields to check the query in order to change and add the 'like' operator only if the query is an event query.

For example:
Before:
file:hashes.'SHA-256' IN ('a48c39cc45efea110a7c8edadcb6719f5d1ebbeebb570b345f47172d393c0821','a9db5aca01499f6ce404db22fb4ba3e4e0dc4b94a41c805c520bd39262df1ddc','92153e88db63016334625514802d0d1019363989d7b3f6863947ce0e490c1006','347e2f0d8332dd2d9294d06544c051a302a2436da453b2ccfa2d7829e3a79944','8ee9141074b48784c89aa5d3cd4010fcf4e6d467b618c8719970f78fcc24a365') OR ipv4-addr:value IN ('94.199.173.6','45.154.24.57','132.148.79.222','129.153.135.83','45.85.235.39') OR url:value IN ('https://45.154.24.57:2078','https://94.199.173.6:2222','https://132.148.79.222:2222','https://45.85.235.39:2078','https://129.153.135.83:2078')] START t'2023-06-07T12:20:01.000Z' STOP t'2023-06-07T13:20:01.000Z'
translates to:
"queries": [ "SELECT applicationid, APPLICATIONNAME(applicationid) as applicationname, CATEGORYNAME(category) as categoryname, credibility, destinationasn, destinationbytes, destinationdscp, destinationflags, destinationip, destinationifindex, destinationpackets, destinationport, destinationprecedence, destinationv6, domainid, fullmatchlist, firstpackettime, flowbias, flowdirection as direction, flowinterfaceid, flowsource, flowtype, geographic, hasoffense, icmpcode, icmptype, flowinterface, intervalid, isduplicate, lastpackettime, partialmatchlist, PROTOCOLNAME(protocolid) as protocol, QIDNAME(qid) as qidname, qid, processorid, relevance, retentionbucket, severity as flowseverity, sourceasn, sourcebytes, sourcedscp, sourceflags, sourceip, sourceifindex, sourcepackets, sourceport, sourceprecedence, sourcev6, starttime, endtime, UTF8(sourcepayload) as flowsourcepayload, UTF8(destinationpayload) as flowdestinationpayload, CASE flowtype WHEN 0 THEN 'Standard Flow' WHEN 3 THEN 'Network Scan (Type A)' WHEN 4 THEN 'DDoS (Type B)' WHEN 5 THEN 'Port Scan (Type C)' ELSE 'Unknown Type' END AS flowtype, \"md5 file hash\" as md5hash, \"sha1 file hash\" as sha1hash, \"sha256 file hash\" as sha256hash, \"file name\" as filename, \"file size\" as filesize, \"file entropy\" as fileentropy, \"http host\" as httphost, \"http referrer\" as httpreferrer, \"http response code\" as httpresponsecode, \"http server\" as httpserver, \"http user agent\" as httpuseragent, \"http version\" as httpversion, \"dns domain name\" as dnsdomainname, \"tls server name indication\" as tlsservernameindication, \"tls ja3 hash\" as tlsja3hash, \"tls ja3s hash\" as tlsja3shash, \"suspect content descriptions\" as suspectcontentdescriptions, flowid, \"content type\" as contenttype FROM flows WHERE (dnsdomainname LIKE '%('https://45.154.24.57:2078', 'https://94.199.173.6:2222', 'https://132.148.79.222:2222', 'https://45.85.235.39:2078', 'https://129.153.135.83:2078')%' OR tlsservernameindication LIKE '%('https://45.154.24.57:2078', 'https://94.199.173.6:2222', 'https://132.148.79.222:2222', 'https://45.85.235.39:2078', 'https://129.153.135.83:2078')%' OR httphost LIKE '%('https://45.154.24.57:2078', 'https://94.199.173.6:2222', 'https://132.148.79.222:2222', 'https://45.85.235.39:2078', 'https://129.153.135.83:2078')%') OR ((sourceip IN ('94.199.173.6', '45.154.24.57', '132.148.79.222', '129.153.135.83', '45.85.235.39') OR destinationip IN ('94.199.173.6', '45.154.24.57', '132.148.79.222', '129.153.135.83', '45.85.235.39')) OR sha256hash IN ('a48c39cc45efea110a7c8edadcb6719f5d1ebbeebb570b345f47172d393c0821', 'a9db5aca01499f6ce404db22fb4ba3e4e0dc4b94a41c805c520bd39262df1ddc', '92153e88db63016334625514802d0d1019363989d7b3f6863947ce0e490c1006', '347e2f0d8332dd2d9294d06544c051a302a2436da453b2ccfa2d7829e3a79944', '8ee9141074b48784c89aa5d3cd4010fcf4e6d467b618c8719970f78fcc24a365')) limit 10000 START 1686140401000 STOP 1686144001000", "SELECT QIDNAME(qid) as qidname, qid, CATEGORYNAME(category) as categoryname, category as categoryid, CATEGORYNAME(highlevelcategory) as high_level_category_name, highlevelcategory as high_level_category_id, logsourceid, devicetype, LOGSOURCETYPENAME(devicetype) as logsourcetypename, LOGSOURCENAME(logsourceid) as logsourcename, starttime, endtime, devicetime, sourceaddress as sourceip, sourceport, sourcemac, destinationaddress as destinationip, destinationport, destinationmac, username, eventdirection as direction, identityip, identityhostname, eventcount, PROTOCOLNAME(protocolid) as protocol, UTF8(payload) as eventpayload, URL as url, magnitude, Filename as filename, \"File Hash\" as filehash, \"SHA1 Hash\" as sha1hash, \"SHA256 Hash\" as sha256hash, \"MD5 Hash\" as md5hash, \"File Path\" as filepath, severity as eventseverity, credibility, relevance, sourcegeographiclocation as sourcegeographic, destinationgeographiclocation as destinationgeographic, \"CRE Name\" as crename, \"CRE Description\" as credescription, creeventlist, rulename(creeventlist) as rulenames, domainid, DOMAINNAME(domainid) as qradardomain, \"DNS Request Domain\" as dnsdomainname, EventID, Image, ParentImage, \"Process CommandLine\" as ProcessCommandLine, ParentCommandLine, TargetImage, Message, \"Registry Value Name\" as RegistryValueName, \"IMP Hash\" as IMPHash, ServiceFileName, \"Registry Key\" as RegistryKey, ObjectName, UrlHost, \"Process Name\" as ProcessName, \"Process ID\" as ProcessId, \"Parent Process ID\" as ParentProcessId, hasoffense, \"Machine ID\" as MachineId, \"Process Guid\" as ProcessGuid FROM events WHERE url IN ('https://45.154.24.57:2078', 'https://94.199.173.6:2222', 'https://132.148.79.222:2222', 'https://45.85.235.39:2078', 'https://129.153.135.83:2078') OR ((sourceip IN ('94.199.173.6', '45.154.24.57', '132.148.79.222', '129.153.135.83', '45.85.235.39') OR destinationip IN ('94.199.173.6', '45.154.24.57', '132.148.79.222', '129.153.135.83', '45.85.235.39') OR identityip IN ('94.199.173.6', '45.154.24.57', '132.148.79.222', '129.153.135.83', '45.85.235.39')) OR sha256hash IN ('a48c39cc45efea110a7c8edadcb6719f5d1ebbeebb570b345f47172d393c0821', 'a9db5aca01499f6ce404db22fb4ba3e4e0dc4b94a41c805c520bd39262df1ddc', '92153e88db63016334625514802d0d1019363989d7b3f6863947ce0e490c1006', '347e2f0d8332dd2d9294d06544c051a302a2436da453b2ccfa2d7829e3a79944', '8ee9141074b48784c89aa5d3cd4010fcf4e6d467b618c8719970f78fcc24a365')) limit 10000 START 1686140401000 STOP 1686144001000" ]

After change:
Translates to:
"queries": [ "SELECT applicationid, APPLICATIONNAME(applicationid) as applicationname, CATEGORYNAME(category) as categoryname, credibility, destinationasn, destinationbytes, destinationdscp, destinationflags, destinationip, destinationifindex, destinationpackets, destinationport, destinationprecedence, destinationv6, domainid, fullmatchlist, firstpackettime, flowbias, flowdirection as direction, flowinterfaceid, flowsource, flowtype, geographic, hasoffense, icmpcode, icmptype, flowinterface, intervalid, isduplicate, lastpackettime, partialmatchlist, PROTOCOLNAME(protocolid) as protocol, QIDNAME(qid) as qidname, qid, processorid, relevance, retentionbucket, severity as flowseverity, sourceasn, sourcebytes, sourcedscp, sourceflags, sourceip, sourceifindex, sourcepackets, sourceport, sourceprecedence, sourcev6, starttime, endtime, UTF8(sourcepayload) as flowsourcepayload, UTF8(destinationpayload) as flowdestinationpayload, CASE flowtype WHEN 0 THEN 'Standard Flow' WHEN 3 THEN 'Network Scan (Type A)' WHEN 4 THEN 'DDoS (Type B)' WHEN 5 THEN 'Port Scan (Type C)' ELSE 'Unknown Type' END AS flowtype, \"md5 file hash\" as md5hash, \"sha1 file hash\" as sha1hash, \"sha256 file hash\" as sha256hash, \"file name\" as filename, \"file size\" as filesize, \"file entropy\" as fileentropy, \"http host\" as httphost, \"http referrer\" as httpreferrer, \"http response code\" as httpresponsecode, \"http server\" as httpserver, \"http user agent\" as httpuseragent, \"http version\" as httpversion, \"dns domain name\" as dnsdomainname, \"tls server name indication\" as tlsservernameindication, \"tls ja3 hash\" as tlsja3hash, \"tls ja3s hash\" as tlsja3shash, \"suspect content descriptions\" as suspectcontentdescriptions, flowid, \"content type\" as contenttype FROM flows WHERE (dnsdomainname IN ('https://45.154.24.57:2078', 'https://94.199.173.6:2222', 'https://132.148.79.222:2222', 'https://45.85.235.39:2078', 'https://129.153.135.83:2078') OR tlsservernameindication IN ('https://45.154.24.57:2078', 'https://94.199.173.6:2222', 'https://132.148.79.222:2222', 'https://45.85.235.39:2078', 'https://129.153.135.83:2078') OR httphost IN ('https://45.154.24.57:2078', 'https://94.199.173.6:2222', 'https://132.148.79.222:2222', 'https://45.85.235.39:2078', 'https://129.153.135.83:2078')) OR ((sourceip IN ('94.199.173.6', '45.154.24.57', '132.148.79.222', '129.153.135.83', '45.85.235.39') OR destinationip IN ('94.199.173.6', '45.154.24.57', '132.148.79.222', '129.153.135.83', '45.85.235.39')) OR sha256hash IN ('a48c39cc45efea110a7c8edadcb6719f5d1ebbeebb570b345f47172d393c0821', 'a9db5aca01499f6ce404db22fb4ba3e4e0dc4b94a41c805c520bd39262df1ddc', '92153e88db63016334625514802d0d1019363989d7b3f6863947ce0e490c1006', '347e2f0d8332dd2d9294d06544c051a302a2436da453b2ccfa2d7829e3a79944', '8ee9141074b48784c89aa5d3cd4010fcf4e6d467b618c8719970f78fcc24a365')) limit 10000 START 1686140401000 STOP 1686144001000", "SELECT QIDNAME(qid) as qidname, qid, CATEGORYNAME(category) as categoryname, category as categoryid, CATEGORYNAME(highlevelcategory) as high_level_category_name, highlevelcategory as high_level_category_id, logsourceid, devicetype, LOGSOURCETYPENAME(devicetype) as logsourcetypename, LOGSOURCENAME(logsourceid) as logsourcename, starttime, endtime, devicetime, sourceaddress as sourceip, sourceport, sourcemac, destinationaddress as destinationip, destinationport, destinationmac, username, eventdirection as direction, identityip, identityhostname, eventcount, PROTOCOLNAME(protocolid) as protocol, UTF8(payload) as eventpayload, URL as url, magnitude, Filename as filename, \"File Hash\" as filehash, \"SHA1 Hash\" as sha1hash, \"SHA256 Hash\" as sha256hash, \"MD5 Hash\" as md5hash, \"File Path\" as filepath, severity as eventseverity, credibility, relevance, sourcegeographiclocation as sourcegeographic, destinationgeographiclocation as destinationgeographic, \"CRE Name\" as crename, \"CRE Description\" as credescription, creeventlist, rulename(creeventlist) as rulenames, domainid, DOMAINNAME(domainid) as qradardomain, \"DNS Request Domain\" as dnsdomainname, EventID, Image, ParentImage, \"Process CommandLine\" as ProcessCommandLine, ParentCommandLine, TargetImage, Message, \"Registry Value Name\" as RegistryValueName, \"IMP Hash\" as IMPHash, ServiceFileName, \"Registry Key\" as RegistryKey, ObjectName, UrlHost, \"Process Name\" as ProcessName, \"Process ID\" as ProcessId, \"Parent Process ID\" as ParentProcessId, hasoffense, \"Machine ID\" as MachineId, \"Process Guid\" as ProcessGuid FROM events WHERE url IN ('https://45.154.24.57:2078', 'https://94.199.173.6:2222', 'https://132.148.79.222:2222', 'https://45.85.235.39:2078', 'https://129.153.135.83:2078') OR ((sourceip IN ('94.199.173.6', '45.154.24.57', '132.148.79.222', '129.153.135.83', '45.85.235.39') OR destinationip IN ('94.199.173.6', '45.154.24.57', '132.148.79.222', '129.153.135.83', '45.85.235.39') OR identityip IN ('94.199.173.6', '45.154.24.57', '132.148.79.222', '129.153.135.83', '45.85.235.39')) OR sha256hash IN ('a48c39cc45efea110a7c8edadcb6719f5d1ebbeebb570b345f47172d393c0821', 'a9db5aca01499f6ce404db22fb4ba3e4e0dc4b94a41c805c520bd39262df1ddc', '92153e88db63016334625514802d0d1019363989d7b3f6863947ce0e490c1006', '347e2f0d8332dd2d9294d06544c051a302a2436da453b2ccfa2d7829e3a79944', '8ee9141074b48784c89aa5d3cd4010fcf4e6d467b618c8719970f78fcc24a365')) limit 10000 START 1686140401000 STOP 1686144001000" ]

@codecov
Copy link

codecov bot commented Jun 27, 2023

Codecov Report

Patch coverage: 100.00% and no project coverage change.

Comparison is base (9e3df37) 85.41% compared to head (1740221) 85.41%.

Additional details and impacted files
@@           Coverage Diff            @@
##           develop    #1521   +/-   ##
========================================
  Coverage    85.41%   85.41%           
========================================
  Files          680      680           
  Lines        54173    54173           
========================================
  Hits         46274    46274           
  Misses        7899     7899           
Impacted Files Coverage Δ
...dules/qradar/stix_translation/query_constructor.py 95.35% <100.00%> (ø)
...dar_stix_to_aql/test_qradar_flows_stix_to_query.py 100.00% <100.00%> (ø)

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

@delliott90 delliott90 merged commit 7058817 into opencybersecurityalliance:develop Jun 29, 2023
KaneBrennan132 added a commit to KaneBrennan132/stix-shifter that referenced this pull request Aug 30, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants