feat(log) no progress when --log-json option #116

sadayuki-matsuno · 2024-07-03T04:56:19Z

If this Pull Request is work in progress, Add a prefix of “[WIP]” in the title.

What did you implement:

no progress when --log-json option

Type of change

Please delete options that are not relevant.

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
This change requires a documentation update

How Has This Been Tested?

before

$ go-cti fetch threat --log-json
{"lvl":"info","msg":"Fetching Cyber Threat Intelligence and CVE-ID to CTI-ID Mappings","t":"2024-07-08T12:26:39.074138311+09:00"}
{"lvl":"info","msg":"Fetching MITRE ATT\u0026CK...","t":"2024-07-08T12:26:39.074401593+09:00"}
{"lvl":"info","msg":"Fetching CAPEC...","t":"2024-07-08T12:26:39.455052872+09:00"}
{"lvl":"info","msg":"Fetching CWE...","t":"2024-07-08T12:26:39.543769525+09:00"}
{"lvl":"info","msg":"Fetching NVD CVE...","t":"2024-07-08T12:26:43.119192893+09:00","year":"recent"}
{"lvl":"info","msg":"Fetching NVD CVE...","t":"2024-07-08T12:26:44.40662475+09:00","year":"modified"}
{"lvl":"info","msg":"Fetching NVD CVE...","t":"2024-07-08T12:26:45.881449584+09:00","year":"2002"}
{"lvl":"info","msg":"Fetching NVD CVE...","t":"2024-07-08T12:26:47.813224678+09:00","year":"2003"}
{"lvl":"info","msg":"Fetching NVD CVE...","t":"2024-07-08T12:26:49.246298873+09:00","year":"2004"}
{"lvl":"info","msg":"Fetching NVD CVE...","t":"2024-07-08T12:26:51.11976077+09:00","year":"2005"}
{"lvl":"info","msg":"Fetching NVD CVE...","t":"2024-07-08T12:26:52.797585367+09:00","year":"2006"}
{"lvl":"info","msg":"Fetching NVD CVE...","t":"2024-07-08T12:26:54.643119212+09:00","year":"2007"}
{"lvl":"info","msg":"Fetching NVD CVE...","t":"2024-07-08T12:26:56.575900409+09:00","year":"2008"}
{"lvl":"info","msg":"Fetching NVD CVE...","t":"2024-07-08T12:26:58.709273585+09:00","year":"2009"}
{"lvl":"info","msg":"Fetching NVD CVE...","t":"2024-07-08T12:27:00.5332743+09:00","year":"2010"}
{"lvl":"info","msg":"Fetching NVD CVE...","t":"2024-07-08T12:27:02.400853891+09:00","year":"2011"}
{"lvl":"info","msg":"Fetching NVD CVE...","t":"2024-07-08T12:27:04.338091103+09:00","year":"2012"}
{"lvl":"info","msg":"Fetching NVD CVE...","t":"2024-07-08T12:27:06.413458123+09:00","year":"2013"}
{"lvl":"info","msg":"Fetching NVD CVE...","t":"2024-07-08T12:27:08.964899933+09:00","year":"2014"}
{"lvl":"info","msg":"Fetching NVD CVE...","t":"2024-07-08T12:27:11.090371338+09:00","year":"2015"}
{"lvl":"info","msg":"Fetching NVD CVE...","t":"2024-07-08T12:27:13.09963404+09:00","year":"2016"}
{"lvl":"info","msg":"Fetching NVD CVE...","t":"2024-07-08T12:27:15.841806394+09:00","year":"2017"}
{"lvl":"info","msg":"Fetching NVD CVE...","t":"2024-07-08T12:27:18.278196477+09:00","year":"2018"}
{"lvl":"info","msg":"Fetching NVD CVE...","t":"2024-07-08T12:27:20.859532896+09:00","year":"2019"}
{"lvl":"info","msg":"Fetching NVD CVE...","t":"2024-07-08T12:27:24.018987033+09:00","year":"2020"}
{"lvl":"info","msg":"Fetching NVD CVE...","t":"2024-07-08T12:27:27.352729083+09:00","year":"2021"}
{"lvl":"info","msg":"Fetching NVD CVE...","t":"2024-07-08T12:27:37.766338382+09:00","year":"2022"}
{"lvl":"info","msg":"Fetching NVD CVE...","t":"2024-07-08T12:27:47.843383073+09:00","year":"2023"}
{"lvl":"info","msg":"Fetching NVD CVE...","t":"2024-07-08T12:27:56.02476123+09:00","year":"2024"}
{"attackers":853,"lvl":"info","mappings":134372,"msg":"Fetched Cyber Threat Intelligence and CVE-ID to CTI-ID Mappings","t":"2024-07-08T12:27:59.627461309+09:00","techniques":1196}
{"db":"sqlite3","lvl":"info","msg":"Insert Cyber Threat Intelligences and CVE-ID to CTI-ID Mappings into go-cti.","t":"2024-07-08T12:27:59.627595184+09:00"}
{"lvl":"info","msg":"Inserting Cyber Threat Intelligences...","t":"2024-07-08T12:27:59.630684957+09:00"}
{"lvl":"info","msg":"Inserting Techniques...","t":"2024-07-08T12:27:59.630730062+09:00"}
1196 / 1196 [-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------] 100.00% 833 p/s
{"lvl":"info","msg":"Inserting CVE-ID to CTI-ID CveToTechniques...","t":"2024-07-08T12:28:01.267319626+09:00"}
134372 / 134372 [--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------] 100.00% 7441 p/s
{"lvl":"info","msg":"Inserting Attackers...","t":"2024-07-08T12:28:19.525911344+09:00"}
853 / 853 [--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------] 100.00% 5132 p/s

after

$ go-cti fetch threat --log-json
{"lvl":"info","msg":"Fetching Cyber Threat Intelligence and CVE-ID to CTI-ID Mappings","t":"2024-07-08T12:29:27.874585022+09:00"}
{"lvl":"info","msg":"Fetching MITRE ATT\u0026CK...","t":"2024-07-08T12:29:27.874691834+09:00"}
{"lvl":"info","msg":"Fetching CAPEC...","t":"2024-07-08T12:29:28.492028426+09:00"}
{"lvl":"info","msg":"Fetching CWE...","t":"2024-07-08T12:29:29.287096166+09:00"}
{"lvl":"info","msg":"Fetching NVD CVE...","t":"2024-07-08T12:29:31.91293125+09:00","year":"recent"}
{"lvl":"info","msg":"Fetching NVD CVE...","t":"2024-07-08T12:29:33.106463894+09:00","year":"modified"}
{"lvl":"info","msg":"Fetching NVD CVE...","t":"2024-07-08T12:29:34.529672045+09:00","year":"2002"}
{"lvl":"info","msg":"Fetching NVD CVE...","t":"2024-07-08T12:29:36.192727447+09:00","year":"2003"}
{"lvl":"info","msg":"Fetching NVD CVE...","t":"2024-07-08T12:29:37.44145785+09:00","year":"2004"}
{"lvl":"info","msg":"Fetching NVD CVE...","t":"2024-07-08T12:29:38.85855944+09:00","year":"2005"}
{"lvl":"info","msg":"Fetching NVD CVE...","t":"2024-07-08T12:29:40.519574104+09:00","year":"2006"}
{"lvl":"info","msg":"Fetching NVD CVE...","t":"2024-07-08T12:29:42.440015848+09:00","year":"2007"}
{"lvl":"info","msg":"Fetching NVD CVE...","t":"2024-07-08T12:29:44.453625678+09:00","year":"2008"}
{"lvl":"info","msg":"Fetching NVD CVE...","t":"2024-07-08T12:29:46.3902562+09:00","year":"2009"}
{"lvl":"info","msg":"Fetching NVD CVE...","t":"2024-07-08T12:29:48.341825283+09:00","year":"2010"}
{"lvl":"info","msg":"Fetching NVD CVE...","t":"2024-07-08T12:29:50.441896972+09:00","year":"2011"}
{"lvl":"info","msg":"Fetching NVD CVE...","t":"2024-07-08T12:29:52.442889104+09:00","year":"2012"}
{"lvl":"info","msg":"Fetching NVD CVE...","t":"2024-07-08T12:29:54.436492469+09:00","year":"2013"}
{"lvl":"info","msg":"Fetching NVD CVE...","t":"2024-07-08T12:29:56.473025341+09:00","year":"2014"}
{"lvl":"info","msg":"Fetching NVD CVE...","t":"2024-07-08T12:29:58.647378813+09:00","year":"2015"}
{"lvl":"info","msg":"Fetching NVD CVE...","t":"2024-07-08T12:30:00.600754217+09:00","year":"2016"}
{"lvl":"info","msg":"Fetching NVD CVE...","t":"2024-07-08T12:30:03.270316962+09:00","year":"2017"}
{"lvl":"info","msg":"Fetching NVD CVE...","t":"2024-07-08T12:30:06.217385423+09:00","year":"2018"}
{"lvl":"info","msg":"Fetching NVD CVE...","t":"2024-07-08T12:30:09.283978718+09:00","year":"2019"}
{"lvl":"info","msg":"Fetching NVD CVE...","t":"2024-07-08T12:30:11.92137863+09:00","year":"2020"}
{"lvl":"info","msg":"Fetching NVD CVE...","t":"2024-07-08T12:30:15.092320612+09:00","year":"2021"}
{"lvl":"info","msg":"Fetching NVD CVE...","t":"2024-07-08T12:30:19.052209778+09:00","year":"2022"}
{"lvl":"info","msg":"Fetching NVD CVE...","t":"2024-07-08T12:30:24.120506955+09:00","year":"2023"}
{"lvl":"info","msg":"Fetching NVD CVE...","t":"2024-07-08T12:30:28.288916675+09:00","year":"2024"}
{"attackers":853,"lvl":"info","mappings":134372,"msg":"Fetched Cyber Threat Intelligence and CVE-ID to CTI-ID Mappings","t":"2024-07-08T12:30:31.371298381+09:00","techniques":1196}
{"db":"sqlite3","lvl":"info","msg":"Insert Cyber Threat Intelligences and CVE-ID to CTI-ID Mappings into go-cti.","t":"2024-07-08T12:30:31.371377531+09:00"}
{"lvl":"info","msg":"Inserting Cyber Threat Intelligences...","t":"2024-07-08T12:30:31.458984546+09:00"}
{"lvl":"info","msg":"Inserting Techniques...","t":"2024-07-08T12:30:31.459039847+09:00"}
{"lvl":"info","msg":"Inserting CVE-ID to CTI-ID CveToTechniques...","t":"2024-07-08T12:30:33.017451109+09:00"}
{"lvl":"info","msg":"Inserting Attackers...","t":"2024-07-08T12:30:48.190946414+09:00"}

$ go-cti fetch threat
INFO[07-08|12:31:00] Fetching Cyber Threat Intelligence and CVE-ID to CTI-ID Mappings 
INFO[07-08|12:31:00] Fetching MITRE ATT&CK... 
INFO[07-08|12:31:00] Fetching CAPEC... 
INFO[07-08|12:31:00] Fetching CWE... 
INFO[07-08|12:31:03] Fetching NVD CVE...                      year=recent
INFO[07-08|12:31:04] Fetching NVD CVE...                      year=modified
INFO[07-08|12:31:05] Fetching NVD CVE...                      year=2002
INFO[07-08|12:31:07] Fetching NVD CVE...                      year=2003
INFO[07-08|12:31:08] Fetching NVD CVE...                      year=2004
INFO[07-08|12:31:10] Fetching NVD CVE...                      year=2005
INFO[07-08|12:31:12] Fetching NVD CVE...                      year=2006
INFO[07-08|12:31:14] Fetching NVD CVE...                      year=2007
INFO[07-08|12:31:16] Fetching NVD CVE...                      year=2008
INFO[07-08|12:31:18] Fetching NVD CVE...                      year=2009
INFO[07-08|12:31:20] Fetching NVD CVE...                      year=2010
INFO[07-08|12:31:22] Fetching NVD CVE...                      year=2011
INFO[07-08|12:31:24] Fetching NVD CVE...                      year=2012
INFO[07-08|12:31:27] Fetching NVD CVE...                      year=2013
INFO[07-08|12:31:29] Fetching NVD CVE...                      year=2014
INFO[07-08|12:31:31] Fetching NVD CVE...                      year=2015
INFO[07-08|12:31:33] Fetching NVD CVE...                      year=2016
INFO[07-08|12:31:36] Fetching NVD CVE...                      year=2017
INFO[07-08|12:31:39] Fetching NVD CVE...                      year=2018
INFO[07-08|12:31:42] Fetching NVD CVE...                      year=2019
INFO[07-08|12:31:45] Fetching NVD CVE...                      year=2020
INFO[07-08|12:31:51] Fetching NVD CVE...                      year=2021
INFO[07-08|12:31:57] Fetching NVD CVE...                      year=2022
INFO[07-08|12:32:03] Fetching NVD CVE...                      year=2023
INFO[07-08|12:32:07] Fetching NVD CVE...                      year=2024
INFO[07-08|12:32:11] Fetched Cyber Threat Intelligence and CVE-ID to CTI-ID Mappings techniques=1196 mappings=134372 attackers=853
INFO[07-08|12:32:11] Insert Cyber Threat Intelligences and CVE-ID to CTI-ID Mappings into go-cti. db=sqlite3
INFO[07-08|12:32:11] Inserting Cyber Threat Intelligences... 
INFO[07-08|12:32:11] Inserting Techniques... 
1196 / 1196 [------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------] 100.00% 2113 p/s
INFO[07-08|12:32:12] Inserting CVE-ID to CTI-ID CveToTechniques... 
134372 / 134372 [--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------] 100.00% 9531 p/s
INFO[07-08|12:32:26] Inserting Attackers... 
853 / 853 [--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------] 100.00% 5133 p/s

Checklist:

You don't have to satisfy all of the following.

Write tests
Write documentation
Check that there aren't other open pull requests for the same issue/feature
Format your source code by make fmt
Pass the test by make test
Provide verification config / commands
Enable "Allow edits from maintainers" for this PR
Update the messages below

Is this ready for review?: YES

Reference

https://blog.github.com/2015-01-21-how-to-write-the-perfect-pull-request/

feat(db) no progress when --log-json option

c67480d

sadayuki-matsuno requested a review from MaineK00n July 3, 2024 06:23

MaineK00n changed the title ~~feat(db) no progress when --log-json option~~ feat(log) no progress when --log-json option Jul 4, 2024

MaineK00n approved these changes Jul 8, 2024

View reviewed changes

MaineK00n merged commit 9193da2 into master Jul 8, 2024
3 checks passed

MaineK00n deleted the no_progress_when_log_json branch July 8, 2024 03:33

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(log) no progress when --log-json option #116

feat(log) no progress when --log-json option #116

sadayuki-matsuno commented Jul 3, 2024 •

edited by MaineK00n

Loading

feat(log) no progress when --log-json option #116

feat(log) no progress when --log-json option #116

Conversation

sadayuki-matsuno commented Jul 3, 2024 • edited by MaineK00n Loading

What did you implement:

Type of change

How Has This Been Tested?

before

after

Checklist:

Reference

sadayuki-matsuno commented Jul 3, 2024 •

edited by MaineK00n

Loading