Report generation from jsonlines broken

[leonjza]

          > Not sure why the results count=0 🤔

Hi,
I was experimenting with gowitness just today and it looks like the --db-uri and --json-file are ignored altogether when generating reports.

For instance, this is what happens when I save data in a jsonl file and then I try to generate a report. Similarly, saving a sqlite3 db with a regular or custom name would work, but loading said DB from a directory other than the current one would fail when generating a report.

$ gowitness scan single -u https://google.com --driver gorod --write-jsonl
2024/09/30 23:39:35 INFO result 🤖 target=https://google.com status-code=200 title=Google have-screenshot=true

$ ls -l
total 680
-rw-r--r--  1 user  staff  345752 30 Sep 23:39 gowitness.jsonl
drwxr-xr-x  3 user  staff      96 30 Sep 23:39 screenshots

$ gowitness report list --json-file ./gowitness.jsonl
╭─────────────────┬────────┬──────┬────────────────────┬────────┬───────┬─────┬─────┬────────┬────────╮
│ When            │ Failed │ Code │ Input URL          │ Title  │ ~Size │ Net │ Con │ Header │ Cookie │
├─────────────────┼────────┼──────┼────────────────────┼────────┼───────┼─────┼─────┼────────┼────────┤
│ Sep 30 23:39:27 │ false  │ 200  │ https://google.com │ Google │ 1kb   │ 38  │ 0   │ 17     │ 2      │
╰─────────────────┴────────┴──────┴────────────────────┴────────┴───────┴─────┴─────┴────────┴────────╯

$ gowitness report generate --json-file ./gowitness.jsonl
2024/09/30 23:40:05 INFO generating HTML report for results count=0
2024/09/30 23:40:05 INFO report zip file generated successfully path=gowitness-report.zip

$ ls -l
total 1304
-rw-r--r--  1 user  staff  234722 30 Sep 23:45 gowitness-report.zip
-rw-r--r--  1 user  staff  356597 30 Sep 23:45 gowitness.jsonl
-rw-r--r--  1 user  staff   69632 30 Sep 23:45 gowitness.sqlite3
drwxr-xr-x  3 user  staff      96 30 Sep 23:45 screenshots

Note the empty gowitness.sqlite3 that gets created after gowitness report generate.

As a workaround, I let gowitness write its default db file within the current directory, then generate a report based on that.

$ gowitness scan single -u https://google.com --driver gorod --write-db
2024/09/30 23:47:55 INFO result 🤖 target=https://google.com status-code=200 title=Google have-screenshot=true
$ gowitness report list
╭─────────────────┬────────┬──────┬────────────────────┬────────┬───────┬─────┬─────┬────────┬────────╮
│ When            │ Failed │ Code │ Input URL          │ Title  │ ~Size │ Net │ Con │ Header │ Cookie │
├─────────────────┼────────┼──────┼────────────────────┼────────┼───────┼─────┼─────┼────────┼────────┤
│ Sep 30 23:47:50 │ false  │ 200  │ https://google.com │ Google │ 1kb   │ 38  │ 0   │ 17     │ 2      │
╰─────────────────┴────────┴──────┴────────────────────┴────────┴───────┴─────┴─────┴────────┴────────╯
$ gowitness report generate
2024/09/30 23:48:08 INFO generating HTML report for results count=1
2024/09/30 23:48:08 INFO report zip file generated successfully path=gowitness-report.zip

Originally posted by @boozezela in #237 (comment)

[boozezela]

Report generation from a sqlite3 file is broken as well.

(I realized this after I wrote the use cases below) it looks like the report generation functionality does not even attempt to open a database in the first place:

$ gowitness report generate --db-uri sqlite://kjdkkdhjdjjdkjd/snjknjsknjks/
2024/10/01 11:42:31 INFO generating HTML report for results count=0
2024/10/01 11:42:31 INFO report zip file generated successfully path=gowitness-report.zip

Typical use cases would be: generating a report from a gowitness.sqlite3 located in a directory different from the current one, or generating a report from a sqlite3 database with an arbitrary name in the current directory.

Case 1: generate a report from gowitness.sqlite3 saved in a directory other than the current one:

$ mkdir foobar

$ gowitness scan single -u https://google.com --driver gorod --write-db --write-db-uri sqlite://./foobar/gowitness.sqlite3
2024/10/01 11:34:26 INFO result 🤖 target=https://google.com status-code=200 title=Google have-screenshot=true

$ ls -l *
foobar:
total 1416
-rw-r--r--  1 user  staff  671744  1 Oct 11:34 gowitness.sqlite3

screenshots:
total 576
-rw-r--r--  1 user  staff  294759  1 Oct 11:34 httpsgoogle.com.jpeg

$ gowitness report list --db-uri sqlite://./foobar/gowitness.sqlite3
╭────────────────┬────────┬──────┬────────────────────┬────────┬───────┬─────┬─────┬────────┬────────╮
│ When           │ Failed │ Code │ Input URL          │ Title  │ ~Size │ Net │ Con │ Header │ Cookie │
├────────────────┼────────┼──────┼────────────────────┼────────┼───────┼─────┼─────┼────────┼────────┤
│ Oct 1 11:34:22 │ false  │ 200  │ https://google.com │ Google │ 1kb   │ 39  │ 0   │ 17     │ 2      │
╰────────────────┴────────┴──────┴────────────────────┴────────┴───────┴─────┴─────┴────────┴────────╯

$ gowitness report generate --db-uri sqlite://./foobar/gowitness.sqlite3
2024/10/01 11:35:55 INFO generating HTML report for results count=0
2024/10/01 11:35:55 INFO report zip file generated successfully path=gowitness-report.zip

Case 2: generate a report from an arbitrarily named sqlite3 database in the current directory (somethingelse.sqlite3):

I will simply rename gowitness.sqlite3 to somethingelse.sqlite3 for simplicity.

$ gowitness scan single -u https://google.com --driver gorod --write-db
2024/10/01 11:16:24 INFO result 🤖 target=https://google.com status-code=200 title=Google have-screenshot=true

$ ls -l
total 1416
-rw-r--r--  1 user  staff  671744  1 Oct 11:16 gowitness.sqlite3
drwxr-xr-x  3 user  staff      96  1 Oct 11:16 screenshots

$ mv gowitness.sqlite3 somethingelse.sqlite3
$ ll
total 1416
drwxr-xr-x  3 user  staff      96  1 Oct 11:16 screenshots
-rw-r--r--  1 user  staff  671744  1 Oct 11:16 somethingelse.sqlite3

$ gowitness report list --db-uri sqlite://somethingelse.sqlite3
╭────────────────┬────────┬──────┬────────────────────┬────────┬───────┬─────┬─────┬────────┬────────╮
│ When           │ Failed │ Code │ Input URL          │ Title  │ ~Size │ Net │ Con │ Header │ Cookie │
├────────────────┼────────┼──────┼────────────────────┼────────┼───────┼─────┼─────┼────────┼────────┤
│ Oct 1 11:16:20 │ false  │ 200  │ https://google.com │ Google │ 1kb   │ 39  │ 0   │ 17     │ 2      │
╰────────────────┴────────┴──────┴────────────────────┴────────┴───────┴─────┴─────┴────────┴────────╯

$ gowitness report generate --db-uri sqlite://somethingelse.sqlite3
2024/10/01 11:19:57 INFO generating HTML report for results count=0
2024/10/01 11:19:57 INFO report zip file generated successfully path=gowitness-report.zip

Last, but not least, the report list command will create an empty sqlite3 database if one does not exist:

$ rm -f ./foobar/*

$ ls -l ./foobar
total 0

$ gowitness report list --db-uri sqlite://foobar/writeme.sqlite3
╭──────┬────────┬──────┬───────────┬───────┬───────┬─────┬─────┬────────┬────────╮
│ When │ Failed │ Code │ Input URL │ Title │ ~Size │ Net │ Con │ Header │ Cookie │
├──────┼────────┼──────┼───────────┼───────┼───────┼─────┼─────┼────────┼────────┤
╰──────┴────────┴──────┴───────────┴───────┴───────┴─────┴─────┴────────┴────────╯

$ ls -l ./foobar
total 136
-rw-r--r--  1 user  staff  69632  1 Oct 11:47 writeme.sqlite3

There are also a couple of use cases where gowitness will complain about out of memory errors when it cannot find and open a sqlite3 database file, or write one (e.g. when the destination directory does not exist).

[leonjza]

Spotted the issue. Will fix this in the morning.