Max messages could be higher (or not exist at all)

[predictiple]

extract_windows.go#L176 causes extraction to skip some potentially useful DLLs.

Without the restriction the extraction seems to max out at 65535 (see logs in my latest PR) which seems pretty reasonable and also preferable to ignoring such DLLs completely.

Is the limit just there to keep the extraction results down to a convenient size or is it a safeguard against some more serious condition?

[scudette]

I think I added that limit because I found some dlls that were doing things like, add 65535 %1 messages in the message table. I guess they were too lazy to make proper messages and were just going to send their messages as an event arg. I don't want to fill up the database with useless messages :-)

…

On Wed, Nov 27, 2019, 05:08 predictiple ***@***.***> wrote: extract_windows.go#L176 <https://github.com/Velocidex/evtx/blob/191dc946afdfe19b5447bc2dadd0f31f3ecb66f0/cmd/extract_windows.go#L176> causes extraction to skip some potentially useful DLLs. Without the restriction the extraction seems to max out at 65535 (see logs in my latest PR) which seems pretty reasonable and also preferable to ignoring such DLLs completely. Is the limit just there to keep the extraction size down or is it a safegaurd against some more serious condition? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#8?email_source=notifications&email_token=AA5NRIV2G4DVCYDV3XH25VLQVVX4VA5CNFSM4JR4RIW2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4H4GU4FQ>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA5NRIU56EFJANR2F77WODTQVVX4VANCNFSM4JR4RIWQ> .

[predictiple]

ok, and yes I have seen such dlls. It's unlikely that any developers are going to craft more than 10k individual messages so probably the 10k constraint is reasonable. I just get nervous when I see things being excluded. For dlls that do just repeat the same values, the sqlite db tables' uniqueness constraint will deduplicate that. So it's really just about wasted CPU cycles.

Not sure if you've noticed the 2nd PR that I've submited for the evtx-data repo? Those db files were all extracted without the 10k limit and there are extraction logs for each where you can see which dlls are hitting 65535.

[predictiple]

that just reminded me to mention that in the logging there's a typo: "Openning"

[scudette]

Yeah that's what I thought. Who really has the need to make 10k specific error messages ? The current implementation doesn't actually deduplicate anything - sometimes I have seen different messages for the same event id and provider. Some even with and without the event data interpolation. I don't think you can dedup much of the 64k messages because they need to deduped on the combination of event id provider and channel. So I'm not sure if we should deduplicate or not?

…

On Wed, Nov 27, 2019, 14:56 predictiple ***@***.***> wrote: ok, and yes I have seen such dlls. It's unlikely that any developers are going to craft more than 10k individual messages so probably the 10k constraint is reasonable. I just get nervous when I see things being excluded. For dlls that do just repeat the same values, the sqlite db tables' uniqueness constraint will deduplicate that. So it's really just about wasted CPU cycles. Not sure if you've noticed the 2nd PR that I've submited for the evtx-data repo? Those db files were all extracted without the 10k limit and there are extraction logs for each where you can see which dlls are hitting 65535. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#8?email_source=notifications&email_token=AA5NRIQYF7RYJUK3VBPDMZLQVX4WXA5CNFSM4JR4RIW2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFIJECI#issuecomment-558928393>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA5NRIRS3L7U26326NOXZVDQVX4WXANCNFSM4JR4RIWQ> .

[predictiple]

It doesn't look like we store channel. I thought event ids are unique (in theory) per provider so the combination of eventID and provider name implies the channel. In my Python script that merges the dbs I apply uniqueness constraint on the rows of each table, so it does deduplicate.