Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ts-free records are excluded from span when mixed with time-having points #2088

Closed
philrz opened this issue Feb 7, 2021 · 2 comments · Fixed by brimdata/zui#2785
Closed
Labels
bug Something isn't working community

Comments

@philrz
Copy link
Contributor

philrz commented Feb 7, 2021

A community user spotted this issue while working with logs from the Zeek SPL-SPT package as described in the join cookbook (#1430). Repro is with a nightly build based on Brim commit 664840a8.

One of the quirks of the SPL-SPT package is that it generates points that lack a ts timestamp field. It's for this reason that I used it as an example in the join cookbook, since it's a good candidate for joining to other records that do have ts timestamps. However, not all users will know that ts-free records currently pose a unique challenge in Brim. They just think of Brim as the app that can handle all Zeek TSV logs you throw at it.

What we observed is that if only the spl.log.gz (attached) is dragged into Brim, the time picker defaults to "epoch zero" time and the points show up ok.

image

image

However, if we drag that spl.log.gz as well as another Zeek log such as conn.log.gz (attached) now the time picker is set to the range of that log and the spl events effectively become invisible.

image

image

If I happen to know what's going on and manually set the start point of the time picker to epoch zero, then I get to see them. But I would not expect our users to know about this.

image

We know we need to do other work to make these kinds of ts-free events easier to work with in the app. But since at least one user has already bumped into it, we might want to see if we can take some interim step to reduce the likelihood of users floating it as a bug.

@philrz philrz transferred this issue from brimdata/zui Feb 9, 2021
@philrz philrz added bug Something isn't working community labels Feb 9, 2021
@philrz
Copy link
Contributor Author

philrz commented Feb 9, 2021

We discussed this one as a group today and it seen as a zqd-side bug. This becomes visible when querying the Space info via zapi. For the case where we imported only the spl.log.gz, the span is shown as <nil>. Consensus was that this was a "zero value problem" being treated the wrong way, e.g. this should have been showing the "epoch zero" 1970 time plus 1 nanosecond.

$ zapi info spl.log.gz
spl.log.gz
  id:           sp_1oG7ttYHuuJuGS5cXEVAFzqQooi
  parent_id:    
  name:         spl.log.gz
  data_path:    file:///Users/phil/Library/Application%20Support/Brim/data/spaces/sp_1oG7ttYHuuJuGS5cXEVAFzqQooi
  storage_kind: filestore
  span:         <nil>
  size:         2.09KB
  pcap_support: false
  pcap_size:    0B
  pcap_path:    

Likewise, for the case where we simultaneously imported spl.log.gz and conn.log.gz, the span should reach all the way back to cover the "epoch zero" range where the spl records live.

$ zapi info cookbook
cookbook
  id:           sp_1oG85qyGCih7ULp8uoL5sjf63a7
  parent_id:    
  name:         cookbook
  data_path:    file:///Users/phil/Library/Application%20Support/Brim/data/spaces/sp_1oG85qyGCih7ULp8uoL5sjf63a7
  storage_kind: filestore
  span:         2020-09-01T20:39:10Z+30m44.654722001s
  size:         10.25KB
  pcap_support: false
  pcap_size:    0B
  pcap_path:    

@philrz philrz changed the title ts-free records are excluded from picker range when mixed with time-having points ts-free records are excluded from span when mixed with time-having points Feb 10, 2021
@philrz
Copy link
Contributor Author

philrz commented Aug 16, 2023

Changes have made it such that this is no longer a problem. Some of the concepts described above are no longer in the software (e.g. "Spaces") but if we up-level, the original symptom of problems when mixing ts-free records with ts-having ones is no longer with us.

The attached video was taken with Zui commit e1126ae and shows the import of the spl.log.gz test data into one pool and then the spl.log.gz and conn.log.gz both into the same pool. When just spl.log.gz is imported it's basically the same as before where the records are shown and there's no stacked bar chart since there's no default ts pool key to form the buckets & X-axis. Then when both spl.log.gz and conn.log.gz are placed in the same pool we see all the records that came from both files. The stacked bar chart is constructed based on the conn.log.gz data that has the ts values that form the pool key, and an info line at the top of the stacked bar chart says "15 missing ts values" to explain why the records that came from spl.log.gz are not shown in the stacked bars. This was covered as a special case thanks to changes that were made in brimdata/zui#2785.

Verify.mp4

Therefore, I'm closing this one as fixed. Thanks @jameskerr!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working community
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant