Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Article describing how to create/use a custom typing schema for Zeek NDJSON ingest #696

Closed
philrz opened this issue Apr 28, 2020 · 2 comments · Fixed by #735
Closed

Article describing how to create/use a custom typing schema for Zeek NDJSON ingest #696

philrz opened this issue Apr 28, 2020 · 2 comments · Fixed by #735
Assignees

Comments

@philrz
Copy link
Contributor

philrz commented Apr 28, 2020

The JSON typing schema that will ship with the next Brim release is based on a "stock" Zeek v3.1.2 configuration. If a user has logs that were generated in a Zeek environment that has been customized at all, their set of log files & fields is likely to vary from what's defined in this schema. If they then attempt to ingest any Zeek NDJSON logs from the environment, they will experience warnings/errors and not all data will be ingested.

To get ahead of Support issues, we'll write an article for the wiki that describes how to use zq to customize the schema and use it in the app. In the article we'll encourage users to come talk to us on Slack if they're doing Zeek NDJSON ingest, since it will help us prioritize how much we invest in further improving the customization experience.

Perhaps we could also link to the article from the error/warning messages in the app.

@philrz philrz self-assigned this Apr 28, 2020
@philrz
Copy link
Contributor Author

philrz commented May 3, 2020

brimdata/super#716 has a doc for the zq side, and after that's approved/merged, I'll write a corresponding doc for the Brim wiki.

@philrz
Copy link
Contributor Author

philrz commented May 4, 2020

#735 has the doc for the Brim wiki.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant