Field mapping validator #8476
Field mapping validator #8476
Conversation
| // ValidateDocument takes a document from Elasticsearch in JSON format | ||
| // and the contents of a Beat's fields.yml and validated the document's | ||
| // fields against it. | ||
| func ValidateDocument(docJson []byte, fieldsYaml []byte) error { |
There was a problem hiding this comment.
func name will be used as validate.ValidateDocument by other packages, and that stutters; consider calling this Document
func parameter docJson should be docJSON
| return seen, nil | ||
| } | ||
|
|
||
| func (m *Mapping) AddField(path string, field Field, required bool) error { |
There was a problem hiding this comment.
exported method Mapping.AddField should have comment or be unexported
adriansr
force-pushed
the
feature-suricata
branch
4 times, most recently
from
9fcdddc to
b113cd5
Compare
| return nil | ||
| } | ||
|
|
||
| func recursiveFattenFields(fields interface{}, prefix Prefix, mapping *Mapping, key string) error { |
There was a problem hiding this comment.
Reminds me a lot of the code in #7972
adriansr
force-pushed
the
feature-suricata
branch
from
6db805b to
3ab125f
Compare
metricbeat/module/php_fpm/url.go
Outdated
| // specific language governing permissions and limitations | ||
| // under the License. | ||
|
|
||
| package php_fpm |
There was a problem hiding this comment.
don't use an underscore in package name
|
|
||
| // +build !integration | ||
|
|
||
| package node_stats |
There was a problem hiding this comment.
don't use an underscore in package name
| ) | ||
| } | ||
|
|
||
| // OrderIntervalLogs, when given a log filename in the form [prefix]-[formattedDate]-n |
There was a problem hiding this comment.
comment on exported function OrderIntervalLogs should be of the form "OrderIntervalLogs ..."
filebeat/magefile.go
Outdated
| var modulesDirGenerated = filepath.Clean("build/packaging/modules") | ||
| var modulesDirGeneratedOSS = filepath.Clean("build/packaging/modules-oss") | ||
| var modulesDirGeneratedXPack = filepath.Clean("build/packaging/modules-x-pack") | ||
| var modules_D_DirGeneratedXpack = filepath.Clean("build/packaging/modules.d-x-pack") |
There was a problem hiding this comment.
don't use underscores in Go names; var modules_D_DirGeneratedXpack should be modulesDDirGeneratedXpack
adriansr
force-pushed
the
feature-suricata
branch
from
4f4727b to
3ab125f
Compare
This patch adds a document validator that compares an Elasticsearch document in JSON format against a Beat fields in Yaml format. It checks that: - All fields in a document have a mapping. - Fields types match the mapping. - Required fields are present. Currently it only supports a limited subset of datatypes. Needs to be extended with more types.
Learned some fixes from running it against current Beats: - Some sub-fields only use the release tag, for documentation purposes. - object used as synonym for group. - type group without fields tag, treat as empty fields.
adriansr
force-pushed
the
feature-suricata
branch
from
ccdb018 to
e341c52
Compare
|
These commits were merged (by mistake) into master in #8313 Luckily this PR was already green and approved when that happened. Sorry about that. |
This patch adds a document validator that compares an Elasticsearch document in JSON format against a Beat fields in Yaml format.
It checks that:
Currently it only supports a limited subset of datatypes. Needs to be extended with more types.