Fix validation of subfields of flattened objects

[jsoriano]

When a document doesn't has subobjects, like when subobjects: false or synthetic mode is used, if an undocumented field is found, we need to check its ancestors to see if it is a member of a flattened object.

This is producing failures for auditd_manager package when LogsDB is enabled: https://buildkite.com/elastic/integrations/builds/15559

test case failed: one or more errors found in documents stored in logs-auditd_manager.auditd-20702 data stream: [0] field "auditd.data.a0" is undefined, could be a multifield
[1] field "auditd.data.a1" is undefined, could be a multifield
[2] field "auditd.data.a2" is undefined, could be a multifield
[3] field "auditd.data.a3" is undefined, could be a multifield

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: f37d7a2

cc @jsoriano

[mrodm]

If it finds an ancestor with a different type of "flattened", should this return false directly here?

I was thinking in the case of the ancestor being something like: ancestor.Type = object and ancestor.ObjectTYpe == "keyword"

[jsoriano]

Not sure, there are packages defining mappings for subfields of flattened objects (for example here). This doesn't make a lot of sense as a mapping, but this is allowed now, and can be used for example to provide documentation.

We probably should not allow to define mappings under flattened fields, but this would be a different task, and checked in a different place. And we would need to provide something for cases where developers want to provide docs for flattened subfields.

[mrodm]

Not sure, there are packages defining mappings for subfields of flattened objects (for example here). This doesn't make a lot of sense as a mapping, but this is allowed now, and can be used for example to provide documentation.

Ok, I didn't know about this kind of usage.

We probably should not allow to define mappings under flattened fields, but this would be a different task, and checked in a different place. And we would need to provide something for cases where developers want to provide docs for flattened subfields.

Agreed!

[mrodm]

Could this be replaced by findParentElementDefinition ?

Suggested change

	i := strings.LastIndex(key, ".")
	key = key[:i]
	ancestor := FindElementDefinition(key, schema)
	ancestor := findParentElementDefinition(key, schema)

[mrodm]

mmm I guess it would be needed to somehow return also the key of the field, or being able to retrieve it from the ancestor struct.

[jsoriano]

Yep, this is the problem I found, we still need to loop over the ancestors.

[mrodm]

LGTM! Checked locally too with logsdb enabled for audtid_manager package.