[DerivedField] object type support in mappings and new settings for derived field

[rishabhmaurya]

Description

Introduces object type definition and following settings -

"properties": {
   <field>: <type>
},
"source_indexed_field": <field_name>
"format": <date-format>
"ignore_malformed": <boolean>

source_indexed_field: This indexed field should be of type text and will be used to filter documents in 2 phased iterator approach before executing the expensive DerivedFieldQuery fetching from source, running script, deriving value, creating memory index. This is to improve the performance for object type.
properties: fieldname and field type can be specified explicity to avoid inferring the type of sub fields within object type.
format: date format to be used when type is inferred. Unrelated to object type and applicable to date type for both nested and non-nested fields.
ignore_malformed: ignores malformed documents for cases when field value cannot be derived or there is a type mismatch. If set to true, the query will ignore such documents and proceeds with rest of the matching documents.

Example -

{
  "mappings": {
    "properties": {
      "specs_json": { "type": "text" }
    },
    "derived": {
      "derived_specs": {
        "type": "object",
        "script": "emit(params._source[\"specs_json\"])",
        "format": "dd-MM-yyyy",
        "ignore_malformed": true,
        "properties": {
                "details.edition": "keyword",
                "details.pages": "long"
        },
        "source_indexed_field": "specs_json"
      }
    }
  }
}

Diff will reduce once preceding PR is merged: #13592

Related Issues

Resolves #13715

#13592

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• API changes companion pull request created.
• Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
• Commits are signed per the DCO using --signoff
• [ ] Commit changes are listed out in CHANGELOG.md file (See: Changelog)
• [ ] Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

❌ Gradle check result for b5fd4b9: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

✅ Gradle check result for 36cae9d: SUCCESS

[msfroh]

This looks good!

Once we merge #13592, I think you'll need to rebase this (to avoid the redundant classes). I'll let you know once I've merged that one, then you can let me know once you've rebased this one.

[github-actions]

✅ Gradle check result for 6e35862: SUCCESS