Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create a logs@custom index template reference #3593

Merged
merged 4 commits into from
Feb 20, 2024

Conversation

mdbirnstiehl
Copy link
Contributor

this PR closes #3393

Create a log@custom index template guide that also references how to update the default_field to message after an update has changed the default behavior to [*].

@mdbirnstiehl mdbirnstiehl self-assigned this Feb 2, 2024
Copy link
Contributor

github-actions bot commented Feb 2, 2024

A documentation preview will be available soon:

Copy link
Contributor

mergify bot commented Feb 2, 2024

This pull request does not have a backport label. Could you fix it @mdbirnstiehl? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

  • backport-/d./d is the label to automatically backport to the /d./d branch. /d is the digit
    NOTE: backport-skip has been added to this pull request.

@mergify mergify bot added the backport-skip Skip notification from the automated backport with mergify label Feb 2, 2024
@mdbirnstiehl mdbirnstiehl added backport-8.12 Automated backport with mergify and removed backport-skip Skip notification from the automated backport with mergify labels Feb 2, 2024
@mdbirnstiehl
Copy link
Contributor Author

@eyalkoren @ruflin I've created this PR as a first iteration for the logs@custom reference to show how to update the default_field. Potentially, we can include other common uses. I think it would be helpful to add the "why" users might want to use the message field over the [*], but I was unsure of that.

Copy link
Member

@ruflin ruflin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@zmoog @gsantoro Can we get your review on this?

docs/en/observability/logs-index-template.asciidoc Outdated Show resolved Hide resolved
docs/en/observability/logs-index-template.asciidoc Outdated Show resolved Hide resolved
docs/en/observability/logs-index-template.asciidoc Outdated Show resolved Hide resolved
docs/en/observability/logs-index-template.asciidoc Outdated Show resolved Hide resolved
* `logs@custom`
* `ecs@mappings`

You can use the `@custom` component template to customize your {es} indices. The `logs@custom` template is not installed by default, but you can create a template named `logs@custom` to override and extend default mappings or settings.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I currently get this error because logs@custom doesn't exist. I'm using this version of the stack 8.13.0-f56d9bd5-SNAPSHOT released on 2024-02-05

{
  "statusCode": 400,
  "error": "Bad Request",
  "message": "index_template [simulate_template_cjga1q24s0swusgjolwsba] invalid, cause [index template [simulate_template_cjga1q24s0swusgjolwsba] specifies component templates [logs@custom] that do not exist]",
  "attributes": {
    "error": {
      "root_cause": [
        {
          "type": "invalid_index_template_exception",
          "reason": "index_template [simulate_template_cjga1q24s0swusgjolwsba] invalid, cause [index template [simulate_template_cjga1q24s0swusgjolwsba] specifies component templates [logs@custom] that do not exist]"
        }
      ],
      "type": "invalid_index_template_exception",
      "reason": "index_template [simulate_template_cjga1q24s0swusgjolwsba] invalid, cause [index template [simulate_template_cjga1q24s0swusgjolwsba] specifies component templates [logs@custom] that do not exist]"
    }
  }
}

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've encountered this at some point (see thread) but couldn't replicate it today after testing several versions (didn't test snapshots)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@gbamparop I could use someone else trying the latest snapshot to see if it is just me or we need to open a bug about this.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@gbamparop I have added some more info to this thread so that someone else can replicate it.

docs/en/observability/logs-index-template.asciidoc Outdated Show resolved Hide resolved
You can update the `default_field` to search in the `message` field instead of all fields using the `logs@custom` component template.

Either create the `logs@custom` template as detailed in the previous section, or edit the existing one to include the following in the *Index settings*:
Either create the `logs@custom` component template as outlined in the previous section, or add the following code to the *Index settings* of the `logs` index template:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ruflin, what do you think about this sentence?

Personally, I wouldn't suggest fiddling with the logs index template but only changing the custom component template.

Thoughts? @gbamparop

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think maybe the initial sentence wasn't clear. My initial intention was to say "Create a logs@custom component template and add the following to the index settings. If you've already created a logs@custom component template, you can edit it and add the following to the index settings." I didn't mean to suggest editing the logs template.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok. sorry it wasn't 100% clear.

I would rephrase as "If you don't already have a logs@custom component template you can create it manually. Once you have it, you can add the following to the settings section"

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for pointing it out. I updated it to be clear and add steps for updating the template. I guess because of the way the search works, you can't search for anything using the "@" symbol? or it throws the error:
image

Not sure if there's an underlying reason for this, but it does make it pretty difficult to find any custom templates.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mdbirnstiehl Yes, I have experienced this. You can't search for "logs@custom". Either for logs or for custom and then iterate through the many pages

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That sounds like a bug especially as we use it in our names. Has anyone reported this? If not, lets do it.

@gsantoro
Copy link
Contributor

@mdbirnstiehl thanks for the changes.

I just have two questions left on this PR, as soon as we have an answer to them we can merge this PR

@gsantoro
Copy link
Contributor

@mdbirnstiehl just to summarize:

  • we need to open a bug for Kibana about not being able to search logs@custom because of the character @. Can you do this?
  • about the other Kibana bug about logs@custom missing, I have provided more info in the linked thread by @gbamparop .

Those bugs are unrelated to this PR. so I have approved this PR now.

@mdbirnstiehl mdbirnstiehl marked this pull request as ready for review February 15, 2024 17:17
@mdbirnstiehl mdbirnstiehl requested a review from a team as a code owner February 15, 2024 17:17
@gsantoro
Copy link
Contributor

/test

@mdbirnstiehl
Copy link
Contributor Author

run docs-build

@mdbirnstiehl mdbirnstiehl merged commit ecf1c57 into elastic:main Feb 20, 2024
4 checks passed
@mdbirnstiehl mdbirnstiehl deleted the custom-logs-template branch February 20, 2024 22:31
mergify bot pushed a commit that referenced this pull request Feb 20, 2024
mdbirnstiehl added a commit that referenced this pull request Feb 20, 2024
(cherry picked from commit ecf1c57)

Co-authored-by: Mike Birnstiehl <114418652+mdbirnstiehl@users.noreply.github.com>
@colleenmcginnis colleenmcginnis added the backport-8.13 Automated backport with mergify label Mar 5, 2024
mergify bot pushed a commit that referenced this pull request Mar 5, 2024
(cherry picked from commit ecf1c57)

# Conflicts:
#	docs/en/observability/logs-checklist.asciidoc
colleenmcginnis added a commit that referenced this pull request Mar 5, 2024
…3649)

* Create a logs@custom index template reference (#3593)

(cherry picked from commit ecf1c57)

# Conflicts:
#	docs/en/observability/logs-checklist.asciidoc

* resolve merge conflicts

* Update docs/en/observability/logs-checklist.asciidoc

---------

Co-authored-by: Mike Birnstiehl <114418652+mdbirnstiehl@users.noreply.github.com>
Co-authored-by: Colleen McGinnis <colleen.mcginnis@elastic.co>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport-8.12 Automated backport with mergify backport-8.13 Automated backport with mergify
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Logs+] Create a logs@custom reference
6 participants