fields starting with underscore ( _ ) doesn't show.

[jeesim2]

hi.

Fields that starting with underscore ( _ ) doesn't show in kibana 4.
for example _class , EventID
I've tried reload fields list but doesn't help.
(neither in visualize panel it doesn't show )

At elasticsearch it really exist at every indices(mapping and data)

Versions.
ES 1.4.0
Kibana 1.4 beta 3

[rashidkpc]

Hi,
We reserve fields beginning with _ for use within elasticsearch, outside of the _source of a document. While elasticsearch does not enforce this rule, we do in Kibana.

[jeesim2]

I've never heard about that rule from any documents..
it is an blocking issue to me for upgrade kibana4.
because very large data in production have fields begginig with _

[shangan]

I do not think this is necessary, underscore field is commonly used in practical process.

[ulir]

We also use underscores, actually as the start for most of our fields, because of naming conventions and dynamic templates that go with it.
A way to configure this away and show those _fields would really help.

[leg100]

The vast majority of the fields I have in elasticsearch have an underscore prefix. We consume systemd journal's JSON output and store it in elasticsearch, and the journal names its fields with an underscore prefix, e.g.:

"_SYSTEMD_CGROUP" : "/", "_MACHINE_ID" : "d85bd5b7c13d4549b3d3d1719d3a344a", "_HOSTNAME" : "ip-172.13.3.3-amazonaws.com", "MESSAGE" : "Starting Generate /run/coreos/motd..."

I'd be interested to hear the justification for excluding them.

[algesten]

This is a terrible decision. We use _-prefixed fields to mean "internally for our own processing, not for public consumption". However those fields are very relevant in making our kibanas.

@rashidkpc
You say

elasticsearch does not enforce this rules, we do in kibana

Could you please elaborate on why kibana should enforce it when elastic doesn't?!

[mjgp2]

Come on guys, this has "it's not a bug, it's a feature" written all over it :)

You can't expect people to re-engineer their valid elasticsearch indexes in order to use kibana. Reading between the lines it looks like you have no way of distinguishing between system fields and object properties, so if there's a collison it's ambiguous (on _id for example).

Surely the aim should be to support any valid elasticsearch index rather than a defined subset of valid indexes according to some criteria you choose? If so this is a bug - it may be low priority, but it's still a bug.

If you want to make "_" prefixes disallowed in ES go ahead - then at least it would be consistent and people wouldn't be scratching their heads wondering why this happens.

[andrewwebber]

i pipe journald JSON logs on every app server directly into elastic search via logstash.

journalctl --boot --no-tail -f -o json | /usr/bin/ncat logstashserver json-codec-port

it was quite nice to view all the systemd properties in kibana3. This is no longer possible to visualize in kibana4. So i should map all journald fields and remove underscores? Some examples:

_MACHINEID
_SYSTEMD_UNIT
_PID
_KERNAL_DEVICE
_KERNEL_SUBSYSTEM
__REALTIME_TIMESTAMP
etc.

Notice some have double underscores :(

[algesten]

Hm. If we could locate where in the code this rule is enforced, it would be worth making a fork.

[algesten]

#4291 and a pull request. Even if it isn't entirely right, I hope we at least can get @rashidkpc or someone to at least enter into a discussion about it.

[andrewwebber]

this was my poor mans work around for systemd however i would need to dynamically generate this as systemd evolves :(

filter {
  mutate {
    rename => { "_MACHINE_ID" => "MACHINE_ID" }
    rename => { "_SYSTEMD_UNIT" => "SYSTEMD_UNIT" }
    rename => { "_SYSTEMD_SLICE" => "SYSTEMD_SLICE" }
    rename => { "_SYSTEMD_CGROUP" => "SYSTEMD_CGROUP" }
    rename => { "_PID" => "PID" }
    rename => { "_EXE" => "EXE" }
    rename => { "_CMDLINE" => "CMDLINE" }
    rename => { "_BOOT_ID" => "BOOT_ID" }
  }
}

I have had to choose between this 'bug' and exporting dashboards it would be the later.

[algesten]

So there is a way forward. #4291 (comment)

[thesmart]

+1 on this, decisions like this make no sense.

[hasJerry]

Is there an open issue for this somewhere? Doesn't seem rational to carry this restriction forward for all time.

[blendle-hubbit]

Heya!

This issue was just mentioned in the Trello card "(3) Store all enriched items in new ES index [MS - 9-6] [blendle-premium]" on the Blendle Data board.

Might be relevant to you, or not...

Happy Hackin'!

[raviv]

+1 for removing this restriction

[andrewthad]

This restriction is a problem for a piece of software I am working on. I'm in favor of removing it.

[ptambe]

+1 for removing this restriction

[tugrulcan]

+1 !

[cabsanchez]

Is this aready fixed in a released version of Kibana?

[lanceblais]

I know this is closed, but I'm also curious in whether or not this has been released?

[alxchk]

This creates an issue with rsyslog elasticsearch plugin

[ychennay]

I can confirm this "feature" is NOT "fixed". Just spent the last couple hours retracing my steps as to why certain fields weren't appearing within Kibana until I found this post.

[ederparaiso]

when this fix will be released?

[joelachance]

'_' isn't a good namespace to privately reserve. It's very frustrating to see this ticket was opened 5 years ago, and there's been zero moment here.

While it appears everyone here can agree this was a terrible decision, any tools to help remedy this could be helpful, which would help a lot of people.

This is also blocking for me.

[paulwilton]

Yes, this really does need to be addressed, not swept under the carpet. Its a blocker for us too.
Given the entire raison d'être of Kibana is to integrate with ES - only working on partial data defeats its primary purpose.
Bonkers.

[tliang03]

confirmed with v6.8. Issue not fixed yet. Anybody has any luck on this?

[SKRYMr]

+1 to fix this, it's a limitation that introduces incongruity between Elasticsearch and Kibana and it's a pain to discover it in production (since it's not mentioned in the docs either) when the workaround would be to migrate all the data to a new structure.

[mattkime]

I think we should be able to do this, but need to verify that we're not exposing unwanted fields to the user.

[elasticmachine]

Pinging @elastic/kibana-app-services (Team:AppServices)

[timroes]

@mattkime my promised brain dump 🧠

I think we have two slightly separate issues here from which we should at least address the first:

Regular fields beginning with underscore are not shown by default

This is most likely simply an artifact of us not being able to make any other distringtion between "meta fields" and regular fields in earlier days of ES than checking if they begin with an underscore. This is no longer the case, since _field_caps actually returns us an metadata_field: boolean property for each field. Thus I think we should by default never filter out fields that begin with an underscore, as long as metadata_field: false on them, and treat them as regular fields always. I think that part won't come with any breakage or problems (since those are normal fields), but will simply solve the cases were we currently accidentally hide fields beginning with underscore.

Example mapping extract (`_event` is a regular field I put in the mapping)

"_event" : {
  "keyword" : {
    "type" : "keyword",
    "metadata_field" : false,
    "searchable" : true,
    "aggregatable" : true
  }
},
"_source" : {
  "_source" : {
    "type" : "_source",
    "metadata_field" : true,
    "searchable" : false,
    "aggregatable" : false
  }
},
"_id" : {
  "_id" : {
    "type" : "_id",
    "metadata_field" : true,
    "searchable" : true,
    "aggregatable" : true
  }
},

Change the meta field include list

Even with the first change made, we can still consider how we want to treat actual meta fields (metadata_field: true). Currently we by default exclude them and only include them if they are part of the metaFields advanced setting. This can be confusing, e.g. when you install a new plugin that adds a _size attribute and wonder why you don't see it in Kibana.

I think we have basically two ways we can handle that in Kibana:

• Continue with an "inclusion" list as of today. Though we can revisit all the current meta fields and see if we might want to extend the default value of it.
• Switch to an "exclusion" list instead, and by default handle all meta fields, unless some specific ones we know we don't want to be treated "as regular fields", because they would cause problems or are handled explicitly in other ways (e.g. _ignored is something I would by default not expect to be treated like a "regular field").

For completion here is a list of all meta fields we currently return via field_caps that we should consider:

• _routing
• _doc_count
• _index
• _feature (Can't find documentation for that)
• _tier
• _ignored
• _seq_no (Can't find documentation for that)
• _field_names
• _data_stream_timestamp (No documentation found)
• _source
• _id
• _version (No documentation found)
• _size (Only when mapper-size plugin is installed)

Looking at that list I have the feeling the most fields really wouldn't benefit from a default inclusion, so we might be fine keep working with an "Inclusion" list instead switching to an "exclusion" one, but might want to add _size into the default value?