Elastic APM Services not shown

[ahmedhkhalil]

Kibana version:
8.15

Elasticsearch version:
8.15

Server OS version:
Linux npr-dos-elk01.t2.local 4.18.0-553.16.1.el8_10.x86_64 #1 SMP Thu Aug 8 17:47:08 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Browser version:

Browser OS version:

Original install method (e.g. download page, yum, from source, etc.):

Describe the bug:

Steps to reproduce:

1. upgrade elastic and kibana from v8.14.3 to v15
2. browse APM Service tab

Expected behavior:
find my APM services
Screenshots (if relevant):

Errors in browser console (if relevant):

Provide logs and/or server output (if relevant):
Aug 12 12:47:13 npr-dos-elk01.t2.local kibana[1775]: [2024-08-12T12:47:13.048+03:00][INFO ][plugins.apm] create static data view apm_static_data_view_id_default
Any additional context:

[ahmedhkhalil]

i found below error also in apm fleet
[elastic_agent.apm_server][error] failed to index documents in '.ds-traces-apm-default-2024.08.11-000328' (document_parsing_exception): [1:2611] failed to parse: field name cannot contain only whitespace: ['.AspNetCore.Antiforgery.VyLW6ORzMgk']

[ahmedhkhalil]

@Kerry350 hi can you please check if you can help here

[Kerry350]

@ahmedhkhalil

This belongs to a different team to logs, I can add the relevant labels.

It looks like you have ingestion issues with formatting and mappings.

[elasticmachine]

Pinging @elastic/obs-ux-infra_services-team (Team:obs-ux-infra_services)

[ahmedhkhalil]

i already solved issue in parsing but still page not loaded

[ahmedhkhalil]

i have inspected Page through browser and i suspect below has issue

http://localhost:6444/internal/apm/entities/services?environment=ENVIRONMENT_ALL&kuery=&start=2024-08-14T14%3A00%3A04.890Z&end=2024-08-15T14%3A00%3A04.890Z
retruing empty brackets
http://localhost:6444/internal/apm/has_data
return false
Despite that there is APM data in elastic

[ahmedhkhalil]

any update

[ahmedhkhalil]

hi can you please check if you could help
@jennypavlova

[smith]

i already solved issue in parsing but still page not loaded

Have you checked out the dataset quality page? https://www.elastic.co/guide/en/observability/current/monitor-datasets.html

If there's an issue with the mappings that happened in the course of the upgrade you might be able to see something there.

I don't think there's a Kibana bug here if APM can't access any data, but we might want to investigate more if something in the upgrade caused this problem.

[ahmedhkhalil]

i have checked and found no issue on data qaulity regarding APM

@smith

[ahmedhkhalil]

@smith if you have time please can you check

[jennypavlova]

Hi @ahmedhkhalil I missed your ping, sorry. It seems the UI can't access the APM data and I don't think it is a UI bug. I will try to help you now.
If you expand some of the Degraded datasets what do you see in the Degraded fields/docs?:

Also can you please check if the indices are correct? You can either check the response of the internal/apm/data_view/index_pattern request or go the APM -> Services click on settings in the right corner and open indices:

Another thing: Do you get any logs from APM? Can you check for any related error logs:

[ahmedhkhalil]

for first question now all APM app data quality is ok
for second question

for third question , yes i am receiving APM data and also alerts tab is working fine

[ahmedhkhalil]

@jennypavlova

[ahmedhkhalil]

below is error from APM integration

failed to index documents in '.ds-traces-apm-default-2024.08.11-000328' (document_parsing_exception): [1:2409] failed to parse: field name cannot contain only whitespace: ['.AspNetCore.Antiforgery.VyLW6ORzMgk']

[jennypavlova]

Hey @ahmedhkhalil,
let's continue the slack discussion here. So the error I see here might be related to the discussed issue here so it's the value starting with a . .

Some guides can help you as well:

• https://www.elastic.co/guide/en/observability/current/apm-app-troubleshooting.html#no-apm-data-found
• https://www.elastic.co/guide/en/observability/8.15/apm-troubleshoot-apm.html

[ahmedhkhalil]

@jennypavlova i have check all linkx in docs and re installed APM from fleet and also rollover current index to let new index has correct values now data quality give me below and still issue not solved

[jennypavlova]

Hey @ahmedhkhalil,
That looks more like an APM server/fleet issue and not a UI issue. I am afraid I am not the right person to help you debug that as my team is responsible for the APM UI. If the troubleshooting steps are not helping please head on over to our official Discuss forum. You will reach a wider audience there.

[tarabiyat]

can we keep it open until confirmed APM issue ?

[jennypavlova]

can we keep it open until confirmed APM issue ?

Hi @tarabiyat,
I don't think it's a bug in Kibana (APM UI), so I guess the right place to reach out for help is the Discuss forum as I already mentioned. There you can track the discussion and find a solution to the problem.
Are you able to reproduce the same issue in your environment? And have you tried the troubleshooting steps I shared here?

[Omar-laz]

Hi @ahmedhkhalil,

any luck fixing this issue ?

[ahmedhkhalil]

no i create new instance