Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Flaky Test]: TestOtelAPMIngestion – there should be apm logs by now #4148

Open
cmacknz opened this issue Jan 25, 2024 · 5 comments
Open

[Flaky Test]: TestOtelAPMIngestion – there should be apm logs by now #4148

cmacknz opened this issue Jan 25, 2024 · 5 comments
Labels
flaky-test Unstable or unreliable test cases. Team:Elastic-Agent Label for the Agent team Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team

Comments

@cmacknz
Copy link
Member

cmacknz commented Jan 25, 2024

Failing test case

TestOtelAPMIngestion

Error message

there should be apm logs by now: map[string]bool{"This is a test debug message 2":false, "This is a test debug message 3":false, "This is a test debug message 4":false, "This is a test error message":false}

Build

https://buildkite.com/elastic/elastic-agent/builds/6574

OS

Linux

Stacktrace and notes

=== RUN   TestOtelAPMIngestion
    fetcher.go:91: Using existing artifact elastic-agent-8.13.0-SNAPSHOT-linux-arm64.tar.gz
    fixture.go:261: Extracting artifact elastic-agent-8.13.0-SNAPSHOT-linux-arm64.tar.gz to /tmp/TestOtelAPMIngestion3888313451/002
    fixture.go:274: Completed extraction of artifact elastic-agent-8.13.0-SNAPSHOT-linux-arm64.tar.gz to /tmp/TestOtelAPMIngestion3888313451/002
    fixture.go:886: All component specifications where removed
    fixture.go:927: Placed component specifications: fake, fake-shipper
    log_watcher.go:40: {"log.level":"info","@timestamp":"2024-01-25T19:19:13.716Z","log.origin":{"function":"github.com/elastic/apm-server/internal/beatcmd.(*Beat).init","file.name":"beatcmd/beat.go","file.line":143},"message":"Home path: [/tmp/TestOtelAPMIngestion3888313451/002/elastic-agent-8.13.0-SNAPSHOT-linux-arm64/data/elastic-agent-d93eb0/components] Config path: [/tmp/TestOtelAPMIngestion3888313451/002/elastic-agent-8.13.0-SNAPSHOT-linux-arm64/data/elastic-agent-d93eb0/components] Data path: [/tmp/TestOtelAPMIngestion3888313451/002/elastic-agent-8.13.0-SNAPSHOT-linux-arm64/data/elastic-agent-d93eb0/components/data] Logs path: [/tmp/TestOtelAPMIngestion3888313451/002/elastic-agent-8.13.0-SNAPSHOT-linux-arm64/data/elastic-agent-d93eb0/components/logs]","service.name":"apm-server","ecs.version":"1.6.0"}
    log_watcher.go:40: {"log.level":"info","@timestamp":"2024-01-25T19:19:13.721Z","log.origin":{"function":"github.com/elastic/apm-server/internal/beatcmd.(*Beat).init","file.name":"beatcmd/beat.go","file.line":150},"message":"Beat ID: 230a04af-7aca-40c8-9e7e-35df589a1c4a","service.name":"apm-server","ecs.version":"1.6.0"}
    log_watcher.go:40: {"log.level":"info","@timestamp":"2024-01-25T19:19:13.721Z","log.logger":"beat","log.origin":{"function":"github.com/elastic/apm-server/internal/beatcmd.logSystemInfo","file.name":"beatcmd/beat.go","file.line":574},"message":"Beat info","service.name":"apm-server","system_info":{"beat":{"path":{"config":"/tmp/TestOtelAPMIngestion3888313451/002/elastic-agent-8.13.0-SNAPSHOT-linux-arm64/data/elastic-agent-d93eb0/components","data":"/tmp/TestOtelAPMIngestion3888313451/002/elastic-agent-8.13.0-SNAPSHOT-linux-arm64/data/elastic-agent-d93eb0/components/data","home":"/tmp/TestOtelAPMIngestion3888313451/002/elastic-agent-8.13.0-SNAPSHOT-linux-arm64/data/elastic-agent-d93eb0/components","logs":"/tmp/TestOtelAPMIngestion3888313451/002/elastic-agent-8.13.0-SNAPSHOT-linux-arm64/data/elastic-agent-d93eb0/components/logs"},"type":"apm-server","uuid":"230a04af-7aca-40c8-9e7e-35df589a1c4a"},"ecs.version":"1.6.0"}}
    log_watcher.go:40: {"log.level":"info","@timestamp":"2024-01-25T19:19:13.721Z","log.logger":"beat","log.origin":{"function":"github.com/elastic/apm-server/internal/beatcmd.logSystemInfo","file.name":"beatcmd/beat.go","file.line":582},"message":"Build info","service.name":"apm-server","system_info":{"build":{"commit":"ccb0b0f72274e38287e86c51344d00cac0eb1f17","time":"2024-01-18T16:14:32.000Z","version":"8.13.0"},"ecs.version":"1.6.0"}}
    log_watcher.go:40: {"log.level":"info","@timestamp":"2024-01-25T19:19:13.721Z","log.logger":"beat","log.origin":{"function":"github.com/elastic/apm-server/internal/beatcmd.logSystemInfo","file.name":"beatcmd/beat.go","file.line":585},"message":"Go runtime info","service.name":"apm-server","system_info":{"go":{"os":"linux","arch":"arm64","max_procs":2,"version":"go1.21.6"},"ecs.version":"1.6.0"}}
    log_watcher.go:40: {"log.level":"info","@timestamp":"2024-01-25T19:19:13.722Z","log.logger":"beat","log.origin":{"function":"github.com/elastic/apm-server/internal/beatcmd.logSystemInfo","file.name":"beatcmd/beat.go","file.line":589},"message":"Host info","service.name":"apm-server","system_info":{"host":{"architecture":"aarch64","boot_time":"2024-01-25T19:13:23Z","containerized":false,"name":"ogc-linux-arm64-ubuntu-2204-default-50f4","ip":["127.0.0.1/8","::1/128","10.128.0.79/32","fe80::4001:aff:fe80:4f/64"],"kernel_version":"6.2.0-1019-gcp","mac":["42:01:0a:80:00:4f"],"os":{"type":"linux","family":"debian","platform":"ubuntu","name":"Ubuntu","version":"22.04.3 LTS (Jammy Jellyfish)","major":22,"minor":4,"patch":3,"codename":"jammy"},"timezone":"UTC","timezone_offset_sec":0,"id":"a44b9775ca1bd237c33e03255b8a4f5b"},"ecs.version":"1.6.0"}}
    log_watcher.go:40: {"log.level":"info","@timestamp":"2024-01-25T19:19:13.722Z","log.logger":"beat","log.origin":{"function":"github.com/elastic/apm-server/internal/beatcmd.logSystemInfo","file.name":"beatcmd/beat.go","file.line":618},"message":"Process info","service.name":"apm-server","system_info":{"process":{"capabilities":{"inheritable":null,"permitted":null,"effective":null,"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","perfmon","bpf","checkpoint_restore"],"ambient":null},"cwd":"/tmp/TestOtelAPMIngestion3888313451/002/elastic-agent-8.13.0-SNAPSHOT-linux-arm64/data/elastic-agent-d93eb0/components","exe":"/tmp/TestOtelAPMIngestion3888313451/002/elastic-agent-8.13.0-SNAPSHOT-linux-arm64/data/elastic-agent-d93eb0/components/apm-server","name":"apm-server","pid":16609,"ppid":16512,"seccomp":{"mode":"disabled","no_new_privs":false},"start_time":"2024-01-25T19:19:13.540Z"},"ecs.version":"1.6.0"}}
    log_watcher.go:40: {"log.level":"info","@timestamp":"2024-01-25T19:19:13.723Z","log.logger":"config","log.origin":{"function":"github.com/elastic/apm-server/internal/beater/config.(*AgentConfig).setup","file.name":"config/agentconfig.go","file.line":70},"message":"using output.elasticsearch for fetching agent config","service.name":"apm-server","ecs.version":"1.6.0"}
    log_watcher.go:40: {"log.level":"info","@timestamp":"2024-01-25T19:19:13.723Z","log.logger":"beater","log.origin":{"function":"github.com/elastic/apm-server/internal/beater.listen","file.name":"beater/http.go","file.line":142},"message":"Listening on: 127.0.0.1:8200","service.name":"apm-server","ecs.version":"1.6.0"}
    log_watcher.go:40: {"log.level":"info","@timestamp":"2024-01-25T19:19:13.723Z","log.origin":{"function":"github.com/elastic/apm-server/internal/beatcmd.(*Beat).Run","file.name":"beatcmd/beat.go","file.line":392},"message":"apm-server started.","service.name":"apm-server","ecs.version":"1.6.0"}
    log_watcher.go:40: {"log.level":"info","@timestamp":"2024-01-25T19:19:13.725Z","log.logger":"beater","log.origin":{"function":"github.com/elastic/apm-server/internal/beater.(*Runner).Run","file.name":"beater/beater.go","file.line":203},"message":"no cgroups detected, falling back to total system memory","service.name":"apm-server","ecs.version":"1.6.0"}
    log_watcher.go:40: {"log.level":"info","@timestamp":"2024-01-25T19:19:13.725Z","log.logger":"beater","log.origin":{"function":"github.com/elastic/apm-server/internal/beater.(*Runner).Run","file.name":"beater/beater.go","file.line":227},"message":"MaxConcurrentDecoders set to 495 based on 80 percent of 4.8gb of memory","service.name":"apm-server","ecs.version":"1.6.0"}
    log_watcher.go:40: {"log.level":"info","@timestamp":"2024-01-25T19:19:13.725Z","log.logger":"beater","log.origin":{"function":"github.com/elastic/apm-server/internal/beater.(*Runner).Run","file.name":"beater/beater.go","file.line":234},"message":"Aggregation.MaxServices set to 4836 based on 4.8gb of memory","service.name":"apm-server","ecs.version":"1.6.0"}
    log_watcher.go:40: {"log.level":"info","@timestamp":"2024-01-25T19:19:13.725Z","log.logger":"beater","log.origin":{"function":"github.com/elastic/apm-server/internal/beater.(*Runner).Run","file.name":"beater/beater.go","file.line":241},"message":"Aggregation.ServiceTransactions.MaxGroups for service aggregation set to 4836 based on 4.8gb of memory","service.name":"apm-server","ecs.version":"1.6.0"}
    log_watcher.go:40: {"log.level":"info","@timestamp":"2024-01-25T19:19:13.726Z","log.logger":"beater","log.origin":{"function":"github.com/elastic/apm-server/internal/beater.(*Runner).Run","file.name":"beater/beater.go","file.line":248},"message":"Aggregation.Transactions.MaxGroups set to 24181 based on 4.8gb of memory","service.name":"apm-server","ecs.version":"1.6.0"}
    log_watcher.go:40: {"log.level":"info","@timestamp":"2024-01-25T19:19:13.726Z","log.logger":"beater","log.origin":{"function":"github.com/elastic/apm-server/internal/beater.(*Runner).Run","file.name":"beater/beater.go","file.line":255},"message":"Aggregation.ServiceDestinations.MaxGroups set to 29181 based on 4.8gb of memory","service.name":"apm-server","ecs.version":"1.6.0"}
    log_watcher.go:40: {"log.level":"info","@timestamp":"2024-01-25T19:19:13.728Z","log.origin":{"function":"github.com/elastic/apm-server/internal/beatcmd.adjustMaxProcs.diffInfof.func2","file.name":"beatcmd/maxprocs.go","file.line":68},"message":"maxprocs: Leaving GOMAXPROCS=2: CPU quota undefined","service.name":"apm-server","ecs.version":"1.6.0"}
    log_watcher.go:40: {"log.level":"info","@timestamp":"2024-01-25T19:19:13.733Z","log.logger":"beater","log.origin":{"function":"github.com/elastic/apm-server/internal/beater.docappenderConfig","file.name":"beater/beater.go","file.line":810},"message":"docappender.DocumentBufferSize set to 3961 based on 4.8gb of memory","service.name":"apm-server","ecs.version":"1.6.0"}
    log_watcher.go:40: {"log.level":"info","@timestamp":"2024-01-25T19:19:13.734Z","log.logger":"beater","log.origin":{"function":"github.com/elastic/apm-server/internal/beater.docappenderConfig","file.name":"beater/beater.go","file.line":824},"message":"docappender.MaxRequests set to 17 based on 4.8gb of memory","service.name":"apm-server","ecs.version":"1.6.0"}
    log_watcher.go:40: {"log.level":"info","@timestamp":"2024-01-25T19:19:13.736Z","log.logger":"handler","log.origin":{"function":"github.com/elastic/apm-server/internal/beater/api.NewMux","file.name":"api/mux.go","file.line":138},"message":"Path / added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
    log_watcher.go:40: {"log.level":"info","@timestamp":"2024-01-25T19:19:13.733Z","log.logger":"beater","log.origin":{"function":"github.com/elastic/apm-server/internal/beater.waitReady","file.name":"beater/waitready.go","file.line":40},"message":"blocking ingestion until all preconditions are satisfied","service.name":"apm-server","ecs.version":"1.6.0"}
    log_watcher.go:40: {"log.level":"info","@timestamp":"2024-01-25T19:19:13.741Z","log.logger":"handler","log.origin":{"function":"github.com/elastic/apm-server/internal/beater/api.NewMux","file.name":"api/mux.go","file.line":138},"message":"Path /config/v1/agents added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
    log_watcher.go:40: {"log.level":"info","@timestamp":"2024-01-25T19:19:13.741Z","log.logger":"handler","log.origin":{"function":"github.com/elastic/apm-server/internal/beater/api.NewMux","file.name":"api/mux.go","file.line":138},"message":"Path /config/v1/rum/agents added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
    log_watcher.go:40: {"log.level":"info","@timestamp":"2024-01-25T19:19:13.741Z","log.logger":"handler","log.origin":{"function":"github.com/elastic/apm-server/internal/beater/api.NewMux","file.name":"api/mux.go","file.line":138},"message":"Path /intake/v2/rum/events added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
    log_watcher.go:40: {"log.level":"info","@timestamp":"2024-01-25T19:19:13.741Z","log.logger":"handler","log.origin":{"function":"github.com/elastic/apm-server/internal/beater/api.NewMux","file.name":"api/mux.go","file.line":138},"message":"Path /intake/v3/rum/events added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
    log_watcher.go:40: {"log.level":"info","@timestamp":"2024-01-25T19:19:13.741Z","log.logger":"handler","log.origin":{"function":"github.com/elastic/apm-server/internal/beater/api.NewMux","file.name":"api/mux.go","file.line":138},"message":"Path /intake/v2/events added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
    log_watcher.go:40: {"log.level":"info","@timestamp":"2024-01-25T19:19:13.742Z","log.logger":"handler","log.origin":{"function":"github.com/elastic/apm-server/internal/beater/api.NewMux","file.name":"api/mux.go","file.line":138},"message":"Path /v1/traces added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
    log_watcher.go:40: {"log.level":"info","@timestamp":"2024-01-25T19:19:13.742Z","log.logger":"handler","log.origin":{"function":"github.com/elastic/apm-server/internal/beater/api.NewMux","file.name":"api/mux.go","file.line":138},"message":"Path /v1/metrics added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
    log_watcher.go:40: {"log.level":"info","@timestamp":"2024-01-25T19:19:13.742Z","log.logger":"handler","log.origin":{"function":"github.com/elastic/apm-server/internal/beater/api.NewMux","file.name":"api/mux.go","file.line":138},"message":"Path /v1/logs added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
    log_watcher.go:40: {"log.level":"info","@timestamp":"2024-01-25T19:19:13.742Z","log.logger":"beater","log.origin":{"function":"github.com/elastic/apm-server/internal/beater.server.run","file.name":"beater/server.go","file.line":215},"message":"Starting apm-server [ccb0b0f72274e38287e86c51344d00cac0eb1f17 built 2024-01-18 16:14:32 +0000 UTC]. Hit CTRL-C to stop it.","service.name":"apm-server","ecs.version":"1.6.0"}
    log_watcher.go:40: {"log.level":"info","@timestamp":"2024-01-25T19:19:13.743Z","log.logger":"beater","log.origin":{"function":"github.com/elastic/apm-server/internal/beater.(*httpServer).start","file.name":"beater/http.go","file.line":94},"message":"RUM endpoints disabled.","service.name":"apm-server","ecs.version":"1.6.0"}
    log_watcher.go:40: {"log.level":"info","@timestamp":"2024-01-25T19:19:13.743Z","log.logger":"beater","log.origin":{"function":"github.com/elastic/apm-server/internal/beater.(*httpServer).start","file.name":"beater/http.go","file.line":104},"message":"SSL disabled.","service.name":"apm-server","ecs.version":"1.6.0"}
    log_watcher.go:40: {"log.level":"info","@timestamp":"2024-01-25T19:19:13.743Z","log.logger":"request","log.origin":{"function":"github.com/elastic/apm-server/internal/beater/api.apmMiddleware.LogMiddleware.func1.1","file.name":"middleware/log_middleware.go","file.line":62},"message":"request ok","service.name":"apm-server","url.original":"/","http.request.method":"GET","user_agent.original":"apm-agent-go/2.4.8 (apm-server)","source.address":"127.0.0.1","http.request.id":"ef646722-6731-498b-9b45-a95765fb501a","event.duration":89840,"http.request.body.bytes":0,"http.response.status_code":200,"ecs.version":"1.6.0"}
    log.go:85: Starting in otel mode
    log.go:96: {"log.level":"info","@timestamp":"2024-01-25T19:19:13.977Z","log.origin":{"file.name":"cmd/run.go","file.line":194},"message":"Elastic Agent started","log":{"source":"elastic-agent"},"process.pid":16610,"agent.version":"8.13.0","ecs.version":"1.6.0"}
    log.go:85: 2024-01-25T19:19:13.987Z	info	service@v0.93.0/telemetry.go:76	Setting up own telemetry...
    log.go:85: 2024-01-25T19:19:13.988Z	info	service@v0.93.0/telemetry.go:146	Serving metrics	{"address": ":8888", "level": "Basic"}
    log.go:85: 2024-01-25T19:19:13.989Z	info	exporter@v0.93.0/exporter.go:275	Development component. May change in the future.	{"kind": "exporter", "data_type": "logs", "name": "debug"}
    log.go:85: 2024-01-25T19:19:13.990Z	info	service@v0.93.0/service.go:139	Starting /tmp/TestOtelAPMIngestion3888313451/002/elastic-agent-8.13.0-SNAPSHOT-linux-arm64/elastic-agent...	{"Version": "8.13.0", "NumCPU": 2}
    log.go:85: 2024-01-25T19:19:13.990Z	info	extensions/extensions.go:34	Starting extensions...
    log.go:85: 2024-01-25T19:19:13.994Z	info	adapter/receiver.go:45	Starting stanza receiver	{"kind": "receiver", "name": "filelog", "data_type": "logs"}
    log.go:85: 2024-01-25T19:19:13.994Z	info	service@v0.93.0/service.go:165	Everything is ready. Begin running and processing data.
    log_watcher.go:40: {"log.level":"info","@timestamp":"2024-01-25T19:19:14.101Z","log.logger":"beater","log.origin":{"function":"github.com/elastic/apm-server/internal/beater.waitReady","file.name":"beater/waitready.go","file.line":68},"message":"no longer blocking ingestion as all precondition checks are now satisfied","service.name":"apm-server","ecs.version":"1.6.0"}
    log.go:85: 2024-01-25T19:19:14.195Z	info	fileconsumer/file.go:268	Started watching file	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer", "path": "/tmp/TestOtelAPMIngestion3888313451/001/content.log"}
    log.go:96: {"log.level":"info","@timestamp":"2024-01-25T19:19:14.246Z","log.origin":{"file.name":"upgrade/rollback.go","file.line":126},"message":"Starting upgrade watcher","log":{"source":"elastic-agent"},"path":"/tmp/TestOtelAPMIngestion3888313451/002/elastic-agent-8.13.0-SNAPSHOT-linux-arm64/elastic-agent","args":["/tmp/TestOtelAPMIngestion3888313451/002/elastic-agent-8.13.0-SNAPSHOT-linux-arm64/elastic-agent","watch","--path.config","/tmp/TestOtelAPMIngestion3888313451/002/elastic-agent-8.13.0-SNAPSHOT-linux-arm64","--path.home","/tmp/TestOtelAPMIngestion3888313451/002/elastic-agent-8.13.0-SNAPSHOT-linux-arm64"],"env":[],"dir":"","ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-01-25T19:19:14.247Z","log.origin":{"file.name":"upgrade/rollback.go","file.line":133},"message":"Upgrade Watcher invoked","log":{"source":"elastic-agent"},"agent.upgrade.watcher.process.pid":16624,"agent.process.pid":16610,"ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-01-25T19:19:14.247Z","log.origin":{"file.name":"upgrade/rollback.go","file.line":121},"message":"releasing watcher 16624","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-01-25T19:19:14.247Z","log.origin":{"file.name":"cmd/run.go","file.line":283},"message":"APM instrumentation disabled","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-01-25T19:19:14.248Z","log.origin":{"file.name":"application/application.go","file.line":64},"message":"Gathered system information","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-01-25T19:19:14.249Z","log.origin":{"file.name":"application/application.go","file.line":70},"message":"Detected available inputs and outputs","log":{"source":"elastic-agent"},"inputs":["fake","fake-apm"],"ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-01-25T19:19:14.249Z","log.origin":{"file.name":"capabilities/capabilities.go","file.line":48},"message":"Capabilities file not found in /tmp/TestOtelAPMIngestion3888313451/002/elastic-agent-8.13.0-SNAPSHOT-linux-arm64/capabilities.yml","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-01-25T19:19:14.249Z","log.origin":{"file.name":"application/application.go","file.line":76},"message":"Determined allowed capabilities","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-01-25T19:19:14.249Z","log.origin":{"file.name":"application/application.go","file.line":91},"message":"Loading baseline config from /tmp/TestOtelAPMIngestion3888313451/001/otel.yml","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-01-25T19:19:14.960Z","log.origin":{"file.name":"application/application.go","file.line":138},"message":"Parsed configuration and determined agent is managed locally","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-01-25T19:19:14.963Z","log.logger":"control","log.origin":{"file.name":"server/server.go","file.line":88},"message":"GRPC control socket listening at unix:///tmp/elastic-agent/oGdudMPcVE4UalXNy7OvIA5fkzYtPgH3.sock","log":{"source":"elastic-agent"},"address":"unix:///tmp/elastic-agent/oGdudMPcVE4UalXNy7OvIA5fkzYtPgH3.sock","ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-01-25T19:19:14.963Z","log.origin":{"file.name":"application/periodic.go","file.line":93},"message":"Configuration changes detected","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-01-25T19:19:14.965Z","log.origin":{"file.name":"upgrade/upgrade.go","file.line":116},"message":"Source URI changed from \"https://artifacts.elastic.co/downloads/\" to \"https://artifacts.elastic.co/downloads/\"","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-01-25T19:19:14.966Z","log.origin":{"file.name":"reload/reload.go","file.line":49},"message":"Starting server","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-01-25T19:19:14.966Z","log.logger":"api","log.origin":{"file.name":"api/server.go","file.line":75},"message":"Starting stats endpoint","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-01-25T19:19:14.967Z","log.origin":{"file.name":"runtime/manager.go","file.line":209},"message":"Starting grpc control protocol listener on port 6789 with max_message_size 104857600","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-01-25T19:19:14.967Z","log.logger":"api","log.origin":{"file.name":"api/server.go","file.line":77},"message":"Metrics endpoint listening on: 127.0.0.1:6791 (configured: http://localhost:6791)","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-01-25T19:19:14.968Z","log.logger":"composable.providers.docker","log.origin":{"file.name":"docker/docker.go","file.line":44},"message":"Docker provider skipped, unable to connect: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-01-25T19:19:15.066Z","log.origin":{"file.name":"coordinator/coordinator.go","file.line":1158},"message":"Updating running component model","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
    log.go:96: {"log.level":"info","@timestamp":"2024-01-25T19:19:15.316Z","log.origin":{"file.name":"coordinator/coordinator.go","file.line":1158},"message":"Updating running component model","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
    log_watcher.go:40: {"log.level":"info","@timestamp":"2024-01-25T19:19:33.496Z","log.logger":"request","log.origin":{"function":"github.com/elastic/apm-server/internal/beater/api.apmMiddleware.LogMiddleware.func1.1","file.name":"middleware/log_middleware.go","file.line":62},"message":"request accepted","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"elasticapm-go/1.15.0 go/go1.21.6","source.address":"127.0.0.1","http.request.id":"c29c9c62-93d4-4e05-a348-4e460aced5a6","event.duration":1838125592,"http.request.body.bytes":1003,"http.response.status_code":202,"ecs.version":"1.6.0"}
    log_watcher.go:40: {"log.level":"info","@timestamp":"2024-01-25T19:19:35.467Z","log.logger":"request","log.origin":{"function":"github.com/elastic/apm-server/internal/beater/api.apmMiddleware.LogMiddleware.func1.1","file.name":"middleware/log_middleware.go","file.line":62},"message":"request accepted","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"elasticapm-go/1.15.0 go/go1.21.6","source.address":"127.0.0.1","http.request.id":"31344d83-c0a3-4f5f-be33-f1b28633b7c4","event.duration":336000,"http.request.body.bytes":843,"http.response.status_code":202,"ecs.version":"1.6.0"}
    log_watcher.go:40: {"log.level":"info","@timestamp":"2024-01-25T19:19:58.885Z","log.logger":"request","log.origin":{"function":"github.com/elastic/apm-server/internal/beater/api.apmMiddleware.LogMiddleware.func1.1","file.name":"middleware/log_middleware.go","file.line":62},"message":"request accepted","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"elasticapm-go/1.15.0 go/go1.21.6","source.address":"127.0.0.1","http.request.id":"e6b1e0d6-49a1-4f0c-857a-c8b1e54b6841","event.duration":10009488401,"http.request.body.bytes":859,"http.response.status_code":202,"ecs.version":"1.6.0"}
    log_watcher.go:40: {"log.level":"info","@timestamp":"2024-01-25T19:20:03.498Z","log.logger":"request","log.origin":{"function":"github.com/elastic/apm-server/internal/beater/api.apmMiddleware.LogMiddleware.func1.1","file.name":"middleware/log_middleware.go","file.line":62},"message":"request accepted","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"elasticapm-go/1.15.0 go/go1.21.6","source.address":"127.0.0.1","http.request.id":"fbb6cfc2-9d73-41ad-86d7-d926fff8fe7a","event.duration":10001020841,"http.request.body.bytes":999,"http.response.status_code":202,"ecs.version":"1.6.0"}
    log_watcher.go:40: {"log.level":"info","@timestamp":"2024-01-25T19:20:28.877Z","log.logger":"request","log.origin":{"function":"github.com/elastic/apm-server/internal/beater/api.apmMiddleware.LogMiddleware.func1.1","file.name":"middleware/log_middleware.go","file.line":62},"message":"request accepted","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"elasticapm-go/1.15.0 go/go1.21.6","source.address":"127.0.0.1","http.request.id":"5a1701ab-6553-4744-bab9-34317f77ebf9","event.duration":10000199067,"http.request.body.bytes":859,"http.response.status_code":202,"ecs.version":"1.6.0"}
    log_watcher.go:40: {"log.level":"info","@timestamp":"2024-01-25T19:20:33.499Z","log.logger":"request","log.origin":{"function":"github.com/elastic/apm-server/internal/beater/api.apmMiddleware.LogMiddleware.func1.1","file.name":"middleware/log_middleware.go","file.line":62},"message":"request accepted","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"elasticapm-go/1.15.0 go/go1.21.6","source.address":"127.0.0.1","http.request.id":"69f083ea-e24d-4736-864f-8fd351fd6dc5","event.duration":10000810867,"http.request.body.bytes":998,"http.response.status_code":202,"ecs.version":"1.6.0"}
    log_watcher.go:40: {"log.level":"info","@timestamp":"2024-01-25T19:20:58.878Z","log.logger":"request","log.origin":{"function":"github.com/elastic/apm-server/internal/beater/api.apmMiddleware.LogMiddleware.func1.1","file.name":"middleware/log_middleware.go","file.line":62},"message":"request accepted","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"elasticapm-go/1.15.0 go/go1.21.6","source.address":"127.0.0.1","http.request.id":"21b63d59-7203-4ee9-8a7d-3a961784cce4","event.duration":10000147307,"http.request.body.bytes":848,"http.response.status_code":202,"ecs.version":"1.6.0"}
    log_watcher.go:40: {"log.level":"info","@timestamp":"2024-01-25T19:21:03.500Z","log.logger":"request","log.origin":{"function":"github.com/elastic/apm-server/internal/beater/api.apmMiddleware.LogMiddleware.func1.1","file.name":"middleware/log_middleware.go","file.line":62},"message":"request accepted","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"elasticapm-go/1.15.0 go/go1.21.6","source.address":"127.0.0.1","http.request.id":"9fa0bb25-5aa3-4bb1-be8b-dc04577a4b37","event.duration":10000301347,"http.request.body.bytes":1005,"http.response.status_code":202,"ecs.version":"1.6.0"}
    log_watcher.go:40: {"log.level":"info","@timestamp":"2024-01-25T19:21:19.788Z","log.logger":"request","log.origin":{"function":"github.com/elastic/apm-server/internal/beater/api.apmMiddleware.LogMiddleware.func1.1","file.name":"middleware/log_middleware.go","file.line":62},"message":"request ok","service.name":"apm-server","url.original":"/config/v1/agents?service.name=cec5d5358a89c6dbbcae8d834ee155b85c35532b","http.request.method":"GET","user_agent.original":"elasticapm-go/1.15.0 go/go1.21.6","source.address":"127.0.0.1","http.request.id":"10851145-a64e-41ec-96c8-ed8932c2024b","event.duration":128880,"http.request.body.bytes":0,"http.response.status_code":200,"ecs.version":"1.6.0"}
    log_watcher.go:40: {"log.level":"info","@timestamp":"2024-01-25T19:21:28.879Z","log.logger":"request","log.origin":{"function":"github.com/elastic/apm-server/internal/beater/api.apmMiddleware.LogMiddleware.func1.1","file.name":"middleware/log_middleware.go","file.line":62},"message":"request accepted","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"elasticapm-go/1.15.0 go/go1.21.6","source.address":"127.0.0.1","http.request.id":"c9b9b6d4-a69a-40e9-bfe3-28c41115d271","event.duration":10000169520,"http.request.body.bytes":858,"http.response.status_code":202,"ecs.version":"1.6.0"}
    log_watcher.go:40: {"log.level":"info","@timestamp":"2024-01-25T19:21:33.501Z","log.logger":"request","log.origin":{"function":"github.com/elastic/apm-server/internal/beater/api.apmMiddleware.LogMiddleware.func1.1","file.name":"middleware/log_middleware.go","file.line":62},"message":"request accepted","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"elasticapm-go/1.15.0 go/go1.21.6","source.address":"127.0.0.1","http.request.id":"eabf4f8c-1b0f-4451-9111-16beb2e0f0aa","event.duration":10000199239,"http.request.body.bytes":995,"http.response.status_code":202,"ecs.version":"1.6.0"}
    log_watcher.go:40: {"log.level":"info","@timestamp":"2024-01-25T19:21:49.788Z","log.logger":"request","log.origin":{"function":"github.com/elastic/apm-server/internal/beater/api.apmMiddleware.LogMiddleware.func1.1","file.name":"middleware/log_middleware.go","file.line":62},"message":"not modified","service.name":"apm-server","url.original":"/config/v1/agents?service.name=cec5d5358a89c6dbbcae8d834ee155b85c35532b","http.request.method":"GET","user_agent.original":"elasticapm-go/1.15.0 go/go1.21.6","source.address":"127.0.0.1","http.request.id":"7a413d50-293a-4fb6-9758-38c41c50dcdb","event.duration":80320,"http.request.body.bytes":0,"http.response.status_code":304,"ecs.version":"1.6.0"}
    log_watcher.go:40: {"log.level":"info","@timestamp":"2024-01-25T19:21:54.448Z","log.logger":"request","log.origin":{"function":"github.com/elastic/apm-server/internal/beater/api.apmMiddleware.LogMiddleware.func1.1","file.name":"middleware/log_middleware.go","file.line":62},"message":"request ok","service.name":"apm-server","url.original":"/config/v1/agents?service.name=integration_test","http.request.method":"GET","user_agent.original":"elasticapm-go/1.15.0 go/go1.21.6","source.address":"127.0.0.1","http.request.id":"3bf7e26d-1ad4-405b-84ab-eebc83a14890","event.duration":86000,"http.request.body.bytes":0,"http.response.status_code":200,"ecs.version":"1.6.0"}
    log_watcher.go:40: {"log.level":"info","@timestamp":"2024-01-25T19:21:58.880Z","log.logger":"request","log.origin":{"function":"github.com/elastic/apm-server/internal/beater/api.apmMiddleware.LogMiddleware.func1.1","file.name":"middleware/log_middleware.go","file.line":62},"message":"request accepted","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"elasticapm-go/1.15.0 go/go1.21.6","source.address":"127.0.0.1","http.request.id":"9b4dbf30-226e-4da9-b561-721dd54c706f","event.duration":10000131119,"http.request.body.bytes":849,"http.response.status_code":202,"ecs.version":"1.6.0"}
    log_watcher.go:40: {"log.level":"info","@timestamp":"2024-01-25T19:22:03.503Z","log.logger":"request","log.origin":{"function":"github.com/elastic/apm-server/internal/beater/api.apmMiddleware.LogMiddleware.func1.1","file.name":"middleware/log_middleware.go","file.line":62},"message":"request accepted","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"elasticapm-go/1.15.0 go/go1.21.6","source.address":"127.0.0.1","http.request.id":"33de17f1-b68f-4e09-bdcd-f6400ac70533","event.duration":10000100280,"http.request.body.bytes":988,"http.response.status_code":202,"ecs.version":"1.6.0"}
    log_watcher.go:40: {"log.level":"info","@timestamp":"2024-01-25T19:22:19.790Z","log.logger":"request","log.origin":{"function":"github.com/elastic/apm-server/internal/beater/api.apmMiddleware.LogMiddleware.func1.1","file.name":"middleware/log_middleware.go","file.line":62},"message":"not modified","service.name":"apm-server","url.original":"/config/v1/agents?service.name=cec5d5358a89c6dbbcae8d834ee155b85c35532b","http.request.method":"GET","user_agent.original":"elasticapm-go/1.15.0 go/go1.21.6","source.address":"127.0.0.1","http.request.id":"5d0dff72-194c-4e78-94e4-6c2bf17f516d","event.duration":112200,"http.request.body.bytes":0,"http.response.status_code":304,"ecs.version":"1.6.0"}
    log_watcher.go:40: {"log.level":"info","@timestamp":"2024-01-25T19:22:24.449Z","log.logger":"request","log.origin":{"function":"github.com/elastic/apm-server/internal/beater/api.apmMiddleware.LogMiddleware.func1.1","file.name":"middleware/log_middleware.go","file.line":62},"message":"not modified","service.name":"apm-server","url.original":"/config/v1/agents?service.name=integration_test","http.request.method":"GET","user_agent.original":"elasticapm-go/1.15.0 go/go1.21.6","source.address":"127.0.0.1","http.request.id":"72f14b80-6ef8-464c-b919-c139bbd63735","event.duration":78440,"http.request.body.bytes":0,"http.response.status_code":304,"ecs.version":"1.6.0"}
    log_watcher.go:40: {"log.level":"info","@timestamp":"2024-01-25T19:22:28.889Z","log.logger":"request","log.origin":{"function":"github.com/elastic/apm-server/internal/beater/api.apmMiddleware.LogMiddleware.func1.1","file.name":"middleware/log_middleware.go","file.line":62},"message":"request accepted","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"elasticapm-go/1.15.0 go/go1.21.6","source.address":"127.0.0.1","http.request.id":"215e8523-2bc9-45b4-844b-e64addcf7de3","event.duration":10008529306,"http.request.body.bytes":849,"http.response.status_code":202,"ecs.version":"1.6.0"}
    log_watcher.go:40: {"log.level":"info","@timestamp":"2024-01-25T19:22:33.504Z","log.logger":"request","log.origin":{"function":"github.com/elastic/apm-server/internal/beater/api.apmMiddleware.LogMiddleware.func1.1","file.name":"middleware/log_middleware.go","file.line":62},"message":"request accepted","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"elasticapm-go/1.15.0 go/go1.21.6","source.address":"127.0.0.1","http.request.id":"a6cf3178-7633-4348-a999-f27ce9b0e0c3","event.duration":10000446371,"http.request.body.bytes":1001,"http.response.status_code":202,"ecs.version":"1.6.0"}
    log_watcher.go:40: {"log.level":"info","@timestamp":"2024-01-25T19:22:49.790Z","log.logger":"request","log.origin":{"function":"github.com/elastic/apm-server/internal/beater/api.apmMiddleware.LogMiddleware.func1.1","file.name":"middleware/log_middleware.go","file.line":62},"message":"not modified","service.name":"apm-server","url.original":"/config/v1/agents?service.name=cec5d5358a89c6dbbcae8d834ee155b85c35532b","http.request.method":"GET","user_agent.original":"elasticapm-go/1.15.0 go/go1.21.6","source.address":"127.0.0.1","http.request.id":"f44ebc4d-5982-44b3-b01b-e031a6a4ff18","event.duration":81400,"http.request.body.bytes":0,"http.response.status_code":304,"ecs.version":"1.6.0"}
    log_watcher.go:40: {"log.level":"info","@timestamp":"2024-01-25T19:22:54.450Z","log.logger":"request","log.origin":{"function":"github.com/elastic/apm-server/internal/beater/api.apmMiddleware.LogMiddleware.func1.1","file.name":"middleware/log_middleware.go","file.line":62},"message":"not modified","service.name":"apm-server","url.original":"/config/v1/agents?service.name=integration_test","http.request.method":"GET","user_agent.original":"elasticapm-go/1.15.0 go/go1.21.6","source.address":"127.0.0.1","http.request.id":"8043ba8e-0a34-4279-9c3a-91d089491eb4","event.duration":65080,"http.request.body.bytes":0,"http.response.status_code":304,"ecs.version":"1.6.0"}
    log_watcher.go:40: {"log.level":"info","@timestamp":"2024-01-25T19:22:58.891Z","log.logger":"request","log.origin":{"function":"github.com/elastic/apm-server/internal/beater/api.apmMiddleware.LogMiddleware.func1.1","file.name":"middleware/log_middleware.go","file.line":62},"message":"request accepted","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"elasticapm-go/1.15.0 go/go1.21.6","source.address":"127.0.0.1","http.request.id":"26b5928c-efce-403f-b0dd-b300fecc0d5e","event.duration":10008929170,"http.request.body.bytes":847,"http.response.status_code":202,"ecs.version":"1.6.0"}
    log_watcher.go:40: {"log.level":"info","@timestamp":"2024-01-25T19:23:03.505Z","log.logger":"request","log.origin":{"function":"github.com/elastic/apm-server/internal/beater/api.apmMiddleware.LogMiddleware.func1.1","file.name":"middleware/log_middleware.go","file.line":62},"message":"request accepted","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"elasticapm-go/1.15.0 go/go1.21.6","source.address":"127.0.0.1","http.request.id":"25272c85-3e07-47e1-8b88-06893bf25cd9","event.duration":10000205371,"http.request.body.bytes":997,"http.response.status_code":202,"ecs.version":"1.6.0"}
    log_watcher.go:40: {"log.level":"info","@timestamp":"2024-01-25T19:23:19.791Z","log.logger":"request","log.origin":{"function":"github.com/elastic/apm-server/internal/beater/api.apmMiddleware.LogMiddleware.func1.1","file.name":"middleware/log_middleware.go","file.line":62},"message":"not modified","service.name":"apm-server","url.original":"/config/v1/agents?service.name=cec5d5358a89c6dbbcae8d834ee155b85c35532b","http.request.method":"GET","user_agent.original":"elasticapm-go/1.15.0 go/go1.21.6","source.address":"127.0.0.1","http.request.id":"1ef9aa1c-37ec-4fbc-a9e5-7e16635cd216","event.duration":66640,"http.request.body.bytes":0,"http.response.status_code":304,"ecs.version":"1.6.0"}
    log_watcher.go:40: {"log.level":"info","@timestamp":"2024-01-25T19:23:24.450Z","log.logger":"request","log.origin":{"function":"github.com/elastic/apm-server/internal/beater/api.apmMiddleware.LogMiddleware.func1.1","file.name":"middleware/log_middleware.go","file.line":62},"message":"not modified","service.name":"apm-server","url.original":"/config/v1/agents?service.name=integration_test","http.request.method":"GET","user_agent.original":"elasticapm-go/1.15.0 go/go1.21.6","source.address":"127.0.0.1","http.request.id":"98dbb5da-9504-4c91-b9d1-c71a4ac67e20","event.duration":64680,"http.request.body.bytes":0,"http.response.status_code":304,"ecs.version":"1.6.0"}
    log_watcher.go:40: {"log.level":"info","@timestamp":"2024-01-25T19:23:28.904Z","log.logger":"request","log.origin":{"function":"github.com/elastic/apm-server/internal/beater/api.apmMiddleware.LogMiddleware.func1.1","file.name":"middleware/log_middleware.go","file.line":62},"message":"request accepted","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"elasticapm-go/1.15.0 go/go1.21.6","source.address":"127.0.0.1","http.request.id":"bc632e9b-2a7a-4a43-967a-26b642ae1a81","event.duration":10009096982,"http.request.body.bytes":851,"http.response.status_code":202,"ecs.version":"1.6.0"}
    log_watcher.go:40: {"log.level":"info","@timestamp":"2024-01-25T19:23:33.506Z","log.logger":"request","log.origin":{"function":"github.com/elastic/apm-server/internal/beater/api.apmMiddleware.LogMiddleware.func1.1","file.name":"middleware/log_middleware.go","file.line":62},"message":"request accepted","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"elasticapm-go/1.15.0 go/go1.21.6","source.address":"127.0.0.1","http.request.id":"b079df7f-ea32-4afe-8aba-aa7c6d0b1f24","event.duration":10000085467,"http.request.body.bytes":999,"http.response.status_code":202,"ecs.version":"1.6.0"}
    log_watcher.go:40: {"log.level":"info","@timestamp":"2024-01-25T19:23:49.792Z","log.logger":"request","log.origin":{"function":"github.com/elastic/apm-server/internal/beater/api.apmMiddleware.LogMiddleware.func1.1","file.name":"middleware/log_middleware.go","file.line":62},"message":"not modified","service.name":"apm-server","url.original":"/config/v1/agents?service.name=cec5d5358a89c6dbbcae8d834ee155b85c35532b","http.request.method":"GET","user_agent.original":"elasticapm-go/1.15.0 go/go1.21.6","source.address":"127.0.0.1","http.request.id":"88fce73f-a369-4117-8424-a6e04480a8f9","event.duration":62480,"http.request.body.bytes":0,"http.response.status_code":304,"ecs.version":"1.6.0"}
    log_watcher.go:40: {"log.level":"info","@timestamp":"2024-01-25T19:23:54.451Z","log.logger":"request","log.origin":{"function":"github.com/elastic/apm-server/internal/beater/api.apmMiddleware.LogMiddleware.func1.1","file.name":"middleware/log_middleware.go","file.line":62},"message":"not modified","service.name":"apm-server","url.original":"/config/v1/agents?service.name=integration_test","http.request.method":"GET","user_agent.original":"elasticapm-go/1.15.0 go/go1.21.6","source.address":"127.0.0.1","http.request.id":"b50e9ab8-6cf2-48b7-8b37-c32556a13f46","event.duration":81200,"http.request.body.bytes":0,"http.response.status_code":304,"ecs.version":"1.6.0"}
    log_watcher.go:40: {"log.level":"info","@timestamp":"2024-01-25T19:23:58.896Z","log.logger":"request","log.origin":{"function":"github.com/elastic/apm-server/internal/beater/api.apmMiddleware.LogMiddleware.func1.1","file.name":"middleware/log_middleware.go","file.line":62},"message":"request accepted","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"elasticapm-go/1.15.0 go/go1.21.6","source.address":"127.0.0.1","http.request.id":"71765de9-67a6-453a-a4c7-b3a9ce9eec70","event.duration":10000174718,"http.request.body.bytes":856,"http.response.status_code":202,"ecs.version":"1.6.0"}
    log_watcher.go:40: {"log.level":"info","@timestamp":"2024-01-25T19:24:03.508Z","log.logger":"request","log.origin":{"function":"github.com/elastic/apm-server/internal/beater/api.apmMiddleware.LogMiddleware.func1.1","file.name":"middleware/log_middleware.go","file.line":62},"message":"request accepted","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"elasticapm-go/1.15.0 go/go1.21.6","source.address":"127.0.0.1","http.request.id":"45bae2c4-6350-416e-9358-efbe6683a362","event.duration":10000766278,"http.request.body.bytes":1010,"http.response.status_code":202,"ecs.version":"1.6.0"}
    otel_test.go:288: 
        	Error Trace:	/home/ubuntu/agent/testing/integration/otel_test.go:288
        	Error:      	Condition never satisfied
        	Test:       	TestOtelAPMIngestion
        	Messages:   	there should be apm logs by now: map[string]bool{"This is a test debug message 2":false, "This is a test debug message 3":false, "This is a test debug message 4":false, "This is a test error message":false}
--- FAIL: TestOtelAPMIngestion (314.43s)
@cmacknz cmacknz added Team:Elastic-Agent Label for the Agent team flaky-test Unstable or unreliable test cases. labels Jan 25, 2024
@elasticmachine
Copy link
Contributor

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

@cmacknz cmacknz mentioned this issue Jan 25, 2024
Merged
7 tasks
@michalpristas
Copy link
Contributor

haven't seen this failing for a while now after latest changes. closing

@michalpristas michalpristas removed their assignment Nov 7, 2024
@michalpristas
Copy link
Contributor

michalpristas commented Nov 7, 2024 •
edited
Loading

reopening, test is flaky again
example: https://buildkite.com/elastic/elastic-agent-extended-testing/builds/4339#01930383-9cbb-4096-a7b9-ee4a66969d8c

@michalpristas michalpristas reopened this Nov 7, 2024
@michalpristas michalpristas added the Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team label Nov 7, 2024
@elasticmachine
Copy link
Contributor

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

@ycombinator
Copy link
Contributor

Another example: https://buildkite.com/elastic/elastic-agent-extended-testing/builds/4512#0193267f-ee00-4cef-9a1f-fd787e474110

@pkoutsovasilis pkoutsovasilis mentioned this issue Nov 21, 2024
Merged
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
flaky-test Unstable or unreliable test cases. Team:Elastic-Agent Label for the Agent team Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@ycombinator @michalpristas @cmacknz @elasticmachine