OmsAgentForLinux is dumping json event files, which walinuxagent cannot process

[johncrim]

I keep seeing these errors from the walinuxagent service:

2019/06/11 23:11:17 ERROR:Traceback (most recent call last):
2019/06/11 23:11:17 ERROR:  File "/usr/sbin/waagent2.0", line 4921, in CollectAndSendWALAEvents
2019/06/11 23:11:17 ERROR:    xmlStr = self.AddSystemInfo(xmlStr)
2019/06/11 23:11:17 ERROR:  File "/usr/sbin/waagent2.0", line 4979, in AddSystemInfo
2019/06/11 23:11:17 ERROR:    eventObject = xml.dom.minidom.parseString(eventData.encode("utf-8")).childNodes[0]
2019/06/11 23:11:17 ERROR:  File "/usr/lib/python2.7/xml/dom/minidom.py", line 1928, in parseString
2019/06/11 23:11:17 ERROR:    return expatbuilder.parseString(string)
2019/06/11 23:11:17 ERROR:  File "/usr/lib/python2.7/xml/dom/expatbuilder.py", line 940, in parseString
2019/06/11 23:11:17 ERROR:    return builder.parseString(string)
2019/06/11 23:11:17 ERROR:  File "/usr/lib/python2.7/xml/dom/expatbuilder.py", line 223, in parseString
2019/06/11 23:11:17 ERROR:    parser.Parse(string, True)
2019/06/11 23:11:17 ERROR:ExpatError: not well-formed (invalid token): line 1, column 1

The source for the waagent file (CollectAndSendWalaEvents) is here:
https://github.com/Azure/WALinuxAgent/blob/v2.2.40/bin/waagent2.0#L4902

As you can see, it expects XML events files. I added some logging to waagent2.0 to see what is going on, and it's receiving XML files for all of the event files that other providers place in /var/lib/waagent/events/, except the OmsAgent files are all JSON and cause errors.

For example (addition of logging is mine):

2019/06/11 23:11:17 Processing file: /var/lib/waagent/events/1560294639671387.tld
2019/06/11 23:11:17 eventData:  {
2019/06/11 23:11:17             "eventId": 1,
2019/06/11 23:11:17             "providerId": "69B669B9-4AF8-4C50-BDC4-6006FA76E975",
2019/06/11 23:11:17             "parameters": [
2019/06/11 23:11:17                         {
2019/06/11 23:11:17                         "name": "Name",
2019/06/11 23:11:17                         "value": "Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux"
2019/06/11 23:11:17                 },
2019/06/11 23:11:17                         {
2019/06/11 23:11:17
2019/06/11 23:11:17                                     "name": "Version",
2019/06/11 23:11:17                                     "value": "1.11.3"
2019/06/11 23:11:17                             },
2019/06/11 23:11:17
2019/06/11 23:11:17                         {
2019/06/11 23:11:17                         "name": "Operation",
2019/06/11 23:11:17                         "value": "ODSIngestion"
2019/06/11 23:11:17                 },
2019/06/11 23:11:17                         {
2019/06/11 23:11:17                         "name": "OperationSuccess",
2019/06/11 23:11:17                         "value": true
2019/06/11 23:11:17                 },
2019/06/11 23:11:17                         {
2019/06/11 23:11:17                         "name": "Message",
2019/06/11 23:11:17                         "value": "Sending success"
2019/06/11 23:11:17                 },
2019/06/11 23:11:17                         {
2019/06/11 23:11:17                         "name": "Duration",
2019/06/11 23:11:17                         "value": 300000
2019/06/11 23:11:17                 }
2019/06/11 23:11:17             ]
2019/06/11 23:11:17             }
2019/06/11 23:11:17 ERROR:Traceback (most recent call last):
2019/06/11 23:11:17 ERROR:  File "/usr/sbin/waagent2.0", line 4921, in CollectAndSendWALAEvents
2019/06/11 23:11:17 ERROR:    xmlStr = self.AddSystemInfo(xmlStr)
2019/06/11 23:11:17 ERROR:  File "/usr/sbin/waagent2.0", line 4979, in AddSystemInfo
2019/06/11 23:11:17 ERROR:    eventObject = xml.dom.minidom.parseString(eventData.encode("utf-8")).childNodes[0]
2019/06/11 23:11:17 ERROR:  File "/usr/lib/python2.7/xml/dom/minidom.py", line 1928, in parseString
2019/06/11 23:11:17 ERROR:    return expatbuilder.parseString(string)
2019/06/11 23:11:17 ERROR:  File "/usr/lib/python2.7/xml/dom/expatbuilder.py", line 940, in parseString
2019/06/11 23:11:17 ERROR:    return builder.parseString(string)
2019/06/11 23:11:17 ERROR:  File "/usr/lib/python2.7/xml/dom/expatbuilder.py", line 223, in parseString
2019/06/11 23:11:17 ERROR:    parser.Parse(string, True)
2019/06/11 23:11:17 ERROR:ExpatError: not well-formed (invalid token): line 1, column 1
2019/06/11 23:11:17 ERROR:

[johncrim]

Also related: Azure/WALinuxAgent#1447

[johncrim]

I should note that this issue seems to be caused by my patch for Azure/WALinuxAgent#1550 . In that patch, I'm using waagent2.0, which causes this. I wasn't aware that waagent2.0 is older code than waagent (because it wasn't documented anywhere).

[hestolz]

Thank you for your submission. As previously announced, the Log Analytics agent has been deprecated and has no support as of August 31, 2024. If you use the Log Analytics agent to ingest data to Azure Monitor, migrate now to the new Azure Monitor agent. As part of repo archival, open issues and pull requests will be closed.