How to get the xml-structured device log?

[otizonaizit]

Hi!

as discussed at length in #119 the logs returned by DeviceInfo1->GetInfo and DeviceInfo1->GetDeviceLog are incomplete and only contain the "Internet"-logs subset.
I realized that AVM is also offering the action DeviceInfo1->X_AVM-DE_GetDeviceLogPath which I tried and which supposedly returns the URL to a lua script which, when called, should return the full log in XML format. It even supports filtering the logs by subsystem.

The only problem is that when I call that action I get the following:

>>> fc = FritzConnection(address=XXX,password=YYY, use_cache=True)
>>> fc.call_action('DeviceInfo:1','X_AVM-DE_GetDeviceLogPath')
{'NewDeviceLogPath': '/devicelog.lua?sid=9c43994fcc802202'}

what can I do with the returned URL? Sticking it into a fc.call_http as-is results in exceptions being raised. I tried to just manualy log into the HTTP interface in a browser and the try to get to the URL from there with our without sid but I get a 404 error
How is the returned URL supposed to be used?

[otizonaizit]

by the way, these guys here seem to support getting the XML-formatted logs, but the code to do that is buried deep into their library and I couldn't extract the relevant parts to make it fit into fritzconnection...

[kbr]

Didn't test it but this looks like something that can be handled by fritzconnection.core.utils.get_xml_root() using the current session. A quick and dirty hack could be:

import fritzconnection
from fritzconnection.core.utils import get_xml_root

fc = fritzconnection.FritzConnection(address="192.168.178.1", use_cache=True)
result = fc.call_action('DeviceInfo:1','X_AVM-DE_GetDeviceLogPath')
url = result["NewDeviceLogPath"]
get_xml_root(url, session=fc.session)

However the action X_AVM-DE_GetDeviceLogPath is unknown on my system: FRITZ!Box 7590, FRITZ!OS: 7.57, so I get a FritzActionError.

[otizonaizit]

hmm, that did not work:

import fritzconnection
from fritzconnection.core.utils import get_xml_root

fc = FritzConnection(address=XXX,password=YYY, use_cache=True)
fc.call_action('DeviceInfo:1','X_AVM-DE_GetDeviceLogPath')
url = result["NewDeviceLogPath"] # -> '/devicelog.lua?sid=d5c44ef3b36219ee' 
get_xml_root(url, session=fc.session)

errors out with

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File ".../fritzconnection/core/utils.py", line 81, in get_xml_root
    with open(source) as fobj:
         ^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/devicelog.lua?sid=d5c44ef3b36219ee'

which is clear, I think, because the url is not complete. If I add the full url:

get_xml_root(fc.address+url, session=fc.session)

I get another error:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File ".../fritzconnection/core/utils.py", line 83, in get_xml_root
    return etree.fromstring(source)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/xml/etree/ElementTree.py", line 1335, in XML
    parser.feed(text)
xml.etree.ElementTree.ParseError: not well-formed (invalid token): line 4, column 21

which is also clear, because what comes back in source is not XML, but an HTML site saying:

The specified site could not be found. Redirecting to the FRITZ!Box "Overview" page.

(I could verify that by just printing source before return in line 83 of /fritzconnection/core/utils.py.

I tried modifying the URL by removing the sid, i.e. by just passing fc.address+'/devicelog.lua', but I still get the same HTML site...

I guess the interface has been added in Fritz!OS 7.90, so you'd need to upgrade to get access to that. I am running Fritz!OS 8.0.

[kbr]

Yes, you are right, the url wasn't complete in my hack. The implementation of FritzHosts.get_hosts_attributes() (located at fritzconnection.lib.fritzhosts) is an example how get_xml_root() is used internally. On returning html with the corresponding header a FritzResourceError should get raised, so currently I don't know what happens in you case.

AVM has told me that support for the 7590 (the first model from over five years ago) should continue in 2025 but may be end thereafter. Unfortunately the newest OS Version I could upgrade to seems to be 7.59

[otizonaizit]

Bingo!!!

>>> from fritzconnection.core.utils import get_xml_root
>>> fc = FritzConnection(address=XXX,password=YYY, use_cache=True)
>>> result = fc.call_action('DeviceInfo:1','X_AVM-DE_GetDeviceLogPath')
>>> url = f'{fc.address}:{fc.port}{result["NewDeviceLogPath"]}'
>>> logs = get_xml_root(url, session=fc.session)
>>> print(''.join(logs[0].itertext()))

506
sys
30.10.24
16:04:01
App belonging to user fritzZZZ logging in from IP address XXX.

So I am getting something! I'll explore later to find out if these logs are indeed more detailed than the ones you get with GetDeviceLog.

Thanks a bunch!

[kbr]

AVM has released OS 8.0 also for the 7590, so I have a chance to dive into this. The action 'X_AVM-DE_GetDeviceLogPath' returns a url that in turn returns a XML datastructure, which is a list of event-nodes enclosed in a device-node. The structure for all event-nodes are the same, like the following one:

<?xml version="1.0" encoding="UTF-8"?>
<devicelog>
<!-- filter :all -->
	<event>
		<id>
			502
		</id>
		<group>
			sys
		</group>
		<date>
			31.10.24
		</date>
		<time>
			17:03:49
		</time>
		<msg>
			The FRITZ!Box settings were changed in the user interface.
		</msg>
	</event>
</devicelog>

This is the kind of datastructure that can be converted to a hierarchical list of Python objects by means of fritzconnection.core.processor. The Phonebook class in fritzconnection.lib.phonebook is an example how to do this.

[otizonaizit]

I wrote a little CLI script to collect the full logs from the fritzbox, format them in a more readable way and store them locally. The script merges the newly collected logs with the ones already stored on disk, so that it becomes possible to store the logs long-term, working around the annoying fact that the fritzbox deletes the logs on reboot... Also, these logs are more complete than the one you can get by using the actions DeviceInfo1->GetInfo and DeviceInfo1->GetDeviceLog.

You can find the script here: https://gist.github.com/otizonaizit/8c06edda6ee5a997c0eede1560340d4c

[kbr]

Glad that it has worked for you.

[kbr]

TR-064 calls returning URLs are worth adding methods to the library. So I have taken this case to add a get_avm_device_log() method to FritzStatus to simplify the access of the device event-log. This code is currently in the development-branch.