import_logs.py broken since 4.4.1

[jsandner]

Importing Apache Logfiles with import_logs.py doesn't work here since upgrade from matomo 4.3.1 to 4.4.1.

Error Message is

Error when connecting to Matomo: 'utf-8' codec can't decode byte 0x80 in position 10: invalid start byte

• Matomo Server runs with Version 4.4.1
• On Import Server I have 2 matomo directories:
  • version 4.3.1 (works)
  • version 4.4.1 (error)

Expected Behavior

Current Behavior

python3 /srv/www/vhosts/ANALYSE/matomo.mydomain/matomo/misc/log-analytics/import_logs.py --url=http://piwik.my.domain --enable-http-errors --enable-http-redirects --enable-bots --recorders=2 --idsite=16 --recorder-max-payload-size=50 --log-format-name=common_complete /srv/import/access_log-20210817

2021-08-17 09:28:16,766: [INFO] Error when connecting to Matomo: 'utf-8' codec can't decode byte 0x80 in position 10: invalid start byte
2021-08-17 09:28:16,767: [INFO] Retrying request, attempt number 2

Possible Solution

import_logs.py

response = opener.open(request, timeout = timeout)
#encoding = response.info().get_content_charset('utf-8')
result = response.read()
response.close()
#return result.decode(encoding)
return result

omitting decoding of result makes it work again.

Steps to Reproduce (for Bugs)

Context

Your Environment

• Matomo Version: 4.4.1
• PHP Version: 7.4.6
• Python Version: 3.6.13
• Server Operating System: SLES 15 SP3
• Additionally installed plugins:

• Browser:
• Operating System:

[sgiehl]

Thanks for the report. I'll move this to the log-importer repo.
Guess this was introduced with #316
Does your apache log file maybe contain any strange characters?
You could maybe try to pass the --encoding parameter and check if that changes something when using something else than utf-8

[jsandner]

using --encoding=ascii or --encoding=iso-8859-1 makes no difference. Still doesn't work.
Possibly there are strange characters in the logfile ... hackers try strange URL-parameters and the called URL is logged including the arguments.

[sgiehl]

could you maybe try to identify the line that let's the importer fail and paste it here, so we can try to reproduce that?

[jsandner]

The importer statement is

python3 /srv/www/vhosts/ANALYSE/matomo.mydomain/matomo/misc/log-analytics/import_logs.py --url=http://piwik.my.domain --enable-http-errors --enable-http-redirects --enable-bots --recorders=2 --encoding=ascii --idsite=16 --recorder-max-payload-size=50 --log-format-name=common_complete apache.log

i get the error, if apache.log contains this line

www.my.domain 10.171.92.193 - - [16/Aug/2021:00:10:03 +0200] "GET /oberbayern/test.rss HTTP/1.0" 200 4496 "-" "axios/0.17.1"

if I replace .rss by .css it works:

www.my.domain 10.171.92.193 - - [16/Aug/2021:00:10:03 +0200] "GET /oberbayern/test.css HTTP/1.0" 200 4496 "-" "axios/0.17.1"

[sgiehl]

if I replace .rss by .css it works:

Guess that's because requests to static files like css are ignored by default.

[metalocator]

Any luck on this issue? We are seeing the same problem

python3 misc/log-analytics/import_logs.py --replay-tracking-expected-tracker-file=piwik.php --replay-tracking --enable-http-errors --url=https://analytics.xxxxxxxxxx.com --debug-tracker --log-format-name=common_complete /var/log/apache2/other_vhosts_access.log.1
0 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
Parsing log /var/log/apache2/other_vhosts_access.log.1...
2021-09-14 23:49:50,278: [INFO] Error when connecting to Matomo: 'utf-8' codec can't decode byte 0x80 in position 10: invalid start byte
2021-09-14 23:49:50,278: [INFO] Retrying request, attempt number 2
961 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
961 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)

[justinvelluppillai]

Probably introduced by the encoding changes here #316

[metalocator]

Thanks for the reply. Is there any possible workaround?

Downgrade or revert this patch?

[EreMaijala]

If you have queued tracking enabled, see pull request #324. That's what seems to have been causing the issue for me.