Logs are not imported when using syslog

[jjmerchante]

Hi! I cannot import the logs from syslog in Matomo. These are the configuration files I am using:

syslog-ng.conf

@version: 3.24
@include "scl.conf"

source s_network {
  network(transport(udp));
};

destination d_matomo {
  program("/log-analytics/matomo.sh" template("$MSG\n"));
};

log {
  source(s_network);
  destination(d_matomo);
};

matomo.sh

#!/bin/sh

python /log-analytics/import_logs.py \
 --url=http://matomo_service:80 \
 --idsite=1 --recorders=2 --enable-http-errors --enable-http-redirects --enable-static --enable-bots \
 --token-auth=xxxxxxxxxxxxxxxxxxx  --log-format-name=common --debug -

I don't get any errors, but Matomo doesn't display anything.

If I change matomo.sh to:

#!/bin/bash
while read line ; do
echo $line >> /tmp/testlog
done

I can see the logs in /tmp/testlog:

172.17.0.1 - - [19/Mar/2020:09:52:58 +0000] "GET /static/css/sb-admin-2.min.css HTTP/1.1" 499 0 "https://localhost:9000/dashboard/1" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:74.0) Gecko/20100101 Firefox/74.0"
172.17.0.1 - - [19/Mar/2020:09:52:59 +0000] "GET /dashboard/1/info HTTP/1.1" 200 52 "https://localhost:9000/dashboard/1" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:74.0) Gecko/20100101 Firefox/74.0"
172.17.0.1 - - [19/Mar/2020:09:52:59 +0000] "GET /dashboard/1/summary HTTP/1.1" 200 80 "https://localhost:9000/dashboard/1" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:74.0) Gecko/20100101 Firefox/74.0"
172.17.0.1 - - [19/Mar/2020:09:53:04 +0000] "GET /dashboard/1/summary HTTP/1.1" 200 80 "https://localhost:9000/dashboard/1" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:74.0) Gecko/20100101 Firefox/74.0"
1
...

If I run the matomo.sh (the first one) by passing the logs obtained from /tmp/testlog:

cat /tmp/testlog | ./matomo.sh

The logs are imported!!!!

I don't know why matomo.sh is not reading the logs from syslog program. But I can import them piping a file

One more detail, when I restart syslog, I can see this POST in Matomo logs:

172.24.0.10 - - [19/Mar/2020:10:58:23 +0000] "POST / HTTP/1.1" 200 719 "-" "Matomo/LogImport"

Dependencies:

• Python 2.7
• The latest version of import_logs.py from the branch 3.x-dev
• Syslog container 3.24 with Python installed and the import_logs file in that container
• Matomo container: matomo-3

[jjmerchante]

I've seen it waits until the number of visits is len(hits) >= config.options.recorder_max_payload_size * len(Recorder.recorders), in my case it's 400, so I haven't reached the limit ... and when I was using a file, it sends all the hits when the file ends.

My bad, I shouldn't open this issue