Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rsyslog + nginx live log import silently fail #285

Open
1000i100 opened this issue Nov 7, 2020 · 1 comment
Open

rsyslog + nginx live log import silently fail #285

1000i100 opened this issue Nov 7, 2020 · 1 comment

Comments

@1000i100
Copy link

1000i100 commented Nov 7, 2020

It probably have some comon point with #258

I've followed the rsyslog part of this :
https://github.com/matomo-org/matomo-log-analytics#how-do-i-setup-nginx-to-directly-import-to-matomo-via-syslog

and after some folder manual creation and the change of /var/cache/nginx/access.socket by /dev/log in nginx.conf and 10-matomo.conf if i uncomment the debug line local0.* /var/tmp/nginx.tmp;matomo, log apear in /var/tmp/nginx.tmp

If i try to import them with matomo.sh it show me the script run, but import nothing.

If i try to run import_logs.py itself, it work.

So i suspect there is something not working in /usr/local/matomo/matomo.sh

 cat /var/tmp/nginx.tmp | ./misc/log-analytics/import_logs.py --url=***** --token-auth=******  --enable-http-errors --enable-http-redirects --enable-static --enable-bots --recorders=4 --log-format-name=nginx_json -

found nothing to import but :

./misc/log-analytics/import_logs.py --url=***** --token-auth=******  --enable-http-errors --enable-http-redirects --enable-static --enable-bots --recorders=4 --log-format-name=nginx_json /var/tmp/nginx.tmp

work

Is there some documentation to fix in README.md about rsyslog /usr/local/matomo/matomo.sh syntax :

echo "${@}" | /***MY_MATOMO_FOLDER***/misc/log-analytics/import_logs.py

or is it something to fix in misc/log-analytics/import_logs.py ?

or anything else i've not spotted ?

@1000i100
Copy link
Author

1000i100 commented Nov 7, 2020

After reading this : #282 (comment)
i've updated import_logs.py to the latest from this repository.
I've change in first line : #!/usr/bin/python to #!/usr/bin/python3
and it now have the same behavor with this 2 way for calling it :

cat /var/tmp/nginx.tmp | ./misc/log-analytics/import_logs.py [...] -
./misc/log-analytics/import_logs.py [...] /var/tmp/nginx.tmp

But it does'nt import anything :

Traceback (most recent call last):
  File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.5/threading.py", line 862, in run
    self._target(*self._args, **self._kwargs)
  File "******/misc/log-analytics/import_logs.py", line 1849, in _run_bulk
    self._record_hits(hits)
  File ""******/misc/log-analytics/import_logs.py", line 1995, in _record_hits
    'requests': [self._get_hit_args(hit) for hit in hits]
  File ""******/misc/log-analytics/import_logs.py", line 1995, in <listcomp>
    'requests': [self._get_hit_args(hit) for hit in hits]
  File ""******/misc/log-analytics/import_logs.py", line 1891, in _get_hit_args
    site_id, main_url = resolver.resolve(hit)
  File ""******/misc/log-analytics/import_logs.py", line 1775, in resolve
    return self._resolve_by_host(hit)
  File ""******/misc/log-analytics/import_logs.py", line 1758, in _resolve_by_host
    site_id = self._resolve(hit)
  File ""******/misc/log-analytics/import_logs.py", line 1726, in _resolve
    res = self._get_site_id_from_hit_host(hit)
  File ""******/misc/log-analytics/import_logs.py", line 1681, in _get_site_id_from_hit_host
    url=hit.host,
  File ""******/misc/log-analytics/import_logs.py", line 1633, in call_api
    return self._call_wrapper(self._call_api, None, None, method, **kwargs)
  File ""******/misc/log-analytics/import_logs.py", line 1585, in _call_wrapper
    response = func(*args, **kwargs)
  File ""******/misc/log-analytics/import_logs.py", line 1574, in _call_api
    return json.loads(res)
  File "/usr/lib/python3.5/json/__init__.py", line 312, in loads
    s.__class__.__name__))
TypeError: the JSON object must be str, not 'bytes'

1365 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
[... Ctrl+C to stop it ...]
1365 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)

Logs import summary
-------------------

    0 requests imported successfully
    17 requests were downloads
    0 requests ignored:
        0 HTTP errors
        0 HTTP redirects
        0 invalid log lines
        0 filtered log lines
        0 requests did not match any known site
        0 requests did not match any --hostname
        0 requests done by bots, search engines...
        0 requests to static resources (css, js, images, ico, ttf...)
        0 requests to file downloads did not match any --download-extensions

Website import summary
----------------------

    0 requests imported to 0 sites
        0 sites already existed
        0 sites were created:

    0 distinct hostnames did not match any existing site:



Performance summary
-------------------

    Total time: 149 seconds
    Requests imported per second: 0.0 requests per second

Processing your log data
------------------------

    In order for your logs to be processed by Matomo, you may need to run the following command:
     ./console core:archive --force-all-websites --force-all-periods=315576000 --force-date-last-n=1000 --url='****'

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant