Better handling of big log files to avoid crash of the viewer

[arnowelzel]

Steps to reproduce

1. Set loglevel to in config/config.php to 0 ('loglevel' => 0).
2. Execute many actions which will trigger a log message to get written.
3. Open the log viewer when the logfile has more than 60000 entires.

Expected behaviour

The log viewer displays the log.

If the log is too big to be displayed, truncate to the newest n entries (where n is a value which can be handled by the frontend - for example 1000 or 5000) or paginate the log.

Actual behaviour

The log viewer crashes, so the page will get blank. It's not clear when exactly this happens. I have seen this with a logfile which counts about 62000 entries, but I believe the actual limit is lower and also depends on what browser and runtime environment you use.

Server configuration

Operating system: Ubuntu 20.04

Web server: Apache 2.4.41-4ubuntu3.10

Database: MariaDB 10.5.15

PHP version: 7.4

Nextcloud version: 24.0.0

Where did you install Nextcloud from: nextcloud.com

List of activated apps:

Enabled:
  - accessibility: 1.10.0
  - activity: 2.16.0
  - admin_audit: 1.14.0
  - apporder: 0.15.0
  - bookmarks: 10.3.1
  - calendar: 3.3.0
  - circles: 24.0.0
  - cloud_federation_api: 1.7.0
  - comments: 1.14.0
  - contacts: 4.1.0
  - contactsinteraction: 1.5.0
  - dashboard: 7.4.0
  - dav: 1.22.0
  - deck: 1.7.0
  - drawio: 1.0.2
  - external: 4.0.0
  - federatedfilesharing: 1.14.0
  - federation: 1.14.0
  - files: 1.19.0
  - files_fulltextsearch: 24.0.0
  - files_fulltextsearch_tesseract: 24.0.0
  - files_pdfviewer: 2.5.0
  - files_rightclick: 1.3.0
  - files_sharing: 1.16.2
  - files_trashbin: 1.14.0
  - files_versions: 1.17.0
  - files_videoplayer: 1.13.0
  - firstrunwizard: 2.13.0
  - forms: 2.5.0
  - fulltextsearch: 24.0.0
  - fulltextsearch_elasticsearch: 24.0.0
  - gpxpod: 4.3.0
  - keeweb: 0.6.9
  - logreader: 2.9.0
  - lookup_server_connector: 1.12.0
  - nextcloud_announcements: 1.13.0
  - notes: 4.3.1
  - notifications: 2.12.0
  - oauth2: 1.12.0
  - onlyoffice: 7.3.4
  - password_policy: 1.14.0
  - phonetrack: 0.7.0
  - photos: 1.6.0
  - polls: 3.6.1
  - privacy: 1.8.0
  - provisioning_api: 1.14.0
  - quota_warning: 1.14.0
  - serverinfo: 1.14.0
  - settings: 1.6.0
  - sharebymail: 1.14.0
  - spreed: 14.0.1
  - support: 1.7.0
  - survey_client: 1.12.0
  - systemtags: 1.14.0
  - tasks: 0.14.4
  - text: 3.5.1
  - theming: 1.15.0
  - theming_customcss: 1.11.0
  - twofactor_backupcodes: 1.13.0
  - updatenotification: 1.14.0
  - user_status: 1.4.0
  - viewer: 1.8.0
  - workflowengine: 2.6.0
Disabled:
  - encryption: 2.2.0
  - files_external: 1.15.0
  - recommendations: 0.4.0
  - user_ldap
  - weather_status: 1.3.0

Nextcloud configuration:

{
    "system": {
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "nextcloud.0x0c.de"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "skeletondirectory": "",
        "overwrite.cli.url": "https:\/\/nextcloud.0x0c.de",
        "dbtype": "mysql",
        "version": "24.0.0.12",
        "installed": true,
        "htaccess.RewriteBase": "\/",
        "maintenance": false,
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "memcache.local": "\\OC\\Memcache\\APCu",
        "knowledgebaseenabled": false,
        "theme": "",
        "simpleSignUpLink.shown": false,
        "loglevel": 2,
        "log_authfailip": true,
        "mysql.utf8mb4": true,
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpmode": "sendmail",
        "mail_smtpauthtype": "LOGIN",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "updater.release.channel": "stable",
        "trashbin_retention_obligation": "auto, 30",
        "app_install_overwrite": [
            "apporder",
            "deck",
            "keeweb",
            "appointments",
            "drawio",
            "bookmarks",
            "spreed"
        ],
        "has_rebuilt_cache": true,
        "onlyoffice": {
            "verify_peer_off": true
        },
        "default_language": "de",
        "default_phone_region": "DE",
        "check_data_directory_permissions": false,
        "log_rotate_size": 52428800
    }
}

Client configuration

Browser:
Vivaldi 5.2.2623.46 (Stable channel) (64-Bit)

Operating system:
Windows 10

Logs

Nextcloud log (data/owncloud.log)

(not relevant)

Browser log

react-dom.production.min.js:216 Error: Minified React error #188; visit https://reactjs.org/docs/error-decoder.html?invariant=188 for the full message or use the non-minified dev environment for full errors and additional helpful warnings.
    at t.findDOMNode (react-dom.production.min.js:289:284)
    at t.transition (CSSTransitionGroupChild.js:129:30)
    at t.r.componentWillEnter (CSSTransitionGroupChild.js:97:15)
    at t.s.performEnter (TransitionGroup.js:83:19)
    at TransitionGroup.js:192:21
    at Array.forEach (<anonymous>)
    at t.componentDidUpdate (TransitionGroup.js:191:17)
    at co (react-dom.production.min.js:219:502)
    at xl (react-dom.production.min.js:259:160)
    at t.unstable_runWithPriority (scheduler.production.min.js:18:343)

Uncaught (in promise) Error: Minified React error #188; visit https://reactjs.org/docs/error-decoder.html?invariant=188 for the full message or use the non-minified dev environment for full errors and additional helpful warnings.
    at t.findDOMNode (react-dom.production.min.js:289:284)
    at t.transition (CSSTransitionGroupChild.js:129:30)
    at t.r.componentWillEnter (CSSTransitionGroupChild.js:97:15)
    at t.s.performEnter (TransitionGroup.js:83:19)
    at TransitionGroup.js:192:21
    at Array.forEach (<anonymous>)
    at t.componentDidUpdate (TransitionGroup.js:191:17)
    at co (react-dom.production.min.js:219:502)
    at xl (react-dom.production.min.js:259:160)
    at t.unstable_runWithPriority (scheduler.production.min.js:18:343)

[adriangalbincea]

I have the same issue, logging page is not loading anything, just spinning forever...

[danir-de]

We can trigger the issue when changing log settings or enabling auto refresh.

[crazycorndog]

I had the very same issue, logging page was not loading anything, just spinning forever...

Nextcloud 24(.0.4)

the logfile was 24M big, with 4554 lines:

-rw-r----- 1 nextcloud nextcloud 24M 18. Aug 00:18 /var/log/nextcloud/nextcloud.log
cat /var/log/nextcloud/nextcloud.log | wc -l
4554

I renamed the original logfile and touch'ed and chown'ed a new logfile, now the logging page is loading just fine (and saying that there are no log entries yet) but the logger doesnt seem to write anything to the new logfile now, even with debug loglevel enabled, the file is still empty

edit: while executing occ maintenance:mimetype:update-js an error happened in background and got logged in the log file, but still no debug level entries or alike

[strugee]

Note: the underlying React error message is Unable to find node on an unmounted component. (mostly adding this so that if folks look up the full React error string and then search GitHub for it, this issue will show up).

[luxifr]

same here with

www-data@2d5e879532e7:~/html/data$ ls -lh nextcloud.log
-rw-r----- 1 www-data www-data 3.0M Jan  1 02:39 nextcloud.log
www-data@2d5e879532e7:~/html/data$ wc -l nextcloud.log
2752 nextcloud.log

[Antreesy]

Quickly checked at Nextcloud 28 with 10k+ rows in a logfile, requests to logfile are chunked by 50 rows,

• no visible freezes with 3k lines shown.
• local file opened with 10k lines - quite slow, but manageable

Frontend handling can be improved with virtual scroller 🚀

https://www.npmjs.com/package/vue-virtual-scroller

[panzerscope]

I am also facing issues with the log page as my browser tab will hang due to the amount of information in the logs. I can replicate this on Chrome, Edge and Firefox. This ticket is surprisingly old and the situation does not seem to be resolved. Is there an ETA on when an improvement could be expected?

[Antreesy]

Hello @panzerscope ,

Thank you for bringing it up again. I agree this would be very useful to fix. Situation was improved already this year with migration to Vue, although stil happening in some cases (high RAM load, not specific to any browser).

Unfortunately, this is not scheduled to fix yet (due to low impact, I might assume, as we do not have a lot of feedback on it from community / enterprise clients).

However, we are welcoming any contributions and can assist with reviewing and completing it. Would you (or anyone else) perhaps be up for implementing this? I left some references in the comment above, this should be a manage-able task for any frontenders interested.

[panzerscope]

Hello @panzerscope ,

Thank you for bringing it up again. I agree this would be very useful to fix. Situation was improved already this year with migration to Vue, although stil happening in some cases (high RAM load, not specific to any browser).

Unfortunately, this is not scheduled to fix yet (due to low impact, I might assume, as we do not have a lot of feedback on it from community / enterprise clients).

However, we are welcoming any contributions and can assist with reviewing and completing it. Would you (or anyone else) perhaps be up for implementing this? I left some references in the comment above, this should be a manage-able task for any frontenders interested.

Happy to try whatever will improve the situation. I assume Vue is not something I would need to implement on the front end as it has already been integrated into Nextcloud already?

[Antreesy]

Yes, I think what missing should be solved by wrapping this component:

logreader/src/components/table/LogTable.vue

Lines 38 to 41 in 61dab73

	<LogTableRow v-for="row, rowNumber in sortedRows"
	:key="rowNumber"
	:row="row"
	@show-details="showDetailsForRow" />

into this wrapper from external lib:

https://github.com/Akryum/vue-virtual-scroller/tree/v1/packages/vue-virtual-scroller#recyclescroller

[panzerscope]

Yes, I think what missing should be solved by wrapping this component:

logreader/src/components/table/LogTable.vue

Lines 38 to 41 in 61dab73

<LogTableRow v-for="row, rowNumber in sortedRows"
:key="rowNumber"
:row="row"
@show-details="showDetailsForRow" />
into this wrapper from external lib:

https://github.com/Akryum/vue-virtual-scroller/tree/v1/packages/vue-virtual-scroller#recyclescroller

I appreciate the feedback but if this is something I need to change/add on my end, I would need a more "dummy" explanation on what steps I need to undertake. Would this also be a one time change made to instance of Nextcloud?

[Antreesy]

Would this also be a one time change made to instance of Nextcloud?

Sorry for misinterpretation here. No, I was talking about a development contributions, which would require:

• setting a dev instance of Nextcloud
• forking of nextloud/logreader repository
• commiting the source code changes locally
• creating a pull request against this repository

Then we review it from our side, merge into codebase and possibly backport, so that everyone can download fixed version ☘
If you don't feel like you're able to do it, that's fine. Any volunteer or someone from engineers could take a look whenever they have time, we'd appreciate that.

[janvlug]

I came to this issue because of the extreme high memory usage in Firefox when I open the Nextcloud log viewer and I scroll and click there a little. Effectively, reading the Nextcloud logs via the Firefox does not work. In the past I did not experience this, as far as I remember. Did something change recently?