[Bug]: Security warning inaccurately reports that X-Robots-Tag header isn't "noindex, nofollow", documentation links broken

[wwklnd]

⚠️ This issue respects the following points: ⚠️

• This is a bug, not a question or a configuration/webserver/proxy issue.
• This issue is not already reported on Github (I've searched it).
• Nextcloud Server is up to date. See Maintenance and Release Schedule for supported versions.
• Nextcloud Server is running on 64bit capable CPU, PHP and OS.
• I agree to follow Nextcloud's Code of Conduct.

Bug description

I updated Nextcloud Server to 26.0.0 today, and faced some odd issues. My setup went from no warnings to this:

• Some files have not passed the integrity check. Further information on how to resolve this issue can be found in the documentation ↗. (List of invalid files… / Rescan…)
• The "X-Robots-Tag" HTTP header is not set to "noindex, nofollow". This is a potential security or privacy risk, as it is recommended to adjust this setting accordingly.
• This instance is missing some recommended PHP modules. For improved performance and better compatibility it is highly recommended to install them.
  • sysvsem

I checked both the .htaccess file and the actual HTTP headers. Firefox reports two X-Robots-Tag headers, one of which reads "none" and one of which reads "noindex, nofollow", so the warning is correct about one header but incorrect overall. I'm not sure where the "none" comes from, though.

When I click the "documentation ↗" link, it takes me to this page which simply shows a "File not found." message in black on white. The same thing happens with the "installation guide ↗" link, here.

Apart from this, after upgrading to 26.0.0 I also noticed that my user avatar had its colours inverted, but only in the top right corner, which seems incredibly odd. Image for reference, showing what the image looks like on my user profile (which renders it properly):

Steps to reproduce

1. Upgrade from latest 25.x.x version to 26.0.0 using the OTA updater on the Administration -> Overview page.
2. After the upgrade, go to the Administration -> Overview page again.

Expected behavior

1. I expect there not to be a warning about the HTTP headers since there is a "noindex, nofollow" header present.
2. I expect the links to the documentation on the Administration -> Overview page to lead to the proper pages.
3. I expect my user avatar to render properly.

Installation method

Community Docker image

Operating system

Debian/Ubuntu

PHP engine version

PHP 8.0

Web server

Nginx

Database engine version

MariaDB

Is this bug present after an update or on a fresh install?

Updated to a major version (ex. 22.2.3 to 23.0.1)

Are you using the Nextcloud Server Encryption module?

Encryption is Disabled

What user-backends are you using?

• Default user-backend (database)
• LDAP/ Active Directory
• SSO - SAML
• Other

Configuration report

{
    "system": {
        "memcache.local": "\\OC\\Memcache\\APCu",
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "cloud.wklnd.me"
        ],
        "dbtype": "mysql",
        "version": "26.0.0.11",
        "overwrite.cli.url": "https:\/\/cloud.wklnd.me",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "mysql.utf8mb4": true,
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "mail_smtpmode": "smtp",
        "mail_smtpsecure": "ssl",
        "mail_sendmailmode": "smtp",
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpauth": 1,
        "mail_smtpauthtype": "LOGIN",
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpport": "465",
        "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
        "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
        "default_phone_region": "SE",
        "loglevel": "2",
        "maintenance": false,
        "theme": "",
        "updater.secret": "***REMOVED SENSITIVE VALUE***"
    }
}

List of activated Apps

Enabled:
  - activity: 2.18.0
  - admin_audit: 1.16.0
  - calendar: 4.3.1
  - circles: 26.0.0
  - cloud_federation_api: 1.9.0
  - comments: 1.16.0
  - contacts: 5.2.0
  - contactsinteraction: 1.7.0
  - dashboard: 7.6.0
  - dav: 1.25.0
  - deck: 1.9.0
  - federatedfilesharing: 1.16.0
  - federation: 1.16.0
  - files: 1.21.1
  - files_external: 1.18.0
  - files_pdfviewer: 2.7.0
  - files_rightclick: 1.5.0
  - files_sharing: 1.18.0
  - files_trashbin: 1.16.0
  - files_versions: 1.19.1
  - firstrunwizard: 2.15.0
  - integration_mastodon: 2.0.1
  - integration_twitter: 1.0.5
  - logreader: 2.11.0
  - lookup_server_connector: 1.14.0
  - mail: 3.0.0
  - nextcloud_announcements: 1.15.0
  - notifications: 2.14.0
  - oauth2: 1.14.0
  - password_policy: 1.16.0
  - photos: 2.2.0
  - privacy: 1.10.0
  - provisioning_api: 1.16.0
  - recommendations: 1.5.0
  - related_resources: 1.1.0-alpha1
  - richdocuments: 8.0.0
  - serverinfo: 1.16.0
  - settings: 1.8.0
  - sharebymail: 1.16.0
  - spreed: 16.0.0
  - support: 1.9.0
  - survey_client: 1.14.0
  - systemtags: 1.16.0
  - tasks: 0.14.5
  - text: 3.7.2
  - theming: 2.1.1
  - twofactor_backupcodes: 1.15.0
  - twofactor_totp: 8.0.0-alpha.0
  - updatenotification: 1.16.0
  - user_status: 1.6.0
  - viewer: 1.10.0
  - weather_status: 1.6.0
  - workflowengine: 2.8.0
Disabled:
  - bruteforcesettings: 2.6.0
  - encryption: 2.14.0
  - files_3dmodelviewer: 0.0.7 (installed 0.0.7)
  - suspicious_login: 4.4.0
  - user_ldap: 1.16.0

Nextcloud Signing status

Technical information
=====================
The following list covers which files have failed the integrity check. Please read
the previous linked documentation to learn more about the errors and how to fix
them.

Results
=======
- core
	- INVALID_HASH
		- core/js/mimetypelist.js

Raw output
==========
Array
(
    [core] => Array
        (
            [INVALID_HASH] => Array
                (
                    [core/js/mimetypelist.js] => Array
                        (
                            [expected] => 4b0a99fbab7bc4824176101150bd0ab2b553e128a390e0f872580db4541462b31aad13099fd41533b6695d1b185dabca26a45e9a78227cfb9479eafa439c7bff
                            [current] => a8e818876a7eff2a2137601eafc6c7d96b99cbf27d94ff9bb59eb237bdab92076ecc2828242bb37551c9c8b6772c0718887bd49cee5b49b3be48d3b9ecaa558f
                        )

                )

        )

)

Nextcloud Logs

No response

Additional info

I installed Nextcloud using the linuxserver/nextcloud docker image, but I don't believe this should be relevant to the problem.

[szaimen]

cc @MichaIng

[solracsf]

@wwklnd you should split your reports in 3 please:

• This one about the Header tag
• a 2nd about docs link
• a 3rd about avatar

This will lead to a better follow up. Thanks 👍

[MichaIng]

Firefox reports two X-Robots-Tag headers, one of which reads "none" and one of which reads "noindex, nofollow", so the warning is correct about one header but incorrect overall.

This was and is the same with all headers Nextcloud checks for: If there are two identical headers, it intentionally prints the warning, even if both contain the same "correct" value. We do not know how (and which) search engines deal with two headers, whether they respect the stronger, the weaker, the first, the second or none. It makes sense IMO to expect a single unambiguous header value, since its about the security and privacy of your data. Find the source for the second header, eliminate it, and the warning will be resolved. Or ignore it and hope for the best.

I guess you use the Apache2 webserver? The .htaccess shipped with Nextcloud assures that no 2 headers are set: It unsets headers from the "onsuccess" table and sets them in the "always" table. And since this is the last (highest priority) config applied by Apache2 for the Nextcloud directory, it should be correct what it sends out. So in this case the culprit would be e.g. on a proxy or CDN in front of your Apache2 webserver.
EDIT: Ah, I just see that you use Nginx. In this case .htaccess is not used at all and you need to check your Nginx configs instead. I didn't know that Nginx is even possible to send our two identical header keys (like Apache2). Proxy or CDN is still a possible culprit.
EDIT2:

Community Docker image

Is is actually supported/intended to update a Nextcloud Docker container via admin panel? I thought, and it makes sense, to update it only with the container itself. Would be quite a logical explanation that the NC25 Docker container's Nginx config still ships the none header, while you need to wait for a NC26 version of the container. The same may or may not be responsible for your avatar issues, and the missing PHP module.

---

I can confirm the broken integrity check shortcut link. Actually the whole NC26 docs version is missing: https://docs.nextcloud.com/server/26/
This should fix it: nextcloud/documentation#9947

[solracsf]

I can confirm that i had the Warning with Nginx, after upgrade from 25.0.4 to 25.0.5, because I had a configuration file setting the header to none.

After changing that file to noindex, nofollow warning is gone.

[wwklnd]

@wwklnd you should split your reports in 3 please:

* This one about the Header tag

* a 2nd about docs link

* a 3rd about avatar

This will lead to a better follow up. Thanks +1

Ah, sorry! I'll do that!

[wwklnd]

Firefox reports two X-Robots-Tag headers, one of which reads "none" and one of which reads "noindex, nofollow", so the warning is correct about one header but incorrect overall.

This was and is the same with all headers Nextcloud checks for: If there are two identical headers, it intentionally prints the warning, even if both contain the same "correct" value. We do not know how (and which) search engines deal with two headers, whether they respect the stronger, the weaker, the first, the second or none. It makes sense IMO to expect a single unambiguous header value, since its about the security and privacy of your data. Find the source for the second header, eliminate it, and the warning will be resolved. Or ignore it and hope for the best.

Ah! That makes sense.

I guess you use the Apache2 webserver? The .htaccess shipped with Nextcloud assures that no 2 headers are set: It unsets headers from the "onsuccess" table and sets them in the "always" table. And since this is the last (highest priority) config applied by Apache2 for the Nextcloud directory, it should be correct what it sends out. So in this case the culprit would be e.g. on a proxy or CDN in front of your Apache2 webserver. EDIT: Ah, I just see that you use Nginx. In this case .htaccess is not used at all and you need to check your Nginx configs instead. I didn't know that Nginx is even possible to send our two identical header keys (like Apache2). Proxy or CDN is still a possible culprit.

This also makes sense, I wasn't sure if .htaccess was relevant since I use Nginx. I'll do some digging in the config files to see what's up.

EDIT2:

Community Docker image

Is is actually supported/intended to update a Nextcloud Docker container via admin panel? I thought, and it makes sense, to update it only with the container itself. Would be quite a logical explanation that the NC25 Docker container's Nginx config still ships the none header, while you need to wait for a NC26 version of the container. The same may or may not be responsible for your avatar issues, and the missing PHP module.

I've always run upgrades from the admin panel before without issue, so I thought that was the proper way to do it. I'm using the linuxserver/nextcloud docker container which is already at 26.0.0 and is as I understand it just a repackaging of the mainline Nextcloud build with Nginx. I might ask them if it might be something that needs fixing on their end. Pulling the latest docker image did fix the PHP module error.

I can confirm the broken integrity check shortcut link. Actually the whole NC26 docs version is missing: https://docs.nextcloud.com/server/26/ This should fix it: nextcloud/documentation#9947

Thank you for the extensive reply! I appreciate it. :)

[wwklnd]

I can confirm that i had the Warning with Nginx, after upgrade from 25.0.4 to 25.0.5, because I had a configuration file setting the header to none.

After changing that file to noindex, nofollow warning is gone.

Thank you for the reply, I went and changed this in the default.conf file and the error went away. Closing this issue since the main issue is resolved, I believe. I'll contact the linuxserver/nextcloud maintainers about the conf file.

[MichaIng]

If this is a Docker container, I guess using the Nextcloud updater is not intended. If I'm not mistaken, it is possible to disable it via config.php, which would then make sense for this Docker image. So any Nextcloud update would come via container update, which allows the maintainers as well to ship updated Nginx configs and such when required.

[wwklnd]

@MichaIng The Nextcloud updater is listed as the first option in the container readme, with the caveat that the latest image should be pulled first. I spoke to one of the linuxserver/nextcloud maintainers on Discord, and they said that it is preferable to use the updater.phar command they provide because it is interactive and shows the full update log.

[MichaIng]

It is however quite an uncommon way of using Docker containers and breaks major benefits/intentions of using them. One of the major points of using Docker containers is that you have a fixed setup which is precisely composed so that all components are assured to work with each other, in this case database, PHP, webserver and web application, possible Redis server and others. Containers are usually not designed/intended to be manipulated by accessing the internal console and change the system. The same way it is usually not intended to update software within the container, but only to update the whole container image and use it like that until a new updated image is available. Also, not only are all your customisations lost with a container upgrade, also the software (Nextcloud) update could be reverted if a new container does not yet ship the latest software version.

Sticking with only container updates would have prevented all your tree issues + the missing sysvsem module, since all this will be addressed with a new container image when Nextcloud 26 is included. Although not sure where the inverted avatar is coming from, which may be a NC26 bug, if the false core/js/mimetypelist.js is not the reason.

[reinob]

I can confirm that i had the Warning with Nginx, after upgrade from 25.0.4 to 25.0.5, because I had a configuration file setting the header to none.

After changing that file to noindex, nofollow warning is gone.

I updated from 25.0.4 to 25.0.5 and also got that warning, which is a bug in itself, as "none" (which is what I had) is equivalent to "noindex, nofollow" (see https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag), to Nextcloud should not warn if the header value is "none".

[MichaIng]

Can you please read through this thread and answer yourself that "none" is NOT equivalent to "noindex, nofollow"?
EDIT: Sorry, I mixed up the issues.

[reinob]

I had to find the relevant commit (5f90b8e) to know the rationale for this change, which I could not find in this thread here (which only mentions the issue of having duplicate headers).

So OK.

[MichaIng]

Ah sorry, I got confused with two other issues about this: #37355 and #37386

As an admin, if you want to stay informed about changes like this, I recommend to subscribe to #34692 and its follow-ups #37039, which are also pinned to the top of the issues page.

[nao982]

for me it is solved
next cloud plugins the truenas core
I changed from "none" to "noindex, nofollow" in the file /mnt/Truenas/iocage/jails/nextcloud/root/usr/local/etc/nginx/conf.d/nextcloud.inc

[MichaIng]

Right, compare with latest docs as well: https://docs.nextcloud.com/server/latest/admin_manual/installation/nginx.html

[funky123]

changing that file to noindex, nofollow warning is gone.

where do i have to set this?

[MichaIng]

Depends on your webserver and config structure. The same place where you set the X-Robots-Tag: none before, replacing it.

[funky123]

NC runs at a Debian Container Apache Webserver behind an other container running Nginx Reverse Proxy manager.

[MichaIng]

With Apache webserver, you just need to take care that NPM forwards the X- headers to clients. The .htaccess shipped with Nextcloud sets the header correctly (of course).

I assume that this Apache webserver generally reads .htaccess files, as this is the default when installing it on Debian from their APT repo.

[Poikilos]

changing that file to noindex, nofollow warning is gone.

For nginx the new setting from the documentation didn't help. I changed my /etc/nginx/sites-enabled/....conf (where ... is my website):

-        add_header X-Robots-Tag                         "none"          always;
+        # add_header X-Robots-Tag                         "none"          always;
+        add_header X-Robots-Tag                         "noindex, nofollow"          always;

Then:

• systemctl reload nginx
• refresh browser (error is still there)
• systemctl restart php8.1-fpm (I ensured that is the version used in /etc/nginx/conf.d/10-fpm.conf)
• systemctl restart nginx
• refresh browser (error is still there)
• log out, log back in (error is still there)

Version: Nextcloud Hub 6 (27.1.0)

[MichaIng]

Verify that the header really is set, e.g. like

curl -I 127.0.0.1

[Poikilos]

Verify that the header really is set, e.g. like

curl -I 127.0.0.1

yes:

HTTP/1.1 200 OK
Server: nginx/1.14.2
Date: Thu, 21 Sep 2023 00:03:29 GMT
Content-Type: text/html
Content-Length: 5779
Last-Modified: Sun, 25 Jul 2021 22:11:39 GMT
Connection: keep-alive
Vary: Accept-Encoding
ETag: "60fde19b-1693"
X-Content-Type-Options: nosniff
X-Download-Options: noopen
X-Frame-Options: SAMEORIGIN
X-Permitted-Cross-Domain-Policies: none
X-Robots-Tag: noindex, nofollow
X-XSS-Protection: 1; mode=block
Accept-Ranges: bytes

[MichaIng]

This is not the Nextcloud instance, is it? Is it within a sub directory, like

curl -IL 127.0.0.1/nextcloud

[Poikilos]

This is not the Nextcloud instance, is it? Is it within a sub directory, like

curl -IL 127.0.0.1/nextcloud

Ugh good point. The setting isn't showing up if I put /nextcloud in the URL after curl -I. I put the setting in the location / { section by mistake. Nginx is so painful...
If I put it in the

location ^~ /nextcloud {

section it is fixed now (Someone make an Nginx configuration GUI, and not just a text editor). Thank you.

[MichaIng]

This is one of the things I do not like about Nginx: Once a location/if/... block contains any add_header directory, none of the parent block add_header directives are added anymore. So naturally it can lead to a lot of repetitiv add_header directives, e.g. when you adjust browser caching per location/file/mime types etc: https://nginx.org/en/docs/http/ngx_http_headers_module.html#add_header

There could be several add_header directives. These directives are inherited from the previous configuration level if and only if there are no add_header directives defined on the current level.