Issues with special chars on Windows and other operating systems

[fabianfrz]

The sync client may have issues with special chars like ":" on Windows. The character is perfectly valid on Linux/BSD systems and allowed by all file systems I know.

The sync client may respect OS limitations in file names by renaming them locally on Windows clients:

For reference - see: opnsense/core#2498

File name template I use in OPNsense for config files: config-<Hostname>.<Domain Name>-<Date>_<Time>.xml

Currently there seems to be no error message on Windows for this, see #1723.

[TwiggyWan]

How about having one list of disallowed characters for all the supported OS and enforcing that?

[fabianfrz]

bad idea, that blocks automation of legit files. there will be always one OS that causes troubles and sometimes you do not even know that os. In my opinion the desktop client should replace all occurrences of special characters by a replacement character like an underscore and handle that internally with a mapping file for example when using json for a .nextclound_filemappings.json in each directory which will be hidden on all operating systems if possible:

{
  "test:1234.json": "test_1234.json"
}

[TwiggyWan]

there will be always one OS that causes troubles and sometimes you do not even know that os
What does that mean? doesn't this project support a finite list of OSes?

Where is it written what characters are forbidden on the server?
I create a demo instance, go in the brower to create a file :

Okay, slashes forbidden, that makes sense.

but now i create 'test\iss'(or 'test\iss', same behavior), i do not get the same pop up telling me the file can't be named that way, but i get an error :

So this file name should be legit because it is legit on linux but is somehow forbidden on the server already, when 'test:iss' is allowed. So from my perspective this looks like character sanitization is already performed.

[Xansala]

Maybe just one thing to add which I mentioned in #1723 : the issue is also that those forbidden characters are not properly reported. At least in the past version, files with the forbidden character had been ignored but the sync was shown as successful (please have a look at the issue, the log file is there). Although it is visible in the log files but which user apart from us freaks is digging through that? ;) It would be very helpful to point non-synced filed clearly to the user including the ones not synced due to forbidden character.

[fabianfrz]

@TwiggyWan you can use NextCloud with any operating system, that has a working TCP stack and supports http. That means that there are also operating systems like OS/2 (for example used at ATMs), FreeDOS (yes that supports networking), Symbian (old nokia phone OS), Tizen (for example used on smart TV), and much more.

All of them can upload files to the server but the sync client needs to be able to handle all of them in some way. Also if there are users of different countries, you might have even some special characters in the file name. Maybe you have a Chinese filename and want to write that to a file system that is not Unicode capable. The only system that has that information is the host running the sync client.

[TwiggyWan]

Okay. AFAIK there is a notion of server capabilities in case of a sync up but nothing regarding client capability in case of a sync down :

desktop/src/libsync/syncengine.cpp

Lines 1094 to 1114 in 78fa448

	QString invalidFilenamePattern = _account->capabilities().invalidFilenameRegex();
	if (invalidFilenamePattern.isNull()
	&& _account->serverVersionInt() < Account::makeServerVersion(8, 1, 0)) {
	// Server versions older than 8.1 don't support some characters in filenames.
	// If the capability is not set, default to a pattern that avoids uploading
	// files with names that contain these.
	// It's important to respect the capability also for older servers -- the
	// version check doesn't make sense for custom servers.
	invalidFilenamePattern = R"([\:?*"<>|])";
	}
	if (!invalidFilenamePattern.isEmpty()) {
	const QRegExp invalidFilenameRx(invalidFilenamePattern);
	for (const auto &syncItem : qAsConst(syncItems)) {
	if (syncItem->_direction == SyncFileItem::Up
	&& isFileModifyingInstruction(syncItem->_instruction)
	&& syncItem->destination().contains(invalidFilenameRx)) {
	syncItem->_errorString = tr("File name contains at least one invalid character");
	syncItem->_instruction = CSYNC_INSTRUCTION_IGNORE;
	}
	}
	}

And this is probably why there is no error message given to the user in #1723 as the error seems to be raised only for sync up.

According to what is in place, there is no notion of replacing characters valid on the client and invalid on the server, the sync just fails. What you suggested above @fabianfrz would mean that for sync down, we would perform renaming, but not for sync up. Wouldn't that just be confusing for the users?

[fabianfrz]

@TwiggyWan in case of sync up I think this can be handled under the hood by the server. Nobody says that the filename on the real file system must match the one stored on disk. For example the server running on windows could just accept a file called test:file.txt, store it as {b0693f9e-9ea8-4886-8c57-7823cd564677} and in the directory listing and on access it can offer it as test:file.txt again. Usually the servers are running on Linux or BSD so usually they have some modern file systems that can handle strange file names. In general, there should be anything allowed that is compliant to the WebDAV standard and any common operating system (no file you can create on Linux, Windows or Mac OS should be rejected because of the file name).

[claell]

I edited the description to reflect what @Aldaris1985 mentioned (no error message on Windows - #1723).

Also changed the title, since this issue seems to be treated like some meta issue now for any special character related problems on any operating system. I'd rather suggest not to do that, since the underlying problems might be pretty different, but for now at least the title correctly reflects again what this issue is about (now).

Also added the labels high (because there is no notificaiton of a failed sync on Windows in that case) and technical debt (since dealing with special characters should have been done directly when implementing, not afterwards).

[Githopp192]

When i downloaded those Files with COLON ":" directly from the NC-Web Gui - Nextcloud did a rename in that Nextcloud Web Gui did replace the ":" (Colon) with "_" (underscore).

The question is ..why not to trigger this behave to the Agent also ?

(in my case Level 1 should had do the job --> replace ":" with "_" on the iPhone App Level)

At least - as this is not fixed - the Nextcloud Agent on Windows (or anywhere else) should report an Error.

(as claell also already commented)

But this will not happen - Windows Agent is ok - so the users will not be aware of that issue and this could be a risk, if users or application will expect those files properly synchronized.

[k9t9]

Reported a similar problem ( trailing DOT - e.g. ". " about a year ago - but was reported on server repo

nextcloud/server#18066

At the moment I installed the latest release yesterday - and now have also problem with HASH symbol - e.g. #

A folder name like #SYNCH# created on the server will be synched like SYNCH ( hash characters stripped ) and if created or existing on local - will be ignored . ( win 10 - client 3.1.1-x64 )

By the look of the other open issues - there seems to be some problems with all kind of characters like HASH # , DOT . WHITESPACE etc..

[Pit-Storm]

I had a related behaviour with a file valid on iOS but not synching to windows client. It is really confusing, that there is no message shown by the sync client showing that some file(s) hadn't been synced...

A notification should basically be implemented. So UX is been guaranteed and you are winning time to implement a overall solution. (My opinion as a private user :-) )

Thanks for that very cool product!

[lucasgruembel]

Hello,

as i can see, this issue not got updated since march.

We have a customer which reports us, that some folders were not syncroniced to they Windows Clients.

The folder were created on the IOS App with colon's in the folder name. On all IOS Devices and in the WebGUI the folder is visible. But on the Windows Client the folder does not get synced. I disconnect the folder Syncronisation and created an new Syncronisation in the Client, without any effekt. After this i noticed in the WebGUI, that the folder has colon's in his name. After removeing this colon's from the folder name, the folder got synced to the Windows Client System.

Expected behaviour

Folders with colon's should be synced to Windows Clients too or the Windows Client should throw an Error, with the explenation that this char is not supported on Windows Clients.

Actual behaviour

Folders with colon in name, not get synced to Windows Clients

Steps to reproduce

1. Create a folder with colon's in the WebGUI or on the IOS App
2. Syncronise this folder to an Windows Client
3. Folder not apperes

Client configuration

Client version: 3.3.5.20210930

Operating system: Windows Server 2019 with RDSH Roll (e.q. Windows 10 1809) Build 17763.2114

OS language: German

Installation path of client: C:\Program Files\Nextcloud

Nextcloud Server 21.0.2
Group folders 9.0.2

[Githopp192]

Dear Nextcloud GmbH

it is really important to fix these errors. All of those has to do with "data consistency" - at least - the Agent should report an error.

[claell]

Please don't forget to vote on the issue description to make this more prominent.

[PhilippSchlesinger]

@claell, @FlexW
is #3991 indeed a duplicate of this bug?

Since there is a resolution dialog for invalid filenames (#3760), I get warnings on every new clients release for hundrets of filenames containing invalid chars (":", ",", "?", ...) as my mother tried to use filenames to describe image content in full sentences ;-)
But after resolving only a few of the listed filename problems, the list of problems disappears after the next sync even if there would be lots of remaining invalid filenames.

I'd be happy to provide more info or logs if needed.

[labor4]

Note on the colon on Macos
":" are illegal on the Macos GUI and prevented from entering, but valid on the Shell (Terminal)
Putting a ":" in Terminal will result in a "/" in the GUI.
Putting a "/" in GUI will result in a ":" in the Terminal.
It may be an inconsitency in iOS to allow entering ":" in iOS GUI.
Background:
Macos GUI useD ":" to communicate path-delimiters
Macos Terminal useS "/" to communicate path-delimiters (as in Linux)
... I could not find a way to enter "/", as in touch oh\/dear.txt, in Macos Terminal.

BTW: There may be many reasons why this worked in GUI for the lady mentioned above.
I've been using Macs since around 1992 and I have seen happen everything with (and to, and because of) Macs. 😀

Note on combining diaeresis (Umlauts)
Further in "Macos Char weirdnesss" I place here a char-related note on
"Macos construction of Umlauts using dual-characters", also present in other "accents" like in "é",
because the specific reason for it proofed hard to find on the net.
It may be though, that this is only present in older filenames (from older Macos'es),
as I could not reconstruct it in Macos Mojave (10.14).
If some Umlaut files cannot be grabbed, it may be for that reason!

https://superuser.com/a/1196604

[mgallien]

Related PR to also make sure we find a way to sync all files despite the limitations of the operating system
#467

[mgallien]

@tobiasKaminsky

[FernandoMarques-Santos]

Since any solution above is not acceptable in every scenario and may cause headaches, we may give each user a choice via user configuration: say, under settings/user/sync-clients ('Mobile & Desktop' config page). Then inside we have these checkboxes:

• Compatibility sync: automatically renames files with illegal characters for Windows on the Server side.
• Send a notification when a rename is triggered

A simple script in a workflow-like rule may rename these files when they arrive.

[sm18lr88]

I keep getting sync errors for my second NTFS partition in Windows 11, because of a hidden folder named $Recycle.bin

I'd appreciate this being solved and not constantly seeing a sync error.

[mgallien]

related PRs
#5783
#5232
#4454
#5801

this cover invalid characters that would be refused by the server and handling of leading/trailing spaces and case clash conflicts
we currently do not download files with invalid names on windows but the desktop client allows direct renaming from the main dialog on activities matching those files with invalid names

[mgallien]

I am closing this one as we may not want to download files with invalid characters in their name and substitute those invalid characters as it would be hard to make this work in every corner cases
alternatively we want to enable the user to rename them
if someone wants to not rename them, windows clients can be configured to ignore those files with invalid names

[Githopp192]

@mgallien : as i wrote once :

at least - the Agent should report an error.

Because, if the agent does not report an error - the user isn't aware of it.

[Xansala]

@mgallien Is there an error message when there is an invalid character? If not, then the issue is not resolved!

[Githopp192]

All of those has to do with "data consistency" (Error Reporting). At the end this means:

1.) Desktop Agents Reports reports an error
2.) Desktop Agent does NOT SHOW GREEN STATUS !

[mgallien]

@Xansala please avoid reopening an issue unless you are prepared to provide convincing arguments