encfs filename too long

[Germar]

Hi, first of all thanks for providing the ssh and ssh+encfs options in backintime!

Unfortunately I can't use the sshfs+encfs mode. I'm getting "File name too long errors". Here is an extract from the logs:

[I] Take snapshot (rsync: JA-tGYW9HxC21cPbI7smjare/eWn4krRY0gwkUKYWeFJGX9u5/x8ew-s9XxzhKAMifGBfx1wqmyR1ewULTBxj-W-ibbYcKJ-/n0lxVMEx19Izm0W4cvbMGy0w/lnNRH9CBUPvVv2rbDQxB6DpMlYcTJbYMoxeeNGFKdvbyo0/XreDMzKpTyUtrpcKxUAzLnRZgfFAeYuNVhHls2ma,Zwnj0)
[I] Take snapshot (rsync: rsync: recv_generator: failed to stat "/share/backup/backintime/UqUO2ytveGGjBq3qzGsgX6Py/F,VDzqn3,cAbylQZJNwRWWYT/yqzTd8DssQsLmqZ5pqMeXtiC/0,m,z4Xv,uVJMJMq,7ElnUsh/dvP0RHayype,rqHv0XXDYFfF/grPaOkZAp08oDalSqosv0rZ0/JA-tGYW9HxC21cPbI7smjare/eWn4krRY0gwkUKYWeFJGX9u5/x8ew-s9XxzhKAMifGBfx1wqmyR1ewULTBxj-W-ibbYcKJ-/n0lxVMEx19Izm0W4cvbMGy0w/lnNRH9CBUPvVv2rbDQxB6DpMlYcTJbYMoxeeNGFKdvbyo0/mzomOpgVLgnB2yA9l5d3VMQ6X3Nu6TFfyMHfpZiyGggUoZv7aD3of4Z2OpbVaq,VH1R-cI5,8koUqXpCUJ84KcjeIH2fdxf2REeYc-nfDgEJkmpFPiwsGIuegSej-PUtZI7N,VOBtJdY-TcIzpOmJ6USxX7o2sx6Ja73KuCwCSJBAJCzYdcC5elNRprcVwIh72ZAL8tuI7lW6K0yt,SJW5iolvpY1jKxCRPiwJZauqjNWHyWv8XZdgSZ0lInsz088v8": File name too long (36))
[E] Error: rsync: recv_generator: failed to stat "/share/backup/backintime/UqUO2ytveGGjBq3qzGsgX6Py/F,VDzqn3,cAbylQZJNwRWWYT/yqzTd8DssQsLmqZ5pqMeXtiC/0,m,z4Xv,uVJMJMq,7ElnUsh/dvP0RHayype,rqHv0XXDYFfF/grPaOkZAp08oDalSqosv0rZ0/JA-tGYW9HxC21cPbI7smjare/eWn4krRY0gwkUKYWeFJGX9u5/x8ew-s9XxzhKAMifGBfx1wqmyR1ewULTBxj-W-ibbYcKJ-/n0lxVMEx19Izm0W4cvbMGy0w/lnNRH9CBUPvVv2rbDQxB6DpMlYcTJbYMoxeeNGFKdvbyo0/mzomOpgVLgnB2yA9l5d3VMQ6X3Nu6TFfyMHfpZiyGggUoZv7aD3of4Z2OpbVaq,VH1R-cI5,8koUqXpCUJ84KcjeIH2fdxf2REeYc-nfDgEJkmpFPiwsGIuegSej-PUtZI7N,VOBtJdY-TcIzpOmJ6USxX7o2sx6Ja73KuCwCSJBAJCzYdcC5elNRprcVwIh72ZAL8tuI7lW6K0yt,SJW5iolvpY1jKxCRPiwJZauqjNWHyWv8XZdgSZ0lInsz088v8": File name too long (36)
[I] Take snapshot (rsync: JA-tGYW9HxC21cPbI7smjare/eWn4krRY0gwkUKYWeFJGX9u5/x8ew-s9XxzhKAMifGBfx1wqmyR1ewULTBxj-W-ibbYcKJ-/n0lxVMEx19Izm0W4cvbMGy0w/lnNRH9CBUPvVv2rbDQxB6DpMlYcTJbYMoxeeNGFKdvbyo0/vnneRbw8oy43kglQEyhYK5SnH9KPGmCgvis7XgPfMTeY8RKZzXCu1l8V8,hVbZSez2qKHJvtGV-ZBIHOed6msQTOZntpIHbjnV7dynYUgkkyJfPfQ4zSPACzeqwrUy7ZUrKp5PZ--qbVlWsu-uc4v0CetsSIk5aEw8IPnhhVBKVHbCxSSOv-bTt61wTquXIM9p2L1gR1AXdCdQx-7aAAaJ1Q4GlsW5sL6aU3d7gz1snCTLLikqqWO1suMojq2Y9vlB9/)
[I] Take snapshot (rsync: rsync: recv_generator: mkdir "/share/backup/backintime/UqUO2ytveGGjBq3qzGsgX6Py/F,VDzqn3,cAbylQZJNwRWWYT/yqzTd8DssQsLmqZ5pqMeXtiC/0,m,z4Xv,uVJMJMq,7ElnUsh/dvP0RHayype,rqHv0XXDYFfF/grPaOkZAp08oDalSqosv0rZ0/JA-tGYW9HxC21cPbI7smjare/eWn4krRY0gwkUKYWeFJGX9u5/x8ew-s9XxzhKAMifGBfx1wqmyR1ewULTBxj-W-ibbYcKJ-/n0lxVMEx19Izm0W4cvbMGy0w/lnNRH9CBUPvVv2rbDQxB6DpMlYcTJbYMoxeeNGFKdvbyo0/vnneRbw8oy43kglQEyhYK5SnH9KPGmCgvis7XgPfMTeY8RKZzXCu1l8V8,hVbZSez2qKHJvtGV-ZBIHOed6msQTOZntpIHbjnV7dynYUgkkyJfPfQ4zSPACzeqwrUy7ZUrKp5PZ--qbVlWsu-uc4v0CetsSIk5aEw8IPnhhVBKVHbCxSSOv-bTt61wTquXIM9p2L1gR1AXdCdQx-7aAAaJ1Q4GlsW5sL6aU3d7gz1snCTLLikqqWO1suMojq2Y9vlB9" failed: File name too long (36))
[E] Error: rsync: recv_generator: mkdir "/share/backup/backintime/UqUO2ytveGGjBq3qzGsgX6Py/F,VDzqn3,cAbylQZJNwRWWYT/yqzTd8DssQsLmqZ5pqMeXtiC/0,m,z4Xv,uVJMJMq,7ElnUsh/dvP0RHayype,rqHv0XXDYFfF/grPaOkZAp08oDalSqosv0rZ0/JA-tGYW9HxC21cPbI7smjare/eWn4krRY0gwkUKYWeFJGX9u5/x8ew-s9XxzhKAMifGBfx1wqmyR1ewULTBxj-W-ibbYcKJ-/n0lxVMEx19Izm0W4cvbMGy0w/lnNRH9CBUPvVv2rbDQxB6DpMlYcTJbYMoxeeNGFKdvbyo0/vnneRbw8oy43kglQEyhYK5SnH9KPGmCgvis7XgPfMTeY8RKZzXCu1l8V8,hVbZSez2qKHJvtGV-ZBIHOed6msQTOZntpIHbjnV7dynYUgkkyJfPfQ4zSPACzeqwrUy7ZUrKp5PZ--qbVlWsu-uc4v0CetsSIk5aEw8IPnhhVBKVHbCxSSOv-bTt61wTquXIM9p2L1gR1AXdCdQx-7aAAaJ1Q4GlsW5sL6aU3d7gz1snCTLLikqqWO1suMojq2Y9vlB9" failed: File name too long (36)
[I] Take snapshot (rsync: *** Skipping any contents from this failed directory ***)
[I] Take snapshot (rsync: JA-tGYW9HxC21cPbI7smjare/eWn4krRY0gwkUKYWeFJGX9u5/x8ew-s9XxzhKAMifGBfx1wqmyR1ewULTBxj-W-ibbYcKJ-/qVhg5WLM4Cu6-m0PjX,yBXlS/)

The backup is written to a QNAP fileserver. (linux-based; filesystem on the RAID is ext4.)
I'm using backintime 1.0.34 as provided by Ubuntu 14.04.

Here is one of the problematic path names broken up by directory/filename:

UqUO2ytveGGjBq3qzGsgX6Py/
F,VDzqn3,cAbylQZJNwRWWYT/
yqzTd8DssQsLmqZ5pqMeXtiC/
0,m,z4Xv,uVJMJMq,7ElnUsh/
dvP0RHayype,rqHv0XXDYFfF/
grPaOkZAp08oDalSqosv0rZ0/
JA-tGYW9HxC21cPbI7smjare/
eWn4krRY0gwkUKYWeFJGX9u5/
x8ew-s9XxzhKAMifGBfx1wqmyR1ewULTBxj-W-ibbYcKJ-/
n0lxVMEx19Izm0W4cvbMGy0w/
lnNRH9CBUPvVv2rbDQxB6DpMlYcTJbYMoxeeNGFKdvbyo0/
vnneRbw8oy43kglQEyhYK5SnH9KPGmCgvis7XgPfMTeY8RKZzXCu1l8V8,hVbZSez2qKHJvtGV-ZBIHOed6msQTOZntpIHbjnV7dynYUgkkyJfPfQ4zSPACzeqwrUy7ZUrKp5PZ--qbVlWsu-uc4v0CetsSIk5aEw8IPnhhVBKVHbCxSSOv-bTt61wTquXIM9p2L1gR1AXdCdQx-7aAAaJ1Q4GlsW5sL6aU3d7gz1snCTLLikqqWO1suMojq2Y9vlB9

As you can see the encrypted filename itself is 259 characters long, which exceeds the max. filename length of ext4.

Just as I'm typing this, I've found it to be an encfs bug:
https://code.google.com/p/encfs/issues/detail?id=157
Until this is fixed, the only workaround seems to be to shorten my filenames.

---

Imported from Launchpad using lp2gh.

• date created: 2014-08-21T14:15:15Z
• owner: shigi
• the launchpad url was https://bugs.launchpad.net/bugs/1359776

[Germar]

(by shigi)
Addendum:
Using
find | egrep '.[^/]{200}'
(with different numeric values) I found that the longest (cleartext) file name was actually a directory name and it's less than 190 characters!
(FWIW the directory name was automatically created by Firefox while saving an offline copy of a web page.)

[Germar]

(by germar)
You could try out to manually create a new encfs config with standard settings but filename encoding set to stream.

From 'man encfs':

Filename Encoding
           New in 1.1. A choice is given between stream encoding of filename and block encoding. 
           The advantage of stream encoding is that the encoded filenames will be as
           short as possible.  If you have a filename with a single letter, it will be very short in the
           encoded form, where as block encoded filenames are always rounded
           up to the block size of the encryption cipher (8 bytes for Blowfish and 16 bytes for AES).

           The advantage of block encoding mode is that filename lenths all come out as a multiple
           of the cipher block size.  This means that someone looking at your
           encrypted data can't tell as much about the length of your filenames.  It is on by default, 
           as it takes a similar amount of time to using the stream cipher.
           However stream cipher mode may be useful if you want shorter encrypted filenames for
           some reason.

           Prior to version 1.1, only stream encoding was supported.

[Germar]

(by shigi)
Thanks, Germar, for your reply. Unfortunately I don't see how I could tell backintime to use non-default encryption parameters without resorting to manually building a new encfs backup destination. It'd take some digging to find the parameters used by backintime and adapting them.

IMHO the default settings shouldn't fail for obscure technical reasons. Maybe the default for new encfs backups could be changed to stream encoding of filenames? Additionally a short security notecould be added informing the user that a more secure filename encryption is possible but that it artifically restricts filename length. (Maximum filename length is even further reduced due to utf-8 encoding. Bytes vs. characters)

I've also noticed that encfs has moved. So here's the updated link to the encfs issue: vgough/encfs#7

[Germar]

(by germar)
You can create a new .encfs6.xml with these commands:

mkdir -p ~/tmp/plain
mkdir -p ~/tmp/enc
encfs --reverse ~/tmp/plain ~/tmp/enc

• select x for expert modus
• type 1 for AES
• type 256 for key length
• hit Enter for default blocksize
• type 3 for stream-cipher
• enter your new password twice

now you can copy the new .encfs6.xml to your destination drive and set up a new profile in BIT for this. Be careful not to override your existing .encfs6.xml as you will loose access to your current backup.

I prefer to keep default with block-cipher. This is a bug in encfs which I suspect to be fixed soon and I don't want to make users backups less secure in the meantime.

[sschwarzmann]

Is there a convenient way to identify the files which cause such issues?

I'm having the File name too long (36) issue and checked the longest basenames of files which are backed up, but the longest files are 133 characters which should be fine.

for file in `find $1 -type f` ; do
    len=`echo "$file" | gawk -F\/ '{printf "%s", $NF}' | wc -c`
    file_base=`basename $file`
    if [ $len -gt $maxlen ]; then
      maxlen=$len
      maxfile=$file
    fi
done

Also had a look at the folder names...nothing that exceeds the limits (~140 chars).

The error log contains the encrypted folder/filenames up to 259 characters, but it can't decode the paths. Also 'backintime --quiet --decode <path> doesn't help to decode. I guess because the encrypted filenames could not be completed.

I'm using backintime version 1.1.24.

[Germar]

Whats the output of backintime --quiet --decode <path>?
Maybe try again with backintime --quiet --decode $(dirname <path>) to get the folder which contain the failing file

[sschwarzmann]

My mistake for two reasons.

First, my few bash lines fromabove suck - I didn't consider whitespaces in filenames correctly (I usually don't use whitespaces and didn't realize that issue).

And second: After some trying, I've realized that the decode command doesn't like the non-encrypted part of the filename which I've been passing as an argument:

$ backintime --quiet decode /my/backintime/path/YiSHcYcKjJlGL2EW0qe,7Rvp/and/so/on
15:41:25 (DirNode.cpp:379) decode err: Filename too small to decode
/my/backintime/path/YiSHcYcKjJlGL2EW0qe,7Rvp/and/so/on

The error message was a bit confusing...it confirmed me in my assumption that the encrypted filename is incomplete and got truncated at 259 characters (all 15 erroneous encrypted filenames have the same length, but they still can get decrypted)!

Only after trying existing foldernames and getting the same error, I realized that I've been passing a bad argument.

Thanks for your help!

[tneo]

I have the same issue that BIT fails due to too long filename. @Germar the --quiet command structure is unclear to me. I have set it up to do SSH encryption, but that throws an error with rsync 5888. BIT gives back the filename is too long and thus the snapshot failed.

[emtiu]

This is a frequent problem with encfs, I've run into it many times. encfs' encrypted filenames are longer than the original ones by design.

We can't change how encfs operates, but we should discuss how backintime can best deal with this problem.

[emtiu]

#1304 is probably a good step towards a solution.

[aryoda]

The EncFS documentation does specify the max file length here:

https://github.com/vgough/encfs/blob/master/encfs/encfs.pod#caveats

One such limitation is filename length. If your underlying filesystem limits you to N characters in a filename, then EncFS will limit you to approximately 3*(N-2)/4. For example if the host filesystem limits to 255 characters, then EncFS will be limited to 189 character filenames. This is because encrypted filenames are always longer than plaintext filenames.

and

Based on an underlying filesystem supporting a maximum of 255 characters in filenames, here is the maximum possible filename length depending on the choosen encoding scheme : stream (189), block (175), block32 (143). Note that we should rather talk about bytes, when filenames contain special (multi-bytes) characters.

[Germar]

EncFS should be treated deprecated because especially in the way BiT is using EncFS the underlying encryption is insecure. I started adding gocryptfs as successor.

The elimination of Host/User/Profile nesting wouldn't help at all with this as the limit is just for the filename, not for the path

[emtiu]

EncFS should be treated deprecated because especially in the way BiT is using EncFS the underlying encryption is insecure. I started adding gocryptfs as successor.

EncFS is still pretty widespread and popular, because it's mature and well-tested. But you're right: it's known to be somewhat insecure under certain cicrumstances. For all I know, gocryptfs is the only somewhat-equal alternative in existence.

The elimination of Host/User/Profile nesting wouldn't help at all with this as the limit is just for the filename, not for the path

You're right, I got that mixed up. Both problems exist, though: EncFS tends to produce too-long filenames and too-long paths.

[Germar]

It is insecure exactly in those conditions made by BiT.

gocryptfs already work on normal encrypted profiles in the above linked branch. But a profile with ssh+encryption need to sync an encrypted view on the filesystem. For this it is necessary to en-/decrypt filenames and paths. This part is still missing in the branch. The founder added a socket in gocryptfs, which will provide en-/decryption of filenames through a json format. But I did not found the time to implement that...

[buhtz]

See #1549 #1734 which is about removing EncFS in the future. So this Issue won't be fixed.

Please note that the _Back In Time_project decided to remove EncFS in
the next five years due to security reasons and maintenance burden.
It is not yet decided if it will be replaced by an alternative (e.g.
GoCrypt) or if BIT recommend using encrypted filesystems (e.g. via LUKS).

Please see Meta-Issue #1734 to monitor the transiation process and join
the discussion.

Best regards,

[buhtz]

Closing this ticket based on the comment above. Feel free to reopen
if the problem still exists. Thank you for your efforts. If you have
any further questions, ideas or encounter any other issues, please
don't hesitate to let us know.

Best regards,
Christian