Long file paths not supported in keter bundles

[ygale]

Keter uses the tar package to decode keter bundles. The tar package does not support long paths that do not fit into the restrictions of the old USTAR tar file format - no paths longer than 256 characters, and even then only if there is a path separator somewhere near the middle.

The tar package is very old, and hasn't been updated for years. AFAIK there is no bug tracker for it, so I am recording this issue here, even though it is really an issue with the tar package. In any case, this issue effectively causes a very serious limitation of keter.

At least the following things need to be updated in the tar package:

Most importantly:

• Support arbitrarily long file paths for formats that support them.

Additionally:

• Support the pax format, which nowadays has become the universally supported standard tar file format.
• For formats that support it, such as pax, support proper handling of Unicode in file paths.
• For formats that support them, support extended file attributes.
• Specify whether GnuFormat means the old GNU format or the new GNU format, and support also the other one.
• Provide a more generic streaming interface without baked-in lazy bytestrings, so that interfaces can be provided using streaming libraries such as conduit and pipes.
• And last but not least - please, please add Show instances for all of the types.

[angerman]

I think @dcoutts is the maintainer of tar, or at least that's what the hackage page says. Did you contact him already regarding this?

[ygale]

@angerman Yes we indeed contacted Duncan.

[ygale]

For the keter issue itself, an easier solution would be to add support for zip archives.

The only subtlety would be reading the first few bytes of the file to check the "magic" and determine the archive type, without accidentally holding on to a reference to the whole file and leaking memory.

@creichert would you be interested in this feature?

[dcoutts]

Sorry folks, I've been away (ICFP etc).

About long paths, yes, the old well-supported tar standards have these path limitations. The newer "pax" standard supports long file names and in principle we could add support for that to the tar package. I'd be perfectly happy with that so long as it was done in the same way we handle the other standards we already support, i.e. in such a way that you can choose (since for some applications it's important to be able to say "ustar compatible only" for example).

[hvr]

fyi, after consultation with @dcoutts tar is gonna move to https://github.com/haskell/tar/issue, feel free to file issues there

the git repo will be populated within the week

[dcoutts]

Incidentally, if it's a big deal, you can actually create pax format tar archives using the tar package as it is. The trick is to make a pax metadata entry after each normal file entry. Pax entries have a special header code and their body just just UTF8 key value pairs (see the Pax spec). So for write-only it should actually be pretty straightforward. Obviously direct support is better, but if you just need it to work and you're in a hurry...