support for "virtual" file metadata changes

[ak2k]

Admittedly, this might come across as a strange goal. I am accessing a ZIP object (over https) via an fsspec filesystem that contains a number of CSV files. I would ideally like to rename/mv/move these contained CSVs within the scope of the session, but then have no attempt made to persist those changes beyond the fsspec session. In essence, I'm hoping to create a view with different file metadata on top of an existing ZIP file, for the open session only and without change to the original.

Is anything like this contemplated/possible in fsspec?

Thank you!

[martindurant]

The ZIP filesystem doesn't allow for this kind of thing, but maybe you could look into ReferenceFileSystem, which maps files on any filesystem backend (including ZIP) to arbitrarily map filesystem names to targets.

[ak2k]

Thank you! Separately, I wondered whether starting a transaction that never completes might attempt to defer the changes. Would that be the best way to try to use ReferenceFileSystem in this way?

…

On Wed, Aug 31, 2022 at 10:35 AM Martin Durant ***@***.***> wrote: The ZIP filesystem doesn't allow for this kind of thing, but maybe you could look into ReferenceFileSystem, which maps files on any filesystem backend (including ZIP) to arbitrarily map filesystem names to targets. — Reply to this email directly, view it on GitHub <#1031 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AESZP3FHUCYEM7OTPJ2BQPLV35USPANCNFSM6AAAAAAQBKEOPE> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[martindurant]

If you are using ReferenceFS, you do not need a transaction, it never writes to the target.

[martindurant]

But I thought you wanted to map the paths for the whole session, so it's enough to set up the right mapping, you shouldn't need any write operations at all.

[ak2k]

Yes, that's perfect. I'll get going in that direction.

(My question regarding transaction was more whether that could be a means for indefinitely deferring the writes if that would make things work with another FS, but it sounds like ReferenceFS is the better approach).

[martindurant]

a means for indefinitely deferring the writes if that would make things work with another FS

It will depend on how temporaries work on that filesystem. On most backends, it amounts to making a temporary file, and moving it to the final location when done. I'm not sure what ZIP doe offhand, but it likely needs some kind of write even while the transaction is live.

[ak2k]

Makes sense, thank you.

[martindurant]

The specification for the references is here: https://fsspec.github.io/kerchunk/spec.html
For your case, you make a JSON file (or dict) with an inner references dict of {path: [target]} where path is how the file is known within the referenceFS, and target is the URL you want to map to. The references are stored in the references attribute of the filesystem instance, so you can even edit it directly in-place, if you wish.