-
Notifications
You must be signed in to change notification settings - Fork 27
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Updating feather file crashes Julia and wipes file #93
Comments
Happened for me on Julia 1.0 and Feather 0.4, Julia 0.6.4 and Feather 0.3.1 work as expected |
What's happening here is that when you do Ideally we certainly would want Feather.jl to catch such an error somehow rather than sticking users with it, but at the moment I can't think of any good options for how to do this. The only solution I can think of would be to actually reference count every feather file that users write or read to or from, which in an ideal world would be supplied by the Julia I'm open to suggestions if anyone has any thoughts. I'm not necessarily opposed to reference counting files within Feather, as the performance hit should only be a small one during initialization, though I haven't carefully thought through what this would look like. There might be some catch I'm not seeing. In the meantime, you should avoid writing over files you currently have open. If you replace your |
How do other implementations handle this? Or don't they use mmap? |
As far as I know the other feather implementations do not use memory mapping in this way. After some thought, I think reference counting would be extremely difficult if not impossible to implement correctly. You'd only be able to do it locally and would not be able to determine if other processes are using it. The right solution is probably to go through the OS. It looks like we might be able to use fuser or |
Maybe |
That's a really good (and simple) idea. I'll experiment with that a bit. I was going to mention that we have FileWatching, but I'm pretty sure that's only useful for actual writes to files. |
In addition to this, you could also use |
See #94. As far as I can tell, that's a rather elegant fix. Indeed, if you have an object in memory that's referring to a file and then you delete that file, it works just fine. |
Reading a feather file followed by a write back to the same file fails:
The file test.feather is now 0 bytes long
The text was updated successfully, but these errors were encountered: