Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pickling/unpickling embeddings object does not work on Windows #260

Closed
austinpatrickm opened this issue Jun 19, 2024 · 2 comments
Closed

Comments

@austinpatrickm
Copy link

austinpatrickm commented Jun 19, 2024

Hi Pavlin,

This is similar to issue #244.

I'll direct you to this thread:

https://stackoverflow.com/questions/70701836/opentsne-pickle-preserve-transformer

I've tested this extensively and on Mac and Linux and it works fine. But on Windows, there is a fundamental issue being invoked when I try to pickle the trained embeddings object, or unpickle the object if it was embedded in a Linux/Mac env. That is: 'WinErr 32 - The process cannot access the file because it is being used by another process: ...\AppData\Local\Temp...\tmp.ann'.

I've found that small datasets do work. So for example embeddings fitted on a dataset of shape (999, 50) did pickle fine, but (1000, 50) and anything above that leads to the same old issue.

@pavlin-policar
Copy link
Owner

Hi, thanks for bringing this up, this has been an ongoing issue that, unfortunately, I don't know how to solve. I don't have access to a Windows machine, so I can't really debug this. I've moved the discussion to #210 so we have all the stuff in one place.

@austinpatrickm
Copy link
Author

Thanks. For what it's worth I did manage to find a workaround by downgrading to 0.6.0 and changing the handling of the annoy file.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants