-
Notifications
You must be signed in to change notification settings - Fork 3
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Quick question #1
Comments
Hey! We'll be releasing the first version of this package tomorrow. |
Unfortunately, we're freezing development of this project due to issues on Linux. Sorry about that. |
Hey @JeremyTCD -- do you see the issues on Linux as fundamental or things we can fix in FASTER? It seems premature to freeze development due to this. Also, are you seeing any of the reported issues on Windows, or is it a Linux specific thing? If you provide us a debuggable repro of all the issues on Linux, I am feeling confident we can come up with a way to get performance back to what it should be, on Linux. |
Hey @badrishc! Apologies about the abrupt freeze. I put a hold on the project because of looming datelines for deliverables (our application runs on Linux VMs). To expand on the issues:
From issues on Faster, it seems like the performance issue is expected. In particular this issue highlights that operations have to run sequentially on non-windows platforms. I'm not sure if .Net Core has resolved the underlying limitation. If it's unresolved I'm not sure how Faster could proceed. I suppose Faster could manually invoke native code to handle I/O if .Net Core still isn't up to scratch, but that seems like a difficult undertaking. Do you think these issues are resolvable? I'd be happy to create a minimal repro and open an issue in the Faster repo. |
My initial gut inclination is to say that these are resolvable. In fact, for the linked issue, we already work around it by keeping multiple file handles for the same file in a pool ( Question: does the slowdown occur only when log compaction is active? How does performance look without log compaction running. That would be the first step before we analyze the impact of GC on performance. PS. We do not have access to MacOS but have Linux. |
Ah okay, wasn't aware of the file handle pooling. Assumed operations were all sequential on non-windows platforms. Will try running with a larger pool.
The benchmarks mentioned in my previous comment all run with compaction disabled. So just plain inserts/reads. Their performance is poor on Linux. I'll create a minimal repro and open an issue on Faster later today.
Noted on MacOS. |
Sounds good. I just ran Linux:
Windows:
|
Hey from your numbers it looks like the issue may be on my end, something to do with how the store uses sessions perhaps. My assumption about sequential operations was wrong. Upon closer inspection it might be a concurrency issue involving SpinLocks. Will update the ReadMe of this project once the issue is resolved/identified. I've created a minimal repro using a session pool, and an issue to continue to discussion: [C#] Non-windows performance. |
Sounds good, thanks. |
Latest in master should have fixed all issues you encountered on Linux. See how to use the latest async Upsert API in our wrapper based sample here: https://github.com/microsoft/FASTER/tree/master/cs/playground/AsyncStress Nuget will be released next week. |
Thanks for the fixes! I've just given the latest commit a go, example (without tweaks) works well on Linux. I tried converting the wrapper's underlying FasterKV to use
Might've been mistakes in my tweaks, haven't has the chance to look into the issue. Any chance you guys could add an example async wrapper for |
This is expected when used with tasks on Linux, Upsert blocks the caller thread (and uses up the thread) waiting for flush to complete, and so the scheduler is unable to schedule the I/O completion. So, for the loaded scenario, using Upsert is not recommended. Use |
The error looks like you (or some unfinished task) is trying to access the store after it is disposed. If you share the code/repro, we should be able to diagnose this. |
Ah okay I think I get it, all threads are blocked waiting for a free thread. A bit like a deadlock situation.
Okay will create a repro when I have some time. |
Sounds good. Just to clarify, it's happening only because the Upserts are being done by task threads. If normal threads issued the Upserts, you wouldn't see the thread starvation issue. Starvation is generally a problem for any C# blocking operation issued on a Task thread. |
The latest in this PR should work well with SpanByte as well: microsoft/FASTER#452 |
Just ran some basic tests with the PR. Works well! |
Will clean up and publish this package some time this week |
I'm going to use it for in-memory game state along with persistence. How to set it with persistence even memory is not full?
Also, releasing it will be great. thanks.
The text was updated successfully, but these errors were encountered: