-
Notifications
You must be signed in to change notification settings - Fork 822
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
sqlite write locks aren't respected in WSL #2395
Comments
@tdsmith - Thanks for your post. Can you share any repro? Or at least some source code reference? Without that it's difficult to say what's going on. |
Here's a Python script that tries to open connections on a sqlite database from separate processes: #!/usr/bin/env python3
from itertools import product
import multiprocessing
import sqlite3
DBNAME = "test.sqlite"
def init_db():
conn = sqlite3.connect(DBNAME)
conn.execute("DROP TABLE IF EXISTS myvalues")
conn.execute("CREATE TABLE myvalues(i, j)")
conn.commit()
conn.close()
def do_work(i):
conn = sqlite3.connect(DBNAME)
my_id = multiprocessing.current_process()._identity[0]
values = product([my_id], range(i, i+1000))
conn.executemany("INSERT INTO myvalues VALUES(?, ?)", values)
conn.commit()
conn.close()
def main():
init_db()
p = multiprocessing.Pool()
p.map(do_work, range(0, 50000, 1000))
p.close()
p.join()
if __name__ == "__main__":
main() On Linux, it succeeds.
On WSL, the script fails, with the exception:
Running the sqlite3 CLI over the file afterwards gives unusual results (this number varies):
If you disable sqlite's journaling by running
|
Weirdly, my home directory is now in a state where I can't even create the database anymore in the parent process -- there aren't any visible lock or journal files, so I don't know what state is interfering:
But I can |
I can confirm it still happens on 16251 build on both LxFs and DrvFS |
A workaround should be to pass an alternative "VFS" name to These are defined in pager.c and include The python3 sqlite module does not seem to support setting an alternative VFS but the apsw module does. |
NixOS seems to suffer from the same problem which can be worked around by disabling WAL mode. NixOS/nix#1203 |
This issue should affect both WAL and regular journals (I was testing with regular journals), though it's possible disabling WAL makes it harder to trigger. |
Any progress on this? |
I think I have stumbled upon this issue and have a small ruby script that creates a sqlite database, imports logs data and fails like this:
My googling suggests the problem is associated with how WSL gets to talk with the sqlite file which lead me here, but I'm rapidly moving out of my depth. If there's interest, I can provide the ruby script and some sample logs so that others can recreate error. I'll watch this issue for replies |
Any update? |
This issue is driving me crazy right now as well, causing all sorts of random errors.
What should happen:
|
I also feel that this bug is a major pain. I've had destroyed subversion repositories (problems with sqllite) and e-mail databases (problem with multiple writers on Xapian indexes) due to this. |
This also affects Fossil (http://fossil-scm.org/) in that it can easily corrupt repository databases. You can reproduce by doing the following: Running commands in two difference terminal processes (
Normally B should fail with a message like |
This issue is blocked by: #1927 ... lxfs doesn't support fcntl locks of any kind, which many programs rely on. |
I've found a workaround to pause for a |
Running into this as well. This affects visual studio which stores information it needs to process in sqlite. If it operates on a project on the shared FS between wsl2 and Windows, then this completely fails (in this case, it just hangs trying to even acquire the proper fs locks). Something isn't working properly with this file system which breaks the entire scenario of having shared projects that can be edited by either the windows or linux side of things. |
It also breaks |
Hit this issue too while trying the parallel optimization with Optuna (https://optuna.readthedocs.io/en/latest/tutorial/distributed.html), it uses sqlite to share optimization progress between the multiple processes, which is broken under WSL... BTW, could this be resolved by upgrading to WSL2? Currently I am using WSL1 with Ubuntu 18.04. |
Yes, WSL2 is not a lightweight emulation layer. It is more of a full
VM, with ext4 as the root filesystem, which supports fcntl.
…On Sat, Dec 21, 2019 at 5:30 PM Lukáš Cerman ***@***.***> wrote:
Hit this issue too while trying the parallel optimization with Optuna (https://optuna.readthedocs.io/en/latest/tutorial/distributed.html), it uses sqlite to share optimization progress between the multiple processes, which is broken under WSL...
BTW, could this be resolved by upgrading to WSL2? Currently I am using WSL1 with Ubuntu 18.04.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or unsubscribe.
|
Any users able to confirm this is unblocked in WSL2? If yes, any other blockers preventing SQLite functioning properly? |
A while ago it was still broken across the OS boundary (e.g. |
The problem seems to be fixed in WSL2. When in WSL1, I was having database lock issues, but after upgrading to WSL 2 the problems all went away, and it worked as expected. My guess is something to do with the lower level kernel code, of Linux vs Windows kernel. |
@willgozlan it is not |
@RahulDey12 |
I use |
Hi! We've identified this issue as a duplicate of another one that already exists in this repository. This specific instance is being closed in favor of tracking the concern over on the referenced thread. Thanks for your report! |
So what is the original issue number.... |
Bad bot! Let's feed it carrots to encourage it to learn to cite duplicates :-) |
i'm trying to run rasa x on wsl2 and it's breaking.! were you able to resolved it .? |
|
Still having problems with subversion on WSL2: $ uname -a |
Under WSL2, anything under /mnt/*/ is like a remote filesystem, that goes over the Plan 9 Filesystem Protocol (9P). There sqlite can't lock it's database files. Everywhere else, outside the shared filesystem, sqlite will work. Under WSL1 it's a bit different, but other breakages means sqlite database files will not work anywhere. |
A workaround I use is to install windows tortoisesvn including the command line utilities, which makes it include the windows version of svn.exe. Then in linux set svn to be aliased to svn.exe via .bashrc:
Bonus is that, it being a windows exe, filesystem performance is better for stuff under /mnt/*. |
thanks.
wsl -l -v
~$ cat test.py
~$ ./test.py |
* zdharma/fast-syntax-highlighting MUST load before zsh-users/zsh-autosuggestions or auto-complate will NOT work + Add `duf` for better `df`
ver
at a Windows Command Prompt)10.0.15063
sqlite's write locking seems to be broken under WSL both inside and outside the /mnt hierarchy; attempting to write to the same database from distinct processes using normal sqlite locking throws I/O errors from sqlite and leads to database corruption.
Only one process should get an exclusive database lock at a time.
Sorry I don't have a minimal reproducer or a deep understanding of how sqlite's locking works. This may or may not have the same root cause as #1927 and #1712.
The text was updated successfully, but these errors were encountered: