Android/Termux doesn't have flock, see yt-dlp solution #496

Leif-W · 2024-07-05T12:10:48Z

TL;DR filelock dependency issue?

Same issue as listed on yt-dlp.

Android/Termux doesn't have flock.
Fallback to lockf.
Possibly fallback to no lock.

scdl -l https://soundcloud.com/user/track
Soundcloud Downloader
Found a track
Downloading track
Traceback (most recent call last):
  File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/filelock/_unix.py", line 46, in _acquire
    fcntl.flock(fd, fcntl.LOCK_EX | fcntl.LOCK_NB)
OSError: [Errno 38] Function not implemented

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/data/data/com.termux/files/usr/bin/scdl", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/scdl/scdl.py", line 304, in main
    download_url(client, **python_args)
  File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/scdl/scdl.py", line 409, in download_url
    download_track(client, item, **kwargs)
  File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/scdl/scdl.py", line 887, in download_track
    with lock:
  File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/filelock/_api.py", line 376, in __enter__
    self.acquire()
  File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/filelock/_api.py", line 332, in acquire
    self._acquire()
  File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/filelock/_unix.py", line 51, in _acquire
    raise NotImplementedError(msg) from exception
NotImplementedError: FileSystem does not appear to support flock; use SoftFileLock instead

The text was updated successfully, but these errors were encountered:

Leif-W · 2024-07-05T12:23:38Z

Redirecting to filelock dependency.

Leif-W · 2024-08-18T11:37:05Z

Actually, regardless of the filelock dependency, this problem exists, and should be addressed. Test if the system can create a file lock, catch the exception, if exception, try a soft file lock, if soft file lock fails, just don't lock. This should not be causing an error. If the filelock lib fixes this, update dependency requirement and remove code.

7x11x13 · 2024-08-18T16:01:56Z

Possibly fixed in v2.12.1, can you see if it works on your system?

Leif-W · 2024-08-18T17:38:51Z

Works for me now. Much appreciated workaround. Can leave open for a bit if anyone else wants to verify.

matte22ladde · 2024-09-30T08:35:11Z

Possibly fixed in v2.12.1, can you see if it works on your system?

Got this error even in v.2.12.1

Traceback (most recent call last):
  File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/filelock/_unix.py", line 46, in _acquire
    fcntl.flock(fd, fcntl.LOCK_EX | fcntl.LOCK_NB)
OSError: [Errno 38] Function not implemented

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/data/data/com.termux/files/usr/bin/scdl", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/scdl/scdl.py", line 426, in main
    download_url(client, typing.cast(SCDLArgs, python_args))
  File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/scdl/scdl.py", line 546, in download_url
    download_track(client, item, kwargs)
  File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/scdl/scdl.py", line 1098, in download_track
    with lock:
  File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/filelock/_api.py", line 376, in __enter__
    self.acquire()
  File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/filelock/_api.py", line 332, in acquire
    self._acquire()
  File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/filelock/_unix.py", line 51, in _acquire
    raise NotImplementedError(msg) from exception
NotImplementedError: FileSystem does not appear to support flock; use SoftFileLock instead

Leif-W · 2024-09-30T20:50:56Z

It DID work for me, and now it DOES NOT. Not sure what else changed.

matte22ladde · 2024-11-02T03:55:11Z

@7x11x13
No longer replicable after updating the python package to 3.12.7
Maybe update the requirement to install?

Traceback (most recent call last):
  File "/data/data/com.termux/files/home/.local/share/pipx/venvs/scdl/lib/python3.12/site-packages/filelock/_unix.py", line 46, in _acquire
    fcntl.flock(fd, fcntl.LOCK_EX | fcntl.LOCK_NB)   OSError: [Errno 38] Function not implemented

The above exception was the direct cause of the following exception:
                                                     Traceback (most recent call last):
  File "/data/data/com.termux/files/home/.local/bin/scdl", line 8, in <module>
    sys.exit(main())                                              ^^^^^^
  File "/data/data/com.termux/files/home/.local/share/pipx/venvs/scdl/lib/python3.12/site-packages/scdl/scdl.py", line 426, in main                                download_url(client, typing.cast(SCDLArgs, python_args))                                                File "/data/data/com.termux/files/home/.local/share/pipx/venvs/scdl/lib/python3.12/site-packages/scdl/scdl.py", line 546, in download_url
    download_track(client, item, kwargs)               File "/data/data/com.termux/files/home/.local/share/pipx/venvs/scdl/lib/python3.12/site-packages/scdl/scdl.py", line 1098, in download_track
    with lock:                                                ^^^^
  File "/data/data/com.termux/files/home/.local/share/pipx/venvs/scdl/lib/python3.12/site-packages/filelock/_api.py", line 376, in __enter__
    self.acquire()
  File "/data/data/com.termux/files/home/.local/share/pipx/venvs/scdl/lib/python3.12/site-packages/filelock/_api.py", line 332, in acquire                         self._acquire()
  File "/data/data/com.termux/files/home/.local/share/pipx/venvs/scdl/lib/python3.12/site-packages/filelock/_unix.py", line 51, in _acquire
    raise NotImplementedError(msg) from exception
NotImplementedError: FileSystem does not appear to support flock; use SoftFileLock instead

Leif-W · 2024-11-02T09:59:44Z

TL;DR: Will ONLY work inside app data portion of filesystem (/data/data/com.termux). cd ~/ (home directory or similar) before running command. Will NOT work on internal storage (/storage/emulated/0) or external storage (/storage/0123-CDEF). Android limitation, no workaround, except to test and soft file lock. We CAN create files, read, write, but can't change mode (silent fail), can't execute. The underlying C fcntl file lock code must bump up against the wrong thing.

Arguably filelock lib should test and return softlock instead, but it doesn't, and probably won't. As previously stated, yt-dlp suffers no problem as they test first. scdl must test first before assuming a hard lock will work. The hard lock does no magic, any program is free to ignore "locks". Not much semantic difference with soft locks. IIUC Neither is thread safe. Both can suffer race conditions.

A basic test case to see the error in Termux:

vim ~/foo.py # file below
chmod 700 ~/foo.py
cd /storage/emulated/0/Download
~/foo.py # Exception OSError function not implemented
rm -f foo.lock

~/foo.py:

#! /usr/bin/env -iS python

import fcntl

lockpath = "/storage/emulated/0/Download/foo.lock"

fh = open(lockpath, "w+")
fd = fh.fileno()
fcntl.flock(fd, fcntl.LOCK_EX | fcntl.LOCK_NB)

7x11x13 · 2024-11-02T16:16:26Z

Might be fixed in v2.12.2

Leif-W · 2024-11-03T07:32:06Z

Still works in the app data area. No error given but silent failure and no file creation in shared/sdcard area.

In shared/sdcard area --debug ends with:

Found a track
Downloading [Track Name]
Could not acquire lock: <scdl.scdl.SafeLock object at 0x701db921b0>. Skipping

The object id varies between runs.

As initially mentioned, need to gracefully degrade, by testing if we can lock, if not, test a soft lock, if not, then no lock.

7x11x13 · 2024-11-03T15:59:34Z

I believe the only reason acquiring the SoftFileLock fails is either not having write permissions in the place where you're trying to download the song, or that a previous SoftFileLock was created there and it thinks another process is downloading the song there. Can you check if there are any lingering .scdl.lock files?

BTW, in the meantime I would recommend switching to yt-dlp. I'm pretty sure it can do everything this script can with some configuration. I'm planning to update this script so it's just a yt-dlp wrapper in the future so I don't have to maintain it anymore.

Leif-W · 2024-11-03T16:40:30Z

I have write permissions, can use shell echo and redirect, vim, wget, yt-dlp, and other programs, including C, Node.js, Python, to create and modify files inside Termux, as well as create and modify documents in other Android apps, or create, rename, copy, move, or delete files in various file managers, all in this Music directory. The only restriction I was aware of was execute permission. There were no temporary lock files or other cruft left behind in this directory.

I ran an strace. After the Downloading [Track Name] C-lib write() call to print to terminal, this happens, when calling from /storage/emulated/0/Music which is on the internal memory at what the user sees as the root level (not actually / root):

mkdirat(AT_FDCWD, ".", 0777)            = -1 EEXIST (File exists)
newfstatat(AT_FDCWD, ".", {st_mode=S_IFDIR|0770, st_size=20480, ...}, 0) = 0
newfstatat(AT_FDCWD, "297676247", 0x7fd2cf9578, AT_SYMLINK_NOFOLLOW) = -1 ENOENT (No such file or directory)
getcwd("/storage/emulated/0/Music", 1024) = 26
newfstatat(AT_FDCWD, "/storage/emulated/0/Music/297676247", 0x7fd2cf9568, 0) = -1 ENOENT (No such file or directory)
mkdirat(AT_FDCWD, "/storage/emulated/0/Music", 0777) = -1 EEXIST (File exists)
newfstatat(AT_FDCWD, "/storage/emulated/0/Music", {st_mode=S_IFDIR|0770, st_size=20480, ...}, 0) = 0
newfstatat(AT_FDCWD, "/storage/emulated/0/Music/297676247.scdl.lock", 0x7fd2cf9368, 0) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/storage/emulated/0/Music/297676247.scdl.lock", O_RDWR|O_CREAT|O_TRUNC|O_CLOEXEC, 0644) = 3
fchmod(3, 0644)                         = 0
flock(3, LOCK_EX|LOCK_NB)               = -1 ENOSYS (Function not implemented)
close(3)                                = 0
newfstatat(AT_FDCWD, "/storage/emulated/0/Music/297676247.scdl.lock", {st_mode=S_IFREG|0660, st_size=0, ...}, 0) = 0
mkdirat(AT_FDCWD, "/storage/emulated/0/Music", 0777) = -1 EEXIST (File exists)
newfstatat(AT_FDCWD, "/storage/emulated/0/Music", {st_mode=S_IFDIR|0770, st_size=20480, ...}, 0) = 0
openat(AT_FDCWD, "/storage/emulated/0/Music/297676247.scdl.lock", O_WRONLY|O_CREAT|O_EXCL|O_TRUNC|O_CLOEXEC, 0644) = -1 EEXIST (File exists)
ioctl(1, TCGETS, {c_iflag=ICRNL|IUTF8, c_oflag=NL0|CR0|TAB0|BS0|VT0|FF0|OPOST|ONLCR, c_cflag=B38400|CS8|CREAD, c_lflag=ISIG|ICANON|ECHO|ECHOE|ECHOK|IEXTEN|ECHOCTL|ECHOKE, ...}) = 0
write(2, "\33[34mCould not acquire lock: <sc"..., 87Could not acquire lock: <scdl.scdl.SafeLock object at 0x77443a2690>. Skipping
) = 87                                                                            madvise(0x78d68d6000, 4096, MADV_DONTNEED) = 0
madvise(0x78d68db000, 4096, MADV_DONTNEED) = 0
madvise(0x794680e000, 4096, MADV_DONTNEED) = 0
madvise(0x7946817000, 28672, MADV_DONTNEED) = 0
madvise(0x7946821000, 131072, MADV_DONTNEED) = 0
newfstatat(AT_FDCWD, "/data/data/com.termux/files/home/.config/scdl", {st_mode=S_IFDIR|0700, st_size=3488, ...}, 0) = 0
openat(AT_FDCWD, "/data/data/com.termux/files/home/.config/scdl", O_RDONLY|O_CLOEXEC|O_DIRECTORY) = 3
getdents64(3, 0xb400007926823570 /* 4 entries */, 4200) = 120
getdents64(3, 0xb400007926823570 /* 0 entries */, 4200) = 0
close(3)                                = 0
unlinkat(AT_FDCWD, "/data/data/com.termux/files/home/.config/scdl/scdl.cfg.scdl.lock", 0) = 0
newfstatat(AT_FDCWD, "/storage/emulated/0/Music", {st_mode=S_IFDIR|0770, st_size=20480, ...}, 0) = 0
openat(AT_FDCWD, "/storage/emulated/0/Music", O_RDONLY|O_CLOEXEC|O_DIRECTORY) = 3 getdents64(3, 0xb400007926823570 /* 20 entries */, 4200) = 1376
getdents64(3, 0xb400007926823570 /* 19 entries */, 4200) = 1416
getdents64(3, 0xb400007926823570 /* 19 entries */, 4200) = 1568
getdents64(3, 0xb400007926823570 /* 17 entries */, 4200) = 1312
getdents64(3, 0xb400007926823570 /* 0 entries */, 4200) = 0
close(3)                                = 0
unlinkat(AT_FDCWD, "/storage/emulated/0/Music/297676247.scdl.lock", 0) = 0
rt_sigaction(SIGINT, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_ONSTACK}, {sa_handler=0x7a36c44a6c, sa_mask=[], sa_flags=SA_ONSTACK}, 8) = 0
madvise(0x794680a000, 4096, MADV_DONTNEED) = 0
madvise(0x794680e000, 4096, MADV_DONTNEED) = 0
madvise(0x7946817000, 172032, MADV_DONTNEED) = 0
madvise(0x79b6806000, 32768, MADV_DONTNEED) = 0
madvise(0x79b6817000, 28672, MADV_DONTNEED) = 0
madvise(0x78e6846000, 12288, MADV_DONTNEED) = 0
madvise(0x78e684c000, 4096, MADV_DONTNEED) = 0
madvise(0x791683f000, 16384, MADV_DONTNEED) = 0

Anyways, fair enough to let yt-dlp handle this. In the past, I don't think it did. Not sure when support was added. It's always nice to have options. But I understand. Thanks for the efforts. Regards.

Leif-W closed this as completed Jul 5, 2024

Leif-W reopened this Aug 18, 2024

7x11x13 closed this as completed Aug 19, 2024

7x11x13 reopened this Oct 22, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Android/Termux doesn't have flock, see yt-dlp solution #496

Android/Termux doesn't have flock, see yt-dlp solution #496

Leif-W commented Jul 5, 2024 •

edited

Loading

Leif-W commented Jul 5, 2024

Leif-W commented Aug 18, 2024

7x11x13 commented Aug 18, 2024

Leif-W commented Aug 18, 2024

matte22ladde commented Sep 30, 2024 •

edited

Loading

Leif-W commented Sep 30, 2024

matte22ladde commented Nov 2, 2024 •

edited

Loading

Leif-W commented Nov 2, 2024

7x11x13 commented Nov 2, 2024

Leif-W commented Nov 3, 2024 •

edited

Loading

7x11x13 commented Nov 3, 2024

Leif-W commented Nov 3, 2024 •

edited

Loading

Android/Termux doesn't have flock, see yt-dlp solution #496

Android/Termux doesn't have flock, see yt-dlp solution #496

Comments

Leif-W commented Jul 5, 2024 • edited Loading

Leif-W commented Jul 5, 2024

Leif-W commented Aug 18, 2024

7x11x13 commented Aug 18, 2024

Leif-W commented Aug 18, 2024

matte22ladde commented Sep 30, 2024 • edited Loading

Leif-W commented Sep 30, 2024

matte22ladde commented Nov 2, 2024 • edited Loading

Leif-W commented Nov 2, 2024

7x11x13 commented Nov 2, 2024

Leif-W commented Nov 3, 2024 • edited Loading

7x11x13 commented Nov 3, 2024

Leif-W commented Nov 3, 2024 • edited Loading

Leif-W commented Jul 5, 2024 •

edited

Loading

matte22ladde commented Sep 30, 2024 •

edited

Loading

matte22ladde commented Nov 2, 2024 •

edited

Loading

Leif-W commented Nov 3, 2024 •

edited

Loading

Leif-W commented Nov 3, 2024 •

edited

Loading