Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

InterProcessLock does not lock #116

Open
jumpojoy opened this issue Jul 17, 2024 · 3 comments
Open

InterProcessLock does not lock #116

jumpojoy opened this issue Jul 17, 2024 · 3 comments

Comments

@jumpojoy
Copy link

jumpojoy commented Jul 17, 2024

It seems that flock and lockf works differently, and lockf does lock file as expected.

There is test code I'm using

import fasteners
import sys
import time
import threading
import fcntl
from oslo_concurrency import lockutils

def test_concurrency(i):
  with lockutils.lock(name='lock-c.file',
            lock_path='/var/lib/openstack/locks/',
            external=True):
    print(time.time())
    print(i)
    time.sleep(10)

def test_concurrency_ext(i):
  with lockutils.external_lock(name='lock-c.file',
                 lock_file_prefix='ext-lock',
                 lock_path='/var/lib/openstack/locks/'):
    print(time.time())
    print(i)
    time.sleep(10)


def test_fcnl_flock(i):

    file_path = "example.txt"
    with open(file_path, "a") as file:
        fcntl.flock(file.fileno(), fcntl.LOCK_EX)

        print(time.time())
        print(i)
        time.sleep(10)

        fcntl.flock(file.fileno(), fcntl.LOCK_UN)

def test_fcnl_lockf(i):

    file_path = "example.txt"
    with open(file_path, "a") as file:
        fcntl.lockf(file.fileno(), fcntl.LOCK_EX)

        print(time.time())
        print(i)
        time.sleep(10)

        fcntl.flock(file.fileno(), fcntl.LOCK_UN)

for tf in [test_concurrency, test_concurrency_ext, test_fcnl_flock, test_fcnl_lockf]:
  print(tf)
  t1 = threading.Thread(target=tf, args=[1])
  t2 = threading.Thread(target=tf, args=[2])
  t1.start()
  t2.start()
  t1.join()
  t2.join()

And it produce following output on Ubuntu 22.04 and on macOS

python3 1.py 
<function test_concurrency at 0x7f4c509fc1f0>
1721200767.7912707
1
1721200777.8025727
2
<function test_concurrency_ext at 0x7f4c4feebaf0>
1721200787.8146389
2
1721200787.8147857
1
<function test_fcnl_flock at 0x7f4c4feebb80>
1721200797.8260574
1

1721200807.832955
2
<function test_fcnl_lockf at 0x7f4c4feebc10>
1721200817.8442693
1
1721200817.8444033
2

Which basically shows that lockutils.external_lock and test_fcnl_lockf does not lock on file as expected. Please help to fix this issue.

@jumpojoy
Copy link
Author

@psarka
Copy link
Collaborator

psarka commented Jul 18, 2024

Is this related to fasteners?

The title of the issue mentions InterProcessLock, but your snippet does not use InterProcessLock or any other fasteners functions. Can you please make a small example which illustrates deficiency of fasteners library?

Also, one thing of note, some interprocess locks do not work with threads. If you are looking for a process lock, test it with multiple processes, not multiple threads.

@jumpojoy
Copy link
Author

@psarka thanks for quick reply!

I'm sorry that I didn't provide enough information, external_lock from lockutils uses InterProcessLock [0] from fasteners. And it seems InterProcessLock is not thread safe which I discovered later [1]. Do you think adding docstrings to both fasteners and oslo.concurrency that will highlight this info will be useful?

[0] https://github.com/openstack/oslo.concurrency/blob/master/oslo_concurrency/lockutils.py#L208
[1] https://stackoverflow.com/questions/28470246/python-lockf-and-flock-behaviour

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants