Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[WIP] Locks + cache for multiple revisions #8510

Closed
wants to merge 68 commits into from
Closed

[WIP] Locks + cache for multiple revisions #8510

wants to merge 68 commits into from

Conversation

jgsogo
Copy link
Contributor

@jgsogo jgsogo commented Feb 16, 2021 •
edited
Loading

Changelog: (Feature | Fix | Bugfix): Describe here your pull request
Docs: https://github.com/conan-io/docs/pull/XXXX

Requirements

  • multiple revisions
  • locking mechanism (threads and processes)
  • LRU implementation
  • use deterministic folders
  • shortest paths as possible
  • sqlite3 rollback mechanism and blocks
  • ...

Keep in mind:

#5513

jgsogo added 30 commits February 2, 2021 16:07
@jgsogo
classes for a Conan cache
03a2833
@jgsogo
implement locks on top of a database
0b1d298
@jgsogo
create a manager and some lockable resource
78d9152
@jgsogo
class per file
15fc970
@jgsogo
Add some testing for basic behavior
602d45a
@jgsogo
Add some testing for basic behavior
c274ad4
@jgsogo
wip
099776f
@jgsogo
work on folders
9e85c3d
@jgsogo
moving folders when the rrev is known
893ee3f
@jgsogo
first running impl
056b452
@jgsogo
add comment
0dcbd71
@jgsogo
add comment
7cc54d1
@jgsogo
small testing
deb051b
@jgsogo
add more tests
9df88a6
@jgsogo
add more tests
062d33f
@jgsogo
check folders are retrieved from database
4454c18
@jgsogo
Merge branch 'feat/cache' of github.com:jgsogo/conan into feat/cache
1850271
@jgsogo
add default path for packages
4b2224d
@jgsogo
working with multiprocessing
ee889b7
@jgsogo
organize fixtures
d068fcc
@jgsogo
use URI to work with memory sqlite3 databases
9117a64
@jgsogo
use URI to work with memory sqlite3 databases
a10a92f
@jgsogo
rename functions
25b15f5
@jgsogo
reuse sqlite3 initializations
d857f53
@jgsogo
replicate scenario to install package
ff85569
@jgsogo
typo
fd2cfc6
@jgsogo
testing using external tools
eecd0a1
@jgsogo
remove files
907b3aa
@jgsogo
fix import
13484d7
@jgsogo
Remove LockableResource (no use case so far)
d8a7b04
memsharded
memsharded reviewed Feb 25, 2021
View reviewed changes
conan/locks/backend_fasteners.py

class RWLock(object):
def __init__(self, resource: str, interprocess_lock: str):
self.w_lock = threading.Lock()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am checking possible multi-threading race conditions, and given the structure of the program, typical operation (parallel uploads, parallel downloads), etc., we can guarantee by the business logic that multithreaded operations within the same process will not collide or produce race conditions with itself. Thus, lets simplify this, leaving only the interprocess lock, the simplicity will pay off

jgsogo added 19 commits March 1, 2021 08:49
@jgsogo
basic operations for table of prefs
6946db8
@jgsogo
basic testing for packages
1b825dc
@jgsogo
tests for packages table
c605699
@jgsogo
latest pref, tests
2bbdc52
@jgsogo
table to store folders
63be035
@jgsogo
manage folders, update timestamp for LRU
cf79d39
@jgsogo
migrating test to new implementation
a38a6be
@jgsogo
add uniqueness checks
d6fba26
@jgsogo
use new database with tables approach
69d067c
@jgsogo
remove old directories table/database
e641c85
@jgsogo
there exists a two-level cache
81df9e4
@jgsogo
use interface to ensure we provide same methods for all cache impleme…
1666cfc 
…ntations
@jgsogo
use db with 'get' and 'get_or_create' API
522b839
@jgsogo
get_or_create need to work at database level (for atomicity)
ea7059c
@jgsogo
provide safe[r] implementation for read-only cache
26fb724
@jgsogo
remove dead code
3879d87
@jgsogo
functions to list, search,... references from one cache
bfc58aa
@jgsogo
Add docu
e67b028
@jgsogo
list packages given a reference (search, list, find package-id)
8079715
@czoido czoido mentioned this pull request Apr 13, 2021
Merged
@czoido czoido mentioned this pull request Jun 9, 2021
Closed
@memsharded
Copy link
Member

The work here is being continued in #8796

@memsharded memsharded closed this Jun 10, 2021
@SSE4 SSE4 mentioned this pull request Jun 19, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@memsharded memsharded memsharded requested changes

Assignees

@jgsogo jgsogo

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@jgsogo @memsharded