local data backend should have file locking for writes and reads #1160

bghira · 2024-11-14T20:40:03Z

so that we do not read partial writes or corrupt multiprocess writes

…that we do not read partial writes or corrupt multiprocess writes

playerzer0x · 2024-11-15T23:13:47Z

Training gets clogged up here:

2024-11-15 05:08:32,527 [INFO] (id=ovrtn_toner-512) Loading bucket manager.
2024-11-15 05:08:32,536 [INFO] (id=ovrtn_toner-512) Refreshing aspect buckets on main process.
2024-11-15 05:08:32,537 [INFO] Discovering new files...
2024-11-15 05:08:32,580 [INFO] Compressed 25 existing files from 9.
2024-11-15 05:08:32,580 [INFO] No new files discovered. Doing nothing.
2024-11-15 05:08:32,581 [INFO] Statistics: {'total_processed': 0, 'skipped': {'already_exists': 25, 'metadata_missing': 0, 'not_found': 0, 'too_small': 0, 'other': 0}}
2024-11-15 05:08:32,589 [WARNING] Key crop_aspect not found in the current backend config, using the existing value 'square'.
2024-11-15 05:08:32,590 [WARNING] Key disable_validation not found in the current backend config, using the existing value 'False'.
2024-11-15 05:08:32,590 [INFO] Configured backend: {'id': 'ovrtn_toner-512', 'config': {'repeats': 0, 'crop': False, 'crop_aspect': 'square', 'crop_style': 'random', 'disable_validation': False, 'resolution': 0.262144, 'resolution_type': 'area', 'caption_strategy': 'textfile', 'instance_data_dir': 'datasets/crrllcrrllxovrtn_subjects/ovrtn_toner', 'maximum_image_size': None, 'target_downsample_size': 0.262144, 'config_version': 1}, 'dataset_type': 'image', 'data_backend': <helpers.data_backend.local.LocalDataBackend object at 0x73cc955cf510>, 'instance_data_dir': 'datasets/crrllcrrllxovrtn_subjects/ovrtn_toner', 'metadata_backend': <helpers.metadata.backends.discovery.DiscoveryMetadataBackend object at 0x73cc955cf550>}
(Rank: 0)  | Bucket     | Image Count (per-GPU)
------------------------------
(Rank: 0)  | 1.0        | 2
(Rank: 0)  | 0.38       | 1
(Rank: 0)  | 1.83       | 1
(Rank: 0)  | 0.78       | 2
(Rank: 0)  | 0.7        | 1
(Rank: 0)  | 0.42       | 1
(Rank: 0)  | 0.36       | 1
(Rank: 0)  | 2.4        | 1
(Rank: 0)  | 0.6        | 1
2024-11-15 05:08:32,592 [INFO] (id=ovrtn_toner-512) Collecting captions.
Loading captions:   0%|                                                                  
             | 0/25 [00:00<?, ?it/s]No images were discovered by the bucket manager in the dataset: ovrtn_toner-512.
Traceback (most recent call last):
  File "/workspace/SimpleTuner/train.py", line 30, in <module>
    trainer.init_data_backend()
  File "/workspace/SimpleTuner/helpers/training/trainer.py", line 655, in init_data_backend
    raise e
  File "/workspace/SimpleTuner/helpers/training/trainer.py", line 629, in init_data_backend
    configure_multi_databackend(
  File "/workspace/SimpleTuner/helpers/data_backend/factory.py", line 906, in configure_multi_databackend
    raise Exception(
Exception: No images were discovered by the bucket manager in the dataset: ovrtn_toner-512.

                                                                                         
2024-11-15 05:08:32,662 [INFO] (id=ovrtn_toner-512) Initialise text embed pre-computation using the textfile caption strategy. We have 25 captions to process.
2024-11-15 05:08:32,667 [INFO] (id=ovrtn_toner-512) Completed processing 25 captions.
2024-11-15 05:08:32,667 [INFO] (id=ovrtn_toner-512) Creating VAE latent cache.

Tried again, errored further down:

(Rank: 0)  | Bucket     | Image Count (per-GPU)
------------------------------
(Rank: 0)  | 0.94       | 1
(Rank: 0)  | 0.78       | 1
2024-11-15 05:40:31,624 [INFO] (id=lthrhrdwn_hairstyle-1024) Collecting captions.        
Loading captions:   0%|                                                                  
                                                                                         
2024-11-15 05:40:31,714 [INFO] (id=lthrhrdwn_hairstyle-1024) Initialise text embed pre-computation using the textfile caption strategy. We have 39 captions to process.
2024-11-15 05:40:31,718 [INFO] (id=lthrhrdwn_hairstyle-1024) Completed processing 39 captions.
2024-11-15 05:40:31,718 [INFO] (id=lthrhrdwn_hairstyle-1024) Creating VAE latent cache.  
2024-11-15 05:40:31,721 [INFO] Configured backend: {'id': 'lthrhrdwn_hairstyle-1024', 'config': {'repeats': 0, 'crop': False, 'crop_aspect': 'square', 'crop_style': 'random', 'disable_validation': False, 'resolution': 1.048576, 'resolution_type': 'area', 'caption_strategy': 'textfile', 'instance_data_dir': 'datasets/crrllcrrllxovrtn_subjects/lthrhrdwn_hairstyle', 'maximum_image_size': None, 'target_downsample_size': 1.048576, 'config_version': 1, 'hash_filenames': True}, 'dataset_type': 'image', 'data_backend': <helpers.data_backend.local.LocalDataBackend object at 0x70808a765090>, 'instance_data_dir': 'datasets/crrllcrrllxovrtn_subjects/lthrhrdwn_hairstyle', 'metadata_backend': <helpers.metadata.backends.discovery.DiscoveryMetadataBackend object at 0x708083fbe090>, 'train_dataset': <helpers.multiaspect.dataset.MultiAspectDataset object at 0x708083fc7990>, 'sampler': <helpers.multiaspect.sampler.MultiAspectSampler object at 0x708083f55d50>, 'train_dataloader': <torch.utils.data.dataloader.DataLoader object at 0x708083fb1290>, 'text_embed_cache': <helpers.caching.text_embeds.TextEmbeddingCache object at 0x708083ff3790>, 'vaecache': <helpers.caching.vae.VAECache object at 0x70808a6339d0>}
[Errno 2] No such file or directory: 'datasets/crrllcrrllxovrtn_subjects/lthrhrdwn_hairstyle/.aspect_ratio_bucket_indices_lthrhrdwn_hairstyle-1024.json.tmp' -> 'datasets/crrllcrrllxovrtn_subjects/lthrhrdwn_hairstyle/aspect_ratio_bucket_indices_lthrhrdwn_hairstyle-1024.json'
[Errno 2] No such file or directory: 'datasets/crrllcrrllxovrtn_subjects/lthrhrdwn_hairstyle/.aspect_ratio_bucket_indices_lthrhrdwn_hairstyle-1024.json.tmp' -> 'datasets/crrllcrrllxovrtn_subjects/lthrhrdwn_hairstyle/aspect_ratio_bucket_indices_lthrhrdwn_hairstyle-1024.json'
2024-11-15 05:40:31,744 [ERROR] [Errno 2] No such file or directory: 'datasets/crrllcrrllxovrtn_subjects/lthrhrdwn_hairstyle/.aspect_ratio_bucket_indices_lthrhrdwn_hairstyle-1024.json.tmp' -> 'datasets/crrllcrrllxovrtn_subjects/lthrhrdwn_hairstyle/aspect_ratio_bucket_indices_lthrhrdwn_hairstyle-1024.json', traceback: Traceback (most recent call last):   
  File "/workspace/SimpleTuner/helpers/training/trainer.py", line 629, in init_data_backend
    configure_multi_databackend(
  File "/workspace/SimpleTuner/helpers/data_backend/factory.py", line 1153, in configure_multi_databackend
    init_backend["metadata_backend"].save_cache()
  File "/workspace/SimpleTuner/helpers/metadata/backends/discovery.py", line 180, in save_cache
    self.data_backend.write(self.cache_file, cache_data_str)
  File "/workspace/SimpleTuner/helpers/data_backend/local.py", line 69, in write
    os.rename(temp_file_path, filepath)
FileNotFoundError: [Errno 2] No such file or directory: 'datasets/crrllcrrllxovrtn_subjects/lthrhrdwn_hairstyle/.aspect_ratio_bucket_indices_lthrhrdwn_hairstyle-1024.json.tmp' -> 'datasets/crrllcrrllxovrtn_subjects/lthrhrdwn_hairstyle/aspect_ratio_bucket_indices_lthrhrdwn_hairstyle-1024.json'

[Errno 2] No such file or directory: 'datasets/crrllcrrllxovrtn_subjects/lthrhrdwn_hairstyle/.aspect_ratio_bucket_indices_lthrhrdwn_hairstyle-1024.json.tmp' -> 'datasets/crrllcrrllxovrtn_subjects/lthrhrdwn_hairstyle/aspect_ratio_bucket_indices_lthrhrdwn_hairstyle-1024.json'
Traceback (most recent call last):
  File "/workspace/SimpleTuner/train.py", line 30, in <module>
    trainer.init_data_backend()
  File "/workspace/SimpleTuner/helpers/training/trainer.py", line 655, in init_data_backend
    raise e
  File "/workspace/SimpleTuner/helpers/training/trainer.py", line 629, in init_data_backend
    configure_multi_databackend(
  File "/workspace/SimpleTuner/helpers/data_backend/factory.py", line 1153, in configure_multi_databackend
    init_backend["metadata_backend"].save_cache()
  File "/workspace/SimpleTuner/helpers/metadata/backends/discovery.py", line 180, in save_cache
    self.data_backend.write(self.cache_file, cache_data_str)
  File "/workspace/SimpleTuner/helpers/data_backend/local.py", line 69, in write
    os.rename(temp_file_path, filepath)
FileNotFoundError: [Errno 2] No such file or directory: 'datasets/crrllcrrllxovrtn_subjects/lthrhrdwn_hairstyle/.aspect_ratio_bucket_indices_lthrhrdwn_hairstyle-1024.json.tmp' -> 'datasets/crrllcrrllxovrtn_subjects/lthrhrdwn_hairstyle/aspect_ratio_bucket_indices_lthrhrdwn_hairstyle-1024.json'

Traceback (most recent call last):
  File "/workspace/SimpleTuner/train.py", line 30, in <module>
    trainer.init_data_backend()
  File "/workspace/SimpleTuner/helpers/training/trainer.py", line 655, in init_data_backend
    raise e
  File "/workspace/SimpleTuner/helpers/training/trainer.py", line 629, in init_data_backend
    configure_multi_databackend(
  File "/workspace/SimpleTuner/helpers/data_backend/factory.py", line 1153, in configure_multi_databackend
    init_backend["metadata_backend"].save_cache()
  File "/workspace/SimpleTuner/helpers/metadata/backends/discovery.py", line 180, in save_cache
    self.data_backend.write(self.cache_file, cache_data_str)
  File "/workspace/SimpleTuner/helpers/data_backend/local.py", line 69, in write
    os.rename(temp_file_path, filepath)
FileNotFoundError: [Errno 2] No such file or directory: 'datasets/crrllcrrllxovrtn_subjects/lthrhrdwn_hairstyle/.aspect_ratio_bucket_indices_lthrhrdwn_hairstyle-1024.json.tmp' -> 'datasets/crrllcrrllxovrtn_subjects/lthrhrdwn_hairstyle/aspect_ratio_bucket_indices_lthrhrdwn_hairstyle-1024.json'

Traceback (most recent call last):
  File "/workspace/SimpleTuner/train.py", line 30, in <module>
    trainer.init_data_backend()
  File "/workspace/SimpleTuner/helpers/training/trainer.py", line 655, in init_data_backend
    raise e
  File "/workspace/SimpleTuner/helpers/training/trainer.py", line 629, in init_data_backend
    configure_multi_databackend(
  File "/workspace/SimpleTuner/helpers/data_backend/factory.py", line 1153, in configure_multi_databackend
    init_backend["metadata_backend"].save_cache()
  File "/workspace/SimpleTuner/helpers/metadata/backends/discovery.py", line 180, in save_cache
    self.data_backend.write(self.cache_file, cache_data_str)
  File "/workspace/SimpleTuner/helpers/data_backend/local.py", line 69, in write
    os.rename(temp_file_path, filepath)
FileNotFoundError: [Errno 2] No such file or directory: 'datasets/crrllcrrllxovrtn_subjects/lthrhrdwn_hairstyle/.aspect_ratio_bucket_indices_lthrhrdwn_hairstyle-1024.json.tmp' -> 'datasets/crrllcrrllxovrtn_subjects/lthrhrdwn_hairstyle/aspect_ratio_bucket_indices_lthrhrdwn_hairstyle-1024.json'

playerzer0x · 2024-11-16T13:11:59Z

Update after trying again: It seems like ST doesn't like one of my dataset folders. It will say it can't find any images for 512, and when I remove it from the data loader, then it says it doesn't like 768. In both cases, training start-up doesn't proceed. It will take 1024 however (with the same "No images were discovered" error) , move to training, start, and then error out with the same "cannot find cache" error after a few hundred steps.

I took that dataset out completely, and training proceeds as normal. At a loss as to what could be the issue with this dataset. Tried a number of things: checked data loader, converted to jpg, converted to png, checked captions, checked resolution (all were 1000px or more), checked count (26), deleting anything that wasn't an image or caption file. All the other datasets were 100s or 1000s of images and processed without issue.

bghira · 2024-11-16T13:15:53Z

you'll have to check debug.log - since this isn't a widespread issue it's likely something like captions not getting found

bghira · 2024-11-16T18:58:45Z

i'm trying this on 3x 4090 and seeing no loss in throughput and it seems like everything is working so i'm going to merge it and assume your issue is now somewhere else 👍

mhirki · 2024-11-17T09:35:45Z

The code could be improved by having each process write to a different temporary file. This would at least solve the issues with the file being missing when os.rename() is called. Adding the process rank after .tmp should suffice. So your temporary files would have .tmp0, .tmp1 etc. added to the filename.

bghira · 2024-11-17T13:27:22Z

@mhirki did you hit an error there? i did not. however, i ran into a situation where none of the pt files were being moved into place during VAE Caching, so i have now resolved that. and implemented your idea.

mhirki · 2024-11-17T14:42:18Z

Nah, I don't have a multi-GPU system for testing this. I was just looking at the errors that @playerzer0x was encountering.

playerzer0x · 2024-11-20T00:23:51Z

Different datasets, similar issue.

Error:

2024-11-20 00:09:41,385 [INFO] Configuring data backend: jmmymrbl_cutup_photography_style-512
2024-11-20 00:09:41,393 [INFO] (id=jmmymrbl_cutup_photography_style-512) Loading bucket manager.
2024-11-20 00:09:41,394 [WARNING] No cache file found, creating new one.
2024-11-20 00:09:41,394 [INFO] (id=jmmymrbl_cutup_photography_style-512) Refreshing aspect buckets on main process.
2024-11-20 00:09:41,395 [INFO] Discovering new files...
2024-11-20 00:09:41,430 [INFO] Compressed 0 existing files from 0.
2024-11-20 00:09:41,674 [INFO] Image processing statistics: {'total_processed': 20, 'skipped': {'already_exists': 0, 'metadata_missing': 0, 'not_found': 0, 'too_small': 0, 'other': 0}}
2024-11-20 00:09:41,683 [INFO] Enforcing minimum image size of 0.262144. This could take a while for very-large datasets.
2024-11-20 00:09:41,689 [INFO] Completed aspect bucket update.
2024-11-20 00:09:41,695 [INFO] Configured backend: {'id': 'jmmymrbl_cutup_photography_style-512', 'config': {'repeats': 0, 'crop': False, 'crop_aspect': 'square', 'crop_style': 'random', 'disable_validation': False, 'resolution': 0.262144, 'resolution_type': 'area', 'caption_strategy': 'textfile', 'instance_data_dir': 'datasets/dsycam/jmmymrbl_cutup_photography_style', 'maximum_image_size': None, 'target_downsample_size': 0.262144, 'config_version': 2}, 'dataset_type': 'image', 'data_backend': <helpers.data_backend.local.LocalDataBackend object at 0x74f06db43910>, 'instance_data_dir': 'datasets/dsycam/jmmymrbl_cutup_photography_style', 'metadata_backend': <helpers.metadata.backends.discovery.DiscoveryMetadataBackend object at 0x74f06da9ec50>}
(Rank: 0)  | Bucket     | Image Count (per-GPU)
------------------------------
(Rank: 0)  | 0.7        | 2                                                                           | 0/23 [00:00<?, ?it/s]
(Rank: 0)  | 0.78       | 1
(Rank: 0)  | 0.6        | 1
2024-11-20 00:09:41,696 [INFO] (id=jmmymrbl_cutup_photography_style-512) Collecting captions.
Loading captions:   0%|                                                                               | 0/20 [00:00<?, ?it/s]No images were discovered by the bucket manager in the dataset: jmmymrbl_cutup_photography_style-512.
Traceback (most recent call last):
  File "/workspace/SimpleTuner/train.py", line 30, in <module>
    trainer.init_data_backend()
  File "/workspace/SimpleTuner/helpers/training/trainer.py", line 661, in init_data_backend
    raise e
  File "/workspace/SimpleTuner/helpers/training/trainer.py", line 635, in init_data_backend
    configure_multi_databackend(
  File "/workspace/SimpleTuner/helpers/data_backend/factory.py", line 906, in configure_multi_databackend
    raise Exception(
Exception: No images were discovered by the bucket manager in the dataset: jmmymrbl_cutup_photography_style-512.

2024-11-20 00:09:41,758 [INFO] (id=jmmymrbl_cutup_photography_style-512) Initialise text embed pre-computation using the textfile caption strategy. We have 20 captions to process.


Write embeds to disk:   0%|                                                                            | 0/3 [00:00<?, ?it/s]
Write embeds to disk:   0%|                                                                            | 0/3 [00:00<?, ?it/s]

                                                                                                                             2024-11-20 00:09:41,999 [INFO] (id=jmmymrbl_cutup_photography_style-512) Completed processing 20 captions.
2024-11-20 00:09:41,999 [INFO] (id=jmmymrbl_cutup_photography_style-512) Creating VAE latent cache.
Exception in thread Thread-4 (batch_write_embeddings):                                                                       uence through the model will result in indexing errors
Traceback (most recent call last):                                                                                           uence through the model will result in indexing errors
  File "/workspace/SimpleTuner/helpers/caching/text_embeds.py", line 229, in batch_write_embeddings
    first_item = self.write_queue.get(timeout=1)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                       | 0/3 [00:00<?, ?it/s]
  File "/usr/lib/python3.11/queue.py", line 179, in get                                                | 0/2 [00:00<?, ?it/s]
    raise Empty
_queue.Empty

During handling of the above exception, another exception occurred:

Traceback (most recent call last):                                                                                           uence through the model will result in indexing errors
  File "/usr/lib/python3.11/threading.py", line 1045, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.11/threading.py", line 982, in run
    self._target(*self._args, **self._kwargs)
  File "/workspace/SimpleTuner/helpers/caching/text_embeds.py", line 250, in batch_write_embeddings
    self.process_write_batch(batch)
  File "/workspace/SimpleTuner/helpers/caching/text_embeds.py", line 269, in process_write_batch
    futures = [pts:   0%|                                                                              | 0/2 [00:00<?, ?it/s]
              ^
  File "/workspace/SimpleTuner/helpers/caching/text_embeds.py", line 270, in <listcomp>
    executor.submit(self.data_backend.torch_save, *args) for args in batch
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/concurrent/futures/thread.py", line 169, in submit
    raise RuntimeError('cannot schedule new futures after '
RuntimeError: cannot schedule new futures after interpreter shutdown

debug.log:

2024-11-20 00:09:41,385 [INFO] (DataBackendFactory) Configuring data backend: jmmymrbl_cutup_photography_style-512
2024-11-20 00:09:41,393 [INFO] (DataBackendFactory) (id=jmmymrbl_cutup_photography_style-512) Loading bucket manager.
2024-11-20 00:09:41,394 [WARNING] (DiscoveryMetadataBackend) No cache file found, creating new one.
2024-11-20 00:09:41,394 [INFO] (DataBackendFactory) (id=jmmymrbl_cutup_photography_style-512) Refreshing aspect buckets on main process.
2024-11-20 00:09:41,395 [INFO] (BaseMetadataBackend) Discovering new files...
2024-11-20 00:09:41,430 [INFO] (BaseMetadataBackend) Compressed 0 existing files from 0.
2024-11-20 00:09:41,674 [INFO] (BaseMetadataBackend) Image processing statistics: {'total_processed': 20, 'skipped': {'already_exists': 0, 'metadata_missing': 0, 'not_found': 0, 'too_small': 0, 'other': 0}}
2024-11-20 00:09:41,683 [INFO] (BaseMetadataBackend) Enforcing minimum image size of 0.262144. This could take a while for very-large datasets.
2024-11-20 00:09:41,689 [INFO] (BaseMetadataBackend) Completed aspect bucket update.
2024-11-20 00:09:41,695 [INFO] (DataBackendFactory) Configured backend: {'id': 'jmmymrbl_cutup_photography_style-512', 'config': {'repeats': 0, 'crop': False, 'crop_aspect': 'square', 'crop_style': 'random', 'disable_validation': False, 'resolution': 0.262144, 'resolution_type': 'area', 'caption_strategy': 'textfile', 'instance_data_dir': 'datasets/dsycam/jmmymrbl_cutup_photography_style', 'maximum_image_size': None, 'target_downsample_size': 0.262144, 'config_version': 2}, 'dataset_type': 'image', 'data_backend': <helpers.data_backend.local.LocalDataBackend object at 0x74f06db43910>, 'instance_data_dir': 'datasets/dsycam/jmmymrbl_cutup_photography_style', 'metadata_backend': <helpers.metadata.backends.discovery.DiscoveryMetadataBackend object at 0x74f06da9ec50>}
2024-11-20 00:09:41,696 [INFO] (DataBackendFactory) (id=jmmymrbl_cutup_photography_style-512) Collecting captions.
2024-11-20 00:09:41,758 [INFO] (DataBackendFactory) (id=jmmymrbl_cutup_photography_style-512) Initialise text embed pre-computation using the textfile caption strategy. We have 20 captions to process.
2024-11-20 00:09:41,999 [INFO] (DataBackendFactory) (id=jmmymrbl_cutup_photography_style-512) Completed processing 20 captions.
2024-11-20 00:09:41,999 [INFO] (DataBackendFactory) (id=jmmymrbl_cutup_photography_style-512) Creating VAE latent cache.

bghira · 2024-11-20T01:14:34Z

think it needs to be an absolute path to the dataset dir

playerzer0x · 2024-11-20T02:19:15Z

think it needs to be an absolute path to the dataset dir

Hm, relative paths are the only paths that have worked for me as long as I've been using SimpleTuner. Only seem to run into issues when training on a dataset with subdirectories, which is often the case when I train multiple subjects, and multi-gpu.

For now, I've solved my issue by removing subdirectories and having all images + caption files live in the same directory. This isn't ideal longterm, b/c I'd like to have more granular control over repeats at the sub-dataset level. Not super important now, but could be soon if I continue building on this training set.

Let me know if fixing this is becoming too onerous for you. Wondering if there's someone we can hire from the Discord to help out as I'm not as technically capable at solving these issues as I seem to be at finding them :). Happy to put resources in that direction in any case.

bghira · 2024-11-20T10:10:28Z

when using subfolders you need abs paths

playerzer0x · 2024-11-21T00:18:15Z

I changed everything to absolute paths and still run into the same issue. If I change to a single GPU, caching and training starts fine, so I think it's pretty isolated to a multi-gpu issue at this point.

I can change TRAINING_NUM_PROCESSES=1 and get through the caching process, but I can't quit and increase the GPU count after caching because I run into the same error starting up again.

My hunch is it has to do with bucketing per GPU. Maybe if there more GPUs than there are bucketed images to go around per GPU, it throws the error?

local data backend should have file locking for writes and reads, so …

78d2d07

…that we do not read partial writes or corrupt multiprocess writes

bghira merged commit 13799b6 into main Nov 16, 2024
1 check passed

bghira deleted the bugfix/file-locking-multigpu branch November 16, 2024 18:58

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

local data backend should have file locking for writes and reads #1160

local data backend should have file locking for writes and reads #1160

bghira commented Nov 14, 2024

playerzer0x commented Nov 15, 2024

playerzer0x commented Nov 16, 2024

bghira commented Nov 16, 2024

bghira commented Nov 16, 2024

mhirki commented Nov 17, 2024

bghira commented Nov 17, 2024

mhirki commented Nov 17, 2024

playerzer0x commented Nov 20, 2024

bghira commented Nov 20, 2024

playerzer0x commented Nov 20, 2024

bghira commented Nov 20, 2024

playerzer0x commented Nov 21, 2024

local data backend should have file locking for writes and reads #1160

local data backend should have file locking for writes and reads #1160

Conversation

bghira commented Nov 14, 2024

playerzer0x commented Nov 15, 2024

playerzer0x commented Nov 16, 2024

bghira commented Nov 16, 2024

bghira commented Nov 16, 2024

mhirki commented Nov 17, 2024

bghira commented Nov 17, 2024

mhirki commented Nov 17, 2024

playerzer0x commented Nov 20, 2024

bghira commented Nov 20, 2024

playerzer0x commented Nov 20, 2024

bghira commented Nov 20, 2024

playerzer0x commented Nov 21, 2024