[FEATURE REQUEST] Add multi-threading/processing to the Dataset Validator

[chavinlo]

It's awfully slow.

I am willing to do it if someone gives me directions

[chinoll]

Did you install xformers? huggingface/diffusers#1343

[chavinlo]

Did you install xformers? huggingface/diffusers#1343

yes but I don't think xformers has anything to do with the validation process...

[cafeai]

You can skip validation, which is what I would suggest. Internally, I'm personally using a preprocessing application written in Rust. This probably isn't something you want to do in Python.

[lopho]

small self plug for a preprocessor written in python and fully parallel:
https://github.com/lopho/parallel_dataprocessor

[lopho]

#60 has parallel validation and migration