Time required for sample_all function?

[clj2567]

The sample_all function never really returns anything. It keeps running in a loop. Is there any solution for this? One instance it ran for 2 hours , still no output.

[JDTheRipperPC]

Thank you for reporting this @clj2567.

To be able to help you, we would need some additional information about the data and metadata you are trying to sample. Also, could you provide us a code snippet that are you using to sample?

[clj2567]

The data is the one that is referenced in the paper https://www.kaggle.com/c/airbnb-recruiting-new-user-bookings/data

This is the metadata file that I am using.meta_sample.txt

Here is the code snippet

from sdv import CSVDataLoader
from sdv import Modeler
from sdv import Sampler

data_loader = CSVDataLoader('data/meta_sample.json')
data_navigator = data_loader.load_data()
transformed_data = data_navigator.transform_data()
modeler = Modeler(data_navigator)
modeler.model_database()

sampler = Sampler(data_navigator, modeler)
sampled_all = sampler.sample_rows('Countries', 10)
sampled_all.to_csv(artifact_path + '/countries_sampled.csv',index=False)
print(sampled_all)
sampled_all = sampler.sample_rows('Users', 1000)
sampled_all.to_csv(artifact_path + '/users_sampled.csv', index=False)
print(sampled_all)
sampled_all = sampler.sample_rows('Sessions', 1000)
sampled_all.to_csv(artifact_path + '/sessions_sampled.csv', index=False)
print(sampled_all)

[csala]

Hi @clj2567

There is an error in the categorical values sampling implementation that provoked this behavior: the time it takes to sample increases exponentially with the number of categorical columns found in the dataset.

This fix for this has been covered in the issue_120_compatibility_with_rdt_issue72 branch and will be released soon.

[clj2567]

Hi @csala ,

Thanks for the update. Is there any timeline around this?

[csala]

Thanks for the update. Is there any timeline around this?

Yes. This will most likely be released this week.

[JDTheRipperPC]

This should have been fixed in PR #121