Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enforce that each synthesizer is unique in benchmark_single_table #190

Closed
npatki opened this issue Jan 20, 2023 · 0 comments · Fixed by #200
Closed

Enforce that each synthesizer is unique in benchmark_single_table #190

npatki opened this issue Jan 20, 2023 · 0 comments · Fixed by #200
Assignees
Labels
feature request Request for a new feature
Milestone

Comments

@npatki
Copy link

npatki commented Jan 20, 2023

Version: 0.8.0 (in development)

Problem Description

Currently, the user can provide synthesizers to benchmark_single_table via the synthesizers or the custom_synthesizers parameters.

import sdgym

sdgym.benchmark_single_table(
    synthesizers=['GaussianCopulaSynthesizer', 'FastMLPreset'],
    custom_synthesizers=[MyCustomSynthesizerClass]
)

It would be good to enforce that each synthesizer the user passes in is unique. Eg. the following would NOT be allowed:

import sdgym

# This should not be allowed
sdgym.benchmark_single_table(
    synthesizers=['GaussianCopulaSynthesizer', 'GaussianCopulaSynthesizer'],
    custom_synthesizers=[MyCustomSynthesizerClass, MyCustomSynthesizerClass]
)

Rationale: The benchmarking framework is designed to work on unique (synthesizer, dataset) pairs. If users want to do multiple iterations, then they can run the script more times.

Expected behavior

Before executing anything else, check to see if the synthesizers are unique. If they are not, throw an error.

sdgym.benchmark_single_table(
    synthesizers=['GaussianCopulaSynthesizer', 'GaussianCopulaSynthesizer'],
    custom_synthesizers=[MyCustomSynthesizerClass, MyCustomSynthesizerClass]
)

BenchmarkingInputError: Synthesizers must be unique. Please remove repeated values in the 'synthesizers' and 'custom_synthesizers' parameters.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature request Request for a new feature
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants