Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make VI work on v4 #4582

Merged
merged 97 commits into from
Feb 25, 2022
Merged

Make VI work on v4 #4582

merged 97 commits into from
Feb 25, 2022

Conversation

ferrine
Copy link
Member

@ferrine ferrine commented Mar 29, 2021

VI in V4

Referencing #4523 Here to work on VI module

Step 1

The first goal of this PR is to fix test errors in

pytest pymc3/tests/test_variational_inference.py

because there are a lot of errors

Step 2

TBD

@ferrine ferrine changed the base branch from master to v4 March 29, 2021 07:26
@twiecki twiecki changed the title V4 4523 Make VI work on v4 Mar 29, 2021
@michaelosthege
Copy link
Member

@ferrine I understand this is still WIP ?

After #4579 (the rebase) replaces v4 branch you'll have to rebase your branch onto v4 again. It won't be a big deal though: You'll get conflicts in all lines that contain aet, but you can just override with your changes and run the script I posted in #4579 to change all the aet occurences.

@ferrine ferrine changed the title Make VI work on v4 [WIP] Make VI work on v4 Mar 29, 2021
@brandonwillard brandonwillard force-pushed the v4 branch 2 times, most recently from 93a096d to c9fa127 Compare March 29, 2021 17:45
@brandonwillard brandonwillard marked this pull request as draft March 29, 2021 18:27
pymc3/variational/opvi.py Outdated Show resolved Hide resolved
pymc3/variational/opvi.py Outdated Show resolved Hide resolved
pymc3/variational/opvi.py Outdated Show resolved Hide resolved
pymc3/variational/opvi.py Outdated Show resolved Hide resolved
@michaelosthege michaelosthege changed the base branch from v4 to main June 19, 2021 16:36
@michaelosthege
Copy link
Member

@ferrine this was one of the last remaining PRs that still had v4 as their target.
I changed it, so we can delete the v4 branch and thereby avoid accidentally merging things into it, as happened recently.
Please rebase onto main.

@michaelosthege
Copy link
Member

I included the VI tests as a standalone job into the pytest workflow, so it's output is not contaminated by flaky tests that still exist on main.

Running locally on Windows:

35 failed, 110 passed, 1 skipped, 19 warnings, 45 errors in 766.25s (0:12:46)
List of test failures
FAILED pymc3/tests/test_variational_inference.py::test_vae - NotImplementedError: not yet ready
FAILED pymc3/tests/test_variational_inference.py::test_fit_oo[FullRankADVI-full] - AssertionError:
FAILED pymc3/tests/test_variational_inference.py::test_fit_oo[FullRankADVI-mini] - AssertionError:
FAILED pymc3/tests/test_variational_inference.py::test_replacements[SVGD] - assert array([1], dtype=int64) == (10,)
FAILED pymc3/tests/test_variational_inference.py::test_rowwise_approx[MeanFieldGroup: {}] - NotImplementedError: not yet ready
FAILED pymc3/tests/test_variational_inference.py::test_rowwise_approx[FullRankGroup: {}] - NotImplementedError: not yet ready
FAILED pymc3/tests/test_variational_inference.py::test_rowwise_approx[NormalizingFlowGroup: {'flow': 'scale'}] - NotImplementedError: not yet ready
FAILED pymc3/tests/test_variational_inference.py::test_rowwise_approx[NormalizingFlowGroup: {'flow': 'loc'}] - NotImplementedError: not yet ready
FAILED pymc3/tests/test_variational_inference.py::test_rowwise_approx[NormalizingFlowGroup: {'flow': 'hh'}] - NotImplementedError: not yet ready
FAILED pymc3/tests/test_variational_inference.py::test_rowwise_approx[NormalizingFlowGroup: {'flow': 'planar'}] - NotImplementedError: not yet ready
FAILED pymc3/tests/test_variational_inference.py::test_rowwise_approx[NormalizingFlowGroup: {'flow': 'radial'}] - NotImplementedError: not yet ready
FAILED pymc3/tests/test_variational_inference.py::test_rowwise_approx[NormalizingFlowGroup: {'flow': 'radial-loc'}] - NotImplementedError: not yet ready
FAILED pymc3/tests/test_variational_inference.py::test_pickle_approx[FullRankGroup: None, MeanFieldGroup: ['one']] - ValueError: Input dimension mismatch. (input[0].shape[1] = 20, input[2].shape[1] = 50)
FAILED pymc3/tests/test_variational_inference.py::test_pickle_approx[MeanFieldGroup: ['one'], FullRankGroup: ['two'], NormalizingFlowGroup: ['three']] - FileNotFoundError: [WinError 206] Der Dateiname oder ...
FAILED pymc3/tests/test_variational_inference.py::test_pickle_approx[MeanFieldGroup: ['one'], FullRankGroup: ['two', 'three']] - ValueError: Input dimension mismatch. (input[0].shape[1] = 20, input[2].shape...
FAILED pymc3/tests/test_variational_inference.py::test_pickle_approx[MeanFieldGroup: ['one'], EmpiricalGroup: ['two', 'three']] - ValueError: Input dimension mismatch. (input[0].shape[1] = 20, input[2].shap...
FAILED pymc3/tests/test_variational_inference.py::test_discrete_not_allowed - FileNotFoundError: [WinError 206] Der Dateiname oder die Erweiterung ist zu lang: 'c:\\users\\osthege\\appdata\\local\\continuum...
FAILED pymc3/tests/test_variational_inference.py::test_var_replacement - FileNotFoundError: [WinError 206] Der Dateiname oder die Erweiterung ist zu lang: 'c:\\users\\osthege\\appdata\\local\\continuum\\min...
FAILED pymc3/tests/test_variational_inference.py::test_empirical_from_trace - FileNotFoundError: [WinError 206] Der Dateiname oder die Erweiterung ist zu lang: 'c:\\users\\osthege\\appdata\\local\\continuum...
FAILED pymc3/tests/test_variational_inference.py::test_flow_det[PlanarFlow] - FileNotFoundError: [WinError 206] Der Dateiname oder die Erweiterung ist zu lang: 'c:\\users\\osthege\\appdata\\local\\continuum...
FAILED pymc3/tests/test_variational_inference.py::test_flow_det[RadialFlow] - FileNotFoundError: [WinError 206] Der Dateiname oder die Erweiterung ist zu lang: 'c:\\users\\osthege\\appdata\\local\\continuum...
FAILED pymc3/tests/test_variational_inference.py::test_flow_det[ScaleFlow] - FileNotFoundError: [WinError 206] Der Dateiname oder die Erweiterung ist zu lang: 'c:\\users\\osthege\\appdata\\local\\continuum\...
FAILED pymc3/tests/test_variational_inference.py::test_flow_det[LocFlow] - FileNotFoundError: [WinError 206] Der Dateiname oder die Erweiterung ist zu lang: 'c:\\users\\osthege\\appdata\\local\\continuum\\m...
FAILED pymc3/tests/test_variational_inference.py::test_flow_det[HouseholderFlow] - FileNotFoundError: [WinError 206] Der Dateiname oder die Erweiterung ist zu lang: 'c:\\users\\osthege\\appdata\\local\\cont...
FAILED pymc3/tests/test_variational_inference.py::test_flow_det_local[PlanarFlow] - FileNotFoundError: [WinError 206] Der Dateiname oder die Erweiterung ist zu lang: 'c:\\users\\osthege\\appdata\\local\\con...
FAILED pymc3/tests/test_variational_inference.py::test_flow_det_local[RadialFlow] - FileNotFoundError: [WinError 206] Der Dateiname oder die Erweiterung ist zu lang: 'c:\\users\\osthege\\appdata\\local\\con...
FAILED pymc3/tests/test_variational_inference.py::test_flow_det_local[ScaleFlow] - FileNotFoundError: [WinError 206] Der Dateiname oder die Erweiterung ist zu lang: 'c:\\users\\osthege\\appdata\\local\\cont...
FAILED pymc3/tests/test_variational_inference.py::test_flow_det_local[LocFlow] - FileNotFoundError: [WinError 206] Der Dateiname oder die Erweiterung ist zu lang: 'c:\\users\\osthege\\appdata\\local\\contin...
FAILED pymc3/tests/test_variational_inference.py::test_flow_det_local[HouseholderFlow] - FileNotFoundError: [WinError 206] Der Dateiname oder die Erweiterung ist zu lang: 'c:\\users\\osthege\\appdata\\local...
FAILED pymc3/tests/test_variational_inference.py::test_flows_collect_chain - FileNotFoundError: [WinError 206] Der Dateiname oder die Erweiterung ist zu lang: 'c:\\users\\osthege\\appdata\\local\\continuum\...
FAILED pymc3/tests/test_variational_inference.py::test_flow_formula[planar-1-order0] - AttributeError: 'ValidatingScratchpad' object has no attribute 'test_value'
FAILED pymc3/tests/test_variational_inference.py::test_flow_formula[planar*2-2-order1] - AttributeError: 'ValidatingScratchpad' object has no attribute 'test_value'
FAILED pymc3/tests/test_variational_inference.py::test_flow_formula[planar-planar-2-order2] - AttributeError: 'ValidatingScratchpad' object has no attribute 'test_value'
FAILED pymc3/tests/test_variational_inference.py::test_flow_formula[planar-planar*2-3-order3] - AttributeError: 'ValidatingScratchpad' object has no attribute 'test_value'
FAILED pymc3/tests/test_variational_inference.py::test_flow_formula[hh-planar*2-3-order4] - AttributeError: 'ValidatingScratchpad' object has no attribute 'test_value'
ERROR pymc3/tests/test_variational_inference.py::test_sample_aevb[MeanFieldGroup: {}] - NotImplementedError: not yet ready
ERROR pymc3/tests/test_variational_inference.py::test_sample_aevb[FullRankGroup: {}] - NotImplementedError: not yet ready
ERROR pymc3/tests/test_variational_inference.py::test_sample_aevb[NormalizingFlowGroup: {'flow': 'scale'}] - NotImplementedError: not yet ready
ERROR pymc3/tests/test_variational_inference.py::test_sample_aevb[NormalizingFlowGroup: {'flow': 'loc'}] - NotImplementedError: not yet ready
ERROR pymc3/tests/test_variational_inference.py::test_sample_aevb[NormalizingFlowGroup: {'flow': 'hh'}] - NotImplementedError: not yet ready
ERROR pymc3/tests/test_variational_inference.py::test_sample_aevb[NormalizingFlowGroup: {'flow': 'planar'}] - NotImplementedError: not yet ready
ERROR pymc3/tests/test_variational_inference.py::test_sample_aevb[NormalizingFlowGroup: {'flow': 'radial'}] - NotImplementedError: not yet ready
ERROR pymc3/tests/test_variational_inference.py::test_sample_aevb[NormalizingFlowGroup: {'flow': 'radial-loc'}] - NotImplementedError: not yet ready
ERROR pymc3/tests/test_variational_inference.py::test_replacements_in_sample_node_aevb[MeanFieldGroup: {}] - NotImplementedError: not yet ready
ERROR pymc3/tests/test_variational_inference.py::test_replacements_in_sample_node_aevb[FullRankGroup: {}] - NotImplementedError: not yet ready
ERROR pymc3/tests/test_variational_inference.py::test_replacements_in_sample_node_aevb[NormalizingFlowGroup: {'flow': 'scale'}] - NotImplementedError: not yet ready
ERROR pymc3/tests/test_variational_inference.py::test_replacements_in_sample_node_aevb[NormalizingFlowGroup: {'flow': 'loc'}] - NotImplementedError: not yet ready
ERROR pymc3/tests/test_variational_inference.py::test_replacements_in_sample_node_aevb[NormalizingFlowGroup: {'flow': 'hh'}] - NotImplementedError: not yet ready
ERROR pymc3/tests/test_variational_inference.py::test_replacements_in_sample_node_aevb[NormalizingFlowGroup: {'flow': 'planar'}] - NotImplementedError: not yet ready
ERROR pymc3/tests/test_variational_inference.py::test_replacements_in_sample_node_aevb[NormalizingFlowGroup: {'flow': 'radial'}] - NotImplementedError: not yet ready
ERROR pymc3/tests/test_variational_inference.py::test_replacements_in_sample_node_aevb[NormalizingFlowGroup: {'flow': 'radial-loc'}] - NotImplementedError: not yet ready
ERROR pymc3/tests/test_variational_inference.py::test_logq_mini_sample_aevb[MeanFieldGroup: {}] - NotImplementedError: not yet ready
ERROR pymc3/tests/test_variational_inference.py::test_logq_mini_sample_aevb[FullRankGroup: {}] - NotImplementedError: not yet ready
ERROR pymc3/tests/test_variational_inference.py::test_logq_mini_sample_aevb[NormalizingFlowGroup: {'flow': 'scale'}] - NotImplementedError: not yet ready
ERROR pymc3/tests/test_variational_inference.py::test_logq_mini_sample_aevb[NormalizingFlowGroup: {'flow': 'loc'}] - NotImplementedError: not yet ready
ERROR pymc3/tests/test_variational_inference.py::test_logq_mini_sample_aevb[NormalizingFlowGroup: {'flow': 'hh'}] - NotImplementedError: not yet ready
ERROR pymc3/tests/test_variational_inference.py::test_logq_mini_sample_aevb[NormalizingFlowGroup: {'flow': 'planar'}] - NotImplementedError: not yet ready
ERROR pymc3/tests/test_variational_inference.py::test_logq_mini_sample_aevb[NormalizingFlowGroup: {'flow': 'radial'}] - NotImplementedError: not yet ready
ERROR pymc3/tests/test_variational_inference.py::test_logq_mini_sample_aevb[NormalizingFlowGroup: {'flow': 'radial-loc'}] - NotImplementedError: not yet ready
ERROR pymc3/tests/test_variational_inference.py::test_logq_aevb[MeanFieldGroup: {}] - NotImplementedError: not yet ready
ERROR pymc3/tests/test_variational_inference.py::test_logq_aevb[FullRankGroup: {}] - NotImplementedError: not yet ready
ERROR pymc3/tests/test_variational_inference.py::test_logq_aevb[NormalizingFlowGroup: {'flow': 'scale'}] - NotImplementedError: not yet ready
ERROR pymc3/tests/test_variational_inference.py::test_logq_aevb[NormalizingFlowGroup: {'flow': 'loc'}] - NotImplementedError: not yet ready
ERROR pymc3/tests/test_variational_inference.py::test_logq_aevb[NormalizingFlowGroup: {'flow': 'hh'}] - NotImplementedError: not yet ready
ERROR pymc3/tests/test_variational_inference.py::test_logq_aevb[NormalizingFlowGroup: {'flow': 'planar'}] - NotImplementedError: not yet ready
ERROR pymc3/tests/test_variational_inference.py::test_logq_aevb[NormalizingFlowGroup: {'flow': 'radial'}] - NotImplementedError: not yet ready
ERROR pymc3/tests/test_variational_inference.py::test_logq_aevb[NormalizingFlowGroup: {'flow': 'radial-loc'}] - NotImplementedError: not yet ready
ERROR pymc3/tests/test_variational_inference.py::test_aevb[ADVI] - AttributeError: 'TensorVariable' object has no attribute 'init_value'
ERROR pymc3/tests/test_variational_inference.py::test_aevb[FullRankADVI] - AttributeError: 'TensorVariable' object has no attribute 'init_value'
ERROR pymc3/tests/test_variational_inference.py::test_aevb[SVGD] - AttributeError: 'TensorVariable' object has no attribute 'init_value'
ERROR pymc3/tests/test_variational_inference.py::test_aevb[ASVGD] - AttributeError: 'TensorVariable' object has no attribute 'init_value'
ERROR pymc3/tests/test_variational_inference.py::test_aevb[NFVI=scale-loc] - AttributeError: 'TensorVariable' object has no attribute 'init_value'
ERROR pymc3/tests/test_variational_inference.py::test_pickle_approx_aevb[MeanFieldGroup: {}] - NotImplementedError: not yet ready
ERROR pymc3/tests/test_variational_inference.py::test_pickle_approx_aevb[FullRankGroup: {}] - NotImplementedError: not yet ready
ERROR pymc3/tests/test_variational_inference.py::test_pickle_approx_aevb[NormalizingFlowGroup: {'flow': 'scale'}] - NotImplementedError: not yet ready
ERROR pymc3/tests/test_variational_inference.py::test_pickle_approx_aevb[NormalizingFlowGroup: {'flow': 'loc'}] - NotImplementedError: not yet ready
ERROR pymc3/tests/test_variational_inference.py::test_pickle_approx_aevb[NormalizingFlowGroup: {'flow': 'hh'}] - NotImplementedError: not yet ready
ERROR pymc3/tests/test_variational_inference.py::test_pickle_approx_aevb[NormalizingFlowGroup: {'flow': 'planar'}] - NotImplementedError: not yet ready
ERROR pymc3/tests/test_variational_inference.py::test_pickle_approx_aevb[NormalizingFlowGroup: {'flow': 'radial'}] - NotImplementedError: not yet ready
ERROR pymc3/tests/test_variational_inference.py::test_pickle_approx_aevb[NormalizingFlowGroup: {'flow': 'radial-loc'}] - NotImplementedError: not yet ready

Hope this helps :)

@ferrine
Copy link
Member Author

ferrine commented Jul 28, 2021

@brandonwillard I have a subset of functionality working. AEVB inference does not work and I start thinking maybe I mark this as NotImplemented?

if aevb: raise NotImplementedError("AEVB is not yet supported, open an issue once you need it https://github.com/pymc-devs/pymc3/issues")

@twiecki
Copy link
Member

twiecki commented Jul 28, 2021 via email

@ferrine
Copy link
Member Author

ferrine commented Jul 28, 2021

Hmm, the issue still appears with grouped inference that might be some shape bug/typo somewhere. I'll keep working on this. This should not be that complicated I hope

@codecov
Copy link

codecov bot commented Sep 8, 2021

Codecov Report

Merging #4582 (ef5f91b) into main (c892317) will increase coverage by 5.76%.
The diff coverage is 92.24%.

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #4582      +/-   ##
==========================================
+ Coverage   81.44%   87.21%   +5.76%     
==========================================
  Files          81       81              
  Lines       14204    14265      +61     
==========================================
+ Hits        11569    12441     +872     
+ Misses       2635     1824     -811     
Impacted Files Coverage Δ
pymc/backends/arviz.py 90.28% <ø> (ø)
pymc/variational/flows.py 83.54% <ø> (+46.75%) ⬆️
pymc/variational/updates.py 92.11% <ø> (+4.43%) ⬆️
pymc/variational/opvi.py 82.24% <90.27%> (+46.02%) ⬆️
pymc/variational/approximations.py 66.06% <91.30%> (+29.75%) ⬆️
pymc/distributions/logprob.py 96.39% <100.00%> (+0.03%) ⬆️
pymc/model.py 86.01% <100.00%> (+0.03%) ⬆️
pymc/sampling.py 87.24% <100.00%> (+0.47%) ⬆️
pymc/variational/inference.py 87.24% <100.00%> (+57.86%) ⬆️
pymc/bart/pgbart.py 94.92% <0.00%> (-0.32%) ⬇️
... and 16 more

@twiecki twiecki marked this pull request as ready for review September 16, 2021 14:48
@twiecki twiecki changed the title [WIP] Make VI work on v4 Make VI work on v4 Sep 16, 2021
@twiecki
Copy link
Member

twiecki commented Sep 16, 2021

/pre-commit-run

@twiecki
Copy link
Member

twiecki commented Sep 24, 2021

@ferrine I rebased this.

@ferrine
Copy link
Member Author

ferrine commented Sep 24, 2021

One weird test is left and it's done. Also I remember to change skip to xfail for one test

@twiecki
Copy link
Member

twiecki commented Sep 26, 2021

Tests are passing 🥳! You need to rebase from the rename, however.

pymc/variational/opvi.py Outdated Show resolved Hide resolved
pymc/variational/opvi.py Outdated Show resolved Hide resolved
pymc/variational/opvi.py Outdated Show resolved Hide resolved
ferrine and others added 5 commits February 23, 2022 10:09
Co-authored-by: Ricardo Vieira <28983449+ricardoV94@users.noreply.github.com>
Co-authored-by: Ricardo Vieira <28983449+ricardoV94@users.noreply.github.com>
Co-authored-by: Ricardo Vieira <28983449+ricardoV94@users.noreply.github.com>
@ricardoV94
Copy link
Member

@ferrine I pushed some changes and closed outdated conversations. Can you go through the remaining open conversations and address/close them? Otherwise I think this is pretty ready to merge, and we can focus on refactoring/deprecating the remaining methods in follow up PRs

@twiecki
Copy link
Member

twiecki commented Feb 23, 2022

@ricardoV94 are we good to merge?

@ricardoV94
Copy link
Member

Only one way to find out...

pymc/distributions/logprob.py Outdated Show resolved Hide resolved
pymc/distributions/logprob.py Outdated Show resolved Hide resolved
pymc/distributions/logprob.py Show resolved Hide resolved
@michaelosthege
Copy link
Member

I commented one instance where detailed type info from the docstring could be annotated in the signature.
Otherwise I'd say this is all good and we should probably merge it before #5474

Co-authored-by: Michael Osthege <michael.osthege@outlook.com>
@michaelosthege
Copy link
Member

/pre-commit run

@twiecki
Copy link
Member

twiecki commented Feb 24, 2022

/pre-commit-run

@michaelosthege
Copy link
Member

@twiecki The same pre-commit error happened in #5474
Maybe an updated pre-commit dependency that changed the rules?

@twiecki
Copy link
Member

twiecki commented Feb 24, 2022

No it's something that should be fixed on main, I merged a PR too wrongly.

@michaelosthege
Copy link
Member

I was about to hit approve when I noticed this. Looks like the linux permissions on that file changed?? Could explain why it's not showing up in the diff view.
grafik

Otherwise I'd say this is ready to be squash-merged

@ferrine
Copy link
Member Author

ferrine commented Feb 25, 2022

I was about to hit approve when I noticed this. Looks like the linux permissions on that file changed?? Could explain why it's not showing up in the diff view. grafik

Otherwise I'd say this is ready to be squash-merged

Yeah, file permissions were weird, I fixed this

@twiecki twiecki merged commit e987950 into main Feb 25, 2022
@twiecki twiecki deleted the v4-4523 branch February 25, 2022 10:27
@twiecki
Copy link
Member

twiecki commented Feb 25, 2022

Incredible -- congrats @ferrine and @ricardoV94!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
v4 VI Variational Inference
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants