[WIP] Test reform post and results #627
Closed
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR mainly addresses issue #615 but was brought up in #602 and #606. It adds a series of scripts that posts a reform file or a reform through the Taxbrain GUI on a local Taxbrain instance using the requests library and simultaneously runs the same reform directly using
taxcalc.dropq.run_nth_year_tax_calc_model
. The results from TaxBrain are retrieved from the local sqlite3 tabletaxbrain_taxsaveinputs
and the results from submitting directly totaxcalc
are formatted into the same structure as the TaxBrain results. These two results are then compared.This effectively tests two parts of TaxBrain:
taxcalc
works as expected.Currently, it takes about 37 minutes to run the 7 tests (each runs for about 5 minutes). I think that the number of tests could be decreased. For example, I don't think the tests involving the Trump and RyanBrady plans test anything that is not tested using r1.json and the reform which posts all of the taxbrain parameters.
The TaxBrain GUI parameters are changed in unexpected ways such as being rounded, casted from floats to integers (1.0 to 1), and from boolean to binary (0-1). Since the dropq algorithm generates a random seed directly from the reform data, this unexpected data manipulation results in using different data sets to estimate the results from using the same reform. Thus, to test the TaxBrain GUI, I tried to compare the reform results within a 2% tolerance.
Further, it does not work when the entire test suite is run. I think that at some point in the test suite the puf file and maybe the requests functions are mocked. To run these tests run
py.test webapp/apps/taxbrain/tests/reform_input_processing/
and make sure the server and supervisord are running in other terminals. To run the test suite without these tests runpy.test webapp/apps/ -m "not requires_pufcsv
Initially, I hoped that this PR would be ready by the 1.0.2 release (#624). However, I don't think it will be ready in time. I think it could still be cleaned up a good bit and maybe some of the functions could be mocked.
cc @martinholmer @brittainhard