Bubble plot visualization

[hayleefay]

This PR includes a bokeh bubble plot that will appear on the results page of the ccc.

[hdoupe]

@hayleefay Thanks for setting up this PR.

First, I think you need to add a blank __init__.py file to the directory apps/btax/bubble_plot.

Second, I tried to run the app locally and got the following error:

Did I do something wrong here? I used your code as is except for adding the __init__.py file. Also, I set the Corporate Income Tax-Rate to 0.2 if that helps.

[hayleefay]

@hdoupe Ah, that is probably a result of the version of bokeh being used. I forgot to include that in the PR. This visualization was built on 0.12.4. I updated requirements.txt and added the __init__.py.

[hdoupe]

@hayleefay Great, thanks!

I see what you mean. So, the javascript calls output_detail and only gives it 15 seconds to deliver an output. How long does it take to create the bubble plots? I increased the waiting time to 30 seconds and it still isn't returning in time.

[hayleefay]

@hdoupe Actually, when I increased the time from 7 to 15 it loads. Is your page not loading at 15?

[martinholmer]

@hayleefay, I have a question about the following line in this pull request:

bokeh==0.12.4

Is this being added because the bubble plot cannot be created using bokeh 0.12.3?
Tax-Calculator is using bokeh>=0.12.3. Please explain.

[hdoupe]

@hayleefay, I realized my problem. model.taxresult[0] only has keys, [u'result_years', 'tooltips', u'asset_coc', u'asset_metr', u'industry_coc', u'row_grouping', u'asset_d', u'btax_version', u'dropq_version', u'taxcalc_version', u'asset_mettr', u'industry_mettr', u'industry_metr', u'industry_d']

I'm not sure what's happening here. Let me look into this some more.

[hayleefay]

@hdoupe In order to get the other keys, you'll need to run the newest B-Tax version from Github. Jason made some changes last week to update what B-Tax is passing.

[hayleefay]

@martinholmer The visualization actually can be loaded on 0.12.3. It just can't be loaded on 0.12.6 and when I run locally that is the version it is defaulting to if I don't pin a version. Will that not be an issue in production?

[hdoupe]

@hayleefay Ah, right. Now it's working. I set up the local version of btax but forgot to remove the btax in my aei_dropq environment. Sorry about the confusion.

[martinholmer]

@hayleefay said:

In order to get the other keys, you'll need to run the newest B-Tax version from Github.
Jason made some changes last week to update what B-Tax is passing.

I'm confused. I thought TaxBrain used packages. The taxcalc package, the btax package, and the ogusa package.

The newest btax package available at anaconda.org is 0.1.8, which is more than two months old. Where are the new btax packages?

@hdoupe

[hdoupe]

@martinholmer said

The newest btax package available at anaconda.org is 0.1.8, which is more than two months old. Where are the new btax packages?

We are using the most up-to-date code in the B-Tax repo. But there will need to be a new B-Tax package on anaconda.org in order to put this on the test and production servers.

[martinholmer]

@hayleefay said:

The visualization actually can be loaded on 0.12.3. It just can't be loaded on 0.12.6 and when I run locally that is the version it is defaulting to if I don't pin a version. Will that not be an issue in production?

This situation indicates a serious problem. I don't know if there has been a regression moving from bokeh 0.12.3 to 0.12.6 or if there is something about your code that is inconsistent with bokeh 0.12.6. Pinning to 0.12.4 is just sweeping the dust under the rug. You need to figure out what enhancement (or bug?) in 0.12.6 prevents your bubble plot from working. This is very important.

[hayleefay]

@martinholmer, okay, I will work on updating the code. I see that 0.12.7 was just released. Should I update the code (if needed) for 0.12.7 or stay on 0.12.6?

[hdoupe]

@hayleefay I don't have much javascript experience, but I think a way to do this would be to leave the 7 second interval as is. Then, when model.tax_result is not None, immediately send a 202 response back with a new wait time of ~15-20 seconds.

Something along these lines:

def output_detail(request, pk):
    """
    This view is the single page of diplaying a progress bar for how
    close the job is to finishing, and then it will also display the
    job results if the job is done. Finally, it will render a 'job failed'
    page if the job has failed.
    """

    try:
        url = BTaxOutputUrl.objects.get(pk=pk)
    except:
        raise Http404

    model = url.unique_inputs
    if model.tax_result:
        return JsonResponse({'eta': exp_num_minutes, 'wait_interval': 20000}, status=202)
        tables = url.unique_inputs.tax_result[0]
        first_year = url.unique_inputs.first_year
        created_on = url.unique_inputs.creation_date
        tables["tooltips"] = {
            "metr": METR_TOOLTIP,
            "mettr": METTR_TOOLTIP,
            "coc": COC_TOOLTIP,
            "dprc": DPRC_TOOLTIP,
        }
        bubble_js, bubble_div, cdn_js, cdn_css = bubble_plot_tabs(model.tax_result[0]['dataframes'])

        inputs = url.unique_inputs
        is_registered = True if request.user.is_authenticated() else False
        context = tables.copy()
        context.update({
            'locals': locals(),
            'unique_url': url,
            'btax_version': BTAX_VERSION,
            'taxcalc_version': TAXCALC_VERSION,
            'table_json': json.dumps(tables),
            'created_on': created_on,
            'first_year': first_year,
            'is_btax': True,
            'bubble_js': bubble_js,
            'bubble_div': bubble_div,
            'cdn_js': cdn_js,
            'cdn_css': cdn_css,
        })
        return render(request, 'btax/results.html', context)

    else:

        job_ids = model.job_ids
        jobs_to_check = model.jobs_not_ready
        if not jobs_to_check:
            jobs_to_check = normalize(job_ids)
        else:
            jobs_to_check = normalize(jobs_to_check)

        try:
            jobs_ready = dropq_compute.dropq_results_ready(jobs_to_check)
        except JobFailError as jfe:
            print jfe
            return render_to_response('taxbrain/failed.html',
                                     context={'is_btax': True})

        if any([j == 'FAIL' for j in jobs_ready]):
            failed_jobs = [sub_id for (sub_id, job_ready) in
                           zip(jobs_to_check, jobs_ready) if job_ready == 'FAIL']

            #Just need the error message from one failed job
            error_msgs = dropq_compute.dropq_get_results([failed_jobs[0]], job_failure=True)
            error_msg = error_msgs[0]
            val_err_idx = error_msg.rfind("Error")
            context = {"error_msg": error_msg[val_err_idx:],
                       "is_btax": True}
            return render(request, 'taxbrain/failed.html', context)



        if all([job == 'YES' for job in jobs_ready]):
            model.tax_result = dropq_compute.btax_get_results(normalize(job_ids))

            model.creation_date = datetime.datetime.now()
            print 'ready'
            model.save()
            return redirect(url)
        else:
            jobs_not_ready = [sub_id for (sub_id, job_ready) in
                                zip(jobs_to_check, jobs_ready) if not job_ready == 'YES']
            jobs_not_ready = denormalize(jobs_not_ready)
            model.jobs_not_ready = jobs_not_ready
            print 'not ready', jobs_not_ready
            model.save()
            if request.method == 'POST':
                # if not ready yet, insert number of minutes remaining
                exp_comp_dt = url.exp_comp_datetime
                utc_now = datetime.datetime.utcnow()
                utc_now = utc_now.replace(tzinfo=pytz.utc)
                dt = exp_comp_dt - utc_now
                exp_num_minutes = dt.total_seconds() / 60.
                exp_num_minutes = round(exp_num_minutes, 2)
                exp_num_minutes = exp_num_minutes if exp_num_minutes > 0 else 0
                if exp_num_minutes > 0:
                    print('here')
                    return JsonResponse({'eta': exp_num_minutes,'wait_interval': 7000}, status=202)
                else:
                    return JsonResponse({'eta': exp_num_minutes,'wait_interval': 7000}, status=200)

            else:
                print "rendering not ready yet"
                not_ready_data = {'eta': '100', 'is_btax': True, 'wait_interval': 7000}
                return render_to_response('btax/not_ready.html', not_ready_data,
                                          context_instance=RequestContext(request))

Then, change the javascript to:
interval = setInterval(ajaxEta, wait_interval);

There are some details that you would have to fill in, but I think something like this would work.

[hayleefay]

@martinholmer, I have updated the code to work on the latest version, 0.12.7. It looks like we have to pin the version in order to load the BokehJS for the widgets: https://bokeh.pydata.org/en/0.12.7/docs/user_guide/embed.html

Or do you know a way around this?

[martinholmer]

@hayleefay said in pull request #634:

I have updated the code to work on the latest [bokeh] version, 0.12.7.
It looks like we have to pin the version in order to load the BokehJS for the widgets:
https://bokeh.pydata.org/en/0.12.7/docs/user_guide/embed.html

Admittedly, I know very little about the bokeh package, but I'm very confused.

Originally, I thought you said your pull request worked with bokeh 0.12.3 or 0.12.4, but now you're saying it needs the very latest version (0.12.7) in order to work. Which is it?

Can you explain which enhancement is required by your pull request on this list of 0.12.7 enhancements? In other words, what is missing from earlier bokeh releases that your pull request needs?

Which commit contains the changes necessary for the plot "to work on the latest [bokeh] version, 0.12.7"?
What did you have to change to make it work with 0.12.7?

And if 0.12.7 is an absolute requirement for this pull request, why does the requirements.txt file contain this additional line?

+bokeh==0.12.4

[hayleefay]

@martinholmer Originally, my PR did work with Bokeh 0.12.4/0.12.3. Because of your feedback, I updated the code to 0.12.7. I have yet to push it because I was unsure of how to update the CDN for the widgets in results.html. Hence, my question in my previous comment. (And since I have not pushed the updated code yet, the line pinning bokeh 0.12.4 is still there.)

Only small changes in handling null values were required to change from 0.12.4 to 0.12.7. However, the code does not work in 0.12.6 due to errors in the handling of Categorical data, which were fixed in the release of 0.12.7.

Can you tell me what you're hoping for as far as versioning?

[brittainhard]

@martinholmer @hayleefay the place to specify this is in https://github.com/OpenSourcePolicyCenter/webapp-public/blob/master/deploy/fab/dropq_environment.yml

This file is used for the dev environment and the deployment environment.

[brittainhard]

Need to remove this line and add specify the version in dropq_environment.yml.

[martinholmer]

OK. Thanks for the helpful info. But then what role does requirements.txt play in TaxBrain?

[brittainhard]

This file is for installing packages from pip. It's mostly Django specific packages. Ideally we could be using the one environment file, and get rid of all these requirements files. environment.yml files can also install pip packages. This would be a good bit of code cleanup, but it would require some refactoring of the deployment scripts.

In the case of Bokeh, since its Continuum run, its best to install it from conda.

[martinholmer]

@brittainhard, Thanks for the explanation.

[hayleefay]

@brittainhard Thanks for the tip, I have made that change!
@martinholmer Great! I made that change.

[hayleefay]

@hdoupe Thanks so much. I implemented the new output_detail and changed the javascript in not_ready.html and all appears to be working.

[hayleefay]

@brittainhard @martinholmer @jdebacker Is this ready to be merged into @brittainhard's PR #634? We would love to have this visualization up on the site as soon as possible. But I know that involves merging #634 and releasing a new version of B-Tax. Any time frame on those things?

[hdoupe]

Great work here @hayleefay. I just have one comment. I don't think we need to re-calculate the wait time here since we know that it will be about 15 seconds. If you want you could remove 327-333 and replace exp_num_minutes with .25.

Sorry I meant to leave this comment a couple days ago.

[hayleefay]

@hdoupe, good idea! I pushed that change.

[hdoupe]

@hayleefay Looks good, thanks!

[jdebacker]

@hayleefay I just tagged a new release of B-Tax (0.1.9) that has the changes you need. I'd have to look back to read over my notes about how to package up a new version. It may be the evening of 9/6 before I can get to this.

[martinholmer]

@jdebacker said:

I just tagged a new release of B-Tax (0.1.9) that has the changes you need. I'd have to look back to read over my notes about how to package up a new version. It may be the evening of 9/6 before I can get to this.

The package creation process in the policybrain_builder repo has changed completely, and much to the better, thanks to the work of @jbcrail. Use his CLI pb to create btax packages. Let me know if you have any pb usage questions. I might not be able to answer them, but try me first rather than asking @jbcrail straightaway.

[jdebacker]

@hayleefay I was swamped with class prep then had some issues building a package (first time I've done it), but I was able to successfully complete the process (I think) and you can find a new version of B-Tax on Anaconda.org now.

[hayleefay]

@jdebacker, no worries! Thanks for getting that done. @brittainhard, when do you plan on deploying the new version of webapp-public?

[brittainhard]

This needs to be set in deploy/fab/drop_environment.yml

[hayleefay]

This is deploy/fab/drop_environment.yml 🙂

[brittainhard]

Oops!

[brittainhard]

@hayleefay the plan is to get PRs merged and do testing today and monday, then deploy to the dev node end of day monday.

Is this ready to be reviewed?

[hayleefay]

@brittainhard, sounds great! And yes it is! Let me know if I need to make any changes.

[brittainhard]

@hayleefay this looks great. What version BTAX do we need for this?

[jdebacker]

@brittainhard 0.1.9

[brittainhard]

@hayleefay @jdebacker merging this.