Add Affiliate Window pulls to finance reporting task #735

bmedx · 2019-06-14T17:29:23Z

Analytics Pipeline Pull Request

Make sure that the following steps are done before merging:

If you have a migration please contact data engineering team before merging.
Before merging run full acceptance tests suite and provide URL for the acceptance tests run.
A member of data engineering team has approved the pull request.

codecov · 2019-06-14T18:09:39Z

Codecov Report

Merging #735 into master will decrease coverage by 0.12%.
The diff coverage is 51.2%.

@@            Coverage Diff             @@
##           master     #735      +/-   ##
==========================================
- Coverage   75.08%   74.95%   -0.13%     
==========================================
  Files         204      206       +2     
  Lines       22985    23110     +125     
==========================================
+ Hits        17258    17322      +64     
- Misses       5727     5788      +61

Impacted Files	Coverage Δ
...ytics/tasks/warehouse/financial/finance_reports.py	`91.66% <100%> (+0.75%)`	⬆️
...tics/tasks/warehouse/financial/affiliate_window.py	`45.54% <45.54%> (ø)`
edx/analytics/tasks/warehouse/financial/fees.py	`73.91% <73.91%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c535eef...4fed889. Read the comment docs.

edx/analytics/tasks/warehouse/financial/affiliate_window.py

codecov-io · 2019-06-17T15:51:18Z

Codecov Report

Merging #735 into master will decrease coverage by 0.13%.
The diff coverage is 49.19%.

@@            Coverage Diff             @@
##           master     #735      +/-   ##
==========================================
- Coverage   74.76%   74.62%   -0.14%     
==========================================
  Files         205      207       +2     
  Lines       23206    23330     +124     
==========================================
+ Hits        17349    17410      +61     
- Misses       5857     5920      +63

Impacted Files	Coverage Δ
...tics/tasks/warehouse/financial/affiliate_window.py	`44% <44%> (ø)`
edx/analytics/tasks/warehouse/financial/fees.py	`70.83% <70.83%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 769cb2e...2ba96e5. Read the comment docs.

pwnage101 · 2019-06-18T20:17:29Z

edx/analytics/tasks/warehouse/financial/fees.py

+
+class LoadFeesToWarehouse(WarehouseMixin, VerticaCopyTask):
+    """
+    Loads Affiliate Window table from Hive into the Vertica data warehouse.


Nit: the class name suggests a general purpose, but the docstring treats Affiliate Window as the main purpose. Is the class correctly named?

Yeah, the comment is just bad. I envisioned this being one place to load all of those random fees we're currently getting from pasted spreadsheets, as we become able to operationalise them. I'll fix the comment.

brianhw

Only nits -- looks great! 👍

brianhw · 2019-06-18T20:22:58Z

edx/analytics/tasks/warehouse/financial/finance_reports.py

@@ -6,6 +6,7 @@
 from edx.analytics.tasks.warehouse.financial.ed_services_financial_report import (
    LoadInternalReportingEdServicesReportToWarehouse
 )
+from edx.analytics.tasks.warehouse.financial.fees import LoadFeesToWarehouse


nit: This is unused.

brianhw · 2019-06-18T20:28:09Z

edx/analytics/tasks/warehouse/financial/affiliate_window.py

+            # If this is the first run, start from the beginning
+            print("Couldn't find last completed date, defaulting to start date: {}".format(self.interval_start))
+
+        self.run_interval = date_interval.Custom(self.interval_start, self.interval_end)


I'm assuming that this is a closed interval, since interval_end is defined as today's date minus one day. Maybe just note that in the description of interval_end, because most of our intervals defined elsewhere are open on the end. That is, we define the interval as ending with today's date, so that it processes events (mostly events) up to midnight as of today.

I went back and forth on this a bit as I was trying to figure out the Affiliate Window approval process, I think I can adjust all of this to just be open on the end so I'll make that change wherever necessary in this PR.

brianhw · 2019-06-18T20:37:25Z

edx/analytics/tasks/warehouse/financial/affiliate_window.py

+    """
+    run_date = luigi.DateParameter(
+        default=datetime.date.today(),
+        description='Default is today.',


nit: should default be yesterday?

The other open end changes should make this ok.

brianhw · 2019-06-18T20:38:50Z

edx/analytics/tasks/warehouse/financial/affiliate_window.py

+
+    Pulls are made for only a single day.
+    """
+    run_date = luigi.DateParameter(


nit: it's not actually the run_date, that is, the date the pull was run. It's the date being pulled. Would "pull_date" or "query_date" be better?

brianhw · 2019-06-18T20:39:55Z

edx/analytics/tasks/warehouse/financial/affiliate_window.py

+        print("Fetching report from Affiliate Window for {}.".format(self.run_date))
+        transactions = self.fetch_report()
+
+        # if there are no transactions in response something is wrong.


Just curious -- is it possible that there are days with nothing to pull? Actually, if there were no transactions, would this actually be a json structure that is empty, but for which "if not transactions" would somehow be false? Or would response.json() really be empty?

There were no days with empty transactions going back to 2019-01-01 so I'm assuming that doesn't happen. The if not should catch any case where there is valid JSON that is an empty string, or empty list. If it's not valid JSON it will error getting it, and if it's something other than a list of dicts it should fail downstream.

If there really are no transactions the API returns an empty list, so that should mean we passed in an invalid date, or our advertiser id changed or some other weird situation like that.

brianhw · 2019-06-18T20:50:06Z

edx/analytics/tasks/warehouse/financial/fees.py

+
+    # Forcing this to overwrite now, as this script will duplicate data
+    # without it. This does not trickle down to the Affiliate Window calls.
+    # TODO: Make this optional once VerticaCopyTask handles multiple input files.


nit: I understand that overwrite=True here means that the Vertica table is being overwritten each time this runs. I don't understand what it means for VerticaCopyTask to handle multiple input files. (And maybe I don't have to. : )

It's entirely possible that I have a misapprehension about how it works, I've made a note to talk to you about this when I get back and either clear it up or make a ticket to make things work the way I think they should.

Add Affiliate Window imports to data warehouse (#735)

bmedx force-pushed the bmedx/add_affiliate_window branch from ebae712 to 52783b3 Compare June 14, 2019 17:59

pwnage101 reviewed Jun 14, 2019

View reviewed changes

edx/analytics/tasks/warehouse/financial/affiliate_window.py Outdated Show resolved Hide resolved

bmedx force-pushed the bmedx/add_affiliate_window branch 3 times, most recently from 4cb6ff4 to 45ad489 Compare June 18, 2019 18:24

Add Affiliate Window pulls to finance reporting task

384eb84

bmedx force-pushed the bmedx/add_affiliate_window branch from 45ad489 to 384eb84 Compare June 18, 2019 18:34

pwnage101 reviewed Jun 18, 2019

View reviewed changes

brianhw approved these changes Jun 18, 2019

View reviewed changes

bmedx added 3 commits June 18, 2019 18:52

fixup! Add Affiliate Window pulls to finance reporting task

17a5ea8

fixup! Add Affiliate Window pulls to finance reporting task

784ca40

fixup! Add Affiliate Window pulls to finance reporting task

2ba96e5

bmedx merged commit 91d5aee into master Jun 19, 2019

bmedx deleted the bmedx/add_affiliate_window branch June 19, 2019 00:54

doctoryes pushed a commit that referenced this pull request Jun 19, 2019

Merge pull request #739 from edx/master

529c1e8

Add Affiliate Window imports to data warehouse (#735)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Affiliate Window pulls to finance reporting task #735

Add Affiliate Window pulls to finance reporting task #735

bmedx commented Jun 14, 2019

codecov bot commented Jun 14, 2019 •

edited

Loading

codecov-io commented Jun 17, 2019 •

edited by codecov bot

Loading

pwnage101 Jun 18, 2019

bmedx Jun 18, 2019

brianhw left a comment

brianhw Jun 18, 2019

brianhw Jun 18, 2019

bmedx Jun 18, 2019

brianhw Jun 18, 2019

bmedx Jun 18, 2019

brianhw Jun 18, 2019

brianhw Jun 18, 2019

bmedx Jun 18, 2019

brianhw Jun 18, 2019

bmedx Jun 18, 2019

Add Affiliate Window pulls to finance reporting task #735

Add Affiliate Window pulls to finance reporting task #735

Conversation

bmedx commented Jun 14, 2019

Analytics Pipeline Pull Request

codecov bot commented Jun 14, 2019 • edited Loading

Codecov Report

codecov-io commented Jun 17, 2019 • edited by codecov bot Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

brianhw left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Jun 14, 2019 •

edited

Loading

codecov-io commented Jun 17, 2019 •

edited by codecov bot

Loading