Add ability to analyse Regression Kink Analysis Designs #264

drbenvincent · 2023-10-20T16:05:21Z

This is my attempt to implement analysis methods for the Regression Kink Design. Happy to get any feedback on the work so far

First draft of notebook/docs
Implement RegressionKink class
- First attempt
- Add docstring for the RegressionKink class
- Add tests
- Refactor. At the moment this whole class is copied/pasted from the RegressionDiscontinuity class, so there is a lot of duplication. The main substantive difference is that we evaluate the function so as to evaluate the change in slopes either side of the kink. Will create an issue about this because it gets into wider refactoring issues.
- Fix summary method
Add entry to Glossary + add reference back to glossary

Closes #223

NOTE: The example notebook does not try to act like an intro/explainer into regression kink designs. It assumes the reader has some background knowledge about what it is and when you'd use it. Happy to get input on whether it should have more introductory info, but there's a Discussion #262 on whether we should add a specific explainer section to the docs which would allow the notebooks to remain concise.

review-notebook-app · 2023-10-20T16:05:26Z

Check out this pull request on

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

codecov · 2023-10-20T16:20:21Z

Codecov Report

Merging #264 (4386f6b) into main (7ebab10) will increase coverage by 1.50%.
The diff coverage is 85.98%.

@@            Coverage Diff             @@
##             main     #264      +/-   ##
==========================================
+ Coverage   74.41%   75.91%   +1.50%     
==========================================
  Files          20       20              
  Lines        1157     1308     +151     
==========================================
+ Hits          861      993     +132     
- Misses        296      315      +19

Files	Coverage Δ
causalpy/tests/test_input_validation.py	`100.00% <100.00%> (ø)`
causalpy/tests/test_integration_pymc_examples.py	`100.00% <100.00%> (ø)`
causalpy/tests/test_pymc_experiments.py	`100.00% <100.00%> (ø)`
causalpy/pymc_experiments.py	`69.11% <73.80%> (+1.32%)`	⬆️

review-notebook-app · 2023-11-05T10:32:48Z

View / edit / reply to this conversation on ReviewNB

NathanielF commented on 2023-11-05T10:32:48Z
----------------------------------------------------------------

This is nice. Good demonstration of the idea.

review-notebook-app · 2023-11-05T10:32:49Z

View / edit / reply to this conversation on ReviewNB

NathanielF commented on 2023-11-05T10:32:49Z
----------------------------------------------------------------

Add a comment above the code where you specify the betas to indicate that this is where 2 is set as the gradient at kink point.

drbenvincent commented on 2023-11-06T10:25:37Z
----------------------------------------------------------------

Done

review-notebook-app · 2023-11-05T10:32:50Z

View / edit / reply to this conversation on ReviewNB

NathanielF commented on 2023-11-05T10:32:49Z
----------------------------------------------------------------

I like this! Didn't realise patsy was so powerful.

review-notebook-app · 2023-11-05T10:32:50Z

View / edit / reply to this conversation on ReviewNB

NathanielF commented on 2023-11-05T10:32:50Z
----------------------------------------------------------------

Nice!

review-notebook-app · 2023-11-05T10:32:51Z

View / edit / reply to this conversation on ReviewNB

NathanielF commented on 2023-11-05T10:32:51Z
----------------------------------------------------------------

Same as above add comment to the code block specifying where you fix the gradient to be recovered

drbenvincent commented on 2023-11-06T10:52:41Z
----------------------------------------------------------------

As far as I can tell, when we use a quadratic, none of the individual parameters corresponds to the change in gradient at the kink point, so I've relied on calculating that numerically. But I have added a comment in about how the betas translate into different equations on the left and right of the kink point.

review-notebook-app · 2023-11-05T10:32:52Z

View / edit / reply to this conversation on ReviewNB

NathanielF commented on 2023-11-05T10:32:51Z
----------------------------------------------------------------

Is the forest plot redundant if you're doing the plor posterior too?

drbenvincent commented on 2023-11-06T10:55:55Z
----------------------------------------------------------------

Yep, I've removed the forest plot

review-notebook-app · 2023-11-05T10:32:52Z

View / edit / reply to this conversation on ReviewNB

NathanielF commented on 2023-11-05T10:32:52Z
----------------------------------------------------------------

Maybe be a bit more explicit that this is not a new data set but that you're just showing basis splines applied to the last example and intending to recover the same parameters again.

drbenvincent commented on 2023-11-06T10:59:04Z
----------------------------------------------------------------

done

review-notebook-app · 2023-11-05T10:32:53Z

View / edit / reply to this conversation on ReviewNB

NathanielF commented on 2023-11-05T10:32:53Z
----------------------------------------------------------------

Can you round the -1.14 to 1.1 to show the same value is being recoverd in the posterior plot and your more customised timeseries one...?

Just a nit.

drbenvincent commented on 2023-11-06T11:07:33Z
----------------------------------------------------------------

Sure. What I've actually done is to show 2 decimal places in the Arviz plot. At the moment, rounding to 1 decimal place in the plot method would require doing that for everything. So I'm just about to add an issue to add a round_to kwarg to the plot methods in causalpy.

NathanielF · 2023-11-05T10:49:13Z

Notebook and experiment design look great. Added a few comments. Mostly nits.

NathanielF

Approved. Feel free to change/address the comments above as you like. Mostly they were small points of clarification. Major work looks great. Cool addition.

NathanielF · 2023-11-05T10:34:51Z

causalpy/pymc_experiments.py

+            {self.running_variable_name: xi, "treated": self._is_treated(xi)}
+        )
+        # self.x_pred = pd.DataFrame({self.running_variable_name: xi})
+        (new_x,) = build_design_matrices([self._x_design_info], self.x_pred)


I'm sure there. is a reason but why is the output wrapped in braces?and then immediately into an array?

build_design_matrices returns both the x and y design matrices as a tuple, so we are just grabbing the x matrix here. And by default these are as dataframes. So you can either provide the return_type='matrix' argument to build_design_matrices, or manually do it yourself with np.asarray as I did here.

NathanielF · 2023-11-05T10:41:37Z

causalpy/tests/test_integration_pymc_examples.py

+    y = reg_kink_function(x, beta, kink) + rng.normal(0, sigma, N)
+    df = pd.DataFrame({"x": x, "y": y, "treated": x >= kink})
+    # run experiment
+    result = cp.pymc_experiments.RegressionKink(


Kind of neat that the kink model calls the linear model!

NathanielF · 2023-11-05T10:45:24Z

causalpy/tests/test_integration_pymc_examples.py

+
+
+@pytest.mark.integration
+def test_rkink_bandwidth():


I don't think you have an example in the notebook with the bandwidth parameter? Maybe worth adding/explaining

NathanielF · 2023-11-05T10:47:55Z

causalpy/pymc_experiments.py

+        A PyMC model
+    :param running_variable_name:
+        The name of the predictor variable that the kink_point is based upon
+    :param epsilon:


I know you suggested not having the notebook as a place for explaining the theory of kink designs, but i feel like the differences between tweaking at least one of epsilon/bandwidth parameters could be mentioned or shown.

It's fine if tweaking them isn't needed for your example, but it'd be good to hint at why they are there.

I've added an example to demonstrate use of the bandwidth parameter. And I've added an admonition box to explain what epsilon does.

drbenvincent · 2023-11-06T10:25:38Z

Done

View entire conversation on ReviewNB

drbenvincent · 2023-11-06T10:52:42Z

As far as I can tell, when we use a quadratic, none of the individual parameters corresponds to the change in gradient at the kink point, so I've relied on calculating that numerically. But I have added a comment in about how the betas translate into different equations on the left and right of the kink point.

View entire conversation on ReviewNB

drbenvincent · 2023-11-06T10:55:57Z

Yep, I've removed the forest plot

View entire conversation on ReviewNB

drbenvincent · 2023-11-06T10:59:05Z

done

View entire conversation on ReviewNB

drbenvincent · 2023-11-06T11:07:34Z

Sure. What I've actually done is to show 2 decimal places in the Arviz plot. At the moment, rounding to 1 decimal place in the plot method would require doing that for everything. So I'm just about to add an issue to add a round_to kwarg to the plot methods in causalpy.

View entire conversation on ReviewNB

… about epsilon

drbenvincent · 2023-11-06T11:49:09Z

Thanks for the feedback @NathanielF. I think I've dealt with all the suggestions. I've also added the regression kink designs to the README.md and index.rst (for readthedocs) which I forgot to do before.

juanitorduz

some minor comments on the core component code style to make the computations more readable (and testable)

causalpy/pymc_experiments.py

drbenvincent · 2023-11-07T14:34:02Z

Thanks for the review @juanitorduz. I've made a stab at modularising the change in gradient calculation code and added some tests. Though I'm not sure if you it hits what you were thinking of.

juanitorduz

Looks much nicer! thanks for the refactor. Letf a single comment :)

causalpy/tests/test_input_validation.py

juanitorduz

Nice addition! Thank you @drbenvincent 🙌

drbenvincent added 3 commits October 20, 2023 13:17

first stab at regression kink design

15ef4ec

first draft of kink design notebook done

81ea342

add notebook to readthedocs

e8905f3

drbenvincent added 9 commits November 2, 2023 10:47

update regression kink design notebook

6b7e0e9

add entry into the glossary

b4436c9

update glossary entry

64c0b99

add link to glossary term in notebook

1aaa028

fix typo in glossary

7847ede

add docstring to RegressionKink class

8ce77a1

add integration tests for regression kink design

9cf6b51

update UML diagram

6e47576

update summary method to make this work + re-run example notebook

5e099c2

drbenvincent changed the title ~~[WIP] Add Regression Kink Analysis~~ Add Regression Kink Analysis Nov 2, 2023

drbenvincent changed the title ~~Add Regression Kink Analysis~~ Add ability to analyse Regression Kink Analysis Designs Nov 2, 2023

drbenvincent added the enhancement New feature or request label Nov 2, 2023

drbenvincent requested review from NathanielF, juanitorduz and AlexAndorra November 2, 2023 11:51

NathanielF approved these changes Nov 5, 2023

View reviewed changes

add comment to explain betas

cc07b94

drbenvincent added 2 commits November 6, 2023 10:53

add comment about beta parameters for the quadratic example

dca2844

remove redundant figure and re-run notebook

e3c37be

clarification about using same dataset in the basis function example

2f89a97

show 2 decimal places in final az.plot_posterior + re-run notebook

3ca5f87

drbenvincent added 3 commits November 6, 2023 11:34

add another example to demonstrate the bandwidth parameter + add note…

59a8b3e

… about epsilon

change title of example notebook

0b70c9e

add regression kink design to README.md and index.rst

732707f

juanitorduz requested changes Nov 6, 2023

View reviewed changes

drbenvincent added 8 commits November 7, 2023 10:17

remove some commented out lines + remove parens

7e982c0

simplify the summary method

1ba0333

add input validation for bandwidth and epsilon parameters

dab61dc

test exceptions are raised for the new input validation

2ad1ba3

make gradient change code more modular

80537c6

Merge branch 'main' into kink

b26a375

update UML diagrams

255ea83

add tests for cp.pymc_experiments.RegressionKink._eval_gradient_change

232944b

juanitorduz requested changes Nov 7, 2023

View reviewed changes

causalpy/tests/test_input_validation.py Outdated Show resolved Hide resolved

remove kwarg being overwritten

4386f6b

juanitorduz approved these changes Nov 8, 2023

View reviewed changes

drbenvincent merged commit 24ad46e into main Nov 9, 2023
10 checks passed

drbenvincent deleted the kink branch November 9, 2023 10:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add ability to analyse Regression Kink Analysis Designs #264

Add ability to analyse Regression Kink Analysis Designs #264

drbenvincent commented Oct 20, 2023 •

edited

Loading

review-notebook-app bot commented Oct 20, 2023

codecov bot commented Oct 20, 2023 •

edited

Loading

review-notebook-app bot commented Nov 5, 2023 •

edited

Loading

review-notebook-app bot commented Nov 5, 2023 •

edited

Loading

review-notebook-app bot commented Nov 5, 2023 •

edited

Loading

review-notebook-app bot commented Nov 5, 2023 •

edited

Loading

review-notebook-app bot commented Nov 5, 2023 •

edited

Loading

review-notebook-app bot commented Nov 5, 2023 •

edited

Loading

review-notebook-app bot commented Nov 5, 2023 •

edited

Loading

review-notebook-app bot commented Nov 5, 2023 •

edited

Loading

NathanielF commented Nov 5, 2023

NathanielF left a comment

NathanielF Nov 5, 2023

drbenvincent Nov 6, 2023

NathanielF Nov 5, 2023

NathanielF Nov 5, 2023

drbenvincent Nov 6, 2023

NathanielF Nov 5, 2023

drbenvincent Nov 6, 2023

drbenvincent commented Nov 6, 2023

drbenvincent commented Nov 6, 2023

drbenvincent commented Nov 6, 2023

drbenvincent commented Nov 6, 2023

drbenvincent commented Nov 6, 2023

drbenvincent commented Nov 6, 2023

juanitorduz left a comment

drbenvincent commented Nov 7, 2023

juanitorduz left a comment

juanitorduz left a comment



		@pytest.mark.integration
		def test_rkink_bandwidth():

Add ability to analyse Regression Kink Analysis Designs #264

Add ability to analyse Regression Kink Analysis Designs #264

Conversation

drbenvincent commented Oct 20, 2023 • edited Loading

review-notebook-app bot commented Oct 20, 2023

codecov bot commented Oct 20, 2023 • edited Loading

Codecov Report

review-notebook-app bot commented Nov 5, 2023 • edited Loading

review-notebook-app bot commented Nov 5, 2023 • edited Loading

review-notebook-app bot commented Nov 5, 2023 • edited Loading

review-notebook-app bot commented Nov 5, 2023 • edited Loading

review-notebook-app bot commented Nov 5, 2023 • edited Loading

review-notebook-app bot commented Nov 5, 2023 • edited Loading

review-notebook-app bot commented Nov 5, 2023 • edited Loading

review-notebook-app bot commented Nov 5, 2023 • edited Loading

NathanielF commented Nov 5, 2023

NathanielF left a comment

Choose a reason for hiding this comment

NathanielF Nov 5, 2023

Choose a reason for hiding this comment

drbenvincent Nov 6, 2023

Choose a reason for hiding this comment

NathanielF Nov 5, 2023

Choose a reason for hiding this comment

NathanielF Nov 5, 2023

Choose a reason for hiding this comment

drbenvincent Nov 6, 2023

Choose a reason for hiding this comment

NathanielF Nov 5, 2023

Choose a reason for hiding this comment

drbenvincent Nov 6, 2023

Choose a reason for hiding this comment

drbenvincent commented Nov 6, 2023

drbenvincent commented Nov 6, 2023

drbenvincent commented Nov 6, 2023

drbenvincent commented Nov 6, 2023

drbenvincent commented Nov 6, 2023

drbenvincent commented Nov 6, 2023

juanitorduz left a comment

Choose a reason for hiding this comment

drbenvincent commented Nov 7, 2023

juanitorduz left a comment

Choose a reason for hiding this comment

juanitorduz left a comment

Choose a reason for hiding this comment

drbenvincent commented Oct 20, 2023 •

edited

Loading

codecov bot commented Oct 20, 2023 •

edited

Loading

review-notebook-app bot commented Nov 5, 2023 •

edited

Loading

review-notebook-app bot commented Nov 5, 2023 •

edited

Loading

review-notebook-app bot commented Nov 5, 2023 •

edited

Loading

review-notebook-app bot commented Nov 5, 2023 •

edited

Loading

review-notebook-app bot commented Nov 5, 2023 •

edited

Loading

review-notebook-app bot commented Nov 5, 2023 •

edited

Loading

review-notebook-app bot commented Nov 5, 2023 •

edited

Loading

review-notebook-app bot commented Nov 5, 2023 •

edited

Loading