Automatic article revisions

[nikhilwoodruff]

Model updates happen frequently, and sometimes this can mean changes to economic impact estimates in articles. For example, let's say that we publish an analysis of a tax cut (a blog post featuring "this would cost £3.1bn in 2023[link]", for example), a year passes, and due to revisions in external data, data updates, or bug fixes, and now that link points to a simulation page displaying £3.4bn.

Updating every article when the model slightly changes is a lot of human work. What would be ideal is if we had an automated Python routine, perhaps run every day or week, that scanned all the Markdown and Jupyter notebook blog post files for some special syntax linking to economic impacts (e.g. text in blog posts like "this would cost {policy#3551-over-#2-in-uk-over-2023.economic_impact.budgetary_impact}), made an API call to check the result, and add a footnote or some clean frontend design that updates the main text and ensures the entire history of the results (and their associated model versions) is updated and shown in the article.

Thoughts @MaxGhenis @anth-volk?

[MaxGhenis]

Yes - especially our documentation-oriented articles like How do we calculate poverty impacts? (coming soon, with baseline poverty trends). I think users will tolerate discrepancies in our analysis articles more, especially if we say Computed with PolicyEngine UK version x which will differ from the app. If we're doing an analysis we may also have more of an interest in preserving the history.

[anth-volk]

I agree with everything said so far. One thing I might caution, though - if we ever push a bad update that breaks some calculation (especially something experimental, like, say, labor impacts), I could see these updates potentially generating bad output.

I would recommend compromising somewhere in the middle by doing the following:

• Indicating the country package version we're calculating the original number with in the original post, as indicated by Max
• Keeping the automatic checking on the lower end, perhaps every week
• Having the automated check flag values where the output would be, say, more than 10% higher or lower than the prior version, so that someone can check to make sure the output value isn't buggy