Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Mock runtime via environment variable #9156

Merged
merged 4 commits into from
Apr 10, 2023
Merged

Mock runtime via environment variable #9156

merged 4 commits into from
Apr 10, 2023

Conversation

cicdw
Copy link
Member

@cicdw cicdw commented Apr 7, 2023

This PR introduces a mechanism for mocking / manually setting runtime attributes via environment variable. This was requested by a user looking to test their code and provide explicit known values for the attributes.

Example

To do this, a user only needs to set an appropriately prefixed environment variable, e.g., PREFECT__RUNTIME__TASK_RUN__NEW_KEY="foobar" and now prefect.runtime.task_run.new_key evaluates to "foobar".

Checklist

  • This pull request references any related issue by including "closes <link to issue>"
    • If no issue exists and your change is not a small fix, please create an issue first.
  • This pull request includes tests or only affects documentation.
  • This pull request includes a label categorizing the change e.g. fix, feature, enhancement, docs.

@cicdw cicdw added the enhancement An improvement of an existing feature label Apr 7, 2023
@cicdw cicdw requested review from a team as code owners April 7, 2023 20:33
@netlify
Copy link

netlify bot commented Apr 7, 2023 •
edited
Loading

✅ Deploy Preview for prefect-docs-preview ready!

Name Link
🔨 Latest commit a4b0bf8
🔍 Latest deploy log https://app.netlify.com/sites/prefect-docs-preview/deploys/64342eb1613d080008d1b69c
😎 Deploy Preview https://deploy-preview-9156--prefect-docs-preview.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site settings.

billpalombi
billpalombi approved these changes Apr 10, 2023
View reviewed changes
Copy link
Contributor

@billpalombi billpalombi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is cool. Suggesting minor copyedits to the tip.

@cicdw @billpalombi
Update docs/concepts/runtime-context.md
a4b0bf8 
Co-authored-by: Bill Palombi <bill@prefect.io>
@zanieb
Copy link
Contributor

zanieb commented Apr 10, 2023

Just wondering, does monkeypatch.setattr not work? That seems more straight-forward than setting an environment variable.

@cicdw
Copy link
Member Author

cicdw commented Apr 10, 2023

@madkinsz it probably does, but requires users to use pytest and be familiar with monkeypatch (both reasonable I admit, but this isn't hard to support either and allows us to document it so that users can discover it easier).

@cicdw cicdw merged commit e41c837 into main Apr 10, 2023
@cicdw cicdw deleted the mock-runtime branch April 10, 2023 19:57
@jcozar87
Copy link
Contributor

I've been using this feature for a while and it works like charm! However I discovered a problem with datetime variables: existing datetime context variables like flow_run.scheduled_start time are of type pendulum.datetime.DateTime. However, if I set a value for PREFECT__RUNTIME__FLOW_RUN__SCHEDULED_START_TIME environment variable it must be a string, and it is not parsed as a pendulum.datetime.DateTime internally. Maybe for existing content variables it is required to cast environment variables to the expected type?

@zanieb
Copy link
Contributor

zanieb commented Apr 25, 2023

@jcozar87 that seems reasonable; feel free to open a PR!

asmundo pushed a commit to asmundo/prefect that referenced this pull request May 11, 2023
@cicdw @billpalombi
Mock runtime via environment variable (PrefectHQ#9156)
a56c7a1 
Co-authored-by: Bill Palombi <bill@prefect.io>
@jcozar87
Copy link
Contributor

Thank you for the feedback, I will do it!

jcozar87 added a commit to jcozar87/prefect that referenced this pull request May 14, 2023
@jcozar87
feat: cast environment variable for known mocked runtime attributes
9cc466f 
The PR PrefectHQ#9156 introduces a mechanism for mocking / manually setting runtime attributes via environment variable.
In commit asmundo@a56c7a1 this functionality is extended to deployment and task_run runtime attributes.

The environment variables substitute the real attribute values in case that environment variable names matches with the name of some flow_run, task_run or deployment attribute.
However, if the type of the attribute is not str, that would lead to a type insconsistency (i.e. flow_run.scheduled_start_time is of type pendulum.DateTime, but environment
variable PREFECT__RUNTIME__FLOW_RUN__SCHEDULED_START_TIME is of type str). Now cast environment variable types to its real attribute type if exists. The functionality has been
extended to flow_run, task_run and deployments, however only flow_run.scheduled_start_time is of a type different than str. Code is ready to adapt for any of these types: bool, int,
float, str and pendulum.DateTime. It would be easy to extend to more attribute types.
@jcozar87 jcozar87 mentioned this pull request May 14, 2023
Merged
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@zanieb zanieb zanieb approved these changes

@billpalombi billpalombi billpalombi approved these changes

Assignees
No one assigned
Labels
enhancement An improvement of an existing feature
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@cicdw @zanieb @jcozar87 @billpalombi