|
1 | 1 | # Prefect Release Notes |
2 | 2 |
|
| 3 | +## Release 2.10.2 |
| 4 | + |
| 5 | +Fixes a bug where deployments were not downloaded from remote storage blocks during flow runs — https://github.com/PrefectHQ/prefect/pull/9138 |
| 6 | + |
| 7 | +### Enhancements |
| 8 | +- Add httpx.ConnectTimeout to the list of retry exceptions in base client — https://github.com/PrefectHQ/prefect/pull/9125 |
| 9 | + |
| 10 | +### Contributors |
| 11 | +* @sorendaugaard made their first contribution in https://github.com/PrefectHQ/prefect/pull/9125 |
| 12 | + |
| 13 | + |
| 14 | +**All changes**: https://github.com/PrefectHQ/prefect/compare/2.10.1...2.10.2 |
| 15 | + |
| 16 | +## Release 2.10.1 |
| 17 | + |
| 18 | +Fixes a bug with accessing project recipes through the CLI. See the [pull request](https://github.com/PrefectHQ/prefect/pull/9132) for implementation details. |
| 19 | + |
| 20 | +## Release 2.10.0 |
| 21 | + |
| 22 | +Prefect deployments often have critical, implicit dependencies on files and build artifacts, such as containers, that are created and stored outside of Prefect. Each of these dependencies is a potential stumbling block when deploying a flow - you need to ensure that they're satisfied for your flow to run successfully. In this release, we're introducing two new beta features, workers and projects, to help you better manage your flow deployment process. Additionally, we're releasing variables for centralized management of management and expanding events and automations to include blocks. There are a lot of highlighted features this week — but we've also made some significant performance improvements alongside a slew of bug fixes and enhancements! |
| 23 | + |
| 24 | +### Workers [Beta] |
| 25 | + |
| 26 | +Workers are next-generation agents, designed from the ground up to interact with [work pools](https://docs.prefect.io/concepts/work-pools/). Each worker manages flow run infrastructure of a specific type and must pull from a work pool with a matching type. Existing work pools are all "agent" typed for backwards compatibility with our agents — but new work pools can be assigned a specific infrastructure type. Specifying a type for a work pool simplifies choosing what kind of infrastructure will be used when creating a flow run. |
| 27 | + |
| 28 | +Work pools expose rich configuration of their infrastructure. Every work pool type has a base configuration with sensible defaults such that you can begin executing work with just a single command. The infrastructure configuration is fully customizable from the Prefect UI. For example, you can now customize the entire payload used to run flows on Kubernetes — you are not limited to the fields Prefect exposes in its SDK. We provide templating to inject runtime information and common settings into infrastructure creation payloads. Advanced users can add _custom_ template variables which are then exposed the same as Prefect's default options in an easy to use UI. |
| 29 | + |
| 30 | +If the work pool’s configuration is updated, all workers automatically begin using the new settings — you no longer need to redeploy your agents to change infrastructure settings. For advanced use cases, you can override settings on a per-deployment basis. |
| 31 | + |
| 32 | +This release includes Process, Kubernetes, and Docker worker types. Additional worker types will be included in subsequent releases. |
| 33 | + |
| 34 | +Creating a Kubernetes work pool: |
| 35 | + |
| 36 | +<img width="1601" alt="Creating a new Kubernetes work pool" src="https://user-images.githubusercontent.com/2586601/230471683-63875a04-f331-4cf1-8b1b-69c2cd0e4e05.png"> |
| 37 | +<img width="1601" alt="Advanced configuration of the work pool infrastructure" src="https://user-images.githubusercontent.com/2586601/230471686-7146e930-34fc-43ae-a946-9e3795c4a27a.png"> |
| 38 | + |
| 39 | +Adding a new variable to the advanced work pool configuration will expose it in the basic config: |
| 40 | + |
| 41 | +<img width="1551" alt="Adding a variable to the advanced config" src="https://user-images.githubusercontent.com/2586601/230475075-b535b158-62a8-4b88-9439-0054f58e8f77.png"> |
| 42 | +<img width="1601" alt="New variables can be adjusted in the basic config" src="https://user-images.githubusercontent.com/2586601/230473701-b8db1973-eb03-4682-86cc-64b698356048.png"> |
| 43 | + |
| 44 | +See the updated [work pool, workers, & agents concepts documentation](https://docs.prefect.io/latest/concepts/work-pools/) for more information. |
| 45 | + |
| 46 | +### Projects [Beta] |
| 47 | + |
| 48 | +A project is a directory of files that define one or more flows, deployments, Python packages, or any other dependencies that your flow code needs to run. If you’ve been using Prefect, or working on any non-trivial Python project, you probably have an organized structure like this already. Prefect projects are minimally opinionated, so they can work with the structure you already have in place and with the containerization, version control, and build automation tools that you know and love. With projects as directories, you can make relative references between files while retaining portability. We expect most projects to map directly to a git repository. In fact, projects offer a first-class way to clone a git repository so they can be easily shared and synced. |
| 49 | + |
| 50 | +Projects also include a lightweight build system that you can use to define the process for deploying flows in that project. That procedure is specified in a new `prefect.yaml` file, in which you can specify steps to build the necessary artifacts for a project's deployments, push those artifacts, and retrieve them at runtime. |
| 51 | + |
| 52 | +Projects are a contract between you and a worker, specifying what you do when you create a deployment, and what the worker will do before it kicks off that deployment. Together, projects and workers bridge your development environment, where your flow code is written, and your execution environment, where your flow code runs. Create your first Prefect project by following [this tutorial](https://docs.prefect.io/latest/tutorials/projects/). |
| 53 | + |
| 54 | +See the new [project concept doc](https://docs.prefect.io/latest/concepts/projects/) for more information or the following pull requests for implementation details: |
| 55 | + |
| 56 | +- https://github.com/PrefectHQ/prefect/pull/8930 |
| 57 | +- https://github.com/PrefectHQ/prefect/pull/9103 |
| 58 | +- https://github.com/PrefectHQ/prefect/pull/9105 |
| 59 | +- https://github.com/PrefectHQ/prefect/pull/9112 |
| 60 | +- https://github.com/PrefectHQ/prefect/pull/9093 |
| 61 | +- https://github.com/PrefectHQ/prefect/pull/9083 |
| 62 | +- https://github.com/PrefectHQ/prefect/pull/9041 |
| 63 | + |
| 64 | +### Variables |
| 65 | + |
| 66 | +Variables enable you to store and reuse non-sensitive bits of data, such as configuration information. Variables are named, mutable string values, much like environment variables. They are scoped to a Prefect Server instance or a single workspace in Prefect Cloud. Variables can be created or modified at any time. While variable values are most commonly loaded during flow runtime, they can be loaded in other contexts, at any time, such that they can be used to pass configuration information to Prefect configuration files, such as project steps. You can access any variable via the Python SDK via the `.get()` method. |
| 67 | + |
| 68 | +```python |
| 69 | +from prefect import variables |
| 70 | + |
| 71 | +# from a synchronous context |
| 72 | +answer = variables.get('the_answer') |
| 73 | +print(answer) |
| 74 | +# 42 |
| 75 | + |
| 76 | +# from an asynchronous context |
| 77 | +answer = await variables.get('the_answer') |
| 78 | +print(answer) |
| 79 | +# 42 |
| 80 | + |
| 81 | +# without a default value |
| 82 | +answer = variables.get('not_the_answer') |
| 83 | +print(answer) |
| 84 | +# None |
| 85 | + |
| 86 | +# with a default value |
| 87 | +answer = variables.get('not_the_answer', default='42') |
| 88 | +print(answer) |
| 89 | +# 42 |
| 90 | +``` |
| 91 | + |
| 92 | +See the new [variables concept doc](https://docs.prefect.io/latest/concepts/variables/) for more information or the [pull request](https://github.com/PrefectHQ/prefect/pull/9088) for implementation details. |
| 93 | + |
| 94 | +### Events |
| 95 | + |
| 96 | +Continuing the rollout of events[https://docs.prefect.io/concepts/events-and-resources/] as the primary unit of observability in Prefect Cloud, Prefect will now emit events for all block method calls by default. These events can be viewed in the Event feed, allowing you to analyze the interactions your flows and tasks have with external systems such as storage locations, notification services, and infrastructure. Additionally, you can trigger automations based on these events. For example, you can create an automation that is triggered when a file is uploaded to a storage location. |
| 97 | + |
| 98 | + |
| 99 | + |
| 100 | +### Versioned documentation |
| 101 | + |
| 102 | +We're releasing a lot of new features every week and we know not everyone is on the latest version of Prefect. We've added versioning to our documentation website to make it easier to find the docs for the version of Prefect that you're using. |
| 103 | + |
| 104 | +Now, when you visit the Prefect documentation site, you'll see a version selector at the top of the page. |
| 105 | + |
| 106 | + |
| 107 | + |
| 108 | + |
| 109 | +### Breaking Changes |
| 110 | +- Unused options for sorting logs have been removed from the API — https://github.com/PrefectHQ/prefect/pull/7873 |
| 111 | + |
| 112 | +### Enhancements |
| 113 | +- Add artifacts view to flow run page — https://github.com/PrefectHQ/prefect/pull/9109 |
| 114 | +- Improve performance of the background event worker — https://github.com/PrefectHQ/prefect/pull/9019 |
| 115 | +- Update deployment flow run creation to default to a SCHEDULED state instead of PENDING — https://github.com/PrefectHQ/prefect/pull/9049 |
| 116 | +- Add `PREFECT_CLIENT_RETRY_EXTRA_CODES` to allow retry on additional HTTP status codes — https://github.com/PrefectHQ/prefect/pull/9056 |
| 117 | +- Improve performance of the background log worker — https://github.com/PrefectHQ/prefect/pull/9048 |
| 118 | +- Update agent cancellation check interval to double the scheduled check interval — https://github.com/PrefectHQ/prefect/pull/9084 |
| 119 | +- Update default agent query interval from 10s to 15s — https://github.com/PrefectHQ/prefect/pull/9085 |
| 120 | +- Add a 10 minute cache to API healthchecks — https://github.com/PrefectHQ/prefect/pull/9069 |
| 121 | +- Improve performance of concurrent task runner — https://github.com/PrefectHQ/prefect/pull/9073 |
| 122 | +- Improve performance of waiting for task submission — https://github.com/PrefectHQ/prefect/pull/9072 |
| 123 | +- Add retry on 502 BAD GATEWAY to client — https://github.com/PrefectHQ/prefect/pull/9102 |
| 124 | +- Update local and remote file systems to return path on write — https://github.com/PrefectHQ/prefect/pull/8965 |
| 125 | +- Add artifacts `/count` route — https://github.com/PrefectHQ/prefect/pull/9022 |
| 126 | +- Improve performance of automatic block registration — https://github.com/PrefectHQ/prefect/pull/8838 |
| 127 | +- Improve performance of log retrieval queries — https://github.com/PrefectHQ/prefect/pull/9035 |
| 128 | +- Improve performance of artifact retrieval — https://github.com/PrefectHQ/prefect/pull/9061 / https://github.com/PrefectHQ/prefect/pull/9064 |
| 129 | +- Add `--type` option to create work-pool CLI — https://github.com/PrefectHQ/prefect/pull/8993 |
| 130 | +- Improve flow run timeline performance - https://github.com/PrefectHQ/prefect-ui-library/pull/1315 |
| 131 | +- Add flow names to sub flows on the flow run timeline graph - https://github.com/PrefectHQ/prefect-ui-library/pull/1304 |
| 132 | + |
| 133 | +### Fixes |
| 134 | +- Fix bug where iterable defaults were treated as mapped parameters — https://github.com/PrefectHQ/prefect/pull/9021 |
| 135 | +- Fix sequential execution with mapped tasks using the SequentialTaskRunner — https://github.com/PrefectHQ/prefect/pull/8473 |
| 136 | +- Fix race condition where futures did not wait for submission to complete — https://github.com/PrefectHQ/prefect/pull/9070 |
| 137 | +- Fix detection of iterables within `quote` annotations while mapping — https://github.com/PrefectHQ/prefect/pull/9095 |
| 138 | +- Fix Dockerfile copy of UI package files on latest Docker version — https://github.com/PrefectHQ/prefect/pull/9077 |
| 139 | + |
| 140 | +### Documentation |
| 141 | +- Add copy to clipboard button in documentation code blocks — https://github.com/PrefectHQ/prefect/pull/9026 |
| 142 | +- Fixed styling of deployments mermaid diagram — https://github.com/PrefectHQ/prefect/pull/9017 |
| 143 | +- Add documentation for database migrations — https://github.com/PrefectHQ/prefect/pull/9044 |
| 144 | +- Adds documentation for BitBucket to flow code storage types — https://github.com/PrefectHQ/prefect/pull/9080 |
| 145 | +- Update rate limit documentation for Cloud — https://github.com/PrefectHQ/prefect/pull/9100 |
| 146 | + |
| 147 | +### Contributors |
| 148 | +- @mianos made their first contribution in https://github.com/PrefectHQ/prefect/pull/9077 |
| 149 | +- @dominictarro made their first contribution in https://github.com/PrefectHQ/prefect/pull/8965 |
| 150 | +- @joelluijmes |
| 151 | +- @john-jam |
| 152 | + |
| 153 | +**All changes**: https://github.com/PrefectHQ/prefect/compare/2.9.0...2.10.0 |
| 154 | + |
3 | 155 | ## Release 2.9.0 |
4 | 156 |
|
5 | 157 | ### Track and manage artifacts |
|
0 commit comments