Skip to content

Commit 6946462

Browse files
authored
Merge branch 'main' into add-count-latest-route
2 parents 98f1543 + 97a23d9 commit 6946462

File tree

112 files changed

+5992
-175
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

112 files changed

+5992
-175
lines changed

.github/codeql-config.yml

+6
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,12 @@ query-filters:
1212
- exclude:
1313
# too many false positives with prefect.runtime
1414
id: py/undefined-export
15+
- exclude:
16+
# sometimes necessary for __init__ files
17+
id: py/import-and-import-from
18+
- exclude:
19+
# we dont need CodeQL quality linting
20+
id: py/mixed-returns
1521

1622
paths-ignore:
1723
- tests/**/test_*.py

Dockerfile

+1-1
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,7 @@ RUN apt-get update && \
2828
RUN npm install -g npm@8
2929

3030
# Install dependencies separately so they cache
31-
COPY ./ui/package*.json .
31+
COPY ./ui/package*.json ./
3232
RUN npm ci install
3333

3434
# Build static UI files

MANIFEST.in

+2
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,8 @@ include requirements-dev.txt
1414
include src/prefect/_version.py
1515
include src/prefect/py.typed
1616
include src/prefect/profiles.toml
17+
include src/prefect/projects/recipes/*/*.yaml
18+
include src/prefect/projects/templates/*.yaml
1719
include src/prefect/.prefectignore
1820
include src/prefect/logging/logging.yml
1921
include src/prefect/cli/templates/*.yaml

RELEASE-NOTES.md

+152
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,157 @@
11
# Prefect Release Notes
22

3+
## Release 2.10.2
4+
5+
Fixes a bug where deployments were not downloaded from remote storage blocks during flow runs — https://github.com/PrefectHQ/prefect/pull/9138
6+
7+
### Enhancements
8+
- Add httpx.ConnectTimeout to the list of retry exceptions in base client — https://github.com/PrefectHQ/prefect/pull/9125
9+
10+
### Contributors
11+
* @sorendaugaard made their first contribution in https://github.com/PrefectHQ/prefect/pull/9125
12+
13+
14+
**All changes**: https://github.com/PrefectHQ/prefect/compare/2.10.1...2.10.2
15+
16+
## Release 2.10.1
17+
18+
Fixes a bug with accessing project recipes through the CLI. See the [pull request](https://github.com/PrefectHQ/prefect/pull/9132) for implementation details.
19+
20+
## Release 2.10.0
21+
22+
Prefect deployments often have critical, implicit dependencies on files and build artifacts, such as containers, that are created and stored outside of Prefect. Each of these dependencies is a potential stumbling block when deploying a flow - you need to ensure that they're satisfied for your flow to run successfully. In this release, we're introducing two new beta features, workers and projects, to help you better manage your flow deployment process. Additionally, we're releasing variables for centralized management of management and expanding events and automations to include blocks. There are a lot of highlighted features this week — but we've also made some significant performance improvements alongside a slew of bug fixes and enhancements!
23+
24+
### Workers [Beta]
25+
26+
Workers are next-generation agents, designed from the ground up to interact with [work pools](https://docs.prefect.io/concepts/work-pools/). Each worker manages flow run infrastructure of a specific type and must pull from a work pool with a matching type. Existing work pools are all "agent" typed for backwards compatibility with our agents — but new work pools can be assigned a specific infrastructure type. Specifying a type for a work pool simplifies choosing what kind of infrastructure will be used when creating a flow run.
27+
28+
Work pools expose rich configuration of their infrastructure. Every work pool type has a base configuration with sensible defaults such that you can begin executing work with just a single command. The infrastructure configuration is fully customizable from the Prefect UI. For example, you can now customize the entire payload used to run flows on Kubernetes — you are not limited to the fields Prefect exposes in its SDK. We provide templating to inject runtime information and common settings into infrastructure creation payloads. Advanced users can add _custom_ template variables which are then exposed the same as Prefect's default options in an easy to use UI.
29+
30+
If the work pool’s configuration is updated, all workers automatically begin using the new settings — you no longer need to redeploy your agents to change infrastructure settings. For advanced use cases, you can override settings on a per-deployment basis.
31+
32+
This release includes Process, Kubernetes, and Docker worker types. Additional worker types will be included in subsequent releases.
33+
34+
Creating a Kubernetes work pool:
35+
36+
<img width="1601" alt="Creating a new Kubernetes work pool" src="https://user-images.githubusercontent.com/2586601/230471683-63875a04-f331-4cf1-8b1b-69c2cd0e4e05.png">
37+
<img width="1601" alt="Advanced configuration of the work pool infrastructure" src="https://user-images.githubusercontent.com/2586601/230471686-7146e930-34fc-43ae-a946-9e3795c4a27a.png">
38+
39+
Adding a new variable to the advanced work pool configuration will expose it in the basic config:
40+
41+
<img width="1551" alt="Adding a variable to the advanced config" src="https://user-images.githubusercontent.com/2586601/230475075-b535b158-62a8-4b88-9439-0054f58e8f77.png">
42+
<img width="1601" alt="New variables can be adjusted in the basic config" src="https://user-images.githubusercontent.com/2586601/230473701-b8db1973-eb03-4682-86cc-64b698356048.png">
43+
44+
See the updated [work pool, workers, & agents concepts documentation](https://docs.prefect.io/latest/concepts/work-pools/) for more information.
45+
46+
### Projects [Beta]
47+
48+
A project is a directory of files that define one or more flows, deployments, Python packages, or any other dependencies that your flow code needs to run. If you’ve been using Prefect, or working on any non-trivial Python project, you probably have an organized structure like this already. Prefect projects are minimally opinionated, so they can work with the structure you already have in place and with the containerization, version control, and build automation tools that you know and love. With projects as directories, you can make relative references between files while retaining portability. We expect most projects to map directly to a git repository. In fact, projects offer a first-class way to clone a git repository so they can be easily shared and synced.
49+
50+
Projects also include a lightweight build system that you can use to define the process for deploying flows in that project. That procedure is specified in a new `prefect.yaml` file, in which you can specify steps to build the necessary artifacts for a project's deployments, push those artifacts, and retrieve them at runtime.
51+
52+
Projects are a contract between you and a worker, specifying what you do when you create a deployment, and what the worker will do before it kicks off that deployment. Together, projects and workers bridge your development environment, where your flow code is written, and your execution environment, where your flow code runs. Create your first Prefect project by following [this tutorial](https://docs.prefect.io/latest/tutorials/projects/).
53+
54+
See the new [project concept doc](https://docs.prefect.io/latest/concepts/projects/) for more information or the following pull requests for implementation details:
55+
56+
- https://github.com/PrefectHQ/prefect/pull/8930
57+
- https://github.com/PrefectHQ/prefect/pull/9103
58+
- https://github.com/PrefectHQ/prefect/pull/9105
59+
- https://github.com/PrefectHQ/prefect/pull/9112
60+
- https://github.com/PrefectHQ/prefect/pull/9093
61+
- https://github.com/PrefectHQ/prefect/pull/9083
62+
- https://github.com/PrefectHQ/prefect/pull/9041
63+
64+
### Variables
65+
66+
Variables enable you to store and reuse non-sensitive bits of data, such as configuration information. Variables are named, mutable string values, much like environment variables. They are scoped to a Prefect Server instance or a single workspace in Prefect Cloud. Variables can be created or modified at any time. While variable values are most commonly loaded during flow runtime, they can be loaded in other contexts, at any time, such that they can be used to pass configuration information to Prefect configuration files, such as project steps. You can access any variable via the Python SDK via the `.get()` method.
67+
68+
```python
69+
from prefect import variables
70+
71+
# from a synchronous context
72+
answer = variables.get('the_answer')
73+
print(answer)
74+
# 42
75+
76+
# from an asynchronous context
77+
answer = await variables.get('the_answer')
78+
print(answer)
79+
# 42
80+
81+
# without a default value
82+
answer = variables.get('not_the_answer')
83+
print(answer)
84+
# None
85+
86+
# with a default value
87+
answer = variables.get('not_the_answer', default='42')
88+
print(answer)
89+
# 42
90+
```
91+
92+
See the new [variables concept doc](https://docs.prefect.io/latest/concepts/variables/) for more information or the [pull request](https://github.com/PrefectHQ/prefect/pull/9088) for implementation details.
93+
94+
### Events
95+
96+
Continuing the rollout of events[https://docs.prefect.io/concepts/events-and-resources/] as the primary unit of observability in Prefect Cloud, Prefect will now emit events for all block method calls by default. These events can be viewed in the Event feed, allowing you to analyze the interactions your flows and tasks have with external systems such as storage locations, notification services, and infrastructure. Additionally, you can trigger automations based on these events. For example, you can create an automation that is triggered when a file is uploaded to a storage location.
97+
98+
![image](https://user-images.githubusercontent.com/26799928/230421783-997e4fda-a02f-4bf4-88e1-f51a2f890cf5.png)
99+
100+
### Versioned documentation
101+
102+
We're releasing a lot of new features every week and we know not everyone is on the latest version of Prefect. We've added versioning to our documentation website to make it easier to find the docs for the version of Prefect that you're using.
103+
104+
Now, when you visit the Prefect documentation site, you'll see a version selector at the top of the page.
105+
106+
![versioned docs](https://user-images.githubusercontent.com/228762/230432235-26fc9406-1390-4c63-9956-b8cdabdfba6f.png)
107+
108+
109+
### Breaking Changes
110+
- Unused options for sorting logs have been removed from the API — https://github.com/PrefectHQ/prefect/pull/7873
111+
112+
### Enhancements
113+
- Add artifacts view to flow run page — https://github.com/PrefectHQ/prefect/pull/9109
114+
- Improve performance of the background event worker — https://github.com/PrefectHQ/prefect/pull/9019
115+
- Update deployment flow run creation to default to a SCHEDULED state instead of PENDING — https://github.com/PrefectHQ/prefect/pull/9049
116+
- Add `PREFECT_CLIENT_RETRY_EXTRA_CODES` to allow retry on additional HTTP status codes — https://github.com/PrefectHQ/prefect/pull/9056
117+
- Improve performance of the background log worker — https://github.com/PrefectHQ/prefect/pull/9048
118+
- Update agent cancellation check interval to double the scheduled check interval — https://github.com/PrefectHQ/prefect/pull/9084
119+
- Update default agent query interval from 10s to 15s — https://github.com/PrefectHQ/prefect/pull/9085
120+
- Add a 10 minute cache to API healthchecks — https://github.com/PrefectHQ/prefect/pull/9069
121+
- Improve performance of concurrent task runner — https://github.com/PrefectHQ/prefect/pull/9073
122+
- Improve performance of waiting for task submission — https://github.com/PrefectHQ/prefect/pull/9072
123+
- Add retry on 502 BAD GATEWAY to client — https://github.com/PrefectHQ/prefect/pull/9102
124+
- Update local and remote file systems to return path on write — https://github.com/PrefectHQ/prefect/pull/8965
125+
- Add artifacts `/count` route — https://github.com/PrefectHQ/prefect/pull/9022
126+
- Improve performance of automatic block registration — https://github.com/PrefectHQ/prefect/pull/8838
127+
- Improve performance of log retrieval queries — https://github.com/PrefectHQ/prefect/pull/9035
128+
- Improve performance of artifact retrieval — https://github.com/PrefectHQ/prefect/pull/9061 / https://github.com/PrefectHQ/prefect/pull/9064
129+
- Add `--type` option to create work-pool CLI — https://github.com/PrefectHQ/prefect/pull/8993
130+
- Improve flow run timeline performance - https://github.com/PrefectHQ/prefect-ui-library/pull/1315
131+
- Add flow names to sub flows on the flow run timeline graph - https://github.com/PrefectHQ/prefect-ui-library/pull/1304
132+
133+
### Fixes
134+
- Fix bug where iterable defaults were treated as mapped parameters — https://github.com/PrefectHQ/prefect/pull/9021
135+
- Fix sequential execution with mapped tasks using the SequentialTaskRunner — https://github.com/PrefectHQ/prefect/pull/8473
136+
- Fix race condition where futures did not wait for submission to complete — https://github.com/PrefectHQ/prefect/pull/9070
137+
- Fix detection of iterables within `quote` annotations while mapping — https://github.com/PrefectHQ/prefect/pull/9095
138+
- Fix Dockerfile copy of UI package files on latest Docker version — https://github.com/PrefectHQ/prefect/pull/9077
139+
140+
### Documentation
141+
- Add copy to clipboard button in documentation code blocks — https://github.com/PrefectHQ/prefect/pull/9026
142+
- Fixed styling of deployments mermaid diagram — https://github.com/PrefectHQ/prefect/pull/9017
143+
- Add documentation for database migrations — https://github.com/PrefectHQ/prefect/pull/9044
144+
- Adds documentation for BitBucket to flow code storage types — https://github.com/PrefectHQ/prefect/pull/9080
145+
- Update rate limit documentation for Cloud — https://github.com/PrefectHQ/prefect/pull/9100
146+
147+
### Contributors
148+
- @mianos made their first contribution in https://github.com/PrefectHQ/prefect/pull/9077
149+
- @dominictarro made their first contribution in https://github.com/PrefectHQ/prefect/pull/8965
150+
- @joelluijmes
151+
- @john-jam
152+
153+
**All changes**: https://github.com/PrefectHQ/prefect/compare/2.9.0...2.10.0
154+
3155
## Release 2.9.0
4156

5157
### Track and manage artifacts

docs/api-ref/prefect/artifacts.md

+8
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
---
2+
description: Prefect Python API for creating artifacts.
3+
tags:
4+
- Python API
5+
- artifacts
6+
---
7+
8+
::: prefect.artifacts

docs/api-ref/prefect/cli/project.md

+11
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
---
2+
description: Prefect command line interface for working with Prefect projects.
3+
tags:
4+
- Python API
5+
- CLI
6+
- projects
7+
- deployments
8+
- storage
9+
---
10+
11+
::: prefect.cli.project

0 commit comments

Comments
 (0)