Skip to content

The easiest way to orchestrate and observe your data pipelines

License

Notifications You must be signed in to change notification settings

Sam-ScottLogic/prefect

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

PyPI

Prefect

Prefect is an orchestrator for data-intensive workflows. It's the simplest way to transform any Python function into a unit of work that can be observed and orchestrated. With Prefect, you can build resilient, dynamic workflows that react to the world around them and recover from unexpected changes. With just a few decorators, Prefect supercharges your code with features like automatic retries, distributed execution, scheduling, caching, and much more. Every activity is tracked and can be monitored with the Prefect server or Prefect Cloud dashboard.

from prefect import flow, task
from typing import List
import httpx


@task(retries=3)
def get_stars(repo: str):
    url = f"https://api.github.com/repos/{repo}"
    count = httpx.get(url).json()["stargazers_count"]
    print(f"{repo} has {count} stars!")


@flow(name="GitHub Stars")
def github_stars(repos: List[str]):
    for repo in repos:
        get_stars(repo)


# run the flow!
github_stars(["PrefectHQ/Prefect"])

After running some flows, fire up the Prefect UI to see what happened:

prefect server start

From here, you can continue to use Prefect interactively or deploy your flows to remote envirnments, running on a scheduled or event-driven basis.

Getting Started

Prefect requires Python 3.7 or later. To install Prefect, run the following command in a shell or terminal session:

pip install prefect

Start by then exploring the core concepts of Prefect workflows, then follow one of our friendly tutorials to learn by example.

Join the community

Prefect is made possible by the fastest growing community of thousands of friendly data engineers. Join us in building a new kind of workflow system. The Prefect Slack community is a fantastic place to learn more abou Prefect, ask questions, or get help with workflow design. The Prefect Discourse is an community-driven knowledge base to find answers to your Prefect-related questions. All community forums, including code contributions, issue discussions, and slack messages are subject to our Code of Conduct.

Contribute

See our documentation on contributing to Prefect.

Thanks for being part of the mission to build a new kind of workflow system and, of course, happy engineering!

About

The easiest way to orchestrate and observe your data pipelines

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Python 97.7%
  • Vue 1.3%
  • TypeScript 0.7%
  • Jinja 0.1%
  • Shell 0.1%
  • Dockerfile 0.1%