Prefect Recipes 🧑‍🍳 🥐

This repository contains common and extensible Prefect patterns to drive efficient workflows — we like to call these patterns our recipes

Here you'll find starter code and more advanced example use cases.

Contributing = Swag 🧢

We're always looking for new contributions! See our existing Recipe Ideas/Issues for inspiration. And check out Contributions to learn how you can add your Prefect 2.0 recipe and earn some swag!

Issues / Bugs 🐛

To report issues, typos, or link fixes, please open an issue. We appreciate it!

Recipe Ideas 👩‍🍳

What are you interested in seeing examples of? Jot down your big idea here.

Table of Contents 📖

• Getting Started
• Subflows
• Control Flow
• Optimization
• Notifications
• Flow Run Observability
• Parameters
• Testing
• Triggering Flow Runs

• Deployments

• Diving Deeper

• Data Engineering / DataOps
• AWS Infrastructure
• Azure Infrastructure
• Helm
• GitHub Actions
• Legacy (Prefect 1.0)
• Issues & Bugs
• Recipe Requests
• Contributions
• Join the Discussion
• Thanks

Getting Started 🍯

Subflows

• Getting Started With Subflows
• Run Multiple Subflows or Child Flows in Parallel
• Subflow with a Different Task Runner Than Parent Flow
• Create a Subflow and Block Until It's Completed
• Running Subflows On Their Own Infrastructure Using a Separate Deployment

Control Flow

• Conditionally Stop a Task Run
• Ensure Tasks Immediately Fail If Upstream Task Fails
• Define State Dependencies Between Tasks

Optimization

• Cache a Task Result To Prevent Recomputation

Notifications

• Send Notifications with a Slack Webhook

Flow Run Observability

• Interact with REST API
• Determine Whether a Flow Run Was Executed Ad Hoc or on a Schedule

Parameters

• Use flow parameters

Testing

• Testing Flows, Subflows, and Tasks

Triggering Flow Runs

• Triggering Flow Runs From a Deployment via API Call or From a Terminal Using Curl
• Event-Driven Data Pipelines with AWS Lambda and GitHub Actions

Flow Deployment

• Deploy Flows to Run as a Local Process, Docker Container or a Kubernetes Job
• Deploy Flows to AWS
• Deploy Flows to GCP
• Deploy Flows to Azure
• Store Flows in GitHub and Execute in a Docker Container
• Python-Based Deployments

Diving Deeper 🍱

Data Engineering / DataOps

• Serverless Real-Time Data Pipelines on AWS with Prefect, ECS and GitHub Actions
• Build a Data Platform with Prefect, dbt, and Snowflake (using blocks)

ELT/ETL

• Export Airbyte Configuration and Load to S3 bucket using blocks, including Python-based deployment
• ELT with Snowflake Using Async and Blocks

AWS Infrastructure

• Deploy a Prefect agent to ECS using the AWS CLI
• Deploy a Prefect agent to ECS with Terraform
• Deploy Flows Using ECSTask Infrastructure Blocks
• Deploy a Prefect agent to ECS Fargate using CloudFormation and GitHub Actions

Azure Infrastructure

• Setup Azure with Prefect
• Deploy Prefect Orion to an AKS Cluster with Azure Blob Storage
• Setup an Azure VM and Run the Prefect Agent
• Deploy Flows Using Azure Container Instances Infrastructure Blocks

Helm

• Deploy Prefect Agent using Helm and Terraform

Github Actions

• Build flow image and Prefect deployment with storage and infra Blocks on push to branch
• Build and Push flow docker image to Google Artifact Registry
• Build / Apply prefect deployment with blocks on change to python files
• Automate Prefect Deployments to AWS ECS Fargate Using GitHub Actions
• Automate Python-Based Deployments with GitHub Actions

Dockerfiles

• Build an image from the latest Python 3.9 base image and your requirements.txt

Django

• Using Django with Prefect 2

Pydantic

• Using Pydantic BaseModel with Prefect 2

Prefect 1.0 Legacy

• Register a Prefect Flow
• Run GraphQL Queries
• Airbyte Orchestration
• ETL with AWS S3 and Snowflake
• Use AWS Lambda for Event-Driven Flows
• Read Secrets into Prefect Cloud tenant
• Handle DBT Model Failures
• S3 Flow Storage on EKS
• Use LocalExecutor to run Dask computations on a Coiled cluster
• Use DaskExecutor to run Prefect tasks in parallel on a Coiled cluster

Join our discussions 🗣️

We use our Slack Community and Discourse to discuss all things Prefect-- such as FAQ, use cases and integrations. Join in the conversation :smile:

Contributions

We're always looking for new contributions! You can add your Prefect 2.0 recipe and earn some swag in a few simple steps:

1. Look through the recipes to ensure your example is unique
2. Clone the prefect-recipes repo:

   git clone git@github.com:PrefectHQ/prefect-recipes.git
   

3. Create and checkout a new branch:

   git branch feat/recipe-name
   git checkout feat/recipe-name
   

4. Add your code under the appropriate category, making sure it is reproducible and easy to understand.
5. Add your recipe to README.
6. Commit and push the code to your remote branch.
7. Create a PR 🤌

Thanks 💙

Thank you for your contributions and efforts to improve prefect-recipes. We're glad to have you in our community!