Diving Deeper 🍱

Data Engineering / DataOps

• Serverless Real-Time Data Pipelines on AWS with Prefect, ECS and GitHub Actions

ELT/ETL

• Export Airbyte Configuration and Load to S3 bucket using blocks, including Python-based deployment
• ELT with Snowflake Using Async and Blocks

AWS Infrastructure

• Deploy a Prefect agent to ECS using the AWS CLI
• Deploy a Prefect agent to ECS with Terraform
• Deploy Flows Using ECSTask Infrastructure Blocks

Azure Infrastructure

• Setup Azure with Prefect
• Deploy Prefect Orion to an AKS Cluster with Azure Blob Storage
• Setup an Azure VM and Run the Prefect Agent

Github Actions

• Build flow image and Prefect deployment with storage and infra Blocks on push to branch
• Build and Push flow docker image to Google Artifact Registry
• Build / Apply prefect deployment with blocks on change to python files
• Automate Prefect Deployments to AWS ECS Fargate Using GitHub Actions

Dockerfiles

• Build an image from the latest Python 3.9 base image and your requirements.txt

Django

• Using Django with Prefect 2

Prefect 1.0 Legacy

• Register a Prefect Flow
• Run GraphQL Queries
• Airbyte Orchestration
• ETL with AWS S3 and Snowflake
• Use AWS Lambda for Event-Driven Flows
• Read Secrets into Prefect Cloud tenant
• Handle DBT Model Failures
• S3 Flow Storage on EKS
• Use LocalExecutor to run Dask computations on a Coiled cluster
• Use DaskExecutor to run Prefect tasks in parallel on a Coiled cluster