Skip to content

Schedule and deploy metadata ingestion flows

Schedule your OpenMetadata ingestion flows with Prefect

Ingesting your data via manually executed scripts is great for initial exploration, but in order to build a reliable metadata platform, you need to run those workflows on a regular cadence. That’s where you can leverage Prefect schedules and deployments.

Here is how you can create a Deployment to ensure that your metadata ingestion flow ingest_metadata in the script myflow.py gets refreshed every 15 minutes:

prefect deployment build -a -n dev myflow.py:ingest_metadata --interval 900

Here is an explanation of the prefect deployment build flags:

  • -a - will automatically register the resulting deployment with the API
  • -n - specifies the name of the deployment - you could use it to differentiate between a deployment for development and production environment
  • myflow.py:ingest_metadata - entrypoint to the flow object, i.e. the Python script and the flow function name
  • --interval 900 - schedule interval in seconds, here: triggering a new flow run every 15 minutes. With the asynchronous scheduling service in Prefect 2.0, you can schedule your flow to run even every 10 seconds if you need your metadata to be updated near real-time.

Deploy your execution layer to run your flows

So far, we’ve looked at how you can create and schedule your workflow, but where does this code actually run? This is a place where the concepts of storage and infrastructure blocks, as well as work queues and agents become important. But don’t worry - all you need to know to get started is running a single CLI command:

prefect agent start -q default

Once you have executed that command, your scheduled deployments (such as the one we defined using myflow.py above) are now scheduled, and Prefect will ensure that your metadata stays up-to-date.

You can observe the state of your metadata ingestion workflows from the Prefect Orion UI. The UI will also include detailed logs showing which metadata got updated to ensure your data platform remains healthy and observable.

Using Prefect in the Cloud

If you want to move beyond this local installation, you can deploy Prefect to run your OpenMetadata ingestion workflows by:

For various deployment options of OpenMetadata, check the Deployment documentation.

Questions about using OpenMetadata with Prefect

If you have any questions about configuring Prefect, post your question on Prefect Discourse or in the Prefect Community Slack.

And if you need support for OpenMetadata, get in touch on OpenMetadata Slack.