Run metadata ingestion

Configure your ingestion spec

In the Install OpenMetadata section, you cloned the prefect-openmetadata repository. This repository contains a directory example-data which you can use to ingest sample data into your OpenMetadata backend using Prefect.

Here is a configuration spec you can use in your flow to ingest that sample data:

source:
  type: sample-data
  serviceName: sample_data
  serviceConnection:
    config:
      type: SampleData
      sampleDataFolder: "example-data"
  sourceConfig: {}
sink:
  type: metadata-rest
  config: {}
workflowConfig:
  openMetadataServerConfig:
    hostPort: http://localhost:8585/api
    authProvider: no-auth

Run ingestion workflow locally

Now you can paste the config from above as a string into your flow and run it:

from prefect_openmetadata.flows import ingest_metadata

config = """
source:
  type: sample-data
  serviceName: sample_data
  serviceConnection:
    config:
      type: SampleData
      sampleDataFolder: "example-data"
  sourceConfig: {}
sink:
  type: metadata-rest
  config: {}
workflowConfig:
  openMetadataServerConfig:
    hostPort: http://localhost:8585/api
    authProvider: no-auth
"""

if __name__ == "__main__":
    ingest_metadata(config)

After running your flow, you should see new users, datasets, dashboards, and other metadata in your OpenMetadata UI. Also, your Prefect UI will display the workflow run and will show the logs with details on which source system has been scanned and which data has been ingested.

Congratulations on building your first metadata ingestion workflow with OpenMetadata and Prefect! Head over to the next section to see how you can run this flow on schedule.