Blocks Catalog

Below is a list of Blocks available for registration in prefect-gcp.

To register blocks in this module to view and edit them on Prefect Cloud, first install the required packages, then

prefect block register -m prefect_gcp

Note, to use the load method on Blocks, you must already have a block document saved through code or saved through the UI.

Credentials Module

GcpCredentials

Block used to manage authentication with GCP. Google authentication is handled via the google.oauth2 module or through the CLI. Specify either one of service account_file or service_account_info; if both are not specified, the client will try to detect the credentials following Google's Application Default Credentials. See Google's Authentication documentation for details on inference and recommended authentication patterns.

To load the GcpCredentials:

from prefect import flow
from prefect_gcp.credentials import GcpCredentials

@flow
def my_flow():
    my_block = GcpCredentials.load("MY_BLOCK_NAME")

my_flow()

For additional examples, check out the Credentials Module under Examples Catalog.

Aiplatform Module

VertexAICustomTrainingJob

Infrastructure block used to run Vertex AI custom training jobs.

To load the VertexAICustomTrainingJob:

from prefect import flow
from prefect_gcp.aiplatform import VertexAICustomTrainingJob

@flow
def my_flow():
    my_block = VertexAICustomTrainingJob.load("MY_BLOCK_NAME")

my_flow()

For additional examples, check out the Aiplatform Module under Examples Catalog.

Bigquery Module

BigQueryWarehouse

A block for querying a database with BigQuery.

Upon instantiating, a connection to BigQuery is established and maintained for the life of the object until the close method is called.

It is recommended to use this block as a context manager, which will automatically close the connection and its cursors when the context is exited.

It is also recommended that this block is loaded and consumed within a single task or flow because if the block is passed across separate tasks and flows, the state of the block's connection and cursor could be lost.

To load the BigQueryWarehouse:

from prefect import flow
from prefect_gcp.bigquery import BigQueryWarehouse

@flow
def my_flow():
    my_block = BigQueryWarehouse.load("MY_BLOCK_NAME")

my_flow()

For additional examples, check out the Bigquery Module under Examples Catalog.

Cloud Run Module

CloudRunJob

Infrastructure block used to run GCP Cloud Run Jobs. Note this block is experimental. The interface may change without notice.

To load the CloudRunJob:

from prefect import flow
from prefect_gcp.cloud_run import CloudRunJob

@flow
def my_flow():
    my_block = CloudRunJob.load("MY_BLOCK_NAME")

my_flow()

For additional examples, check out the Cloud Run Module under Examples Catalog.

Cloud Storage Module

GcsBucket

Block used to store data using GCP Cloud Storage Buckets.

Note! GcsBucket in prefect-gcp is a unique block, separate from GCS in core Prefect. GcsBucket does not use gcsfs under the hood, instead using the google-cloud-storage package, and offers more configuration and functionality.

To load the GcsBucket:

from prefect import flow
from prefect_gcp.cloud_storage import GcsBucket

@flow
def my_flow():
    my_block = GcsBucket.load("MY_BLOCK_NAME")

my_flow()

For additional examples, check out the Cloud Storage Module under Examples Catalog.

Secret Manager Module

GcpSecret

Manages a secret in Google Cloud Platform's Secret Manager.

To load the GcpSecret:

from prefect import flow
from prefect_gcp.secret_manager import GcpSecret

@flow
def my_flow():
    my_block = GcpSecret.load("MY_BLOCK_NAME")

my_flow()

For additional examples, check out the Secret Manager Module under Examples Catalog.