Skip to content

prefect_docker.deployments.steps

Prefect deployment steps for building and pushing Docker images.

These steps can be used in a prefect.yaml file to define the default build steps for a group of deployments, or they can be used to define the build step for a specific deployment.

Example

Build a Docker image before deploying a flow: 

build:
    - prefect_docker.deployments.steps.build_docker_image:
        id: build-image
        requires: prefect-docker
        image_name: repo-name/image-name
        tag: dev

push:
    - prefect_docker.deployments.steps.push_docker_image:
        requires: prefect-docker
        image_name: "{{ build-image.image_name }}"
        tag: "{{ build-image.tag }}"

Classes

BuildDockerImageResult

Bases: TypedDict

The result of a build_docker_image step.

Attributes:

Name Type Description
image_name str

The name of the built image.
tag str

The tag of the built image.
image str

The name and tag of the built image.
image_id str

The ID of the built image.
additional_tags Optional[str]

The additional tags on the image, in addition to tag.
Source code in prefect_docker/deployments/steps.py 
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
class BuildDockerImageResult(TypedDict):
    """
    The result of a `build_docker_image` step.

    Attributes:
        image_name: The name of the built image.
        tag: The tag of the built image.
        image: The name and tag of the built image.
        image_id: The ID of the built image.
        additional_tags: The additional tags on the image, in addition to `tag`.
    """

    image_name: str
    tag: str
    image: str
    image_id: str
    additional_tags: Optional[str]

PushDockerImageResult

Bases: TypedDict

The result of a push_docker_image step.

Attributes:

Name Type Description
image_name str

The name of the pushed image.
tag str

The tag of the pushed image.
image str

The name and tag of the pushed image.
additional_tags Optional[str]

The additional tags on the image, in addition to tag.
Source code in prefect_docker/deployments/steps.py 
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
class PushDockerImageResult(TypedDict):
    """
    The result of a `push_docker_image` step.

    Attributes:
        image_name: The name of the pushed image.
        tag: The tag of the pushed image.
        image: The name and tag of the pushed image.
        additional_tags: The additional tags on the image, in addition to `tag`.
    """

    image_name: str
    tag: str
    image: str
    additional_tags: Optional[str]

Functions

build_docker_image

Builds a Docker image for a Prefect deployment.

Can be used within a prefect.yaml file to build a Docker image prior to creating or updating a deployment.

Parameters:

Name Type Description Default
image_name str

The name of the Docker image to build, including the registry and repository.

 required
dockerfile str

The path to the Dockerfile used to build the image. If "auto" is passed, a temporary Dockerfile will be created to build the image.

 'Dockerfile'
tag Optional[str]

The tag to apply to the built image.

 None
push bool
False
credentials Optional[Dict]

A dictionary containing the username, password, and URL for the registry to push the image to.

 None
additional_tags Optional[List[str]]

Additional tags on the image, in addition to tag, to apply to the built image.

 None
**build_kwargs

Additional keyword arguments to pass to Docker when building the image. Available options can be found in the docker-py documentation.

 {}

Returns:

Type Description
BuildDockerImageResult

A dictionary containing the image name and tag of the built image.
Example

Build and push a Docker image prior to creating a deployment: 

build:
    - prefect_docker.deployments.steps.build_docker_image:
        requires: prefect-docker
        image_name: repo-name/image-name
        tag: dev

Build and push a Docker image with multiple tags: 

build:
    - prefect_docker.deployments.steps.build_docker_image:
        requires: prefect-docker
        image_name: repo-name/image-name
        tag: dev
        additional_tags:
            - v0.1.0,
            - dac9ccccedaa55a17916eef14f95cc7bdd3c8199

Build a Docker image using an auto-generated Dockerfile: 

build:
    - prefect_docker.deployments.steps.build_docker_image:
        requires: prefect-docker
        image_name: repo-name/image-name
        tag: dev
        dockerfile: auto
        push: false

Build a Docker image for a different platform: 

build:
    - prefect_docker.deployments.steps.build_docker_image:
        requires: prefect-docker
        image_name: repo-name/image-name
        tag: dev
        dockerfile: Dockerfile
        push: false
        platform: amd64

Source code in prefect_docker/deployments/steps.py 
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
@deprecated_parameter(
    "push",
    when=lambda y: y is True,
    start_date="Jun 2023",
    help="Use the `push_docker_image` step instead.",
)
def build_docker_image(
    image_name: str,
    dockerfile: str = "Dockerfile",
    tag: Optional[str] = None,
    push: bool = False,
    credentials: Optional[Dict] = None,
    additional_tags: Optional[List[str]] = None,
    **build_kwargs,
) -> BuildDockerImageResult:
    """
    Builds a Docker image for a Prefect deployment.

    Can be used within a `prefect.yaml` file to build a Docker
    image prior to creating or updating a deployment.

    Args:
        image_name: The name of the Docker image to build, including the registry and
            repository.
        dockerfile: The path to the Dockerfile used to build the image. If "auto" is
            passed, a temporary Dockerfile will be created to build the image.
        tag: The tag to apply to the built image.
        push: DEPRECATED: Whether to push the built image to the registry.
        credentials: A dictionary containing the username, password, and URL for the
            registry to push the image to.
        additional_tags: Additional tags on the image, in addition to `tag`, to apply to the built image.
        **build_kwargs: Additional keyword arguments to pass to Docker when building
            the image. Available options can be found in the [`docker-py`](https://docker-py.readthedocs.io/en/stable/images.html#docker.models.images.ImageCollection.build)
            documentation.

    Returns:
        A dictionary containing the image name and tag of the
            built image.

    Example:
        Build and push a Docker image prior to creating a deployment:
        ```yaml
        build:
            - prefect_docker.deployments.steps.build_docker_image:
                requires: prefect-docker
                image_name: repo-name/image-name
                tag: dev
        ```

        Build and push a Docker image with multiple tags:
        ```yaml
        build:
            - prefect_docker.deployments.steps.build_docker_image:
                requires: prefect-docker
                image_name: repo-name/image-name
                tag: dev
                additional_tags:
                    - v0.1.0,
                    - dac9ccccedaa55a17916eef14f95cc7bdd3c8199
        ```

        Build a Docker image using an auto-generated Dockerfile:
        ```yaml
        build:
            - prefect_docker.deployments.steps.build_docker_image:
                requires: prefect-docker
                image_name: repo-name/image-name
                tag: dev
                dockerfile: auto
                push: false
        ```


        Build a Docker image for a different platform:
        ```yaml
        build:
            - prefect_docker.deployments.steps.build_docker_image:
                requires: prefect-docker
                image_name: repo-name/image-name
                tag: dev
                dockerfile: Dockerfile
                push: false
                platform: amd64
        ```
    """  # noqa
    auto_build = dockerfile == "auto"
    if auto_build:
        lines = []
        base_image = get_prefect_image_name()
        lines.append(f"FROM {base_image}")
        dir_name = os.path.basename(os.getcwd())

        if Path("requirements.txt").exists():
            lines.append(
                f"COPY requirements.txt /opt/prefect/{dir_name}/requirements.txt"
            )
            lines.append(
                f"RUN python -m pip install -r /opt/prefect/{dir_name}/requirements.txt"
            )

        lines.append(f"COPY . /opt/prefect/{dir_name}/")
        lines.append(f"WORKDIR /opt/prefect/{dir_name}/")

        temp_dockerfile = Path("Dockerfile")
        if Path(temp_dockerfile).exists():
            raise ValueError("Dockerfile already exists.")

        with Path(temp_dockerfile).open("w") as f:
            f.writelines(line + "\n" for line in lines)

        dockerfile = str(temp_dockerfile)

    build_kwargs["path"] = build_kwargs.get("path", os.getcwd())
    build_kwargs["dockerfile"] = dockerfile
    build_kwargs["pull"] = build_kwargs.get("pull", True)
    build_kwargs["decode"] = True
    build_kwargs["labels"] = {**build_kwargs.get("labels", {}), **IMAGE_LABELS}
    image_id = None

    with docker_client() as client:
        try:
            events = client.api.build(**build_kwargs)

            try:
                for event in events:
                    if "stream" in event:
                        sys.stdout.write(event["stream"])
                        sys.stdout.flush()
                    elif "aux" in event:
                        image_id = event["aux"]["ID"]
                    elif "error" in event:
                        raise BuildError(event["error"])
                    elif "message" in event:
                        raise BuildError(event["message"])
            except docker.errors.APIError as e:
                raise BuildError(e.explanation) from e

        finally:
            if auto_build:
                os.unlink(dockerfile)

        if not isinstance(image_id, str):
            raise BuildError("Docker did not return an image ID for built image.")

        if not tag:
            tag = slugify(pendulum.now("utc").isoformat())

        image: Image = client.images.get(image_id)
        image.tag(repository=image_name, tag=tag)

        additional_tags = additional_tags or []
        for tag_ in additional_tags:
            image.tag(repository=image_name, tag=tag_)

        if push:
            if credentials is not None:
                client.login(
                    username=credentials.get("username"),
                    password=credentials.get("password"),
                    registry=credentials.get("registry_url"),
                    reauth=credentials.get("reauth", True),
                )
            events = client.api.push(
                repository=image_name, tag=tag, stream=True, decode=True
            )
            try:
                for event in events:
                    if "status" in event:
                        sys.stdout.write(event["status"])
                        if "progress" in event:
                            sys.stdout.write(" " + event["progress"])
                        sys.stdout.write("\n")
                        sys.stdout.flush()
                    elif "error" in event:
                        raise OSError(event["error"])
            finally:
                client.api.remove_image(image=f"{image_name}:{tag}", noprune=True)

    return {
        "image_name": image_name,
        "tag": tag,
        "image": f"{image_name}:{tag}",
        "image_id": image_id,
        "additional_tags": additional_tags,
    }

push_docker_image

Push a Docker image to a remote registry.

Parameters:

Name Type Description Default
image_name str

The name of the Docker image to push, including the registry and repository.

 required
tag Optional[str]

The tag of the Docker image to push.

 None
credentials Optional[Dict]

A dictionary containing the username, password, and URL for the registry to push the image to.

 None
additional_tags Optional[List[str]]

Additional tags on the image, in addition to tag, to apply to the built image.

 None

Returns:

Type Description
PushDockerImageResult

A dictionary containing the image name and tag of the pushed image.

Examples:

Build and push a Docker image to a private repository: 

build:
    - prefect_docker.deployments.steps.build_docker_image:
        id: build-image
        requires: prefect-docker
        image_name: repo-name/image-name
        tag: dev
        dockerfile: auto

push:
    - prefect_docker.deployments.steps.push_docker_image:
        requires: prefect-docker
        image_name: "{{ build-image.image_name }}"
        tag: "{{ build-image.tag }}"
        credentials: "{{ prefect.blocks.docker-registry-credentials.dev-registry }}"

Build and push a Docker image to a private repository with multiple tags 

build:
    - prefect_docker.deployments.steps.build_docker_image:
        id: build-image
        requires: prefect-docker
        image_name: repo-name/image-name
        tag: dev
        dockerfile: auto
        additional_tags: [
            v0.1.0,
            dac9ccccedaa55a17916eef14f95cc7bdd3c8199
        ]

push:
    - prefect_docker.deployments.steps.push_docker_image:
        requires: prefect-docker
        image_name: "{{ build-image.image_name }}"
        tag: "{{ build-image.tag }}"
        credentials: "{{ prefect.blocks.docker-registry-credentials.dev-registry }}"
        additional_tags: "{{ build-image.additional_tags }}"

Source code in prefect_docker/deployments/steps.py 
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
def push_docker_image(
    image_name: str,
    tag: Optional[str] = None,
    credentials: Optional[Dict] = None,
    additional_tags: Optional[List[str]] = None,
) -> PushDockerImageResult:
    """
    Push a Docker image to a remote registry.

    Args:
        image_name: The name of the Docker image to push, including the registry and
            repository.
        tag: The tag of the Docker image to push.
        credentials: A dictionary containing the username, password, and URL for the
            registry to push the image to.
        additional_tags: Additional tags on the image, in addition to `tag`, to apply to the built image.

    Returns:
        A dictionary containing the image name and tag of the
            pushed image.

    Examples:
        Build and push a Docker image to a private repository:
        ```yaml
        build:
            - prefect_docker.deployments.steps.build_docker_image:
                id: build-image
                requires: prefect-docker
                image_name: repo-name/image-name
                tag: dev
                dockerfile: auto

        push:
            - prefect_docker.deployments.steps.push_docker_image:
                requires: prefect-docker
                image_name: "{{ build-image.image_name }}"
                tag: "{{ build-image.tag }}"
                credentials: "{{ prefect.blocks.docker-registry-credentials.dev-registry }}"
        ```

        Build and push a Docker image to a private repository with multiple tags
        ```yaml
        build:
            - prefect_docker.deployments.steps.build_docker_image:
                id: build-image
                requires: prefect-docker
                image_name: repo-name/image-name
                tag: dev
                dockerfile: auto
                additional_tags: [
                    v0.1.0,
                    dac9ccccedaa55a17916eef14f95cc7bdd3c8199
                ]

        push:
            - prefect_docker.deployments.steps.push_docker_image:
                requires: prefect-docker
                image_name: "{{ build-image.image_name }}"
                tag: "{{ build-image.tag }}"
                credentials: "{{ prefect.blocks.docker-registry-credentials.dev-registry }}"
                additional_tags: "{{ build-image.additional_tags }}"
        ```
    """  # noqa
    with docker_client() as client:
        if credentials is not None:
            client.login(
                username=credentials.get("username"),
                password=credentials.get("password"),
                registry=credentials.get("registry_url"),
                reauth=credentials.get("reauth", True),
            )
        events = list(
            client.api.push(repository=image_name, tag=tag, stream=True, decode=True)
        )
        additional_tags = additional_tags or []
        for i, tag_ in enumerate(additional_tags):
            event = list(
                client.api.push(
                    repository=image_name, tag=tag_, stream=True, decode=True
                )
            )
            events = events + event

        for event in events:
            if "status" in event:
                sys.stdout.write(event["status"])
                if "progress" in event:
                    sys.stdout.write(" " + event["progress"])
                sys.stdout.write("\n")
                sys.stdout.flush()
            elif "error" in event:
                raise OSError(event["error"])

    return {
        "image_name": image_name,
        "tag": tag,
        "image": f"{image_name}:{tag}",
        "additional_tags": additional_tags,
    }