​Docker

Run custom Python, R, Julia, Node.js and Shell scripts in isolated containers.

Tasks running in Docker containers

Many tasks in Kestra will run in dedicated Docker containers, including among others:

• Script tasks running Python, Node.js, R, Julia, Bash, and more
• Singer tasks running data ingestion syncs
• dbt tasks running dbt jobs

Kestra uses Docker for those tasks to ensure that they run in a consistent environment and to avoid dependency conflicts.

Defining a Docker task runner

To run a task in a Docker container, set the taskRunner property with the type io.kestra.plugin.scripts.runner.docker.Docker:

taskRunner:
  type: io.kestra.plugin.scripts.runner.docker.Docker

Many tasks, including Python, use the io.kestra.plugin.scripts.runner.docker.Docker task runner by default.

Using the containerImage property, you can define the Docker image for the task. You can use any image from a public or private container registry, as well as a custom local image built from a Dockerfile. You may even build a Docker image using the Docker plugin in one task, and reference the built image by the tag in a downstream task.

containerImage: ghcr.io/kestra-io/pydata:latest

The taskRunner property also allows you to configure credentials with nested username and password properties to authenticate a private container registry.

id: private_docker_image
namespace: company.team

tasks:
  - id: custom_image
    type: io.kestra.plugin.scripts.python.Script
    taskRunner:
      type: io.kestra.plugin.scripts.runner.docker.Docker
      credentials:
        username: your_username
        password: "{{ secret('GITHUB_ACCESS_TOKEN') }}"
    containerImage: ghcr.io/your_org/your_package:tag
    script: |
        print("this runs using a private container image")

The task documentation of each script task provides more details about available taskRunner properties.

For more examples of running scripts in Docker containers, check the Script tasks page.

Next steps

Congrats! 🎉 You've completed the tutorial.

Next, you can dive into:

• Architecture
• Key concepts
• Developer Guide that includes a more detailed explanation of everything covered in this tutorial
• Plugins to integrate with external systems
• Deployments to deploy Kestra to production.