Namespace Files

This page covers the concept of Namespace Files and how to use them in your flows.

What are Namespace Files

Namespace Files are files tied to a given namespace. You can think of Namespace Files as equivalent of a project in your local IDE or a copy of your Git repository. Namespace Files can hold Python modules, R or Node.js scripts, SQL queries, configuration files, and many more. You can synchronize your Git repository with a specific namespace to orchestrate dbt, Terraform or Ansible, or any other project that contains code and configuration files.

Once you add any file to a namespace, you can reference it in your flows using the read() function in EVERY task or trigger from the same namespace! For instance, if you add a SQL query called my_query.sql to the queries directory in the company.team namespace, you can reference it in any Query task or any JDBC Trigger like so: {{ read('queries/my_query.sql') }}. Here is an example showing how you can use the read() function in a ClickHouse Trigger to read a SQL query stored as a Namespace File:

yaml

id: jdbc_trigger
namespace: company.team

tasks:
  - id: for_each_row
    type: io.kestra.plugin.core.flow.EachSequential
    value: "{{ trigger.rows }}"
    tasks:
      - id: return
        type: io.kestra.plugin.core.debug.Return
        format: "{{ json(taskrun.value) }}"

triggers:
  - id: query_trigger
    type: io.kestra.plugin.jdbc.clickhouse.Trigger
    interval: "PT5M"
    url: jdbc:clickhouse://127.0.0.1:56982/
    username: ch_user
    password: ch_passwd
    sql: "{{ read('queries/my_query.sql') }}" # 🚀 The read() function reads the content of the file as a string!
    fetch: true

Note how in the above example, we didn't have to use the namespaceFiles.enabled: true property — that property is only required to inject the entire directory of files from the namespace into the working directory of a script (e.g. a Python task). More on that in the subsequent sections of this page.

Why use Namespace Files

Namespace Files offer a simple way to organize your code and configuration files. Before Namespace Files, you had to store your code and configuration files in a Git repository and then clone that repository at runtime using the git.Clone task. With Namespace Files, you can store your code and configuration files directly in the Kestra's internal storage backend. That storage backend can be your local directory or an S3 bucket to ensure maximum security and privacy.

Namespace Files make it easy to:

orchestrate Python, R, Node.js, SQL, and more, without having to worry about code dependencies, packaging and deployments — simply add your code in the embedded Code Editor or sync your Git repository with a given namespace
manage your code for a given project or team in one place, even if those files are stored in different Git repositories, or even different Git providers
share your code and configuration files between workflows and team members in your organization
orchestrate complex projects that require the code to be separated into multiple scripts, queries or modules.

How to add Namespace Files

Embedded Code Editor

The easiest way to get started with Namespace Files is to use the embedded Code Editor. This embedded IDE allows you to easily add custom scripts, queries and configuration files along with your flow YAML configuration files.

Get started by selecting a namespace from the dropdown menu. If you type a name of a namespace that doesn't exist yet, Kestra will create it for you.

Then, add a new file, e.g., a Python script. Add a folder named scripts and a file called hello.py with the following content:

python

print("Hello from the Editor!")

Once you added a file, you can use it in your flow. Try adding a new flow named editor.yml in the _flowsdirectory and paste the following content (adjust the name of the namespace if needed):

yaml

id: editor
namespace: company.team

tasks:
  - id: hello
    type: io.kestra.plugin.scripts.python.Script
    namespaceFiles:
      enabled: true
    script: "{{ read('scripts/hello.py') }}"

Note that if your Python script contains any Pebble expressions like e.g. reading Secrets, you need to wrap the read function with render function like so:

yaml

{{ render(read('scripts/hello.py')) }}

To make the example more concrete, here is a simple weather.py script that reads a secret to talk to a Weather Data API:

python

import requests
api_key = '{{ secret("WEATHER_DATA_API_KEY") }}'
url = f"https://api.openweathermap.org/data/2.5/weather?q=Paris&APPID={api_key}"
weather_data = requests.get(url)
print(weather_data.json())

And here is the flow that uses the script:

yaml

id: weather_data
namespace: company.team
tasks:
  - id: get_weather
    type: io.kestra.plugin.scripts.python.Script
    namespaceFiles:
      enabled: true
    docker:
      image: ghcr.io/kestra-io/pydata:latest
    script: "{{ render(read('scripts/weather.py')) }}"

The read function allows you to read any file from the namespace as a string. You can also execute your flows directly from the Code Editor by clicking on the Execute button.

The Execute button allows you to run your flow directly from the Code Editor. Click on the Execute button to run your flow. You should then see the Execution being created in a new browser tab and once you navigate to the Logs tab, you should see a friendly message Hello from the Editor! in the logs.

Note that if you use the Brave browser, you may need to disable the Brave Shields to make the Editor work as expected. Specifically, to view the task documentation, you need to set the Block cookies option to Disabled in the Shields settings: brave://settings/shields.

Brave cookies

GitHub Actions CI/CD

You can leverage our official GitHub Action called deploy-action to synchronize your Git repository with a given namespace. This is useful if you want to orchestrate complex Python modules, dbt projects, Terraform or Ansible infrastructure, or any other project that contains code and configuration files with potentially multiple nested directories and files.

Here is a simple example showing how you can deploy all scripts from the scripts directory in your Git branch to the prod namespace:

yaml

name: Kestra CI/CD
on:
  push:
    branches:
      - main
jobs:
  prod:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: deploy-scripts-to-prod
        uses: kestra-io/deploy-action@master
        with:
          resource: namespace_files
          namespace: prod
          directory: ./scripts # directory in the Git repository
          to: ./scripts # remote directory in the namespace
          server: https://demo.kestra.io/
          user: your_username
          password: ${{secrets.KESTRA_PASSWORD}}

When creating a service account role for the GitHub Action in the Enterprise Edition, you need to grant the FLOWS permission to the Role.

Terraform Provider

You can use the kestra_namespace_file resource from the official Kestra Terraform Provider to deploy all your custom script files from a specific directory to a given Kestra namespace.

Here is a simple example showing how you can synchronize an entire directory of scripts from the directory src with the company.team namespace using Terraform:

hcl

resource "kestra_namespace_file" "prod_scripts" {
  for_each  = fileset(path.module, "src/**")
  namespace = "company.team"
  filename   = each.value # or "/${each.value}"
  content   = file(each.value)
}

Deploy Namespace Files from Git via CLI

You can also use the Kestra CLI to deploy all your custom script files from a specific directory to a given Kestra namespace. Here is a simple example showing how you can synchronize an entire directory of local scripts with the prod namespace using the Kestra CLI:

bash

./kestra namespace files update prod /Users/anna/gh/KESTRA_REPOS/scripts --server=http://localhost:8080 --user=rick:password

In fact, you can even use that command directly in a flow. You can attach a schedule or a webhook trigger to automatically execute that flow anytime you push/merge changes to your Git repository, or on a regular schedule.

Here is an example of a flow that synchronizes an entire directory of local scripts with the prod namespace:

yaml

id: ci
namespace: company.team

variables:
  host: http://host.docker.internal:28080/

tasks:
  - id: deploy
    type: io.kestra.plugin.core.flow.WorkingDirectory
    tasks:
      - id: clone
        type: io.kestra.plugin.git.Clone
        url: https://github.com/kestra-io/scripts
        branch: main

      - id: deploy_files
        type: io.kestra.plugin.scripts.shell.Commands
        warningOnStdErr: false
        taskRunner:
          type: io.kestra.plugin.core.runner.Process
        commands:
          - /app/kestra namespace files update prod . . --server={{vars.host}}

Note that the two dots in the command /app/kestra namespace files update prod . . indicate that we want to sync an entire directory of files cloned from the Git repository to the root directory of the prod namespace. If you wanted to e.g. sync that repository to the scripts directory, you would use the following command: /app/kestra namespace files update prod . scripts. The syntax of that command follows the structure:

bash

/app/kestra namespace files update <namespace> <local_directory> <remote_directory>

To reproduce that flow, start Kestra using the following command:

bash

docker run --pull=always --rm -it -p 28080:8080  kestra/kestra:latest-full  server local

Then, open the Kestra UI at http://localhost:28080 and create a new flow with the content above. Once you execute the flow, you should see the entire directory from the scripts repository being synchronized with the prod namespace.

How to use Namespace Files in your flows

There are multiple ways to use Namespace Files in your flows. You can use the read() function to read the content of a file as a string, or you can point to the file path in the supported tasks.

Usually, pointing to a file location, rather than reading the file's content, is required when you want to use a file as an input to a CLI command, e.g. in a Commands task such as io.kestra.plugin.scripts.python.Commands or io.kestra.plugin.scripts.node.Commands. In all other cases, the read() function can be used to read the content of a file as a string e.g. in Query or Script tasks.

You can also use the io.kestra.plugin.core.flow.WorkingDirectory task to read namespace files there and then use them in child tasks that require reading the file path in CLI commands e.g. python scipts/hello.py.

The `read()` function

Note how the script in the first section used the read() function to read the content of the scripts/hello.py file as a string using the expression "{{ read('scripts/hello.py') }}". It'a important to remeber that this function reads the content of the file as a string. Therefore, you should use that function only in tasks that expect a string as an input, e.g., io.kestra.plugin.scripts.python.Script or io.kestra.plugin.scripts.node.Script, rather than io.kestra.plugin.scripts.python.Commands or io.kestra.plugin.scripts.node.Commands.

The read() function allows you to read the content of a Namespace File stored in the Kestra's internal storage backend. The read() function takes a single argument, which is the absolute path to the file you want to read. The path must point to a file stored in the same namespace as the flow you are executing.

Was this page helpful?

Developer GuideErrors and retries

Developer GuideVersion Control with Git

​Namespace ​Files

Namespace Files