PluginsDatabricksTasksClusterCreateCluster

CreateCluster​Create​Cluster

CreateCluster CreateCluster

yaml
type: "io.kestra.plugin.databricks.cluster.CreateCluster"

Create a Databricks cluster.

Examples

Create a Databricks cluster with one worker.

yaml
id: databricks_create_cluster
namespace: company.team

tasks:
  - id: create_cluster
    type: io.kestra.plugin.databricks.cluster.CreateCluster
    authentication:
      token: <your-token>
    host: <your-host>
    clusterName: kestra-demo
    nodeTypeId: n2-highmem-4
    numWorkers: 1
    sparkVersion: 13.0.x-scala2.12

Properties

clusterName

  • Type: string
  • Dynamic: ✔️
  • Required: ✔️

The name of the cluster.

sparkVersion

  • Type: string
  • Dynamic: ✔️
  • Required: ✔️

The Spark version.

accountId

  • Type: string
  • Dynamic: ✔️
  • Required: ❌

Databricks account identifier.

authentication

Databricks authentication configuration.

This property allows to configure the authentication to Databricks, different properties should be set depending on the type of authentication and the cloud provider. All configuration options can also be set using the standard Databricks environment variables. Check the Databricks authentication guide for more information.

autoTerminationMinutes

  • Type: integer
  • Dynamic: ❌
  • Required: ❌

If set, the cluster will be terminated automatically after this time period.

configFile

  • Type: string
  • Dynamic: ✔️
  • Required: ❌

Databricks configuration file, use this if you don't want to configure each Databricks account properties one by one.

host

  • Type: string
  • Dynamic: ✔️
  • Required: ❌

Databricks host.

maxWorkers

  • Type: integer
  • Dynamic: ❌
  • Required: ❌

The maximum number of workers.

Use this property along with minWorkers to use autoscaling. Otherwise, set a fixed number of workers using numWorkers.

minWorkers

  • Type: integer
  • Dynamic: ❌
  • Required: ❌

The minimum number of workers.

Use this property along with maxWorkers for autoscaling. Otherwise, set a fixed number of workers using numWorkers.

nodeTypeId

  • Type: string
  • Dynamic: ✔️
  • Required: ❌

The type of node, the value depends on the cloud provider.

numWorkers

  • Type: integer
  • Dynamic: ❌
  • Required: ❌

The fixed number of workers.

You must set this property unless you use the minWorkers and maxWorkers properties for autoscaling.

Outputs

clusterId

  • Type: string
  • Required: ❌

The cluster identifier.

clusterState

  • Type: string
  • Required: ❌
  • Possible Values:
    • ERROR
    • PENDING
    • RESIZING
    • RESTARTING
    • RUNNING
    • TERMINATED
    • TERMINATING
    • UNKNOWN

The cluster state.

clusterURI

  • Type: string
  • Required: ❌
  • Format: uri

The cluster URI on the Databricks console.

Definitions

io.kestra.plugin.databricks.AbstractTask-AuthenticationConfig

Properties

authType
  • Type: string
  • Dynamic: ✔️
  • Required: ❌
azureClientId
  • Type: string
  • Dynamic: ✔️
  • Required: ❌
azureClientSecret
  • Type: string
  • Dynamic: ✔️
  • Required: ❌
azureTenantId
  • Type: string
  • Dynamic: ✔️
  • Required: ❌
clientId
  • Type: string
  • Dynamic: ✔️
  • Required: ❌
clientSecret
  • Type: string
  • Dynamic: ✔️
  • Required: ❌
googleCredentials
  • Type: string
  • Dynamic: ✔️
  • Required: ❌
googleServiceAccount
  • Type: string
  • Dynamic: ✔️
  • Required: ❌
password
  • Type: string
  • Dynamic: ✔️
  • Required: ❌
token
  • Type: string
  • Dynamic: ✔️
  • Required: ❌
username
  • Type: string
  • Dynamic: ✔️
  • Required: ❌

Was this page helpful?