Skip to main content

Dagster+ Hybrid deployment

In a Dagster+ Hybrid deployment, the orchestration control plane is run by Dagster+ while your Dagster code is executed within your environment. Hybrid deployments are containerized: your code runs inside containers managed by an agent you operate. Kubernetes is a common choice, but not required. The Docker agent provides a lighter-weight alternative for teams that don't need Kubernetes orchestration.

For an overview of the Hybrid design, including security considerations, see Dagster+ Hybrid architecture.

Getting started​

To get started with a Hybrid deployment, you'll need to:

  1. Create a Dagster+ organization.
  2. Install a Dagster+ Hybrid agent in your environment:
    • Kubernetes (supports AKS on Azure, EKS on AWS, GKE on GCP, and other clusters)
    • AWS ECS
    • Docker (runs containers directly without Kubernetes; can be paired with Azure Container Registry (ACR) on an Azure VM for a lightweight Azure setup)
    • Microsoft Azure
  3. Create a Dagster project and add it to your deployment, typically using a Git repository and CI/CD.
tip

If you're not sure which agent to use, we recommend the Dagster+ Kubernetes agent in most cases.

note

If you are migrating from from Dagster+ Serverless, see the Dagster+ Serverless to Hybrid migration guide.

Run isolation​

Dagster+ Hybrid deployments support both isolated and non-isolated runs. Isolated runs execute in their own container with dedicated resources, while non-isolated runs use a shared container for faster iteration during development. For details, see Run isolation.

Best practices​

  • Agent container - Start at 0.25 vCPU core and 1 GB RAM, then scale with concurrent runs and the number and size of projects.
  • Code server container - Budget for imports, plus the definition graph, and any heavy initialization. We recommend starting with 0.25 vCPU cores and 1GB RAM.
  • Runs: 4 vCPU cores, 8-16 GB of RAM depending on the workload

For compute-heavy jobs, increase memory and/or CPU where the run workers are (i.e. Kubernetes pods or ECS tasks), not just the code server.

Security​

To make your Dagster+ Hybrid deployment more secure, you can:

Frequently asked questions​

Can the Dagster+ agent and code location versions be different?​

Yes. The agent and code location versions can be updated independently, but they must follow a specific upgrade order:

  • Always upgrade the agent before upgrading code locations. The agent maintains backward compatibility with older code versions.
  • Never use a code location version that is newer than your agent version, as this can cause compatibility issues.