permanent_depoyments.md

The Two-Step Approach to Dynamic Dagster+ Deployments

Creating a new persistent deployment requires coordinating two systems: the Dagster+ API and your Kubernetes agent configuration.

1. Dagster+ API (Create the Deployment Resource)

The deployment must exist in Dagster+ as a logical entity. This is done via the GraphQL API:

import httpx

def create_deployment(org: str, token: str, name: str):
    """Create deployment in Dagster+ via GraphQL."""
    
    mutation = """
    mutation CreateDeployment($input: CreateDeploymentInput!) {
        createDeployment(input: $input) {
            ... on CreateDeploymentSuccess {
                deployment { deploymentName }
            }
        }
    }
    """
    
    response = httpx.post(
        f"https://{org}.dagster.cloud/prod/graphql",
        headers={"Dagster-Cloud-Api-Token": token},
        json={
            "query": mutation,
            "variables": {"input": {"deploymentName": name}}
        }
    )
    return response.json()

This tells Dagster+ "a deployment called sandbox-alice exists" — but no agent is serving it yet.

2. Flux HelmRelease (Tell the Agent to Serve It)

Your Kubernetes agent only serves deployments listed in its config. The agent reads from the deployments list in the HelmRelease:

# Before
spec:
  values:
    dagsterCloud:
      deployments:
        - data-eng-prod
        - data-eng-staging

# After (add the new deployment)
spec:
  values:
    dagsterCloud:
      deployments:
        - data-eng-prod
        - data-eng-staging
        - sandbox-alice  # ← Added

The code to do this:

import yaml

def add_deployment_to_helmrelease(helmrelease_path: str, deployment_name: str):
    """Add deployment to Flux HelmRelease config."""
    
    with open(helmrelease_path) as f:
        hr = yaml.safe_load(f)
    
    deployments = hr["spec"]["values"]["dagsterCloud"]["deployments"]
    
    if deployment_name not in deployments:
        deployments.append(deployment_name)
        deployments.sort()  # Keep it tidy
        
        with open(helmrelease_path, "w") as f:
            yaml.dump(hr, f)
    
    # Then commit and push to trigger Flux reconciliation
    # git add, git commit, git push...

The Full Flow

┌─────────────────┐     ┌──────────────────┐     ┌─────────────────┐
│  Git Tag Push   │────►│  GitHub Actions  │────►│  Dagster+ API   │
│ deploy/sandbox  │     │  (or script)     │     │  (create)       │
└─────────────────┘     └────────┬─────────┘     └─────────────────┘
                                 │
                                 ▼
                        ┌──────────────────┐     ┌─────────────────┐
                        │  GitOps Repo     │────►│  Flux           │
                        │  (update YAML)   │     │  (reconcile)    │
                        └──────────────────┘     └────────┬────────┘
                                                          │
                                                          ▼
                                                 ┌─────────────────┐
                                                 │  K8s Agent      │
                                                 │  (now serves    │
                                                 │  sandbox-alice) │
                                                 └─────────────────┘

Why Both Steps?

Step	What it does	Without it...
Dagster+ API	Registers deployment in Dagster+ control plane	UI shows nothing, can't deploy code to it
Flux/Agent config	Tells your agent to serve this deployment	Agent ignores it, no compute available

They're complementary — the API creates the "logical" deployment, the agent config provides the "physical" compute to run it.

Quick Reference: Manual Steps

Create a deployment

# 1. Create in Dagster+ (via CLI or API call)
python scripts/cli.py create --name sandbox-alice

# 2. Or via git tag (triggers GitHub Actions)
git tag -a deploy/sandbox-alice -m "owner=alice@example.com"
git push origin deploy/sandbox-alice

Delete a deployment

# 1. Delete via CLI
python scripts/cli.py delete --name sandbox-alice

# 2. Or via git tag deletion
git push origin --delete deploy/sandbox-alice

Key Files

File	Purpose
utils/dagster_cloud_api.py	Dagster+ GraphQL client
utils/flux_updater.py	HelmRelease YAML management
k8s/base/helmrelease.yaml	Base Flux configuration
.github/workflows/deploy-on-tag.yaml	CI automation