Cloud Composer Costs Too Much, So We Built It Ourselves on a $20 VM

I spent two weeks last month arguing with our DevOps team about whether we should migrate our Airflow setup to the cloud. The conversation went in circles: "Cloud Composer is managed, so it's safer." "But we're paying $350/month for something that just schedules tasks." "Yeah, but we don't want to manage it ourselves." Sound familiar? I stumbled onto an article that basically showed me we were overthinking this entirely.

The team that wrote about moving their Airflow to GCP made a choice that stuck with me: they did the math, realized Cloud Composer was bleeding them dry, and decided to just run Airflow on a single Compute Engine VM. Not just any VM, one that costs about $150 a month total. Reading through their approach, I realized I'd been accepting the "managed = better" narrative without actually questioning whether it applied to our workload.

The Cloud Composer Trap (And Why Smart Teams Skip It)

Here's the thing about Cloud Composer that nobody really talks about: it charges you an hourly compute fee whether your pipelines are running or not. If you're like most teams I know, where Airflow is an orchestrator that triggers work elsewhere (Vertex AI, BigQuery, Lambda functions)-you're paying for an idle cluster most of the time.

The original article nailed this. Their team was running machine learning pipelines, but the actual heavy computation happened in Vertex AI. Airflow was just the conductor, not the orchestra. In that scenario, a $350/month managed service feels like buying a full restaurant kitchen when you just need a hot plate.

I started mapping our own usage and realized we had the same pattern. We have scheduled data jobs that run queries, trigger some transformations, and maybe kick off an analysis. Airflow sits there deciding when things happen, not doing the computation. That's a scheduler's job, not a cluster's.

Self-Managed Airflow: The Unglamorous Reality

The team's second iteration was where things got interesting. They stood up Airflow 3.x on a single Compute Engine VM, added PostgreSQL for metadata, and suddenly their monthly bill dropped below $150. The kicker? They're a technical team that already runs infrastructure, so the operational overhead wasn't actually overhead, it was just work they already knew how to do.

This is the part where I had to be honest with myself. The "you don't want to manage Airflow" argument only holds if you can't manage Airflow. But if your team has people who understand Linux, Python, and infrastructure-as-code, the self-managed route becomes not just cheaper, but also more flexible. You control the Airflow version. You control the upgrade schedule. You control what gets installed.

What impressed me most was their architecture around the configuration system. They decoupled it entirely from Airflow, a Flask app called config_hq writes configuration changes to a GCS bucket, and their DAGs just read from that bucket. If the config service is down, the pipelines still run with the last known good configuration. That's not something you get easily with a fully managed solution.

The Security Question I Almost Missed

Here's where I think the article really shines: they didn't just pick a cheaper option and call it done. They layered security thoughtfully. The config app sits behind an HTTPS load balancer with Identity-Aware Proxy. The VM has no public IP address. SSH access requires two separate IAM gates. Every session gets logged.

This is the moment I realized "self-managed" doesn't mean "insecure." It means your team is responsible for security, which honestly, I prefer. We can audit our own setup end-to-end, rather than trusting Google's defaults for something that orchestrates our data pipelines.

What I'd Actually Do Differently

Reading this, I'd probably make one change: I'd be more aggressive about containerizing the entire setup, even on a single VM. Docker containers for Airflow, PostgreSQL, and the config service would give me more portability if we ever need to move or scale. But that's implementation detail, the core philosophy is sound.

The real question this raises for me is: how many teams are paying premium prices for "managed" when they actually need "self-managed-but-good"? There's a middle ground here that doesn't get enough attention.

Your Turn

If you're running Airflow in the cloud, what's your actual monthly spend? Are you using Cloud Composer, or did you already take the self-managed path? I'm genuinely curious whether the cost numbers hold up at different scales.

Source: This post was inspired by "Moving Your Local Airflow to GCP for under $150 a month" by Dev.to. Read the original article