Databricks Cost Series Part 3: DLT, Monitoring & Photon: Hidden Cost Multipliers

This post is Part 3 of a 5-part series on cost-aware architecture in Databricks, published by Qubika. In this series, we share how our teams make architectural and compute decisions with cost-efficiency in mind, without sacrificing speed, flexibility, or maintainability.

Part	Title	Status
1	Cost-First Design	Published: read here
2	Serverless vs Classic Compute	Published: read here
3	DLT, Monitoring & Photon	→You are here
4	From Design to Numbers	Published: read here
5	Cost Governance in Practice	Published: read here

Features ≠ Free

It’s easy to think of DLT, Monitoring, and Photon as just “features”, you toggle them on and gain some benefits.

But these features run on compute. Some spin up jobs. Some scan your data. Some materialize views periodically.

They consume DBUs and generate downstream costs. If you don’t account for them, your estimates will be off.

---

DLT: Premium Reliability, Premium Cost

Delta Live Tables (DLT) simplifies pipeline management, adds lineage and reliability, and supports automatic retries and observability.

But:

• DLT runs on its own serverless SKU with a higher DBU rate than regular jobs.

• The Enhanced and Advanced modes increase this further.

• Even if your tables are small, refresh frequency × number of tables can add up.

Recommendation:

• Use DLT when the pipeline needs auto-healing, auditable lineage, or CI/CD integration.

• For ultra-simple, low-frequency jobs → regular Jobs compute is likely cheaper.

---

Lakehouse Monitoring: Great Visibility, Hidden Footprint

Lakehouse Monitoring automates data quality checks, profiling, and drift detection.

It’s valuable, but it creates:

• Monitoring jobs, which scan tables.

• Checkpoints and results, which are written to storage.

• Regular refreshes, based on your configuration.

Cost tips:

• Monitor only critical tables.

• Set daily or weekly cadence unless real-time is needed.

• Track the DBUs of monitoring jobs in system.billing.usage.

---

Photon: Worth It When It Pays for Itself

Photon is a vectorized engine that speeds up SQL and DataFrame workloads.

It often reduces runtime, but comes with a premium DBU rate.

Break-even analysis is key:

• If Photon cuts runtime from 10 min to 5 min → it’s likely cheaper overall.

• If speedup is minor, the higher rate may not be justified.

Pro tips:

• Serverless always includes Photon.

• For SQL-heavy pipelines, benchmark with/without Photon enabled.

• Use Photon when runtime reduction > 30–40%.

---

Design Rule: Features Should Be Intentional, Not Default

When planning a pipeline:

1. Start with design (volume, frequency, complexity).

2. Add features intentionally – only if they bring value.

3. Estimate and monitor their individual cost contributions.

---

Example: Medium-Sized DLT Pipeline

A pipeline refreshes 4 tables hourly via DLT in Advanced mode.

• Each refresh takes ~5 minutes.

• DLT serverless SKU = $0.25/DBU.

• Observed usage = 4 DBUs per run.

→ Monthly DBUs = 4 tables × 24 runs/day × 30 days × 4 DBUs = 11,520 DBUs/month
→ Monthly cost = 11,520 × $0.25 = $2,880

This may be fine if you’re saving hours of engineer time and downtime, but not if the same could be done via Jobs for $300/month.

Note: This is an illustrative cost scenario based on public DBU pricing as of this writing. Actual costs depend on your cloud provider, pricing plan, and runtime configuration.
Use the Databricks Pricing Calculator or your account-specific console to generate up-to-date estimates.

---

Bonus: Feature Cost Checklist

---

Keep Learning

In Part 4, we explore cover how to translate architecture into a quantitative cost estimate, and how to create defensible forecasts using usage data.

→ Read here: Part 4: From Design to Numbers – Estimating and Governing Databricks Costs