Every payment this company processes goes through a fraud check. The models behind that check ask questions like “how many times has this card been used in the last 30 days?” or “when did we first see this email?”, and they need the answers while the payment is still in flight. The fraud service has 250 ms at p99 to get them.
This is the story of how we moved the features that answer those questions from a streaming stack on AWS to Databricks, and what the numbers look like today.
The challenge
The company’s fraud models rely on about 1,600 to 1,700 long-window features: aggregations over one day up to the full history, across 13 entities such as card, email, document, phone, device, merchant and several composite keys. They ran on the same infrastructure as the rest of the fraud platform: Flink aggregations, a Redis cache for the hot path and DynamoDB filled by batch backfills.
That infrastructure was out of room. Redis had reached its memory limit, and long-window features were the largest part of it. The fleet hosting the models was also close to its ceiling, which made the move more urgent.
Long-window features also change slowly, and the agreed SLA allowed hourly freshness. The company was paying for streaming freshness that the models didn’t use.
One requirement sat above the rest: the values used to train the models had to match the values the fraud service reads online. A feature that drifts between training and serving is a different feature, and the models would have to be retrained around it.
What we built
The design follows one rule: freshness where it matters, batch everywhere else.
Long windows moved to batch on Databricks. Raw events land in Delta through Auto Loader. PySpark jobs compute the features into Silver tables in Unity Catalog on a schedule tied to window length: hourly for windows up to 7 days, every 6 hours up to 60 days, daily beyond that. Synced tables in triggered mode push the results to Lakebase, and the fraud service reads them from a Lakebase read replica over JDBC, like any Postgres database.
Short windows keep their sub-second path. Features that change by the second still need streaming. Their path on Databricks is Spark Real-Time Mode writing to Lakebase, which we covered in our Real-Time Mode post.
Each feature is a short YAML entry (entity keys, aggregation, filters, window) compiled into Spark SQL. Four operation types cover the whole catalog: count, sum, age and ratio. Adding a feature means adding an entry and running the tests.
Each Databricks component has a specific job:
| Component | Role in the solution |
|---|---|
| PySpark on Delta Lake | Batch compute over the full event history. Hourly freshness makes batch simpler and cheaper than a stateful streaming engine. |
| Unity Catalog | One access model across raw events, features and environments. This is the first of several feature domains, so governance is solved once. |
| Lakebase | The online store. Synced tables move features from Unity Catalog to Postgres with no sync pipeline to maintain outside the platform. |
| MLflow | An audit trail for every compute run, which a migration needs when correctness has to be proven against a legacy system. |
The read path got shorter as we measured it. Serving features through Unity Catalog endpoints landed at 300 to 400 ms, over the budget. A Databricks App in front added a fixed 1.2 to 1.5 seconds per request. Direct JDBC from the fraud service to Lakebase removed both layers, and it became the read path in September.
The results
The fraud service measured latency itself, from its own code through JDBC to Lakebase, with 100% of production traffic over 18 minutes and hundreds of thousands of queries per minute:
| Metric | Result | Target |
|---|---|---|
| Average | 11.7 ms | |
| p95 | 41 ms | 60 ms (client SLA) |
| p99 | 70.5 ms | 120 ms (client SLA), 250 ms (fraud service budget) |
| p99.9 | 162 ms |
On correctness:
- The first wave of 124 features was signed off by the client’s data science team with zero value differences against ground truth. A second wave of 102 features is scoped.
- A full month of data, 156 million rows, matched the legacy source with zero mismatches.
- Data scientists now build their own training datasets. The datasets model the delay between a compute finishing and its values reaching Lakebase, so offline values match what the fraud service saw at decision time.
On infrastructure, the Redis footprint shrinks with every feature family that moves, and the streaming stack keeps only what needs sub-second freshness.
Where it stands today: since mid-October the fraud service calls Lakebase on every transaction in shadow mode, which means the lookup runs on real traffic but doesn’t change the fraud decision yet. The next gates are the end-to-end approval and model-score parity. Moving the models themselves to Databricks is a separate track.
How we delivered it
Migration by mechanism, delivery by wave. The catalog looked uniform from the outside. Behind it, two different legacy engines computed the features, with different rules for windows and keys. We grouped features by the mechanism that computes them to plan the port, and delivered them in waves of feature families that the client’s data science lead prioritized and approved.
A three-layer validation gate. A feature counts as migrated only when it passes data quality checks, runs in shadow mode on real traffic, and produces the same model scores as the legacy feature.
Traffic ramps in shadow mode. The first ramp hit timeouts past 60% of traffic. We raised the Lakebase compute range, the fraud team tuned their side, and the second ramp reached 100% the next day with a p99 of 70.5 ms. Shadow mode let us find that limit without touching a single fraud decision.
AI-assisted engineering, with a senior engineer on every change. The team worked with Claude Code on top of Databricks’ AI Dev Kit and the Qubika AI Dev Kit (our conventions, project setup and safety checks), plus skills written for this project. In the first six weeks, engineers logged about 150 hours saved, or 85 hours counting only tasks with a stated manual baseline. The share of commits co-authored with AI went from 17% in the early weeks to about 90% by October. These are the engineers’ own estimates, not a controlled study, and every change still went through review.
Three lessons from production
Measure from the client. A lab test gave a p99 of 4 ms. A check from another region gave 115 ms at p50, mostly network. The number that matters is the 70.5 ms the fraud service measured on its own traffic.
Put every setting in code. A compute range raised by hand was reset by a later infrastructure deploy, and latency went up tenfold the next day.
Size for writes as well as reads. Once reads were fast, sync throughput became the number to plan around, and the read replica needs capacity to replay every write.
Where else this applies
The same pattern fits any decision that has to read historical aggregates in milliseconds: credit risk scoring, anti-money laundering checks, quota and limit tracking, and live application state. If your features live in Unity Catalog and your service needs them during a transaction, Lakebase gives you a governed serving layer without a separate cache to run.
If you’re weighing a similar move, talk to our Data team. We’ll share what we measured and what we’d do differently.



