Stop the Pipeline Pain
From painful debugging to unwieldy DAGs, the data pipeline is its most difficult when things go wrong. Turn days of problem investigation into immediate problem identification without needing complicated configuration software.
Moving Beyond the DAG
PUFF is built on the concept of incremental computing.
While a DAG involves configuring a graph to compute only what needs changing, incremental computing does this at every operation in the code.
Joins, mathematical operations, sorts and filters. Each one is only recomputed when the output might actually change.
This functionality is built in, without your engineers needing to define tasks or draw the graph themselves.
The same graph enables efficient multithreading so your code runs in a fraction of the time.
All without your engineer needing to configure either concept. It’s built to work by default.
Incremental Computing
If the code has run successfully once and
The code hasn’t changed
The inputs haven’t changed
The data hasn’t changed
Don’t Rerun It.
Case Study: The numbers are very wrong
A fraud detection model that runs on live transaction data. Every day it flags a certain number of transactions it identifies as ‘suspicious’ and sends them to the relevant team to investigate. The model brings together data from multiple sources, both internal and external, to come up with a final solution.
One day someone flagged that the fraud detection model was producing a fraud investigation rate of 3000%. That meant either:
Thousands of legitimate customer transactions were getting blocked, losing real customers and revenue and generating a mass of complaints.
Or, if the model was turned off, fraudulent transactions would be sailing through undetected with the potential for fines a real risk.
The number was wrong and needed fixing immediately. But no one knew why.
The Pipeline Visualised
The model, especially if it’s on a dag, looks like a sequence of nodes like this, where the nodes at the top are the data coming in, the middle are things like the checking the data, cleaning it, combining it, etc and the last node at the bottom is your final result.
In this case the problem was the supplier had changed how they passed the risk-scoring data. Something totally outside of the client’s control and not the fault of their data scientist or engineers unless they managed to think of every possible edge case which is just unreasonable.
All they knew was that the numbers coming out of the final model were nonsensical.
The Current World:
The data engineer checks the pipeline and tries to work out what the issue is.
After doing a full analysis and finding nothing failed, they send it to the data scientist to check the model.
The data scientist waits for repro to confirm the issue.
They take a look at the code to make sure it’s working as intended.
The kernel dies from holding too much big data in memory. They accidentally step over the exact line that would show them the bug. They put a break point in the wrong point of entry into the DAG. Lots of waiting for iterative runs.
They work out the model was not the problem and the upstream data was wrong. They send it back to the data engineer to take a look.
The data engineer checks for issues in the pipeline data processing, with the same iteration challenges.
They see that the risk-scoring data has changed day over day.
They contact the supplier to find out what changed, why and what is the right fix.
Hours, if not days wasted.
The Investigation
In PUFF:
The data engineer checks the incremental computing graph which identifies that the risk scoring data changed.
They contact the supplier to find out what changed, why and what is the right fix.
Days of debugging saved.
When the issue is the code, not the data?
Data Engineer checks the incremental computing logs.
Sees where the code changed.
Sends to the relevant party to fix.
Days of debugging saved again.
LLMs aren’t Superheroes
They burn tokens through hallucinations
They spend as long waiting for the code to run as a human
They ignore guardrails telling them not to read your sensitive data
They make any attempt at an audit trail impossible
They make changes to the code that make future bugs harder to trace and fix
LLMs don’t fix the problem. But they do when combined with PUFF.
How PUFF Enables AI Agents
Domain specificity to reduce hallucinations
Faster computation through incremental computing and multithreading
Metadata propagation forcing strict access controls
Full lineage built from the ground up
Impossible for malicious code injection through hacked .md files. PUFF can only execute data science code.