How to Implement Real-Time Fraud Detection

According to the FTC, US businesses lost $5.6 billion to fraud in 2021 — a 70% increase from 2020. That number jumped a further 30% to $8.8 billion in 2022. As the threat continues to grow, businesses need fraud detection that can stop bad actors before they inflict financial damage.

In this article, we’ll explain how to prevent fraud with the latest methods and technologies. Learn how to implement real-time fraud detection with an operational data warehouse.

From banks, to fintechs, to mom-and-pop businesses with high transaction volumes, fraud is an acute threat to the bottom lines of most companies. Some of the most common types of financial fraud that businesses experience include:

• New account fraud: A scammer opens fraudulent accounts or lines of credit by stealing someone’s name, address, email, phone number, or social security number. Suspicious patterns in transactions reveal this type of fraud.

• Credit card fraud: A scammer maliciously acquires a physical credit card and uses it to make purchases in stores. Sudden, abnormal spending patterns can flag this type of fraud.

• Account takeover (ATO): A scammer acquires a user’s credentials and hijacks the account. Ramp initially spent over an hour to detect ATO. This was due to the limitations of batch processing in Ramp’s cloud data warehouse. After adopting an operational data warehouse, Ramp can now identify fraud in 1-3 seconds.  

These are some of the most popular kinds of fraud, but different methods are developed by bad actors on a consistent basis. Developing fraud detection that is quick, consistent, and agile is key to avoiding heavy financial losses. And in order to do this, businesses need a responsive, real-time data infrastructure. 

Not all fraud detection is the same. If you detect fraud two hours after it occurs, bad actors can make off with thousands or millions of dollars before you have a chance to stop them. 

That’s why real-time fraud detection is so important. Real-time fraud detection allows you to identify fraud as it happens, so you can stop fraudsters in their tracks. 

Real-time fraud detection automatically flags suspicious activity as soon as it occurs. Detecting fraud as early as possible saves the cost of manual investigation and remediation. 

There is also an important reputational benefit to real-time fraud detection. End users have greater trust in a platform that will alert them to fraudulent activity on their account. 

Receiving immediate notifications saves customers from the painful and time-consuming process of reclaiming their identity and lost funds.

In addition to being faster, algorithmic systems are more accurate. When implemented well, they pick up on subtle, suspicious behavior patterns that humans overlook.

However, real-time fraud detection is not workable for many financial and enterprise companies. They often utilize analytic data warehouses built on batch processing. These DWHs are more suited for historical data analysis, rather than operational use cases that power critical business processes.  

With analytic data warehouses, SQL queries and business logic that detect fraud are executed every few hours or days, as opposed to every few seconds. But, as discussed, it’s not helpful to learn about fraud several hours after the fact.

In order to perform real-time fraud detection, companies need continuous SQL transformation based on streaming data.  

To detect suspicious transactions, businesses typically assign a “fraud” score to each transaction, and deny transactions above a certain threshold. They generate the score algorithmically, often by querying financial transaction data.  

This approach is flexible since businesses can adjust the scoring criteria to combat new fraud tactics. But to produce helpful fraud scores, businesses first need the right data infrastructure. 

Businesses that want to create adaptive fraud models must wrestle with key considerations when building their data stacks. These businesses must: 

1. Choose the right data warehouse for analyzing raw transaction data

2. Ingest real-time transaction data from operational systems, like PostgreSQL or Kafka stream, into the data warehouse

3. Produce materialized views that greatly reduce the time and cost of running fraud detection queries

4. Define detection logic and alerts to operationalize real-time, anti-fraud insights that reduce financial losses

Businesses need the right tooling to address all of the challenges above. That’s where Materialize comes in.  

Real-time fraud detection requires real-time data. This data is often located in an operational database such as PostgreSQL.

But operational databases are designed primarily for transactional work. These databases would almost certainly crash if subjected to the intensive computation that fraud detection requires.

Because of this, many companies have started implementing real-time functionality in their existing data warehouses. However, conventional data warehouses are limited in a different way. They are designed to compute and ingest data in batches. 

As a result, data transformation only occurs intermittently, delaying the computation of a fraud score. This gives fraudsters hours to rip off customers and businesses. By the time the fraud score is computed, fraudsters have already inflicted significant monetary losses.    

These technological shortcomings create difficult trade-offs. In some cases, businesses compute micro-batches multiple times a day, or even several times in the same hour, to power fraud detection.

This is cost prohibitive, since re-running queries demands extensive compute resources, leading to higher costs. Even with this workaround, fraud scores are still not calculated fast enough, giving fraudsters enough time to escape with their loot.   

Real-time fraud detection requires a new kind of data infrastructure: an operational data warehouse. Operational data warehouses such as Materialize continuously compute new streaming data as it arrives, and stores results in materialized views that are always up to date. 

An operational data warehouse offers the real-time latency of a streaming solution, abstracted into a data warehouse for easy access to the fresh data via SQL. This provides the ease, trust, and scale required of a critical function such as fraud detection.

Your first step for real-time fraud detection, then, is to launch an operational data warehouse like Materialize. Here’s how to get Materialize up and running. 

To ingest data into Materialize, you’ll simply need to create a connection, or a source. Sources can include PostgreSQL, Kafka, Webhooks, and more. 

You can use Kafka and Debezium to connect to other operational databases. A source operates in real-time on a compute cluster, ensuring that data is ingested and ready for analysis as soon as it arrives.

Since you’re using an operational data warehouse, you can access real-time data directly by leveraging materialized views.

Materialized views are cached snapshots of a view. Since materialized views are cached in memory, they don’t rerun their query computations when you query them. Caching results in this way can save significantly on computing time. 

With a cached version of the transaction data, you can run fraud detection queries on top of it.

Of course, the downside of materialized views is that they get stale. Since computations are cached, materialized views will no longer reflect the dataset if the underlying data changes. 

Refreshing the entire view solves the freshness issue, but it is just as expensive as a direct query. The key to making real-time fraud detection fast and cost-effective is incremental updates.

Incremental updates of a materialized view only refresh data that has changed, leaving unmodified data in the cache.

An incrementally updated materialized view will update in real-time, but only compute the new data, leaving the existing cache unchanged and significantly reducing processing latency. 

Materialize is designed to provide incrementally updated materialized views of this kind, ensuring that you can query transactions in SQL inexpensively, with sub-second latency.

With data streaming into a materialized view, you leverage them in your SQL queries. Since Materialize supports SQL, you can now develop and test fraud metrics, feeding a decision engine that raises a red flag on suspicious transactions.

The particulars of this will depend on your specific transaction system. Regardless, with an operational data warehouse, you can respond to fraud scores in near real-time, stopping fraudsters before they can do damage. 

What does fraud detection look like with an operational data warehouse? Let’s take a look at our previous customer example, Ramp. 

Ramp had a fraud detection system running on their analytic data warehouse. Although their analytic data warehouse was able to deliver updates every hour, they still had many instances of fraud that would occur before this window of time.

They found that Materialize would better suit their real-time fraud detection service. Here are a few of the benefits Ramp experienced after switching to Materialize:  

• ATO attacks fell by 60%, and 50% of hacked accounts were flagged at no cost

• The response time for the Materialize data warehouse offered better, cheaper support for their real-time fraud detection 

• Since Materialize supports SQL, Ramp could transition their anti-fraud system quickly using a familiar language

The result: faster, more accurate fraud detection results, at a much cheaper rate. Ramp could act on fraud scores in real-time, stopping fraudsters in their tracks. This generated significant cost-savings in both fraud prevention and data analysis.

Fraud detection needs to occur in real-time. Otherwise, businesses are left with out-of-date fraud signals that allow fraudsters to act fast, and flee with ill-gotten gains. 

However, traditional data warehouses cannot perform fraud detection effectively, due to the constraints such as batch processing and excessive compute costs.

That’s where an operational data warehouse comes in. An ODW allows you to perform fraud scoring in real-time, based on streaming data and incremental materialized views.

Now you can stamp out fraud in all of its forms. Build instantaneous fraud detection systems that stop bad actors, and save tremendous costs, using your weapon of choice: SQL.  

Do you want to build anti-fraud technology? Try Materialize today.