At Marble, our mission is to help financial institutions combat money laundering and financial crime through real-time transaction monitoring. Our AML platform processes transactions through sophisticated rule engines that score risk factors, and one critical component of this system is screening counterparties against sanctions lists, adverse media databases, and politically exposed persons (PEP) registries.
For the past year, we've relied on Yente, an excellent open-source project by Open Sanctions that provides an API for screening entities against comprehensive sanctions data. Yente has and still serve us well, and it remains the reference implementation for this type of screening. However, as Marble's customer base grew and our processing volumes increased, we encountered performance challenges that led us to explore an alternative approach: Motiva, a reimplementation of Yente's core matching algorithms designed specifically for high-concurrency environments.
The Challenge: Performance at Scale
Financial crime prevention operates under strict constraints. When a transaction is being processed, every millisecond of latency matters. Customers expect near-instantaneous responses, and regulatory requirements demand thorough screening. Our screening rules need to query sanctions lists and return results fast enough to sit on the critical path of transaction processing.
Yente, built in Python with nomenklatura as its scoring engine, performs admirably for most use cases. The algorithms are sophisticated, the data quality is excellent, and the API is well-designed. However, Python's runtime characteristics (particularly around concurrency and memory management) began to show limitations as our request volumes grew. We observed increasing tail latencies under high load and growing memory consumption that required horizontal scaling across multiple instances.
This wasn't a criticism of Yente's design; it was simply a recognition that our specific requirements had evolved beyond what the Python implementation was optimized for. We needed:
- Consistent low latency even under extreme concurrency
- High throughput for both real-time screening and batch monitoring
- Stable memory footprint regardless of load
- Efficient resource utilization to minimize infrastructure costs
The future requirements made this even more pressing. Beyond real-time transaction screening, our roadmap includes the implementation of continuous transaction monitoring across our entire user base (from hundreds of thousands to millions of entities that need to be regularly screened against updated sanctions lists). This kind of scale would multiply our concurrency requirements by orders of magnitude.
Why We Built Motiva
Motiva was created by Antoine Popineau, a Marble engineer, as a personal project developed on his own time. Recognizing its potential to address our scaling challenges, Marble has since sponsored its continued development and adopted it for production use.
We took inspiration from Yente's excellent design and reimplemented its core matching and scoring algorithms. The goal wasn't to replace Yente as the reference implementation (we actually still use it under the hood) but to create a specialized variant optimized for our high-throughput use case.
We chose to build Motiva using Rust. This was a pragmatic engineering decision based on the language's characteristics that align with our performance requirements:
- Memory efficiency: Predictable memory usage patterns help maintain stable performance under load
- No garbage collection: Deterministic memory management avoids unpredictable latency spikes
- Excellent concurrency primitives: Native async/await support and threading models designed for high-concurrency scenarios
- Fast execution: Compiled code with minimal runtime overhead
- Mature ecosystem: Comprehensive libraries for everything we needed
The result is Motiva, delivered as two components: libmotiva, a library implementing the matching algorithms that can be used independently, and motiva, an HTTP API server that wraps the library and provides Yente-compatible endpoints.
Crucially, we didn't invent new algorithms. All the matching families we implemented (NameBased, NameQualified, and LogicV1) were taken directly from nomenklatura and reimplemented with careful attention to producing nearly equivalent scores. We also leverage data directly from Open Sanctions' work, including word lists and normalization rules from their 'rigour' library, used for cleaning and normalizing entity text before comparison.
We also took the liberty to tailor-fit the processing to what Marble requires, adding some opinionated changes to the way scoring works (some are very small changes, some might produce slightly different results depending on input).
How Motiva Works
At its core, Motiva's job is straightforward: take a query entity (like a payment counterparty), search an Elasticsearch index containing millions of sanctioned entities, retrieve promising candidates, and score each one to determine the likelihood of a match.
Performance Optimizations
While we've avoided getting too technical, a few key optimizations are worth mentioning because they directly address our scaling challenges:
Efficient memory allocation: We use arena allocation for temporary data during scoring. Every entity scoring operation allocates and deallocates many small strings and only lasts from microseconds to a few milliseconds. Instead of allocating and freeing memory for every comparison, we allocate once and reset the arena between entities. This dramatically reduces allocation overhead when processing thousands of comparisons per second.
Lazy initialization: Heavy data structures like company type mappings and address normalizers are loaded once into optimized Aho-Corasick data structures and shared across all operations, rather than being rebuilt repeatedly.
Parallel processing: When a request contains multiple entities to screen, each is processed in parallel across available CPU cores. This is particularly important for batch operations.
Early termination: If a comparison's score cannot possibly exceed the required cutoff threshold (considering remaining features), we skip the rest of the computation. This saves significant work on obvious non-matches.
Ensuring Correctness
Building a reimplementation raised an important question: how do we ensure our scores match the reference implementation? We took this seriously because incorrect matching could mean either missing sanctions hits (compliance risk) or generating false positives (incurring operational burden).
Our test suite includes integration tests that compare Motiva's scores directly against nomenklatura's reference implementation. We screen identical entities through both systems and verify scores are within a small margin (0.01) of each other. These tests run automatically on every code change, ensuring we maintain compatibility as both codebases evolve.
Real-World Impact
Deploying Motiva in production has delivered measurable improvements to our screening infrastructure:
Latency stability: P99 latencies remain consistent even during traffic spikes that previously caused elevated tail latencies. This reliability is critical for payment processing where delays cascade through dependent systems.
On a sample of screening requests over 24 hours, P99 latency went from 249 ms to 160 ms, and maximum latency from 1.26 seconds to 254 ms. Overall, and without mentioning raw performance, latency is more stable.
Memory efficiency: Memory consumption stays stable regardless of concurrent request count. We no longer need to scale instances proactively based on anticipated load spikes, simplifying operations and reducing costs.
Motiva now occupies a stable 3 % of its available memory of 512 MB, whereas it used to consumes around 50% of 4 GB before the migration.
Cost reduction: More efficient resource utilization means we handle higher throughput on fewer instances. The operational cost savings compound over time as our transaction volumes grow.
These improvements weren't just theoretical. We built observability into Motiva from the start, with OpenTelemetry tracing and Prometheus metrics that allowed us to measure and validate these performance gains in production. This instrumentation gives us detailed visibility into scoring latencies, memory usage patterns, and request throughput, ensuring we can continue optimizing as our requirements evolve.
Building on Open Sanctions' Work
It's important to emphasize that Motiva is built on the shoulders of giants: the excellent work of the Open Sanctions team. The matching algorithms, normalization rules, schema definitions, and even much of the test data come from their projects. We didn't invent new approaches; we reimplemented proven ones with a focus on performance.
We also continue to rely on Yente and Open Sanctions' infrastructure for critical functions:
- Data quality: Open Sanctions' curated and licensed datasets remain our data source. Access to this data requires appropriate licensing from Open Sanctions for commercial use.
- Infrastructure dependency: Motiva requires Yente to run alongside it on the infrastructure, both for the Elasticsearch index and the Yente indexer that keeps sanctions data up to date.
- Algorithm validation: We validate our implementations against nomenklatura as the reference.
Motiva isn't a replacement for Yente; it's designed to complement it. It's a specialized tool optimized for a particular use case. Yente remains the comprehensive, full-featured platform that handles data ingestion, index management, and the complete lifecycle of sanctions data. For many organizations, Yente is absolutely the right choice.
We built Motiva because our specific requirements (extremely high concurrency, predictable low latency, and stable memory usage at scale) called for optimization at a level that required rethinking the implementation runtime characteristics.
The Path Forward
Motiva remains a work in progress. We've focused on the LogicV1 algorithm and the endpoints we need immediately: entity matching, entity retrieval, and catalog access. Other algorithms like LogicV2 and additional features remain unimplemented.
As our monitoring requirements evolve (particularly as we scale to continuous screening of our entire user base), we'll continue developing Motiva to meet those needs. But we'll do so while maintaining our commitment to compatibility with the reference implementation and transparency in how our scoring works.
We've open-sourced Motiva at https://github.com/apognu/motiva because we believe others facing similar scaling challenges might benefit from our approach. Financial crime prevention is fundamentally collaborative; it requires sharing intelligence, tools, and techniques across organizations and borders. If our work helps other institutions more effectively screen against sanctions lists, we all benefit from a more robust compliance ecosystem.
The fight against financial crime requires both breadth and depth: comprehensive data from organizations like Open Sanctions and performant infrastructure that can apply that data at scale. Motiva is our contribution to the latter, built with gratitude for those who enable the former.
Acknowledgments
Motiva would not exist without the foundational work of the Open Sanctions team, particularly their nomenklatura matching library and rigour text processing utilities. The algorithms, normalization rules, and domain expertise embedded in their code informed every aspect of our implementation. We're grateful for their commitment to open-source tooling for financial crime prevention and their excellent documentation that made reimplementation feasible.
