arrow left facebook twitter linkedin medium menu play circle

Managing Thousands of Spark Workers in the Cloud: DataVisor Presents at SAIS 2018

By Claire Zhou June 18, 2018

Photo of Claire Zhou

about Claire Zhou
Claire is a Product Marketing Manager at DataVisor with over 5 years of marketing experience in security and fin-tech. She is passionate about empowering enterprise customers with AI-based solutions. Her expertise spans data analytics, cybersecurity, and fraud prevention. Claire has an MBA from UCLA.

DataVisor’s Yuhao Zheng, Tech Lead Manager, and Boduo Li, Senior Research Scientist, Infrastructure, discuss how DataVisor leverages Spark’s scalability and portability to protect 4 Billion accounts from fraud abuse and money laundering. Watch the video to learn about managing 2000+ Spark workers in clusters and as well as DataVisor’s proprietary SparkGenerator, and automated Spark management platform that optimally balances cost and resource allocation.

Click here to get more information about the session >>

Session Abstract:

At DataVisor, we fight online fraud, abuse, and money laundering using unsupervised machine learning approach that clusters millions of users. In order to support the computationally intensive workload, DataVisor uses Spark as the mainstay of its computation infrastructure. The scalability and portability of our Spark infrastructure is critical to our company when we expand our business. In this talk, we will present our story of how we manage our Spark infrastructure at scale.

At peak time, we have 2000+ Spark workers online, and we group these workers into ~50 clusters of various size. The benefits of this, on one hand, is data isolation, which is critical to DataVisor as we are processing multi-customer data. On the other hand, this is for cost and performance consideration, as we want to provide just enough resources to each Spark application. When under-provision, Spark application will fail due to out-of-memory or out-of-disk. However we want to avoid unnecessary over-provision as it dramatically increases our cloud cost.

Next, we will present our DataVisor SparkGenerator (DSG), which is designed to automatically manage our Spark infrastructure. The responsibility of DSG includes (a) launching and shutting down Spark cluster, to maximize concurrency and minimize cost, (b) assigning Spark applications to the proper clusters intelligently, according to the Spark application profile, and (c) managing the dependency among Spark applications, to make our pipeline run smoothly and efficiently, and (d) running all of the Spark worker on Spot instances, reducing the cloud computation cost versus on-demand by over 80%.


Popular Posts

Intelligent solutions. Informed decisions. Unrivaled results.

DataVisor Fraud Index Report: Q2 2019

Learn More

Customers online want convenience, ease, and access. Fortunately, your business offers it all. Unfortunately, that’s…

Customers online want convenience, ease, and access. Fortunately, your business offers it all. Unfortunately, that’s what fraudsters want too. To a cyber criminal, those features means vulnerabilities. To bring you the very latest and most actionable insights about where the risks are and what you…

Dumb & Dumber vs Ocean’s 11

Learn More

Understand the range of modern fraud attacks to ensure complete coverage for your organization.

Complex and coordinated fraud attacks that are extensively planned, hard to detect, and highly scalable are the new normal for online platforms. Explore and understand the full spectrum of fraud attacks—from simple to sophisticated—and learn how you can defend against each type in this…

Diagnose and Defeat Application Fraud with the Latest AI-Powered Tools

Learn More

Learn how leading financial institutions are using ML to proactively detect card application fraud.

In this insightful webinar, you’ll explore how organizations are leveraging AI-powered fraud management solutions to get tangible, real-world benefits as they work to proactively detect and defeat sophisticated modern fraud attacks. Plus, you’ll discover strategies for empowering cross-team…


Protect your business, your customers, and your data.

Request Demo