From Batch to Streaming: The Real-time Data Revolution
Our Investment in DeltaStream
In a world where milliseconds matter, data, and especially real-time data, is the lifeblood of the modern enterprise, enabling critical insights and timely actions. DeltaStream is to real-time, what the last generation big data and analytics companies were to batch processing, by harnessing the power of Apache Flink and real-time streaming. So why did we jump on board? Simply put, in our view DeltaStream is set to disrupt the post-modern data processing world, and we’re excited to be part of the journey.
A Market Hungry for Speed and Scale
Real-time data processing is not just a buzzword. The rate at which we generate data – across volume, velocity, and variety – has grown exponentially, with 90% of existing data created in the past two years. According to Statista, the global data creation is expected to reach more than 180 zettabytes by 2025, up from just 33 zettabytes in 2018.
Consider AI and machine learning— currently most AI models are trained on static datasets, processed in large batches over long periods. This method, while effective for use cases like chatbots, becomes insufficient as AI advances into AI agent systems in dynamic environments like autonomous vehicles, robots, or virtual assistants, which require real-time decision-making and interaction. Real-time data streaming allows AI models to update its knowledge continuously, learning from new patterns, feedback, or even contextual shifts as they happen.
This capability is also critical in the financial services— ingesting from various data sources and identifying fraudulent transactions as they occur can save millions and protect customer trust. In the case of high-frequency trading or DeFi, real-time data feeds and decision making are required to adjust interest rates, execute smart contracts and process trades on the fly based on continuous insights.
In the world of digital advertising, real-time data processing is a game-changer for delivering personalized and targeted ads to users. Think about a scenario where a user is browsing an e-commerce website and showing interest in a specific product. With real-time data processing, the advertiser can instantly detect this behavior and push a targeted ad for a related product to the user as they continue browsing social media or another website. This seamless transition from intent to advertisement happens within milliseconds, making the user much more likely to engage with the ad when it is highly relevant to their immediate interest.
DeltaStream: The Complete Stream Processing Platform
While Spark revolutionized big data processing with its efficient batch processing, it falls short when true real-time, low-latency requirements come into play. Spark’s micro-batching approach, where it collects small batches of data before processing, introduces inherent latency, often in the range of seconds. This might be acceptable for some applications, but for applications where even a few seconds of delay is unacceptable, such as fraud detection, autonomous vehicles, or real-time bidding in advertising, the current solution simply doesn’t deliver the speed needed for instant decisions.
Furthermore, sometimes batch processing can be more costly than streaming due to the infrastructure overhead required to handle large volumes of data all at once. This approach also demands significant resources for job scheduling and management. While modern batch systems often include fault-tolerance mechanisms, failures may still lead to partial or full reprocessing, resulting in wasted time and computing power.
DeltaStream Streaming Platform vs. Traditional Batch Processing
What Databricks and Snowflake did for stored data, DeltaStream does for streaming data powered by Apache Flink. Unlike Spark, which needs to pause to gather micro-batches, Flink is designed from the ground up for real-time stream processing by handling data in smaller, manageable chunks as soon as it arrives. As a result, it offers latency in the range of milliseconds. This stark difference makes Flink the superior and popular choice for high-frequency event data where every second counts, and the industry has taken note of this—reflected in Flink’s growing GitHub Stars and adoption across industries that rely on speed and precision.
DeltaStream takes the power of Flink and wraps it in an accessible, extensible, and enterprise-grade platform that integrates seamlessly with existing data ecosystems. It's designed to simplify complex configurations, allowing companies to deploy and manage real-time data pipelines effortlessly without the overhead of building and maintaining the infrastructure themselves. It allows entrepreneurs and managers to free up the engineering resources so that the development and business teams can focus their time on building business logics and applications that generate direct commercial impact.
A Winning Team with a Proven Track Record
Behind DeltaStream is a team of industry rockstars from the post-modern data and compute world, including Hojjat Jafarpour, the brain behind ksqlDB for Kafka, and Krishna Raman, a co-creator of OpenShift, a popular Kubernetes-based open-source platform that offers developers a streamlined experience for container orchestration. These are the people who don’t just follow trends—they set them. With such expertise, we’re confident that DeltaStream is not only capable of delivering on its promises but also of leading the next wave of innovation in data streaming.
DeltaStream's platform already boasts blue-chip clients who could potentially process up to thousands of Flink jobs, which uses DeltaStream to streamline operations and optimize performance. These kinds of relationships illustrate the significant potential for a land-and-expand model, where initial use cases can quickly grow into broader applications within an organization. By focusing on mission-critical tasks, DeltaStream positions itself to unlock substantial long-term value from its business partners, driving deeper integration and more significant partnerships over time.
Riding the Real-Time Revolution
DeltaStream fits perfectly into Galaxy Interactive’s investment thesis of supporting underlying infrastructure that form the backbone of the post-modern Distributed Computing Stack – from Data, Compute, to Tooling and Applications.
How DeltaStream Fits Into Our Tech Stack Thesis
As the demand for real-time data, insights and actions continues to grow, companies that can harness the power of real-time data will be the ones to thrive. DeltaStream is at the forefront of this transformation, and we’re proud to support them as they lead the way.
*DeltaStream is a Galaxy Interactive portfolio company.
Legal Disclosure:
This document, and the information contained herein, has been provided to you by Galaxy Digital Holdings LP and its affiliates (“Galaxy Digital”) solely for informational purposes. This document may not be reproduced or redistributed in whole or in part, in any format, without the express written approval of Galaxy Digital. Neither the information, nor any opinion contained in this document, constitutes an offer to buy or sell, or a solicitation of an offer to buy or sell, any advisory services, securities, futures, options or other financial instruments or to participate in any advisory services or trading strategy. Nothing contained in this document constitutes investment, legal or tax advice or is an endorsementof any of the digital assets or companies mentioned herein. You should make your own investigations and evaluations of the information herein. Any decisions based on information contained in this document are the sole responsibility of the reader. Certain statements in this document reflect Galaxy Digital’s views, estimates, opinions or predictions (which may be based on proprietary models and assumptions, including, in particular, Galaxy Digital’s views on the current and future market for certain digital assets), and there is no guarantee that these views, estimates, opinions or predictions are currently accurate or that they will be ultimately realized. To the extent these assumptions or models are not correct or circumstances change, the actual performance may vary substantially from, and be less than, the estimates included herein. None of Galaxy Digital nor any of its affiliates, shareholders, partners, members, directors, officers, management, employees or representatives makes any representation or warranty, express or implied, as to the accuracy or completeness of any of the information or any other information (whether communicated in written or oral form) transmitted or made available to you. Each of the aforementioned parties expressly disclaims any and all liability relating to or resulting from the use of this information. Certain information contained herein (including financial information) has been obtained from published and non-published sources. Such information has not been independently verified by Galaxy Digital and, Galaxy Digital, does not assume responsibility for the accuracy of such information. Affiliates of Galaxy Digital may have owned or may own investments in some of the digital assets and protocols discussed in this document. Except where otherwise indicated, the information in this document is based on matters as they exist as of the date of preparation and not as of any future date, and will not be updated or otherwise revised to reflect information that subsequently becomes available, or circumstances existing or changes occurring after the date hereof. This document provides links to other Websites that we think might be of interest to you. Please note that when you click on one of these links, you may be moving to a provider’s website that is not associated with Galaxy Digital. These linked sites and their providers are not controlled by us, and we are not responsible for the contents or the proper operation of any linked site. The inclusion of any link does not imply our endorsement or our adoption of the statements therein. We encourage you to read the terms of use and privacy statements of these linked sites as their policies may differ from ours. The foregoing does not constitute a “research report” as defined by FINRA Rule 2241 or a “debt research report” as defined by FINRA Rule 2242 and was not prepared by Galaxy Digital Partners LLC. For all inquiries, please email [email protected]. ©Copyright Galaxy Digital Holdings LP 2024. All rights reserved.