Qubole, the open date lake company, today launched Qubole Pipelines Service, a new offering that makes it much easier and faster for customers to build robust, scalable streaming data pipelines and capitalize on the rapid growth of real-time data in their businesses. With Qubole Pipelines Service, data teams can now build, test, deploy, monitor and manage hundreds of streaming data pipelines from a single platform, resulting in increased productivity, greater innovation and reduced operating costs.
By 2022, more than half of all new major business systems will adopt continuous intelligence systems that use real-time data in order to drive important business decisions, according to Gartner. As data-driven innovations such as AI/ML and Internet of Things (IoT) increasingly drive competitive advantage for businesses, troves of streaming data are continuously generated from multiple internal and external sources. A rapidly growing number of organizations need a solution that makes it faster and easier for developers to unlock the value of streaming data, and Qubole Pipelines Service fully addresses this need.
With Qubole Pipelines Service, businesses can complement their existing data lake with advanced features that help them instantly capture streaming data from various sources, accelerate development of streaming applications and run highly reliable and observable production applications at the lowest cost. All in a managed environment via the public cloud of their choice. Salient new features include:
● Accelerated Development Cycle: Numerous built-in connectors, code generation wizard, dry run framework, and quick-start options that help accelerate development lifecycles. A pipeline can be developed within minutes without writing even a single line of code and be deployed instantly.
● Robust and Cost-Efficient Stream Processing Engine: Leveraging Apache Spark Structured Streaming, Qubole added several enhancements including Rocksdb state storage, direct writes and memory pressure scheduling, among others for reliably building and deploying long-running streaming applications.
● Comprehensive Operational Management: Qubole Pipelines Service includes a broad set of APIs and user interfaces for engineers to holistically manage the lifecycle of streaming applications and get continuous operational insights.
● Data Management and Consistency: The new Pipelines service uses Qubole’s ACID framework to efficiently compact small files in the background while allowing concurrent read / write operations, without impacting performance..
“Today, organizations continue to struggle with stream processing at scale, and we found that they needed a comprehensive solution that would tackle the most complex pain points around reliability, scalability and cost efficiencies,” said Joydeep Sarma, CTO and co-founder at Qubole. “Through our extensive discussions with Qubole’s customer and partner ecosystems, we knew that solving these problems was a large undertaking. The arrival of Pipelines Service not only equips data teams and engineers with the most comprehensive solution to quickly build streaming data pipelines and analyze massive streams of data And it also underscores Qubole’s longstanding tradition of providing businesses with an open data lake platform for batch and streaming analytics.”
“MiQ has developed a series of predictive targeting solutions that allow our customers to gather large volumes of data about real-world events, generate insights, and then take appropriate actions—all in real time,” said Rohit Srivastava, Engineering Manager at MiQ. “We use Qubole Pipelines Service to build these solutions because it provides our teams with the ease of use and scalability they need to quickly build and deploy the high-performance targeting applications our customers have come to expect.”
