Data management trends in 2021: more money, more cloud
It was a year like no other for data, as organizations of all sizes continued to grapple with the continuing impact of the coronavirus pandemic.
With the proliferation of working from home and the pressing need for organizations to optimize their operations in a resource-constrained environment, data has become the fuel of the digital transformation effort and business survival.
The need for data meant growth for databases of all types in the cloud, an accelerated movement to better utilize cloud data lakes with different technologies and efforts to improve the observability and quality of data.
The demand for data capacity in 2021 has also resulted in unprecedented financial activity, with providers raising funds from private investors to help meet growing demand and opportunities.
Money for data providers and their cloud data lakes
The explosive Snowflake IPO at the end of 2020, sparked interest from the venture capital community in cloud data technology that drove activity in early 2021.
Data lake query engine platform provider Dremio hit the first on Jan.6, raising $ 135 million for its technology that helps organizations more easily query data in data lakes.
Days later Starburst said it raised $ 100 million. Starburst is the main commercial supplier behind the Trino open source data query platform, which before 2021 was known as PrestoSQL.
Dremio and Starburst both raised funds and developed their technologies to enable organizations to use cloud data lakes for data analysis and business intelligence as demand increased throughout the year. year.
No technology vendor has benefited financially from the interest in the cloud data lake market as much as Databricks, which pioneered the concept of a data lakehouse.
The concept of a data lakehouse is a data warehouse that runs on a cloud data lake. Databricks grabbed several funding rounds in 2021, including the biggest venture capital round ever for a data provider, raising $ 1.6 billion in August.
Demand for streaming event data increases
Another big trend that developed in 2021 has been the continued demand for streaming and event data.
The biggest tech in the industry is Apache Kafka, which has led to big results for its main business sponsor, Confluent.
Confluent went public in June, listed on the NASDAQ stock exchange and marking a milestone for growth and demand for what the provider calls moving data.
With Kafka, organizations are looking to make it easier to work with real-time data, which can often be problematic due to scale and complexity.
Open source led by Confluent ksqlDB The project continued to expand in 2021, using Kafka data with data queries.
Meanwhile, the ability to query streaming data is something the Materialize startup is also looking to capitalize on, as the provider creates a cloud-based streaming database that can handle Kafka sources.
Kafka is far from the only open source event streaming technology that has developed in 2021.
Apache Pulsar has also had its share of growth.
In January, open source mainstay DataStax acquired private event streaming provider Kesque, bringing Pulsar-based technology to DataStax’s portfolio. Also among Pulsar’s backers is StreamNative, which raised $ 23 million for its Pulsar-based commercial data streaming technology in September.
Database providers go serverless in the cloud
Database as a Service (DBaaS) as a Trend started before 2021, but has undergone significant changes during the year.
Previously, DBaaS vendors primarily provided database technologies that still required organizations to manage certain aspects of cloud infrastructure, including compute and storage.
What has become increasingly visible in 2021 is the trend towards serverless DBaaS, with fully managed database services that do not require organizations to have fixed cloud deployments with specific amounts of resources. calculation and storage. The promise of serverless DBaaS is that the organization can quickly get started with a service that consumes only the necessary resources.
A myriad of vendors launched serverless DBaaS systems in 2021, including DataStax Astra in February and PlanetScale in May, while CockroachDB serverless and Capella sofa both were released in October.
While on-premises data will continue to exist for years to come, the growth in data lake, DBaaS, and cloud event streaming technologies that occurred in 2021 is a trend that cannot be ignored.
There’s no denying that the cloud in 2021 and beyond will increasingly be the default choice for data efforts.