Elastic’s new Search AI Lake enables highly scalable vector search for generative AI apps

Enterprise search technology provider Elastic N.V. is hoping to make the data essential for generative artificial intelligence and security and observability workloads more accessible with the launch of its new Search AI Lake service.

Announced today, Elastic Search AI Lake is a radically different offering from existing Elasticsearch deployments, as it decouples storage from compute – an approach that allows it to scale search across exponentially larger data volumes with rapid query performance for both traditional structured data and unstructured information represented as vectors. This latter capability makes Search AI Lake an especially useful data store for generative AI services, the company believes.

Elastic Search AI Lake is based on the company’s popular Elasticsearch technology, which in turn is built on the open-source Apache Lucene project. That platform is widely used by enterprises to store, search and analyze enormous volumes of data in real time, and sits at the heart of millions of applications globally that demand comprehensive search capabilities.

In addition to powering general search, Elasticsearch also caters to use cases such as threat detection and application observability, providing tools that can visualize networks and monitor their performance. More recently, Elastic launched the Elasticsearch Relevance Engine that merges vector search with its traditional search engine, enabling it to support unstructured data types such as audio, image and video files.

However, those older offerings are hindered by the fact that they couple storage with compute, which creates a barrier to scalability. By decoupling the two elements, that’s no longer a problem, said Elastic Chief Product Officer Ken Exner.

“To meet the requirements of more AI and real-time workloads, it’s clear a new architecture is needed that can handle compute and storage at enterprise speed and scale – not one or the other,” he said. “Search AI Lake pours cold water on traditional data lakes that have tried to fill this need but are simply incapable of handling real-time applications. This new architecture and the serverless projects it powers are precisely what’s needed for the search, observability, and security workloads of tomorrow.”

In an interview with VentureBeat, Elastic Chief Executive Ash Kulkarni said Search AI Lake is built on a very different architecture to traditional data lake offerings such as Databricks Inc.’s Delta Lake and Snowflake Inc.’s cloud data warehouse. Unlike those platforms, it brings its search functionality into the data lake, enabling real-time exploration and queries of the information within, without the need for any predefined schemas.

Kulkarni added that Search AI Lake also offers dense vectors, hybrid search, faceted search and relevance ranking features that are suited to generative AI models and retrieval-augmented generation techniques

Another difference is that the Elastic Search AI Lake doesn’t use traditional data table formats such as Apache Iceberg or Apache Hudi. That’s because those architectures can hinder data exploration, Kulkarni said. When data is put into a data lake table, the user must also add metadata and make it possible to search for that metadata, or it becomes almost impossible to search for it. Search AI Lake, on the other hand, uses the Elastic Common Schema format and relies on the Elasticsearch Query Language, which enables data to be explored in a federated way across Elastic clusters.

With Search AI Lake, Elastic is trying to position itself as the data platform for generative AI models, which can be dramatically enhanced with highly scalable vector search. By supporting these capabilities, large language models can enhance their knowledge by searching for the most recent and relevant data as it’s made available in real time, improving their responses.

Search AI Lake is available as a standalone platform and also powers a new offering called Elastic Cloud Serverless, with both being made available in technology preview from today.

Image: SiliconANGLE/Microsoft Designer