Full-text search vs vector search

Navigating today’s ocean of information can be both a marvel and a maze. Full-text and vector search enable us to build search experiences empowering users to find relevant products, content, and more. As our quest for precision and context in search evolves, a question arises: can we balance the lexical flexibility of full-text search with the semantic depth of vector search?

Let’s explore together the pros and cons of each solution and discover the synergies that are redefining modern search & discovery experiences.

Full-text search

Full-text search refers to matching some or all of a text query with documents stored in a database. Compared to traditional database queries, full-text search provides results even in case of partial matches. It allows building more flexible search interfaces for users, thus enabling them to find accurate results more quickly.

In practice, an efficient full-text search solution comes with tolerance to typos, synonyms, prefix search, and fuzzy matching. These features enable end users to find what they’re looking for even when the information at their disposal is limited. Allowing searches with incomplete information makes discovery faster.

Although modern SQL and NoSQL databases come with some full-text search capabilities, they often fall short compared to search-focused databases. For a deeper dive on this topic, consider reading on areas where Postgres falls short for full-text search.

Vector search

Vector search is the AI-powered approach to search. Instead of looking for documents matching a text query, it allows finding documents that have a similar semantic meaning. This is achieved by building upon the semantic understanding of text that Large Language Models (LLMs) provide.

LLMs can process database records and generate vector embeddings—the numeric representation of the semantic meaning of documents. These embeddings are then stored in vector databases. As the name suggests, vector search works by comparing similar vectors.

By performing vector comparisons, vector search effectively enables semantic search. Despite relying on different underlying technologies, it also solves a similar purpose as full-text search: empowering users by improving the discovery aspect of search. It notably enables multi-modal search—searching with non-textual input, like in Google Images.

Comparative analysis: full-text search vs vector search

Full-text search works by finding words in documents. That makes it good at understanding the lexical intent behind a user query. But it falls short when it has to work with vague queries that cannot be fuzzy-matched. The lack of semantic understanding of documents makes it unable to understand context and make generalizations.

To summarize, full-text search:

• ✅ Corrects typos — e.g. reutrn of the jedi
• ✅ Handles precise queries — e.g. an exact product name
• ✅ Handles incomplete queries — e.g. return of the j
• ❌ Cannot handle vague queries — e.g. folks fighting with lightsabers
• ❌ Cannot understand context — e.g. winter clothes

On the other hand, vector search works by finding documents with similar meanings. That makes it good at understanding the semantic intent behind a user query. It’s great at understanding synonyms and descriptions of documents. It’s also adapted to the discovery of similar documents.

To summarize, vector search:

• ✅ Handles vague queries — e.g. first released star wars movie
• ✅ Understands context — e.g. winter clothes
• ✅ Can suggest similar documents
• ❌ Cannot handle precise queries
• 👎 Requires more resources

As we can see, neither solution is perfect. They each come with clear advantages for given use cases. They complement each other. Building state-of-the-art search experiences would require combining the benefits of both full-text and vector search.

Indeed, can we have both?

The case for hybrid search

Hybrid search combines the strengths of full-text search and vector search. It builds upon the accessible, search-as-you-type experience of full-text search and integrates the enhanced discovery capabilities that AI search enables.

Solutions like Meilisearch’s hybrid search bring the best of both worlds without compromise. This allows developers to use a unified API for all types of searches. The code sample below demonstrates a usage of such API.

// Hybrid search with Meilisearch

const results = await client.multiSearch({
	queries: [{ 
		indexUid: 'movies', 
		q: 'batman',
		hybrid: { embedder: 'default', semanticRatio: 0.5 }
	}]
})

This code uses a Meilisearch client to search for the batman query in the movies index. It uses hybrid search, which is configured via the hybrid parameter. Let’s review the fields in the hybrid object and see what they enable.

The embedder field allows choosing which embedder will process the query (Meilisearch allows configuring multiple embedders for a given index.)

The semanticRatio field allows us to balance the importance of results coming from semantic search. A higher value will allow documents that are further distanced in the semantic space to make it to the final search results.

The semanticRatio value works as follows:

• Semantic ratio = 0 — it’s a full-text search
• Semantic ratio = 1 — it’s a vector search
• 0 < semantic ratio < 1 — it’s a hybrid search ✨

As semantic search will always return some documents, it’s important to strike a good balance to keep the overall search results relevant. The demo below allows you to search for movies and play with the semantic ratio to see how it affects results.

Hybrid search is coming to Meilisearch Cloud, join the waitlist:

Conclusion

As engineers will tell you, the question of “What is the best tool for X?” is always: it depends. On their own, neither full-text search nor vector search can meet all the criteria to build a fast, relevant search experience. Hybrid search merges the accessibility of full-text search with improved discovery that AI enables.

Hybrid search is the modern approach to search, unifying behind a single API the most advanced search capabilities. This hybrid approach blends full-text search with semantic search, enhancing both the accuracy and comprehensiveness of search results. It is a modern, flexible solution adapted to today’s search use cases.

Meilisearch is an open-source search engine enabling developers to build state-of-the-art experiences while enjoying simple, intuitive DX. 

For more things Meilisearch, you can join the community on Discord or subscribe to the newsletter. You can learn more about the product by checking out its roadmap and participating in product discussions.