What’s new in v0.29

Carolina Ferreira -

This article looks at some of the main changes in Meilisearch’s latest release. For more details, read the full changelog on GitHub.

New feature: matching strategy

v0.29 introduces a new search parameter: matchingStrategy. This parameter lets you define, at search time, how Meilisearch should match query terms:

  • last: returns documents that contain all query terms first. Additional results are then retrieved by decreasing number of query terms present in a match: Meilisearch will remove one query term at a time, starting from the end of the query, and include any new matches to the list of results. This has been the behavior since v0.21 and will remain the default strategy.
  • all: only returns documents containing all query terms.

Example:

curl \\
  -X POST '<http://localhost:7700/indexes/movies/search>' \\
  -H 'Content-Type: application/json' \\
  --data-binary '{ "q": "american ninja", "matchingStrategy": "all" }'

With the above request, Meilisearch will only return documents containing both "american" AND "ninja".

We hope this new feature will encourage those still working with v0.20—the last official Meilisearch release using an all matching strategy by default—to update to our latest release and benefit from the new features introduced since then: sorting, geosearch, granular API key rights and many more! If you’re still on v0.20 and don’t think you will upgrade just yet, we’d be happy to know what’s refraining you.

You can test this feature with the matching strategy demo, and learn more about it in the dedicated blog post.

Improvement: indexing performance

We have sped up indexing, especially for the following operations:

  • Facet indexing: we have modified the algorithm to increase the indexing speed of filterable attributes
  • Document deletion and update:
  • We have introduced soft deletion. This is a technique in which a flag is used to mark data as deleted, rendering it inaccessible to all users without actually removing it from the database. Documents are deleted at a later date, before they take up too much space on the device. Thus, the time taken to delete records on disk is postponed and doesn’t delay the indexing process.
  • when reindexing, Meilisearch now detects if the exact same version of the document already exists and, if that is the case, does not index it again

Behavior change: auto-batching

In v0.26, we introduced an experimental feature: auto-batching. When activated, auto-batching automatically combines consecutive document addition requests, significantly speeding up the indexing process.

We are happy to announce auto-batching is now the default indexing behavior and no longer experimental  🥳

This means you don’t need to use the --enable-auto-batching CLI flag to enable it anymore. Using --enable-auto-batching will now throw the following error:

error: Found argument '--enable-auto-batching' which wasn't expected, or isn't valid in this context

	Did you mean '--disable-auto-batching'?

	If you tried to supply `--enable-auto-batching` as a value rather than a flag, use `-- --enable-auto-batching`

USAGE:
    meilisearch --disable-auto-batching

The following auto-batching customization options have also been removed and will throw errors if used, so make sure your application is not using any of them before upgrading to v0.29:

  • --debounce-duration-sec
  • --max-batch-size
  • --max-documents-per-batch

If you hadn't activated the auto-batching feature, you have nothing to do but enjoy the increased indexing speed 🏎️

🧐 You may have noticed it in the error message above: if for some reason you do not want Meilisearch to automatically batch requests, a new CLI flag is available to deactivate this feature: --disable-auto-batching.

👋
We are continuously working on improving our search engine, and your feedback is essential. If you experience any issues with document indexing speed, we would be very grateful if you could let us know in this discussion.

Improvement: filters

We have added two new filter operators: IN and EXISTS

  • The IN operator selects all documents whose specified field contains at least one of the given values
  • The EXISTS operator checks for the existence of a field. Beware that fields with empty or null values still count as existing

Given the following book dataset:

[
  { "id": 1, "title": "Pride and Prejudice", "genres": ["romance"] },
  { "id": 2, "title": "Le Petit Prince", "genres": [] },
  { "id": 3, "title": "Alice In Wonderland" },
  { "id": 4, "title": "The Hobbit", "genres": ["adventure", "fantasy"] },
  { "id": 5, "title": "A Game of Thrones", "genres": ["fantasy"] },
]

The expression genres IN [fantasy, adventure] will return documents 4 and 5.

The expression genres EXISTS will return documents 1, 2, 4, and 5.

Both operators can be combined with the  NOT (!=) operator.

Behavior change: != (NOT) filter operator

The != (NOT) filter operator returns documents that do not satisfy a condition.

Considering the following documents:

{ "id": 1, "product": "T-shirt", "price": 20 }
{ "id": 2, "product": "T-shirt" }
{ "id": 3, "product": "T-shirt", "price": 30 }

NOT price = 20 should return all documents whose price attribute is not equal to 20.

With v0.29, the != (NOT) filter matches documents missing the specified attributes. In the above example, this means Meilisearch returns documents 2 and 3.

Before v0.29, Meilisearch did not include results missing the price attribute. In the above example, v0.28 would have returned only document 3.

You can make sure a filter only returns documents containing the specified attribute by using the != (NOT) operator with the new  EXISTS operator: price EXISTS AND price != 20 will only return document 3.

Improvement: key management

When managing keys, using the * wildcard character in the actions array gives access to all endpoints:

{
      "name": "Default Admin API Key",
      "description": "Use it for anything that is not a search operation. Caution! Do not expose it on a public frontend",
      "key": "380689dd379232519a54d15935750cc7625620a2ea2fc06907cb40ba5b421b6f",
      "uid": "20f7e4c4-612c-4dd1-b783-7934cc038213",
      "actions": ["*"],
      "indexes": ["*"],
      "expiresAt": null,
      "createdAt": "2021-08-11T10:00:00Z",
      "updatedAt": "2021-08-11T10:00:00Z"
 }

Now, thanks to our contributors, it can also be used as a wildcard to express all endpoints for a single particular route 🙌

Instead of having to list each and every single endpoint, you can append the * character to the desired route. For example, documents.* authorizes access to all document-related endpoints.

Improvement: Thai support

Our dear contributor @aFluffyHotdog has introduced optimized support for Thai 🥳

We are always looking for ways to make Meilisearch’s language support even better. The more feedback we get from native speakers, the easier it is for us to understand how to improve performance and search relevancy for those languages.

Want to help us support your language? Contribute! Don’t hesitate to reach out if you need guidance or help; we are looking forward to working with you!

Contributors

Meilisearch could not exist as it does today without the support from our community. You're really, truly great and we can't thank you enough for all the effort, suggestions, comments, and time you dedicate to Meilisearch. This month, we want to send a special thank you to @evpeople, @Mcdostone, @pavo-tusker, @phdavis1027, @ryanrussell, and @VasiliySoldatkin for their help with Meilisearch, @GraDKh for his help with Milli, and @aFluffyHotdog, @matthias-wright, and @mosuka for their help with our tokenizer Charabia.

And that's all, folks! Remember to check out the changelog for the full release notes, and see you next time!