#community-help

Discussing Typesense's Tokenization Feature

TLDR Roshan seeks to understand typesense's tokenization feature. Kishore Nallan explains that it tokenizes on spaces and suggests using a special character as a separator.

21mo

Join the chat

Feb 18, 2022 (21 months ago)

Photo of md5-34813b759bbed85074c3a458c4a2a053

Roshan

11:06 AM

Kishore Nallan Kishore Akash Ankur Does typesense has tokenization feature? I want to split search query into tokens(for grouping of text etc)

11:07

Roshan

11:07 AM

If it doesn't have this feature natively, how can we use tokenization while querying and indexing?

Photo of md5-4e872368b2b2668460205b409e95c2ea

Kishore Nallan

11:53 AM

Can you give me an example of tokenization you wish to do?

11:54

Kishore Nallan

11:54 AM

Typesense tokenizes on space and also if you define custom separators, it considers those as well.

Roshan

12:09 PM

ok, like if I want to tokenize "Hello word everyone"
text, then I want Hello world to be one and everyone to be another token Kishore Nallan

Kishore Nallan

12:10 PM

We don't have a way to customize that behavior. You can always choose to tokenize before indexing by combining words using a symbol like "hello_world" and likewise do the same with the query as well.

Roshan

12:16 PM

ok, suppose I have my own tokenizer , then how can i pass those tokenized list of words while indexing to typesense? and how it will work on search? Kishore Nallan

Kishore Nallan

12:29 PM

You've to use a special character as a separator rather than space. Then add that character as symbols_to_index configuration.

Typesense

Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI

Indexed 2779 threads (79% resolved)

Join Our Community Manage subscription

Similar Threads

Resolving Typesense Search Issues

Conversation started by Maximilian about Typesense search behavior led to Users Kishore Nallan and Mike discussing and suggesting workaround, with Kishore Nallan promising an official solution soon. No final confirmation of resolution provided.

19mo

Restricting `token_separators` to a Specific Field in Typesense

Loic asked Jason about applying `token_separators` to a specific field in Typesense. Jason suggested opening a github issue to add this feature.

15mo

Tokenization and Indexing Fields with Typesense

kam wanted to understand how to control tokenization and indexing for certain fields. Jason explained that tokenization is applied during search queries and not during the indexing phase, and shared how to delete a document using an indexed unique value under `id`.

1mo

Handling Two-Word Queries with Custom Separators

Dima proposes adding a parameter to API for handling two-word queries. Jason suggests opening a GitHub issue for the feature request.

3mo

Issues with Repeated Words and Hyphen Queries in Typesense API

JinW discusses issues with repeated word queries and hyphen-containing queries in Typesense. Kishore Nallan offers possible solutions. During the discussion, Mr seeks advice on `token_separators` and how to send custom headers. Issues remain with repeated word queries.

23mo