#community-help

Special Character Impact on Indexing and Querying

TLDR Sergio suspects a special character may be affecting Typesense indexing and querying. Kishore Nallan requests a code snippet and advises creating an issue on GitHub.

7mo

Join the chat

Mar 01, 2023 (7 months ago)

Photo of md5-19856b8e92142bdd0747d7a3706736c8

Sergio

01:47 PM

Hi Team, could it be that a "special character" is messing up indexing or querying?
Although documentation mention that "Typesense will remove special characters", could be that this special character got through, and querying was returning empty values.

01:47

Sergio

01:47 PM

Typesense Docs
https://typesense.org/docs/guide/tips-for-searching-common-types-of-data.html#default-behavior-2

01:50

Sergio

01:50 PM

We indexed Apple TV 128GB 4K as a string which contained some weird split between Apple and TV. (Not visible in Slack).
And later querying by Apple TV the document was not being returned.

Image 1 for We indexed `Apple TV 128GB 4K` as a string which contained some weird split between _Apple_ and _TV_. (Not visible in Slack).
And later querying by Apple TV the document was not being returned.

Photo of md5-4e872368b2b2668460205b409e95c2ea

Kishore Nallan

01:55 PM

Interesting, is it possible for you to post a code snippet that I can run to index that data?

Sergio

02:14 PM

02:15

Sergio

02:15 PM

Pasting the code here seems to clean it 😕

02:16

Sergio

02:16 PM

Maybe it's a edge case

Kishore Nallan

02:16 PM

Ok we might have to handle this explicitly, can you please create an issue on GH?

Sergio

02:17 PM

Do you have where do you sanitize the input? Maybe I can take a look

Kishore Nallan

02:19 PM

I Just checked, we explicitly split on space and newline. Because non visible white space is not differentiated from regular unicode which we can't split on because that's valid.

Typesense

Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI

Indexed 2779 threads (79% resolved)

Join Our Community Manage subscription

Similar Threads

Issue with Typesense Search and Special Characters

gab faced an issue with Typesense when searching with special characters. Jason and Kishore Nallan provided support and identified the issue as a bug. The problem was resolved after upgrading to `0.25.0.rc44`.

3mo

Modification of Special Characters Indexing in Typesense

A inquired about a Typesense update concerning search/input of specialized data. Kishore Nallan reported the update as priority and upcoming.

27mo

Handling Special Characters in Search Queries

Sidharth asks how to handle special characters in search queries. Kishore Nallan suggests using `symbols_to_index` configuration and provides an example to help.

4mo

Revisiting Typesense for Efficient DB Indexing and Querying

kopach experienced slow indexing and crashes with Typesense. The community suggested to use batch import and check the server's resources. Improvements were made but additional support was needed for special characters and multi-search queries.

7mo

Phrase Search Relevancy and Weights Fix

Jan reported an issue with phrase search relevancy using Typesense Instantsearch Adapter. The problem occurred when searching phrases with double quotes. The team identified the issue to be related to weights and implemented a fix, improving the search results.

111

6mo