#community-help

Sanitizing Queries with Special Characters in Typesense

TLDR Daniel asked about sanitizing queries containing special characters in Typesense. Kishore Nallan recommended using backticks for literal string values and fixing the schema for field names with special characters, sticking to alphanumeric and underscores.

Powered by Struct AI
white_check_mark1
13
5mo
Solved
Join the chat
Apr 25, 2023 (5 months ago)
Daniel
Photo of md5-a7605b093448e06efb0a1c150153cad6
Daniel
03:19 PM
We're running into issues where our filter_by properties sometimes contain open parenthesis, but no closing parenthesis (so something like "Cup Size (mm" rather than "Cup Size (mm)"), and this is leading to issues.
Does Typesense have a good way of sanitizing the queries so that characters like this don't become a factor? Or do we have some way of boxing filters so that this isn't an issue?
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
03:23 PM
👋 Wrap string values with backticks to treat as literal. E.g.

title: `Cup size (mm`
Daniel
Photo of md5-a7605b093448e06efb0a1c150153cad6
Daniel
03:24 PM
Ah, thanks! We'll give it a try
03:55
Daniel
03:55 PM
Does it work for the fields, too? We're doing something like
'filter_by': '`descriptors.Cup Size (mm`:=`4 x 5`'

(The field itself is typo'd, but we're finding that we're having issues even if there is both open and close parenthesis)
Image 1 for Does it work for the fields, too? We're doing something like<br><pre><code>'filter_by': '`descriptors.Cup Size (mm`:=`4 x 5`'</code></pre><br>(The field itself is typo'd, but we're finding that we're having issues even if there is both open and close parenthesis)
03:56
Daniel
03:56 PM
This is giving us the following error, if any parentheses are in the filter_by field, from the looks of it
Image 1 for This is giving us the following error, if any parentheses are in the filter_by field, from the looks of it
03:57
Daniel
03:57 PM
If it matters, this is on v0.25, so maybe a quirk of the new version? (and using a wildcard for the facets?)
04:01
Daniel
04:01 PM
If there's limitations on what we can use as field names in documents, we could use some kind of placeholder value (like underscores, etc.) to get around it
04:02
Daniel
04:02 PM
(though obviously it'll be easiest for us to just use what we already have in the collection, but it's not too late to redo things like this)
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
04:24 PM
Sorry, this won't help with field names. You have fix your schema.
04:25
Kishore Nallan
04:25 PM
We have traditionally not validated against field names. We cannot enforce it now because it will break backward compatibility.
Daniel
Photo of md5-a7605b093448e06efb0a1c150153cad6
Daniel
05:20 PM
Ahh ok
Apr 26, 2023 (5 months ago)
Daniel
Photo of md5-a7605b093448e06efb0a1c150153cad6
Daniel
01:29 AM
Kishore Nallan In that case, if there are letters we should avoid, is there a list so that we can work on some kind of converter? (And then something to convert it back to normal text)

I take it that parenthesis are bad, but are things like = and : also an issue?
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
03:09 AM
Best to stick to alphanumeric and underscores.
white_check_mark1