#community-help

Understanding Search Parameter `pre_segmented_query`

TLDR Robert asked about the behavior of the search parameter pre_segmented_query and Kishore Nallan clarified that it results in space-based splitting regardless of token_separators.

Powered by Struct AI
3
11mo
Solved
Join the chat
Nov 19, 2022 (11 months ago)
Robert
Photo of md5-6384d24e1825271b2c37ad8afa24a899
Robert
03:39 AM
Does setting the search parameter pre_segmented_query to true still only split on space even if token_separators was specified at collection creation? Or even with pre_segmented_query set to true will TS still split on the separators specified in token_separators?
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
03:46 AM
Only space based splitting will happen. The idea is that once that flag is used, it's assumed that all logic for tokenization happens outside so we don't want to interfere in any way apart from blindly splitting on space.
Robert
Photo of md5-6384d24e1825271b2c37ad8afa24a899
Robert
03:48 AM
k, thanks.