Resolving Special Character Search Errors
TLDR suraj had issues searching for data containing special characters. Kishore Nallan resolved the issue by advising suraj to remove the parameters for 'preSegmentedQuery' and 'tokenSeparators'.
Powered by Struct AI
1
48
3mo
Solved
Jun 22, 2023 (3 months ago)
suraj
suraj
07:22 AMHello Team,
I have trouble while searching for data which contains the speical character like &.
below is my snap shot dummy json data:
{"company_id":"11609817","parent_id":"365","track_id":"1","security_id":"12",
"symbol":"Ram&Madhu","company_nm":"Mahindra & Mahindra","short_name":"M&M"
Whenever I search this data it gives me bad request error, intially when I created json file special character are encoded into UTF-8
still that was not coming into search.
Can anybody please suggest a way here.
I have trouble while searching for data which contains the speical character like &.
below is my snap shot dummy json data:
{"company_id":"11609817","parent_id":"365","track_id":"1","security_id":"12",
"symbol":"Ram&Madhu","company_nm":"Mahindra & Mahindra","short_name":"M&M"
Whenever I search this data it gives me bad request error, intially when I created json file special character are encoded into UTF-8
still that was not coming into search.
Can anybody please suggest a way here.
Jason
Jason
04:29 PMCould you share a set of curl commands like this that replicates the issue: https://gist.github.com/jasonbosco/7c3432713216c378472f13e72246f46b
04:29
Jason
04:29 PMI suspect it might be because of URL encoding
Jun 23, 2023 (3 months ago)
suraj
suraj
04:24 AMSo while creating batch document I should use text/plain, is that right?
Kishore Nallan
Kishore Nallan
06:22 AMHi Suraj, if you can just post a snippet that i can run to see the actual error, it would be easy for me to figure out what's going wrong.
suraj
suraj
11:49 AMHi Kishor..
class SearchParameters {
q: m&m
queryBy: security_nm,symbol,brand.brandName
queryByWeights: 3,2,1
prefix: true,true,true
infix: null
maxExtraPrefix: null
maxExtraSuffix: null
filterBy: null
sortBy: null
facetBy: null
maxFacetValues: null
facetQuery: null
numTypos: null
page: null
perPage: 100
groupBy: null
groupLimit: null
includeFields: null
excludeFields: null
highlightFullFields: null
highlightAffixNumTokens: null
highlightStartTag: null
highlightEndTag: null
snippetThreshold: null
dropTokensThreshold: null
typoTokensThreshold: null
pinnedHits: null
hiddenHits: null
highlightFields: null
splitJoinTokens: null
preSegmentedQuery: true
enableOverrides: null
prioritizeExactMatch: false
maxCandidates: null
prioritizeTokenPosition: null
exhaustiveSearch: null
searchCutoffMs: null
useCache: null
cacheTtl: null
minLen1typo: null
minLen2typo: null
}
class SearchParameters {
q: m&m
queryBy: security_nm,symbol,brand.brandName
queryByWeights: 3,2,1
prefix: true,true,true
infix: null
maxExtraPrefix: null
maxExtraSuffix: null
filterBy: null
sortBy: null
facetBy: null
maxFacetValues: null
facetQuery: null
numTypos: null
page: null
perPage: 100
groupBy: null
groupLimit: null
includeFields: null
excludeFields: null
highlightFullFields: null
highlightAffixNumTokens: null
highlightStartTag: null
highlightEndTag: null
snippetThreshold: null
dropTokensThreshold: null
typoTokensThreshold: null
pinnedHits: null
hiddenHits: null
highlightFields: null
splitJoinTokens: null
preSegmentedQuery: true
enableOverrides: null
prioritizeExactMatch: false
maxCandidates: null
prioritizeTokenPosition: null
exhaustiveSearch: null
searchCutoffMs: null
useCache: null
cacheTtl: null
minLen1typo: null
minLen2typo: null
}
11:50
suraj
11:50 AMcurl --location --request GET 'http://localhost:8080/search/M & M'
<!doctype html><html lang="en"><head><title>HTTP Status 400 โ Bad Request</title><style type="text/css">body {font-family:Tahoma,Arial,sans-serif;} h1, h2, h3, b {color:white;background-color:#525D76;} h1 {font-size:22px;} h2 {font-size:16px;} h3 {font-size:14px;} p {font-size:12px;} a {color:black;} .line {height:1px;background-color:#525D76;border:none;}</style></head><body><h1>HTTP Status 400 โ Bad Request</h1></body></html>
Kishore Nallan FYR
<!doctype html><html lang="en"><head><title>HTTP Status 400 โ Bad Request</title><style type="text/css">body {font-family:Tahoma,Arial,sans-serif;} h1, h2, h3, b {color:white;background-color:#525D76;} h1 {font-size:22px;} h2 {font-size:16px;} h3 {font-size:14px;} p {font-size:12px;} a {color:black;} .line {height:1px;background-color:#525D76;border:none;}</style></head><body><h1>HTTP Status 400 โ Bad Request</h1></body></html>
Kishore Nallan FYR
Jun 26, 2023 (3 months ago)
suraj
suraj
04:14 AM Jason Kishore Nallan Can you please suggest a way here
Kishore Nallan
Kishore Nallan
10:51 AMThe HTML response above is from your web server? I need to see the actual response from the Typesense API.
10:51
Kishore Nallan
10:51 AMIf you are using the GET search method, you might also want to try the using POST
/multi_search end-point.suraj
suraj
12:07 PMNow I am not getting any error as of now but with & it is not giving me result. it is sending me empty result
12:07
suraj
12:07 PMfor query=m&m
12:08
suraj
12:08 PMfor this json data:{"company_id":"11609817","parent_id":"365","track_id":"1","security_id":"12",
"symbol":"Ram&Madhu","company_nm":"Mahindra & Mahindra","short_name":"M&M"
"symbol":"Ram&Madhu","company_nm":"Mahindra & Mahindra","short_name":"M&M"
12:08
suraj
12:08 PM Kishore Nallan
03:46
suraj
03:46 PMI have used tokenSeparators and symbolsToIndex as well, providing special character but still it sdoes not search with keyword like M&M
Kishore Nallan
Kishore Nallan
03:50 PMWhen I tried locally on a small sample set, querying with
Please share a sample dataset so I can try it, for e.g. like this (as Jason shared earlier): https://gist.github.com/jasonbosco/7c3432713216c378472f13e72246f46b
m&m does fetch that record for me.Please share a sample dataset so I can try it, for e.g. like this (as Jason shared earlier): https://gist.github.com/jasonbosco/7c3432713216c378472f13e72246f46b
Jun 27, 2023 (3 months ago)
suraj
suraj
06:26 AMThis is my jsol file , below is data
06:26
suraj
06:26 AM06:27
suraj
06:27 AMThis is how I am creating collection in java
06:29
suraj
06:29 AM06:30
suraj
06:30 AMSo, when I search data for M&M, it return me blank data
06:30
suraj
06:30 AM Kishore Nallan
Kishore Nallan
Kishore Nallan
06:34 AMPlease post the search Java snippet also. Again, you need to post a fully reproduceable example so I run and check. Otherwise it's hard for me to help. Please post a full program in Java client that creates collection, indexes document and queries that show cases the issue.
suraj
suraj
06:39 AM06:40
suraj
06:40 AMHave added code for collection create, document insert and search, please check the same Kishore Nallan, let lme know in case anything else is required
Kishore Nallan
Kishore Nallan
06:41 AMWill check and get back to you in an hour or so.
suraj
suraj
06:42 AMThanks Kishore Nallan for your quick response, appreciate๐
Kishore Nallan
Kishore Nallan
11:55 AM suraj That M&M document is not even getting indexed. If you check the response of the import call, you will see this error:
Field `security` has been declared in the schema, but is not found in the document.11:56
Kishore Nallan
11:56 AMThe import function always returns true because only some of the documents could fail (like in this case), so you have to check the response.
suraj
suraj
02:29 PM Kishore Nallan below is the right jsonl file, on which collection is created
{"company_id":"3005","parent_id":"16795","exchange_id":"2","security":"","symbol":"M&MFIN","company_nm":"Mahindra & Mahindra Financial Services Ltd","short_company_nm":"M & M Fin. Serv.","code":"INE774D0102","series":"","brand":[{"heading":"Financial Services","brandName":"MAHINDRA FINANCE FINSMART"}]}
{"company_id":"3004","parent_id":"27242","exchange_id":"2","security":"","symbol":"BLKASHYAP","company_nm":"B.L.Kashyap & Sons Ltd","short_company_nm":"B.L.Kashyap","code":"INE350H1032","series":"","brand":[{"heading":"Construction & Civil Engineering","brandName":"BLK"}]}
{"company_id":"4354","parent_id":"28","exchange_id":"2","security":"","symbol":"ARUNAHTEL","company_nm":"Aruna Hotels Ltd","short_company_nm":"Aruna Hotels","code":"INE95701019","series":"","brand":[{"heading":"HotelBusiness","brandName":"ARUNA"}]}02:29
suraj
02:29 PMI am able to search with other symbol like BLKASHYAP
Jun 28, 2023 (3 months ago)
suraj
suraj
09:03 AMWhen I search keyword like Mahindra , I am getting record of M&MFIN as well, hence it is stored in collection properly, only when i search M&M, id do not work Kishore Nallan
Kishore Nallan
Kishore Nallan
09:08 AMI'm yet to look. Since I juggle supporting multiple customers it's always useful if you can share a fully reproduceable example that I can just run in one go. When you share snippets like above I still need to put it together into a Java program and run it and test it. That takes a lot of my time.
suraj
suraj
09:09 AMI Understand, let me put the entire code base here then
Kishore Nallan
Kishore Nallan
09:10 AMJust share a single Java program in static void main that we both can run and get the exact results.
09:10
Kishore Nallan
09:10 AMIt will prevent issues like earlier where the documents didn't even get indexed properly .
suraj
suraj
09:10 AMOkay got it..
09:15
suraj
09:15 AMjava code and jsonl file
09:16
suraj
09:16 AM09:17
suraj
09:17 AMhave added this line query =URLEncoder.encode(query, StandardCharsets.UTF_8.toString()); in code just to check if with this special character is working or not
Kishore Nallan
Kishore Nallan
09:20 AMAgain you are adding several snippets instead of a single program which creates, collection, indexes the docs, queries and prints the result. ๐
suraj
suraj
09:38 AMWhole code.. jsonl file already sent to you
09:39
suraj
09:39 AM Kishore Nallan Have sent you consolidated code.. Please have a look.. jsonl file is already sent
Kishore Nallan
Kishore Nallan
09:40 AMThanks will get back to you
1
10:45
Kishore Nallan
10:45 AMRemove
preSegmentedQuery param and it should work.10:50
Kishore Nallan
10:50 AMAlso you probably don't need
tokenSeparators during collection creation as well.suraj
suraj
01:59 PM Kishore Nallan After removing
tokenSeparators and preSegmentedQuery *it is wokring fine now.*Thank you for for help.:pray:Kishore Nallan
Kishore Nallan
02:01 PMWelcome!
Typesense
Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI
Indexed 2776 threads (79% resolved)
Similar Threads
Cold Start Problem with Dynamic Collections
Adrian reported cold start issues with dynamic collections. Jason suggested using wildcard `*` for query_by parameters, upgrading to `0.25.0.rc34`, and clarified conventions. Adrian's issues were resolved but they reported a limitation that will potentially be addressed.
39
4mo
Solved
Large JSONL Documents Import Issue & Resolution
Suraj was having trouble loading large JSONL documents into Typesense server. After several discussions and attempts, it was discovered that the issue was due to data quality. Once the team extracted the data again, the upload process worked smoothly.
4
94
6mo
Solved
Troubleshooting Issues with DocSearch Hits and Scraper Configuration
Rubai encountered issues with search result priorities and ellipsis. Jason helped debug the issue and suggested using different versions of typesense-docsearch.js, updating initialization parameters, and running the scraper on a Linux-based environment. The issues related to hits structure and scraper configuration were resolved.
131
6mo
Solved
Utilizing Vector Search and Word Embeddings for Comprehensive Search in Typesense
Bill sought clarification on using vector search with multiple word embeddings in Typesense and using them instead of OpenAI's embedding. Kishore Nallan and Jason informed him that their development version 0.25 supports open source embedding models. They also resolved Bill's concerns regarding search performance, language support, and limitations in the search parameters.
225
2mo
Solved
Querying and Indexing Multiple Elements Issues
Krish queried fields with multiple elements, which Kishore Nallan suggested checking `drop_tokens_threshold`. Krish wished to force OR mode for token, but Kishore Nallan admitted the feature was missing. Krish was able to resolve the issue with url encoding.
34
10mo
Solved