Hello <@U01PL2YSG8L> <@U01NZ4D0LDD> - Good Morning...
# community-help
r
Hello @Kishore Nallan @Jason Bosco - Good Morning/Evening. Currently I'm facing an issue when trying to filter documents using Starts With (ID_KEY:=CS*) along with other filters (STATUS:=[
ACTIVE
] && ARCHIVED:=[
NO
]); I've tried with and without max_filter_by_candidates parameter; Still no luck; But, If I've filetered with Starts with (ID_KEY:=CS*) and max_filter_by_candidates then it returns records; If I've filtered with (STATUS:=[
ACTIVE
] && ARCHIVED:=[
NO
]) alone then also I'm getting the reccords When I've combined both, I'm not getting the expected results; But, I'm sure that, we have records for the given condition. Tested with Version 29.0 Please let me know if I've missing something here; Your advice on this would be really appreciated. Looking forward to hear from you. Here is the sample payload.
Copy code
{
  "searches": [
    {
      "query_by": "*",
      "sort_by": "ID_KEY:asc",
      "highlight_full_fields": "*",
      "collection": "CASE_DOCS",
      "q": "*",
      "filter_by": "STATUS:=[`ACTIVE`] && ARCHIVED:=[`NO`] && ID_KEY:=CS*",
      "max_filter_by_candidates": 10000,
      "page": 1,
      "per_page": 10
    }
  ]
}
h
Hi, Could you post a small reproducible example like this?
r
Hello @Hayden Hoang - Thank you for your response. I'll prepare and share the sample soon. During further investigation, I've created sample dataset with 20 records; Applied Filter: "filter_by": "STATUS:=[
ACTIVE
] && ARCHIVED:=[
NO
] && ID_KEY:=CS*" Expected Hits Count : 6 1. Without passing max_filter_by_candidates, I'm getting the Hits count as 1 2. max_filter_by_candidates 5, I'm getting the Hits count as 1 3. max_filter_by_candidates 8, I'm getting the Hits count as 2 4. max_filter_by_candidates 10, I'm getting the Hits count as 3 5. max_filter_by_candidates 15, I'm getting the Hits count as 4 6. max_filter_by_candidates 19, I'm getting the Hits count as 5 7. max_filter_by_candidates 20 and above, I'm getting the Hits count as 6 It appears to be max_filter_by_candidates playing critical role on this. Any thoughts ? @Kishore Nallan @Jason Bosco - Please note.
Hello @Hayden Hoang @Kishore Nallan @Jason Bosco - FYI: Herewith I've provided the reproducible sample; When I ran with 500K dataset for the given filter, I'm not getting the expected results; Please review and advise. Thank You.
Copy code
### Run Typesense via Docker ########################################
set -x

export TYPESENSE_API_KEY=xyz
export TYPESENSE_HOST=http://localhost:8108

docker stop typesense-repro 2>/dev/null
docker rm typesense-repro 2>/dev/null
rm -rf "$(pwd)"/typesense-data-dir-repro
mkdir "$(pwd)"/typesense-data-dir-repro

# Run Typesense
docker run -d -p 8108:8108 --name typesense-repro \
  -v"$(pwd)"/typesense-data-dir-repro:/data \
  typesense/typesense:29.0.rc31 \
  --data-dir /data \
  --api-key=$TYPESENSE_API_KEY \
  --enable-cors

# Wait till Typesense is ready
until curl -s -o /dev/null -w "%{http_code}" \
  "$TYPESENSE_HOST/health" \
  -H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}" | grep -q "200"; do
  sleep 2
done

curl -s "$TYPESENSE_HOST/debug" \
  -H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}" | jq


### Create CASE_DOCS collection ###############################
curl -s "$TYPESENSE_HOST/collections" \
  -X POST \
  -H "Content-Type: application/json" \
  -H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}" \
  -d '
  {
    "name": "CASE_DOCS",
    "fields": [
      { "name": "ID_KEY", "type": "string" },
      { "name": "STATUS", "type": "string", "facet": true },
      { "name": "ARCHIVED", "type": "string", "facet": true },
      { "name": "CASE_TITLE", "type": "string" },
      { "name": "DESCRIPTION", "type": "string" },
      { "name": "PRIORITY", "type": "string", "facet": true },
      { "name": "OWNER", "type": "string", "facet": true },
      { "name": "CREATED_AT", "type": "int64", "facet": true }
    ],
    "default_sorting_field": "CREATED_AT"
  }' | jq


### Import sample CASE_DOCS documents #########################
curl -s "$TYPESENSE_HOST/collections/CASE_DOCS/documents/import?action=create" \
  -H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}" \
  -H "Content-Type: text/plain" \
  -X POST \
  -d '
{"ID_KEY":"CS200001","STATUS":"ACTIVE","ARCHIVED":"NO","CASE_TITLE":"Login Failure","DESCRIPTION":"User cannot log in.","PRIORITY":"HIGH","OWNER":"agent_01","CREATED_AT":1704067200}
{"ID_KEY":"CS200002","STATUS":"ACTIVE","ARCHIVED":"YES","CASE_TITLE":"Payment Issue","DESCRIPTION":"Duplicate charge detected.","PRIORITY":"CRITICAL","OWNER":"agent_02","CREATED_AT":1704153600}
{"ID_KEY":"CS200003","STATUS":"CLOSED","ARCHIVED":"NO","CASE_TITLE":"Email Not Received","DESCRIPTION":"Verification email missing.","PRIORITY":"MEDIUM","OWNER":"agent_03","CREATED_AT":1704240000}
{"ID_KEY":"CS200004","STATUS":"CLOSED","ARCHIVED":"YES","CASE_TITLE":"UI Bug","DESCRIPTION":"Alignment issue on mobile.","PRIORITY":"LOW","OWNER":"agent_04","CREATED_AT":1704326400}
{"ID_KEY":"CS200005","STATUS":"ACTIVE","ARCHIVED":"NO","CASE_TITLE":"Slow Dashboard","DESCRIPTION":"Dashboard loads slowly.","PRIORITY":"MEDIUM","OWNER":"agent_05","CREATED_AT":1704412800}
{"ID_KEY":"CS200006","STATUS":"ON_HOLD","ARCHIVED":"NO","CASE_TITLE":"API Error","DESCRIPTION":"500 error from API.","PRIORITY":"HIGH","OWNER":"agent_01","CREATED_AT":1704499200}
{"ID_KEY":"CS200007","STATUS":"ON_HOLD","ARCHIVED":"YES","CASE_TITLE":"Subscription Downgrade","DESCRIPTION":"Downgrade not applied.","PRIORITY":"LOW","OWNER":"agent_02","CREATED_AT":1704585600}
{"ID_KEY":"CS200008","STATUS":"ACTIVE","ARCHIVED":"NO","CASE_TITLE":"File Upload Error","DESCRIPTION":"Upload fails for large files.","PRIORITY":"HIGH","OWNER":"agent_03","CREATED_AT":1704672000}
{"ID_KEY":"CS200009","STATUS":"CLOSED","ARCHIVED":"NO","CASE_TITLE":"Invoice Mismatch","DESCRIPTION":"Invoice total incorrect.","PRIORITY":"MEDIUM","OWNER":"agent_04","CREATED_AT":1704758400}
{"ID_KEY":"CS200010","STATUS":"ACTIVE","ARCHIVED":"YES","CASE_TITLE":"2FA Issue","DESCRIPTION":"Authentication code invalid.","PRIORITY":"CRITICAL","OWNER":"agent_05","CREATED_AT":1704844800}
{"ID_KEY":"CS200011","STATUS":"ACTIVE","ARCHIVED":"NO","CASE_TITLE":"Search Not Working","DESCRIPTION":"No results returned.","PRIORITY":"HIGH","OWNER":"agent_01","CREATED_AT":1704931200}
{"ID_KEY":"CS200012","STATUS":"CLOSED","ARCHIVED":"YES","CASE_TITLE":"Report Timeout","DESCRIPTION":"Report generation fails.","PRIORITY":"MEDIUM","OWNER":"agent_02","CREATED_AT":1705017600}
{"ID_KEY":"CS200013","STATUS":"ON_HOLD","ARCHIVED":"NO","CASE_TITLE":"Data Sync Issue","DESCRIPTION":"Records not syncing.","PRIORITY":"HIGH","OWNER":"agent_03","CREATED_AT":1705104000}
{"ID_KEY":"CS200014","STATUS":"ACTIVE","ARCHIVED":"NO","CASE_TITLE":"Notification Delay","DESCRIPTION":"Push notifications delayed.","PRIORITY":"LOW","OWNER":"agent_04","CREATED_AT":1705190400}
{"ID_KEY":"CS200015","STATUS":"CLOSED","ARCHIVED":"NO","CASE_TITLE":"Account Locked","DESCRIPTION":"Account locked after attempts.","PRIORITY":"HIGH","OWNER":"agent_05","CREATED_AT":1705276800}
{"ID_KEY":"CS200016","STATUS":"ACTIVE","ARCHIVED":"YES","CASE_TITLE":"Export Failure","DESCRIPTION":"CSV export fails.","PRIORITY":"MEDIUM","OWNER":"agent_01","CREATED_AT":1705363200}
{"ID_KEY":"CS200017","STATUS":"ON_HOLD","ARCHIVED":"NO","CASE_TITLE":"Role Permission Issue","DESCRIPTION":"Incorrect access rights.","PRIORITY":"HIGH","OWNER":"agent_02","CREATED_AT":1705449600}
{"ID_KEY":"CS200018","STATUS":"CLOSED","ARCHIVED":"YES","CASE_TITLE":"Language Bug","DESCRIPTION":"Wrong translation shown.","PRIORITY":"LOW","OWNER":"agent_03","CREATED_AT":1705536000}
{"ID_KEY":"CS200019","STATUS":"ACTIVE","ARCHIVED":"NO","CASE_TITLE":"Session Expiry","DESCRIPTION":"Session expires quickly.","PRIORITY":"MEDIUM","OWNER":"agent_04","CREATED_AT":1705622400}
{"ID_KEY":"CS200020","STATUS":"ON_HOLD","ARCHIVED":"YES","CASE_TITLE":"Webhook Failure","DESCRIPTION":"Webhook not triggered.","PRIORITY":"HIGH","OWNER":"agent_05","CREATED_AT":1705708800}
' | jq


### Multi-search (CORRECT VERSION) ############################
curl -s "$TYPESENSE_HOST/multi_search" \
  -X POST \
  -H "Content-Type: application/json" \
  -H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}" \
  -d '
  {
  "searches": [
    {
      "collection": "CASE_DOCS",
      "q": "*",
      "query_by": "CASE_TITLE,DESCRIPTION",
      "filter_by": "STATUS:=[ACTIVE] && ARCHIVED:=[NO] && ID_KEY:=CS*",
      "sort_by": "CREATED_AT:desc",
      "page": 1,
      "per_page": 10,
      "highlight_full_fields": "CASE_TITLE,DESCRIPTION",
      "max_filter_by_candidates": 10
    }
  ]
}' | jq


docker stop typesense-repro
docker rm typesense-repro
f
Hey @Raja K , When you use a wildcard filter like
ID_KEY:=CS*
, Typesense searches for tokens that start with "CS". It uses a parameter called
max_filter_by_candidates
to limit how many candidate tokens it considers. The default is 4. So when processing
ID_KEY:=CS*
, it finds tokens matching "CS" but only considers the top 4 candidates. Each candidate becomes a posting list, and those lists determine which documents can match. With only 4 candidates, you can only get up to 4 documents back, even if 6 actually match. That's why you see
out_of: 6
but only 4 results. Typesense knows 6 match, but it only looked at 4 candidate tokens. When you increase
max_filter_by_candidates
to 20, it considers more candidate tokens, so it can find all 6 matching documents. The progression you saw (1 result at 5, 2 at 8, 3 at 10, etc.) shows it finding more candidates as the limit increases. This is a performance optimization. Without a limit, a broad prefix like "CS*" could match many tokens, which can be slow. The default of 4 keeps things fast for typical cases, but you can raise it when you need complete results from wildcard filters. For your case with 6 documents, setting
max_filter_by_candidates
to 20 or higher ensures you get all of them.
r
Hello @Fanis Tharropoulos - Thank you for your clarification; I have agreed that, wild card search is costly; I thought, Typesense would 1. Apply all the filters (except wildcard) to get eligible records first (that is matching records of STATUS:=[
ACTIVE
] && ARCHIVED:=[
NO
]) 2. And then apply the wildcard filter (ID_KEY:=CS*) with
max_filter_by_candidates
to get the final results It appears to be, it's not designed in that way; In our usecase, the user selected the STATUS and ARCHIEVED facets and then try to find some records that started with CS. As we have around 500K documents, setting the
max_filter_by_candidates
to 500K leads to performance issue. Is there any workaround to achieve it; Thanks
Hello @Fanis Tharropoulos @Hayden Hoang @Kishore Nallan @Jason Bosco - Could you please review my above message and guide me to address the issue ? Your advice on this would be greatly appreciated. Looking forward to hear from you. Thanks
h
Hi @Raja K, what is the
search_time_ms
when you set
max_filter_by_candidates
to 500K? I think one way to work around this is to add a new field to your collection which holds the first two letters of
ID_KEY
. And then in search just do an exact-match filtering
new_field:=CS
.
👍 1
f
+1 for the ID KEY approach
r
Hello @Hayden Hoang @Fanis Tharropoulos - Thank you for your response. It's taking around 30 to 35 seconds when I set
max_filter_by_candidates
to 500K. As, we have given a free form input text field for the users to perform the prefix based search, the user can type-in any number of characters. And we will not be able to restrict them to enter ONLY 2 chars and do exact match on the new field. Is there any way to specify the filter execution order something like below. 1. Apply all the filters (except wildcard) to get eligible records first (that is matching records of STATUS:=[
ACTIVE
] && ARCHIVED:=[
NO
]) 2. And then apply the wildcard filter (ID_KEY:=CS*) with
max_filter_by_candidates
to get the final results Looking forward to hear from you. Thanks
f
Why not use the
q
parameter instead of prefix filtering? It will be the best in terms of performance!
r
Hello @Fanis Tharropoulos - Thanks again. I understand that. It's a client requirement. We have a global search text field and a Grid to populate the results. The text entered in the global search text field will be passed in the q parameter to search across the fields. And we have grid column level filters (Starts with, Equal, Not Equal) for each grid column which will be passed in the filter_by parameter. The issue that we have posted (ID_KEY:=CS*) is the Grid column (starts with) filter applied by the user without entering anything in the global search (q) parameter. Hope you have understand the requirement. Thanks
Hello @Fanis Tharropoulos @Hayden Hoang @Kishore Nallan @Jason Bosco - Could you please review my above response and guide me ? Looking forward to hear from you. Thanks
f
Typesense doesn't currently support explicit control over filter execution order. When you use a filter query like STATUS:=[ACTIVE] && ARCHIVED:=[NO] && ID_KEY:=CS*, all filters are evaluated together as part of the filter tree. Typsesense will optimize AND filter subtrees by estimated result size, but it doesn't prioritize non-wildcard filters over wildcard filters. The wildcard filter ID_KEY:=CS* is expensive because it uses fuzzy search with max_filter_by_candidates, which can scan many candidates. You can try reordering the filter query to put non-wildcard filters first, but this doesn't guarantee non-wildcard-first evaluation. Another option is to split the query. First apply the non-wildcard filters (STATUS:=[ACTIVE] && ARCHIVED:=[NO]), then use those results as included_ids in a second query with the wildcard filter. This requires client-side coordination and adds complexity. If ID_KEY patterns are predictable, consider indexing strategies like a computed field or a separate indexed field to avoid wildcard matching.
r
Thank you for your response and suggestions @Fanis Tharropoulos