https://typesense.org logo
Join Slack
Powered by
Hi, I’m getting incorrect item counts when using m...
# community-help
a
Hi, I’m getting incorrect item counts when using my filter query with hybrid search. For example, when I search for a word, I see a filter option like 
"men"
with a count of 
1
. But after applying that filter, I actually get more than one result. Here’s the query I’m using:
Copy code
{
  "searches": [
    {
      "collection": "products",
      "q": "modern night dress",
      "facet_by": "price,brand,product_category",
      "max_facet_values": 10000,
      "page": 1,
      "per_page": 10,
      "query_by": "title,tags,product_types,embedding",
      "exclude_fields": "embedding",
      "split_join_tokens": "always",
      "vector_query": "embedding:([],k:10000, distance_threshold:0.5)"
    },
    {
      "collection": "products",
      "q": "modern night dress",
      "facet_by": "price",
      "max_facet_values": 10000,
      "page": 1,
      "per_page": 10,
      "query_by": "title,tags,product_types,embedding",
      "exclude_fields": "embedding",
      "split_join_tokens": "always",
      "vector_query": "embedding:([],k:10000, distance_threshold:0.5)"
    },
    {
      "collection": "products",
      "q": "modern night dress",
      "facet_by": "brand",
      "max_facet_values": 10000,
      "page": 1,
      "per_page": 10,
      "query_by": "title,tags,product_types,embedding",
      "exclude_fields": "embedding",
      "split_join_tokens": "always",
      "vector_query": "embedding:([],k:10000, distance_threshold:0.5)"
    },
    {
      "collection": "products",
      "q": "modern night dress",
      "facet_by": "product_category",
      "max_facet_values": 10000,
      "page": 1,
      "per_page": 10,
      "query_by": "title,tags,product_types,embedding",
      "exclude_fields": "embedding",
      "split_join_tokens": "always",
      "vector_query": "embedding:([],k:10000, distance_threshold:0.5)"
    }
  ]
}
f
Here's an answer from our FAQ section in the docs: https://typesense.org/docs/guide/faqs.html#keyword-search
a
Is it possible to turn off this setting?
Or can i get the correct count in filters?
f
You can increase the 
max_candidates
value for Typesense to look for more prefix matches when no filters are applied, but this will have an impact on performance
a
There is no changes even after setting 
max_candidates=100000
and i have only 10000 documents in the collection.
f
This may be related to your vector query as well. Could you post a reproducible example like this one? https://gist.github.com/jasonbosco/7c3432713216c378472f13e72246f46b
a
Sure here is the example
Copy code
### Run Typesense via Docker ########################################
set -x

export TYPESENSE_API_KEY=xyz
export TYPESENSE_HOST=<http://localhost:8108>

docker stop typesense-repro 2>/dev/null
docker rm typesense-repro 2>/dev/null
rm -rf "$(pwd)"/typesense-data-dir-repro
mkdir "$(pwd)"/typesense-data-dir-repro

# Wait for Typesense to be ready
docker run -d -p 8108:8108 --name typesense-repro \
            -v"$(pwd)"/typesense-data-dir-repro:/data \
            typesense/typesense:29.0.rc31 \
            --data-dir /data \
            --api-key=$TYPESENSE_API_KEY \
            --enable-cors

# Wait till typesense is ready.
until curl -s -o /dev/null -w "%{http_code}" "$TYPESENSE_HOST/health" -H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}" | grep -q "200"; do
  sleep 2
done

curl -s "$TYPESENSE_HOST/debug" \
       -H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}" | jq


curl -s "$TYPESENSE_HOST/collections" \
  -X POST \
  -H "Content-Type: application/json" \
  -H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}" \
  -d '{
        "name": "products",
        "fields": [
          {"name": "title", "type": "string" },
          {"name": "tags", "type": "string[]", "facet": true, "optional": true},
          {"name": "product_types", "type": "string", "facet": true,"optional": true },
          {"name": "category", "type": "string", "facet": true,"optional": true },
          {"name": "gender", "type": "string", "facet": true,"optional": true },
          {"name": "embedding","type":"float[]","embed": {"from":["title","tags","product_types","category","gender"],"model_config":{"model_name": "ts/all-MiniLM-L12-v2"}}
          }
        ]
      }' | jq

curl -s "$TYPESENSE_HOST/collections/products/documents/import?action=create" \
  -H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}" \
  -H "Content-Type: text/plain" \
  -X POST \
  --data-binary @- <<EOF | jq
{"id": "1","title": "Bobble LED","tags":["LED lamp","bobble lamp","white lamp","modern lamp","decorative lamp","table lamp","night light","bedroom lamp","living room lamp","accent lighting","textured lamp","unique lamp","bulb lamp","small lamp"],"product_types":"Table Lamp","category":"Home Accessories"}
{"id": "2","title": "Mink Traditional Nightie","tags":["mink nightie","traditional nightie","long sleeve nightgown","button down nightgown","collared nightgown","nightdress","sleepwear","womens nightwear","pajama dress","comfortable nightgown","cozy nightgown"],"product_types":"nightgown","gender":"Women","category":"Womens Night Wear"}
{"id": "3","title": "Ditsy Print Nightie","tags":["ditsy print","nightie","short sleeve","knee length","pink nightgown","lace trim","comfortable","lightweight","summer nightwear","womens sleepwear"],"product_types":"nightgown","gender":"Women","category":"Womens Night Wear"}
{"id": "4","title": "Star Print Nightie","tags":["star print","nightgown","night dress","sleepwear","short sleeve","casual nightie","star pattern","women's nightwear","lightweight nightgown"],"product_types":"nightgown","gender":"Women","category":"Womens Night Wear"}
{"id": "5","title": "Floral Nightie","tags":["floral nightgown","floral nightdress","short sleeve nightgown","lightweight nightgown","spring nightgown","summer nightgown","pastel nightgown","women's nightwear","women's sleepwear","comfortable nightgown"],"product_types":"nightgown","gender":"Women","category":"Womens Night Wear"}
{"id": "6","title": "Ditsy Floral Nightie","tags":["floral nightgown","ditsy print nightie","short sleeve nightgown","midi nightgown","lightweight nightgown","summer nightgown","cotton nightgown","comfortable nightgown","blue floral nightgown","white nightgown"],"product_types":"nightgown","gender":"Women","category":"Womens Night Wear"}
{"id": "7","title": "Floral Nightie","tags":["floral nightie","v-neck nightie","sleeveless nightie","short nightie","slip nightie","floral print nightie","lightweight nightie","summer nightie","spring nightie","casual nightie","comfortable nightie","feminine nightie","pastel nightie"],"product_types":"nightie","gender":"Women","category":"Womens Night Wear"}
{"id": "8","title": "Floral Soft Touch Nightie","tags":["floral nightie","soft touch nightie","black and white nightie","lace trim nightie","sleeveless nightie","spaghetti strap nightie","women's sleepwear","nightgown","night dress","summer nightie"],"product_types":"nightie","gender":"Women","category":"Womens Night Wear"}
{"id": "9","title": "Floral Chemise Nightie","tags":["floral nightie","chemise nightgown","mint green nightie","lace trim nightie","sleeveless nightgown","summer nightwear","lightweight nightie","floral print chemise","comfortable sleepwear","women's nightwear"],"product_types":"chemise","gender":"Women","category":"Womens Night Wear"}
{"id": "10","title": "Bow Print Nightie","tags":["bow print","nightie","short sleeve","t-shirt dress","casual dress","navy dress","womens nightwear","sleepwear","night dress","knee length dress"],"product_types":"nightie","gender":"Women","category":"Womens Night Wear"}
{"id": "11","title": "Stripe Cotton Shirt Nightie","tags":["stripe nightie","cotton nightgown","striped sleepwear","long sleeve nightie","button down nightgown","light blue nightie","casual nightwear","comfortable nightie","classic nightgown","striped night shirt","cotton sleep shirt"],"product_types":"nightgown","gender":"Women","category":"Womens Night Wear"}
{"id": "12","title": "Floral Satin Chemise Nightie","tags":["floral chemise","satin nightgown","lace trim nightie","sleeveless nightdress","floral nightie","satin chemise","nightie","floral","satin","lace","nightwear","sleepwear"],"product_types":"chemise","gender":"Women","category":"Womens Night Wear"}
{"id": "13","title": "Floral Satin Dressing Gown","tags":["floral dressing gown","satin robe","floral robe","long dressing gown","women's robe","loungewear","sleepwear","floral sleepwear","satin loungewear","patterned robe","lightweight robe","spring robe","summer robe","beige floral robe","casual robe","kimono robe"],"product_types":"robe","gender":"Women","category":"Womens Night Wear"}
{"id": "14","title": "Stripe Nightie","tags":["stripe nightgown","striped sleep dress","lace trim nightie","spaghetti strap nightgown","sleeveless nightgown","summer nightwear","lightweight nightie","comfortable sleepwear","casual nightgown","striped nightwear","gray and white nightie"],"product_types":"nightgown","gender":"Women","category":"Womens Night Wear"}
{"id": "15","title": "Floral Soft Touch Nightie","tags":["floral nightgown","soft touch nightie","lace trim nightie","sleeveless nightgown","floral print sleepwear","knee length nightie","summer nightgown","casual nightgown","comfortable sleepwear","lightweight nightgown","women's sleepwear","sage green nightgown"],"product_types":"nightgown","gender":"Women","category":"Womens Night Wear"}
{"id": "16","title": "Ditsy Heart Nightie","tags":["nightie","nightgown","sleepwear","womens nightwear","heart print","ditsy print","short sleeve","knee length","navy blue","casual nightgown","cotton nightgown"],"product_types":"nightgown","gender":"Women","category":"Womens Night Wear"}
{"id": "17","title": "Star Nightie","tags":["star nightie","star dress","night dress","nightgown","sleepwear","womens nightie","casual dress","t-shirt dress","star print dress","white dress","brown star","short sleeve nightie"],"product_types":"nightie","gender":"Women","category":"Womens Night Wear"}
{"id": "18","title": "Satin Stripe Pyjama Set","tags":["satin pyjamas","stripe pyjamas","black pyjama set","long sleeve pyjamas","womens pyjamas","nightwear","loungewear","heart detail pyjamas","contrast piping","button-up pyjamas"],"product_types":"pyjama sets","gender":"Women","category":"Womens Night Wear"}
{"id": "19","title": "Stripe Star Nightie","tags":["striped nightie","star nightie","t-shirt dress","casual dress","nightgown","sleepwear","comfortable nightwear","short sleeve nightie","knee length nightie","black and white stripes","star pattern","easy wear","summer nightie"],"product_types":"nightie","gender":"Women","category":"Womens Night Wear"}
{"id": "20","title": "Satin Stripe Pyjama Set","tags":["satin pyjama set","striped pyjamas","navy pyjama set","long sleeve pyjamas","button-down pyjamas","heart embroidered pyjamas","women's pyjamas","nightwear","sleepwear","loungewear","satin pajamas","striped pajamas"],"product_types":"pyjama set","gender":"Women","category":"Womens Night Wear"}
{"id": "3951","title": "Girls Black Print Coat","tags": ["kids table and chair set","childrens furniture","plastic table set","playroom furniture","toddler table","junior table and chair","step2 furniture","colorful furniture","kids activity table","preschool furniture","small table and chairs","modern kids furniture"],"product_types": "nightgown","gender": "Girls","category": "Girls"}
{"id": "229086","title": "Kids Girl Home dress","tags": ["girls nightgown","kids pajamas","girls sleepwear","childrens nightdress","cartoon print","kuromi","ruffle hem","short sleeve","night dress","home wear"],"product_types": "nightgown","gender": "Girls","category": "Girls"}
EOF

# Query without filter
curl -s "$TYPESENSE_HOST/multi_search" \
        -X POST \
        -H "Content-Type: application/json" \
        -H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}" \
        -d '
          {
            "searches": [
              {
                "collection": "products",
                "q": "modern night dress",
                "query_by": "title, tags, product_types, gender, category, embedding",
                "exclude_fields": "embedding",
                "include_fields": "title",
                "facet_by":"category",
                "vector_query": "embedding:([], distance_threshold:0.50)"
              }
            ]
          }'  | jq


# Query with filter
curl -s "$TYPESENSE_HOST/multi_search" \
        -X POST \
        -H "Content-Type: application/json" \
        -H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}" \
        -d '
          {
            "searches": [
              {
                "collection": "products",
                "q": "modern night dress",
                "query_by": "title, tags, product_types, gender, category, embedding",
                "exclude_fields": "embedding",
                "include_fields": "title",
                "facet_by":"category",
                "filter_by":"category:Girls",
                "vector_query": "embedding:([], distance_threshold:0.50)"
              }
            ]
          }'  | jq

docker stop typesense-repro
docker rm typesense-repro



curl -H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}" \
     -X GET \
    "<http://localhost:8108/collections/products>"
f
There are 2 found with the filter applied, while 15 for no filters
a
Step 1: Run the initial query
Copy code
curl -s "$TYPESENSE_HOST/multi_search" \
  -X POST \
  -H "Content-Type: application/json" \
  -H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}" \
  -d '{
    "searches": [
      {
        "collection": "products",
        "q": "modern night dress",
        "query_by": "title, tags, product_types, gender, category, embedding",
        "exclude_fields": "embedding",
        "include_fields": "title",
        "facet_by": "category",
        "vector_query": "embedding:([], distance_threshold:0.50)"
      }
    ]
  }' | jq
👉 In the response, under 
facet
, you’ll see a facet named 
category
 with the value 
Girls
and count 
1
. Step 2: Run the query with filter applied
Copy code
curl -s "$TYPESENSE_HOST/multi_search" \
  -X POST \
  -H "Content-Type: application/json" \
  -H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}" \
  -d '{
    "searches": [
      {
        "collection": "products",
        "q": "modern night dress",
        "query_by": "title, tags, product_types, gender, category, embedding",
        "exclude_fields": "embedding",
        "include_fields": "title",
        "facet_by": "category",
        "filter_by": "category:Girls",
        "vector_query": "embedding:([], distance_threshold:0.50)"
      }
    ]
  }' | jq
This time i'm expecting only 1 result, since the facet count was 1. But i'm actually getting 2 results.
@Fanis Tharropoulos
f
The way faceting works after v29.0 due to the performance increase is there's a larger approximation of results. The number isn't always exact because of that
Open in Slack
PreviousNext
Powered by