Hi <@U01V0MWR0DU>! Since full-text search on refe...
# community-help
g
Hi @Harpreet Sangar! Since full-text search on referenced fields is not available yet, could you share what the Typesense team considers the recommended approach for handling this use case today? Specifically: • Is denormalizing related fields into the main collection the preferred method? • Or is a multi-step search (search vendors → filter products) the approach you recommend? Any guidance or examples on best practices would be greatly appreciated!
h
Hey @Georgi Nachev Both of these solutions are viable and choosing one out of them depends upon your use case. Denormalizing would mean greater memory usage and multi-step search will require greater processing time. Can you share your schema and the number of documents per collection and the query so I can give you a definite answer?
g
Copy code
{
  "name": "products",
  "enable_nested_fields": true,
  "fields": [
    {
      "name": "id",
      "type": "string"
    },
    {
      "name": "name",
      "type": "string",
      "sort": true,
      "locale": "your-locale"
    },
    {
      "name": "description",
      "type": "string",
      "locale": "your-locale",
      "optional": true
    },
    {
      "name": "bundle",
      "type": "bool",
      "facet": true
    },
    {
      "name": "vendor",
      "type": "string",
      "locale": "your-locale",
      "optional": true
    },
    {
      "name": "categories",
      "type": "string[]",
      "locale": "your-locale"
    },
    {
      "name": "category_properties",
      "type": "object[]",
      "locale": "your-locale"
    }
  ]
}
Copy code
{
  "name": "variants",
  "enable_nested_fields": true,
  "fields": [
    {
      "name": "id",
      "type": "string"
    },
    {
      "name": "name",
      "type": "string",
      "sort": true,
      "locale": "your-locale"
    },
    {
      "name": "url_handles",
      "type": "string[]",
      "locale": "your-locale"
    },
    {
      "name": "description",
      "type": "string",
      "locale": "your-locale",
      "optional": true
    },
    {
      "name": "vendor",
      "type": "string",
      "locale": "your-locale",
      "optional": true
    },
    {
      "name": "categories",
      "type": "string[]",
      "locale": "your-locale"
    },
    {
      "name": "category_properties",
      "type": "object[]",
      "locale": "your-locale"
    },
    {
      "name": "item_id",
      "type": "string",
      "facet": true
    },
    {
      "name": "price",
      "type": "int32",
      "optional": true
    },
    {
      "name": "created_at",
      "type": "int32"
    },
    {
      "name": "category_ids",
      "type": "int32[]",
      "facet": true
    },
    {
      "name": "vendor_id",
      "type": "int32",
      "optional": true,
      "facet": true
    },
    {
      "name": "new",
      "type": "bool",
      "facet": true
    },
    {
      "name": "featured",
      "type": "bool",
      "facet": true
    },
    {
      "name": "sale_.*",
      "type": "bool",
      "facet": true
    },
    {
      "name": "product_to_discount_.*",
      "type": "int32",
      "optional": true
    },
    {
      "name": "sort_price_.*",
      "type": "int32",
      "optional": true,
      "facet": true
    },
    {
      "name": "collection_ids_.*",
      "type": "int32[]",
      "facet": true
    },
    {
      "name": "brand_model_.*",
      "type": "int32[]",
      "facet": true
    },
    {
      "name": "tag_ids",
      "type": "int32[]",
      "facet": true
    },
    {
      "name": "product_type",
      "type": "string",
      "facet": true
    },
    {
      "name": "category_properties_.*",
      "type": "int32[]",
      "facet": true
    },
    {
      "name": "parameters_.*",
      "type": "int32",
      "optional": true,
      "facet": true
    },
    {
      "name": "enable_sell",
      "type": "bool",
      "sort": true
    },
    {
      "name": "url",
      "type": "string",
      "index": false
    },
    {
      "name": "image",
      "type": "string",
      "index": false
    },
    {
      "name": "image_id",
      "type": "int32",
      "optional": true,
      "index": false
    },
    {
      "name": "sku",
      "type": "string",
      "optional": true,
      "locale": "your-locale"
    },
    {
      "name": "barcode",
      "type": "string",
      "optional": true,
      "locale": "your-locale"
    },
    {
      "name": "property_1",
      "type": "string",
      "optional": true,
      "locale": "your-locale"
    },
    {
      "name": "property_2",
      "type": "string",
      "optional": true,
      "locale": "your-locale"
    },
    {
      "name": "property_3",
      "type": "string",
      "optional": true,
      "locale": "your-locale"
    },
    {
      "name": "quantity",
      "type": "float",
      "optional": true
    },
    {
      "name": "total_reviews",
      "type": "int32",
      "optional": true
    },
    {
      "name": "average_rating",
      "type": "float",
      "optional": true,
      "facet": true
    },
    {
      "name": "order_by_sale_.*",
      "type": "int64",
      "sort": true
    },
    {
      "name": "order_by_featured_.*",
      "type": "int64",
      "sort": true
    },
    {
      "name": "order_by_id",
      "type": "int64",
      "sort": true
    },
    {
      "name": "sort_order",
      "type": "int64",
      "sort": true,
      "optional": true
    },
    {
      "name": "order_by_random",
      "type": "int64",
      "sort": true
    }
  ],
  "default_sorting_field": "order_by_id"
}
Copy code
{
  "name": "categories",
  "enable_nested_fields": true,
  "fields": [
    {
      "name": "id",
      "type": "string"
    },
    {
      "name": "parent_id",
      "type": "int32",
      "optional": true
    },
    {
      "name": "name",
      "type": "string",
      "sort": true,
      "locale": "bg"
    },
    {
      "name": "url",
      "type": "string",
      "index": false
    },
    {
      "name": "image",
      "type": "string",
      "index": false
    }
  ],
  "default_sorting_field": "name"
}
Copy code
{
  "name": "vendors",
  "enable_nested_fields": true,
  "fields": [
    {
      "name": "id",
      "type": "string"
    },
    {
      "name": "name",
      "type": "string",
      "sort": true,
      "locale": "bg"
    },
    {
      "name": "url",
      "type": "string",
      "index": false
    },
    {
      "name": "image",
      "type": "string",
      "index": false
    }
  ],
  "default_sorting_field": "name"
}
Hi Harpreet, thanks for the help! Here is more context about my use case: • We have two main searchable collections: products and variants. • Variants belong to products (via
item_id
), and the UI requires filtering, grouping, and sorting on the variants collection, while still allowing full-text search on product fields (
name
,
description
,
categories
,
vendor.name
etc.). • Grouping is done by
item_id
and optionally by dynamic
parameters_*
fields. • The number of documents ranges from tens of thousands up to millions in each collection. My goal is: • Use the variants collection for filtering/sorting/grouping • But also allow text search across product-level fields (product name, description, vendor, category, etc.) Given this structure, do you recommend: 1. Denormalizing product fields into each variant (name, description, categories, vendor name), 2. or 3. Using multi-step search (e.g., search products first → filter variants by product IDs)? Which approach would scale better at “tens of thousands → millions” of documents? Thanks again!
@Harpreet Sangar please help 🙂
h
Since there can be millions of documents, I'll suggest you to use the multi-step approach in the interim and keep an eye on this https://github.com/typesense/typesense/issues/1657
g
Thanks
h
You should use the export api for searching the products collection since it will return all the matches in one go.
Sorry, I think you can't use the export api with a query search.
@Kishore Nallan Does the export api support querying?
k
No the export API does not support query parameters. It only supports filtering.
h
@Georgi Nachev I think the multi-step approach won't be feasible then.
g
Got it, thank you very much for the clarification. In that case, my remaining option would be to denormalize and include the categories, vendor (manufacturer), and main product searchable fields directly into the variants collection for full-text search.
h
Yes.
g
Thanks again 🙂
👍🏼 1
Hi @Harpreet Sangar @Kishore Nallan, Sorry to bother you again, but I ran into another issue. In my variants schema, I have: { "name": "parameters_.*", "type": "int32", "optional": true, "facet": true } When I make a search request with grouping: "group_by": "item_id,parameters_57,parameters_59,parameters_60" and have
validate_field_names: false
, I get the error: "Could not parse the filter query: unbalanced
&&
operands." How can I avoid this? Creating explicit
parameters_N
fields for each parameter is not a feasible option because they are very dynamic and the schema would have to be constantly updated. Thanks for any guidance!
h
I think it has to do with the value of your
filter_by
parameter
g
Here is my all paameters:
Copy code
"q" => "yellow"
    "query_by" => "name,short_description,description,categories,vendor,property_1,property_2,property_3,sku,barcode,category_properties"
    "filter_by" => "active:=true && draft:=false && hidden:=false"
    "per_page" => 9
    "page" => 0
    "highlight_start_tag" => "<mark>"
    "highlight_end_tag" => "</mark>"
    "snippet_threshold" => 30
    "exhaustive_search" => false
    "use_cache" => false
    "cache_ttl" => 60
    "prioritize_exact_match" => true
    "enable_overrides" => true
    "highlight_affix_num_tokens" => 4
    "prefix" => true
    "validate_field_names" => false
    "group_by" => "item_id,parameters_57,parameters_59,parameters_60"
    "group_limit" => 1
    "query_by_weights" => "10,2,2,5,5,5,5,5,5,5,5"
    "collection" => "4793_products_variants"
    "include_fields" => "id,name,description,price,sale_2,sort_price_2,url,image,barcode,sku,category_properties,categories,brand_model"
  ]
if i change group_by to has only item_id, the problem is gone
h
Which version are you using?
g
Copy code
30.0.rc11
h
Can you try again with the latest rca build. I think we fixed a similar bug in v30
g
Yes, i test
that version
Copy code
30.0.rca33
?
h
Yes
g
Yes, it works 🙂
Thanks
👍🏼 1