I'm having trouble indexing product SKUs / part nu...
# community-help
r
I'm having trouble indexing product SKUs / part numbers that have two or more hyphens. I'm using Typesense Cloud version 29.0. My collection schema config for this field is
Copy code
{
  "created_at": 1770839153,
  "default_sorting_field": "createdAt",
  "fields": [
    {
      "facet": false,
      "index": true,
      "infix": false,
      "locale": "",
      "name": "sku",
      "optional": true,
      "sort": false,
      "stem": false,
      "stem_dictionary": "",
      "store": true,
      "symbols_to_index": [
        "-",
        "_"
      ],
      "type": "string"
    }
  ],
  "name": "global",
  "num_documents": 878462,
  "symbols_to_index": [],
  "token_separators": []
}
notice I've included the hyphen and underscore for
symbols_to_index
. Unfortunately, searching for SKUs with three hyphens like
N22-9-2SV-1
doesn't work, but I get results if I type the prefix up to
N22-9-2
. The only way I've been able to make this work is if I set
"facet": true
but that seems like a hack. I've read this Typesense Support Article on SKUs, but I don't believe enabling infix or adding the hyphen as a token separator is the move here. Does anybody know how I can get these SKUs indexed?
f
Could you try upgrading to 30.1? We've touched
symbols_to_index
and hyphen-related queries a bunch since 29.0
r
@Fanis Tharropoulos thanks for the response. I upgraded to 30.1, and I'm still not getting the results I expect. If I make "sku" the first
query_by
parameter, then it works. If it's not the first, then it doesn't work. See these screenshots for when sku is the first
query_by
param:
@Fanis Tharropoulos And then when it's not the first param:
@Fanis Tharropoulos I also want to note that after upgrading I truncated the index and re-added all of the documents, so the index should be fresh.
f
Hey @Richard Scotten , Is it okay with you if I copy a subset of your data on one of our debug clusters so I can check them?
r
sure
Does Typesense have a token analyzer like Elasticsearch?
f
Not in the same sense that Elasticsearch does it. You can configure parts of the Tokenization process. Could you share some details on how it would be useful for your use-case?
Btw, could you share your cluster's id either here on in DMs? The clusters associated with your email have been terminated for a while
r
a token analyzer allows you to examine how typesense tokenized a specific document. So you would go onto Typesense cloud, add a document ID into an input field, and the response would be all the tokens for that document. Then I can see how the SKU is tokenized. Then I can either fix my collection config file so that it is tokenized correctly, or I can report a bug. It would help in my situation where the SKUs might not be tokenized the way I expect them to be. For example, the tokens need to preserve hyphens for my SKUs.
is this thread public?
I shared the cluster ID in a DM
f
Hey Richard, In queries, the most weighted field's
symbols_to_index
and
token_separators
are used during the Tokenization process of the query. That's why it works for
sku
fields and not for
idNumber
r
Is the most weighted field the first
query_by
field? So if my query is
query_by: 'sku, idNumber'
then the search query is tokenized using the
symbols_to_index
defined for
sku
in my collection config file?
f
Yes, exactly. Either the first
query_by
field defined, or the one with the most weight in
query_by_weights
.
r
well, hmm. Thanks for clarifying. This is sort of an unexpected gotcha. I don't want my query tokenized with the hyphen as a symbol to index for all of my other query fields, which include product title, description, and tags. In these fields, I want the query tokenized without the hyphen. I'm switching from both Algolia and Elasticsearch and I never had this problem with those search engines.
The intuitive, expected behavior is that the query would be tokenized separately for each
query_by
field according to the
symbols_to_index
and
token_separators
of that field.
f
Hey Richard, We are aware of this issue but we're still thinking about how to fix it, since it touches multiple parts of the search pipeline
r
okay, thanks for letting me know. Is this behavior documented anywhere?
Hey @Fanis Tharropoulos, I'm getting more complaints from my customers with hyphens in their SKUs. Queries for SKUs with just one hyphen only return 4 or 5 results when 21 exist. For example, if my query is "300-", I only get 4 results. But if I start adding numbers to the query like "300-1" or "300-2" then I start to see the missing results. Please see this demo video of the problem: https://www.loom.com/share/5ff6b006ca804045995cbe74d442e0b1
f
Hey Richard, Could you provide. small curl request so I can see the results myself?
r
Can I just give you access to my Typesense Cloud account instead? The behavior is repeatable there.
f
It would be appreciated to know the exact parameters used in the request instead. You can copy the request as curl from your browser's dev tools
r
Sending as a dm
Just sent it as a dm
@Fanis Tharropoulos Hi Fanis, I just want to bump this issue of the "the most weighted field's
symbols_to_index
and
token_separators
are used during the Tokenization process of the query." This is breaking a lot of my searches and is making debugging difficult. Is there an estimation on when this might get fixed?
@Fanis Tharropoulos like what's the point of having a
symbols_to_index
at the field level when the query is tokenized using different token separators. It only works if you query one field at a time, or if all fields have the same token separators and symbols to index. I never had this problem with Algolia or Elasticsearch.
f
Hey @Richard Scotten , There’s no timeline on if and when it will be addressed. This design is tied to deeper internals in parsing, synonym handling, token dropping, and ranking, so we do not have a near-term ETA for changing it safely. The reliable workaround is to split search strategies: • Run a dedicated SKU query where SKU is first or only in query_by, and run a separate text query for name or description, then merge results via union search. • Use a normalized SKU field can also help for consistent matching across separator formats.
r
@Fanis Tharropoulos Thanks for your response. Running a union/multi search is kinda ugly but I guess I can live with that workaround.
Hi @Fanis Tharropoulos I'm trying to implement a union
multiSearch
(as you recommended) and it appears the
facet_by
and
max_facet_values
params are not being recognized. Is that a bug? I can't find that exception documented in https://typesense.org/docs/30.1/api/federated-multi-search.html or https://typesense.org/docs/30.1/api/search.html#faceting-parameters And if not a bug, how do I get the facets? Do I need to run a third query (regular not
multiSearch
) to get the facets?
f
Union faceting is implemented on the v31 branch, including support for facet_by and max_facet_values in union multi searches. However, v31 RC builds are not available yet, so this is not something you can use in Typesense Cloud today. For the build you’re currently on, what you’re seeing is expected. Union multi-search won’t reliably return merged facets yet. Until v31 is available, if you need facets alongside a union-style result set, the practical workaround is to run a separate faceting query. If avoiding an extra network round trip is important, you could also use federated multi_search and include an additional faceting search in the same request, but that would still be a separate search operation, not facets computed from the union response itself. Using a normalized value of the SKUS, in order for the prefix match to take place will be the most straight-forward approach to this.
r
@Fanis Tharropoulos Thanks for your detailed response. Glad to see facet_by support will be included in the v31 release.
@Fanis Tharropoulos Regarding normalized values for SKUs, I'm guessing the normalized version of a SKU would be to remove the hyphens such that SKU "ABC-123" would be "ABC123"? But wouldn't that be the same normalized SKU for "AB-C1-23" and "A-B-C-1-2-3". These of course are extreme examples but the point is to show that normalizing SKUs by removing hyphens creates new problems in returning a single exact-match SKU for a given query.
f
Removing hyphens naively, like turning ABC-123 into ABC123, can create collisions between different SKUs, so it should not replace the original SKU field. The safer approach is to keep the original SKU field for exact matching, and use a second normalized field / array string field with different SKU permutations only as a fallback for more tolerant matching. That way, exact SKU lookups still use the real value, and the normalized version only helps catch cases where punctuation handling would otherwise miss a result.
👍 1