I was wondering if all queries are diacritic-insen...
# community-help
b
I was wondering if all queries are diacritic-insensitive with default locales of "en" both in the query and in the search index. I am noticing that I am not getting results that don't exactly match diacritics even though I never set the locale of my fields.
f
Could you post a small reproducible example like this one here?
👀 1
b
Copy code
### Run Typesense via Docker ########################################
set -x

export TYPESENSE_API_KEY=xyz
export TYPESENSE_HOST=http://localhost:8108

docker stop typesense-repro 2>/dev/null
docker rm typesense-repro 2>/dev/null
rm -rf "$(pwd)"/typesense-data-dir-repro
mkdir "$(pwd)"/typesense-data-dir-repro

# Wait for Typesense to be ready
docker run -d -p 8108:8108 --name typesense-repro \
            -v"$(pwd)"/typesense-data-dir-repro:/data \
            typesense/typesense:29.0.rc31 \
            --data-dir /data \
            --api-key=$TYPESENSE_API_KEY \
            --enable-cors

# Wait till typesense is ready.
until curl -s -o /dev/null -w "%{http_code}" "$TYPESENSE_HOST/health" -H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}" | grep -q "200"; do
  sleep 2
done

curl -s "$TYPESENSE_HOST/debug" \
       -H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}" | jq


curl -s "$TYPESENSE_HOST/collections" \
       -X POST \
       -H "Content-Type: application/json" \
       -H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}" \
       -d '
          {
             "name": "companies",
             "fields": [
               {"name": "company_name", "type": "string" },
               {"name": "num_employees", "type": "int32" },
               {"name": "country", "type": "string", "facet": true }
             ],
             "default_sorting_field": "num_employees"
           }' | jq

curl -s "$TYPESENSE_HOST/collections/companies/documents/import?action=create" \
        -H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}" \
        -H "Content-Type: text/plain" \
        -X POST \
        -d '{"id": "124","company_name": "Kṛṣṇa Industries","num_employees": 5215,"country": "USA"}
            {"id": "125","company_name": "Krsna Corp","num_employees": 2133,"country": "CA"}'  | jq

curl -s "$TYPESENSE_HOST/multi_search" \
        -X POST \
        -H "Content-Type: application/json" \
        -H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}" \
        -d '
          {
            "searches": [
              {
                "collection": "companies",
                "q": "Krsna",
                "query_by": "company_name"
              }
            ]
          }'  | jq

docker stop typesense-repro
docker rm typesense-repro

### Documentation ######################################################################################
# Visit the API reference section: https://typesense.org/docs/28.0/api/collections.html
# Click on the "Shell" tab under each API resource's docs, to get shell commands for other API endpoints
Only returns 1 hit instead of the expected 2
@Fanis Tharropoulos
Copy code
+-zsh:699> export TYPESENSE_API_KEY=xyz
+-zsh:700> export TYPESENSE_HOST=http://localhost:8108
+-zsh:702> docker stop typesense-repro
+-zsh:703> docker rm typesense-repro
+-zsh:704> pwd
+-zsh:704> rm -rf /Users/balaramdb/typesense-data-dir-repro
+-zsh:705> pwd
+-zsh:705> mkdir /Users/balaramdb/typesense-data-dir-repro
+-zsh:708> pwd
+-zsh:708> docker run -d -p 8108:8108 --name typesense-repro -v/Users/balaramdb/typesense-data-dir-repro:/data typesense/typesense:29.0.rc31 --data-dir /data '--api-key=xyz' --enable-cors
6fe058251ce788136ac13db26e8fafb855f9d736c76eb9b56055db810477e886
+-zsh:716> curl -s -o /dev/null -w '%{http_code}' http://localhost:8108/health -H 'X-TYPESENSE-API-KEY: xyz'
+-zsh:716> grep '--color=auto' '--exclude-dir=.bzr' '--exclude-dir=CVS' '--exclude-dir=.git' '--exclude-dir=.hg' '--exclude-dir=.svn' '--exclude-dir=.idea' '--exclude-dir=.tox' -q 200
+-zsh:717> sleep 2
+-zsh:716> curl -s -o /dev/null -w '%{http_code}' http://localhost:8108/health -H 'X-TYPESENSE-API-KEY: xyz'
+-zsh:716> grep '--color=auto' '--exclude-dir=.bzr' '--exclude-dir=CVS' '--exclude-dir=.git' '--exclude-dir=.hg' '--exclude-dir=.svn' '--exclude-dir=.idea' '--exclude-dir=.tox' -q 200
+-zsh:717> sleep 2
+-zsh:716> curl -s -o /dev/null -w '%{http_code}' http://localhost:8108/health -H 'X-TYPESENSE-API-KEY: xyz'
+-zsh:716> grep '--color=auto' '--exclude-dir=.bzr' '--exclude-dir=CVS' '--exclude-dir=.git' '--exclude-dir=.hg' '--exclude-dir=.svn' '--exclude-dir=.idea' '--exclude-dir=.tox' -q 200
+-zsh:720> curl -s http://localhost:8108/debug -H 'X-TYPESENSE-API-KEY: xyz'
+-zsh:721> jq
{
  "state": 1,
  "version": "29.0.rc31"
}
+-zsh:724> curl -s http://localhost:8108/collections -X POST -H 'Content-Type: application/json' -H 'X-TYPESENSE-API-KEY: xyz' -d $'\n          {\n             "name": "companies",\n             "fields": [\n               {"name": "company_name", "type": "string" },\n               {"name": "num_employees", "type": "int32" },\n               {"name": "country", "type": "string", "facet": true }\n             ],\n             "default_sorting_field": "num_employees"\n           }'
+-zsh:737> jq
{
  "created_at": 1765971573,
  "default_sorting_field": "num_employees",
  "enable_nested_fields": false,
  "fields": [
    {
      "facet": false,
      "index": true,
      "infix": false,
      "locale": "",
      "name": "company_name",
      "optional": false,
      "sort": false,
      "stem": false,
      "stem_dictionary": "",
      "store": true,
      "type": "string"
    },
    {
      "facet": false,
      "index": true,
      "infix": false,
      "locale": "",
      "name": "num_employees",
      "optional": false,
      "sort": true,
      "stem": false,
      "stem_dictionary": "",
      "store": true,
      "type": "int32"
    },
    {
      "facet": true,
      "index": true,
      "infix": false,
      "locale": "",
      "name": "country",
      "optional": false,
      "sort": false,
      "stem": false,
      "stem_dictionary": "",
      "store": true,
      "type": "string"
    }
  ],
  "name": "companies",
  "num_documents": 0,
  "symbols_to_index": [],
  "token_separators": []
}
+-zsh:739> curl -s 'http://localhost:8108/collections/companies/documents/import?action=create' -H 'X-TYPESENSE-API-KEY: xyz' -H 'Content-Type: text/plain' -X POST -d $'{"id": "124","company_name": "Kṛṣṇa Industries","num_employees": 5215,"country": "USA"}\n            {"id": "125","company_name": "Krsna Corp","num_employees": 2133,"country": "CA"}'
+-zsh:744> jq
{
  "success": true
}
{
  "success": true
}
+-zsh:746> curl -s http://localhost:8108/multi_search -X POST -H 'Content-Type: application/json' -H 'X-TYPESENSE-API-KEY: xyz' -d $'\n          {\n            "searches": [\n              {\n                "collection": "companies",\n                "q": "Krsna",\n                "query_by": "company_name"\n              }\n            ]\n          }'
+-zsh:759> jq
{
  "results": [
    {
      "facet_counts": [],
      "found": 1,
      "hits": [
        {
          "document": {
            "company_name": "Krsna Corp",
            "country": "CA",
            "id": "125",
            "num_employees": 2133
          },
          "highlight": {
            "company_name": {
              "matched_tokens": [
                "Krsna"
              ],
              "snippet": "<mark>Krsna</mark> Corp"
            }
          },
          "highlights": [
            {
              "field": "company_name",
              "matched_tokens": [
                "Krsna"
              ],
              "snippet": "<mark>Krsna</mark> Corp"
            }
          ],
          "text_match": 578730123365187705,
          "text_match_info": {
            "best_field_score": "1108091338752",
            "best_field_weight": 15,
            "fields_matched": 1,
            "num_tokens_dropped": 0,
            "score": "578730123365187705",
            "tokens_matched": 1,
            "typo_prefix_score": 0
          }
        }
      ],
      "out_of": 2,
      "page": 1,
      "request_params": {
        "collection_name": "companies",
        "first_q": "Krsna",
        "per_page": 10,
        "q": "Krsna"
      },
      "search_cutoff": false,
      "search_time_ms": 0
    }
  ]
}
+-zsh:761> docker stop typesense-repro
typesense-repro
+-zsh:762> docker rm typesense-repro
typesense-repro
+precmd:1> [ 3 ']'                                                                                                         
+precmd:3> timer_show=7 
+precmd:4> export ZSH_THEME_PROMPT_TIMER=7s
+precmd:5> unset timer
+warp_set_title_idle_on_precmd:1> [[ '' != true ]]
+warp_set_title_idle_on_precmd:2> [[ 1 == 1 ]]
+warp_set_title_idle_on_precmd:3> warp_title '%n@%m:%~'
+warp_title:1> DISABLE_AUTO_TITLE=true 
+warp_title:2> setopt localoptions nopromptsubst
+warp_title:3> [[ -n '' ]]
+warp_title:4> title='%25<..<%n@%m:%~' 
+warp_title:5> print -Pn '\e]0;%25\<..\<%n@%m:%~\a'
+omz_termsupport_precmd:1> [[ true != true ]]
+omz_termsupport_precmd:1> return
+warp_precmd:1> local exit_code=0
+warp_precmd:2> warp_send_json_message '{"hook": "CommandFinished", "value": {"exit_code": 0, "next_block_id": "precmd-17659715652488-2"}}'
+warp_send_json_message:1> warp_hex_encode_string '{"hook": "CommandFinished", "value": {"exit_code": 0, "next_block_id": "precmd-17659715652488-2"}}'
+warp_hex_encode_string:1> printf %s '{"hook": "CommandFinished", "value": {"exit_code": 0, "next_block_id": "precmd-17659715652488-2"}}'
+warp_hex_encode_string:1> od -An -v -tx1
+warp_hex_encode_string:1> tr -d ' \n'
+warp_send_json_message:1> local msg=7b22686f6f6b223a2022436f6d6d616e6446696e6973686564222c202276616c7565223a207b22657869745f636f6465223a20302c20226e6578745f626c6f636b5f6964223a2022707265636d642d31373635393731353635323438382d32227d7d
+warp_send_json_message:2> [ false '=' true ']'
+warp_send_json_message:5> printf %b%b%s%b $'\C-[P$' d 7b22686f6f6b223a2022436f6d6d616e6446696e6973686564222c202276616c7565223a207b22657869745f636f6465223a20302c20226e6578745f626c6f636b5f6964223a2022707265636d642d31373635393731353635323438382d32227d7d $'\M-\C-\'
Don't get this issue for some special chars like ā or ī. But doesn't seem to work for ṇ, ṛ, etc.
The unicode normalizer only works for latin-extended chars but not for the special ones I am working with
a
Hey @Balaram Behera When using the default locale (en), Typesense won't always be able to tokenize diacritics, which makes it index them as it is. Therefore, if the query is not exactly the same (that is, also containing diacritics), it won't match. We don't have a specific locale for sanskrit, but using the russian locale worked on your example.
Yeah, when using EN locale it will use ASCII//TRANSLIT, while on locales with special characters it will use ICU's transliteration.
b
Thanks for that info! Is there a way to create a custom tokenizer? Or does that require forking the codebase?
I would prefer English tokenizer and stemmer because most words are in English with some Sanskrit transliteration in there too.
@Alan Martini
Is it possible to change the English normalization to ICU here? Or does it need to be iconv (I see a comment already there recognizing that this is possible).
a
I'm not sure on the technical decision that led to not use ICU in locale EN. Will defer this one to @Kishore Nallan You can create a fork and make a pull request, maybe add a double locale logic (en and ru)
👍 1
b
I would prefer "en" stemming even for the sanskrit words, so I don't think the "ru" direction is needed for me. Just want the index to be diacritic insensitive.
Lmk if you want me to make a PR, not super familiar with C++
a
Hmmm I don't think is viable for us to make such a agressive change. If we would implement it, would be as a additional path, not overwritting the current one. CC: @Kishore Nallan
b
Makes sense, can i add a locale of "en+" for this section?
a
As a side note, thanks for contributing, it's always cool to receive prs
❤️ 1
I think is a good path! But you might want to wait our CTO @Kishore Nallan take on this, might be a dead end
👍 1
b
Will do, thanks!
k
ICU is very slow for
en
so we cannot use that. For
en
locale we handle some common latin characters for diacritic removal (within ascii set), but we need preserve other unicode characters outside that. We don't have plans to change this behavior. If you want additional customization you can pre-process your text and query on your application.
b
Got it, thanks