Hello Typesense Community, I'm running a bit into ...
# community-help
m
Hello Typesense Community, I'm running a bit into a problem with trying to find ideal index behavior for an email field. I want to use token_separators to get hits on just a domain name, but we also have some people that copy paste an entire email that with token_separators set up doesn't return a result anymore (I'm not really sure I get why). Would it be better to just use infix instead of token_separators? Can I somehow combine it with symbols_to_index? It's a bit of a rabbit hole it seems for such a common pattern 😕
a
Hey Mirko, Yeah, the approach will be using both token_separators and symbols_to_index.
m
So that means if I have an email like :
foo.bar+test@test-foobar.com
If I have token-separators
[".","+","@","-"]
and `symbols_to_index as the same it will create the following tokens:
foo.bar+test@test-foobar.com
foo
,
bar
,
test
,
foobar
,
com
? Or also all combinations? 🤔
It doesn't seem to work that way unfortunately 😞
a
Hmmm, could you create a reproducible example using the below gist as example? I can then do a thorough investigation on the issue! https://gist.github.com/auth/github?return_to=https%3A%2F%2Fgist.github.com%2Fjasonbosco%2F7c3432713216c378472f13e72246f46b
m
Copy code
### Run Typesense via Docker ########################################
set -x

export TYPESENSE_API_KEY=xyz
export TYPESENSE_HOST=http://localhost:8108

docker stop typesense-repro 2>/dev/null
docker rm typesense-repro 2>/dev/null
rm -rf "$(pwd)"/typesense-data-dir-repro
mkdir "$(pwd)"/typesense-data-dir-repro

# Wait for Typesense to be ready
docker run -d -p 8108:8108 --name typesense-repro \
            -v"$(pwd)"/typesense-data-dir-repro:/data \
            typesense/typesense:29.0.rc31 \
            --data-dir /data \
            --api-key=$TYPESENSE_API_KEY \
            --enable-cors

# Wait till typesense is ready.
until curl -s -o /dev/null -w "%{http_code}" "$TYPESENSE_HOST/health" -H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}" | grep -q "200"; do
  sleep 2
done

curl -s "$TYPESENSE_HOST/debug" \
       -H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}" | jq


curl -s "$TYPESENSE_HOST/collections" \
       -X POST \
       -H "Content-Type: application/json" \
       -H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}" \
       -d '
          {
             "name": "companies",
             "fields": [
               {"name": "company_name", "type": "string" },
               {"name": "num_employees", "type": "int32" },
               {"name": "country", "type": "string", "facet": true },
               {"name": "email", "type": "string", "token_separators": [".","+","@","-"], "symbols_to_index": [".","+","@","-"]}
             ],
             "default_sorting_field": "num_employees"
           }' | jq

curl -s "$TYPESENSE_HOST/collections/companies/documents/import?action=create" \
        -H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}" \
        -H "Content-Type: text/plain" \
        -X POST \
        -d '{"id": "124","company_name": "Stark Industries","num_employees": 5215,"country": "USA", "email": "foo.bar+test@test-foobar.com"}
            {"id": "125","company_name": "Acme Corp","num_employees": 2133,"country": "CA", "email": "foo.bar2+test@test-foobar.com"}'  | jq

curl -s "$TYPESENSE_HOST/multi_search" \
        -X POST \
        -H "Content-Type: application/json" \
        -H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}" \
        -d '
          {
            "searches": [
              {
                "collection": "companies",
                "q": "foo.bar+test@test-foobar.com",
                "query_by": "email"
              }
            ]
          }'  | jq

docker stop typesense-repro
docker rm typesense-repro

### Documentation ######################################################################################
# Visit the API reference section: https://typesense.org/docs/28.0/api/collections.html
# Click on the "Shell" tab under each API resource's docs, to get shell commands for other API endpoints
Copy code
+ curl -s -o /dev/null -w '%{http_code}' http://localhost:8108/health -H 'X-TYPESENSE-API-KEY: xyz'
+ grep -q 200
+ sleep 2
+ curl -s -o /dev/null -w '%{http_code}' http://localhost:8108/health -H 'X-TYPESENSE-API-KEY: xyz'
+ grep -q 200
+ sleep 2
+ curl -s -o /dev/null -w '%{http_code}' http://localhost:8108/health -H 'X-TYPESENSE-API-KEY: xyz'
+ grep -q 200
+ curl -s http://localhost:8108/debug -H 'X-TYPESENSE-API-KEY: xyz'
+ jq
{
  "state": 1,
  "version": "29.0.rc31"
}
+ curl -s http://localhost:8108/collections -X POST -H 'Content-Type: application/json' -H 'X-TYPESENSE-API-KEY: xyz' -d '
          {
             "name": "companies",
             "fields": [
               {"name": "company_name", "type": "string" },
               {"name": "num_employees", "type": "int32" },
               {"name": "country", "type": "string", "facet": true },
               {"name": "email", "type": "string", "token_separators": [".","+","@","-"], "symbols_to_index": [".","+","@","-"]}
             ],
             "default_sorting_field": "num_employees"
           }'
+ jq
{
  "created_at": 1765369216,
  "default_sorting_field": "num_employees",
  "enable_nested_fields": false,
  "fields": [
    {
      "facet": false,
      "index": true,
      "infix": false,
      "locale": "",
      "name": "company_name",
      "optional": false,
      "sort": false,
      "stem": false,
      "stem_dictionary": "",
      "store": true,
      "type": "string"
    },
    {
      "facet": false,
      "index": true,
      "infix": false,
      "locale": "",
      "name": "num_employees",
      "optional": false,
      "sort": true,
      "stem": false,
      "stem_dictionary": "",
      "store": true,
      "type": "int32"
    },
    {
      "facet": true,
      "index": true,
      "infix": false,
      "locale": "",
      "name": "country",
      "optional": false,
      "sort": false,
      "stem": false,
      "stem_dictionary": "",
      "store": true,
      "type": "string"
    },
    {
      "facet": false,
      "index": true,
      "infix": false,
      "locale": "",
      "name": "email",
      "optional": false,
      "sort": false,
      "stem": false,
      "stem_dictionary": "",
      "store": true,
      "symbols_to_index": [
        ".",
        "+",
        "@",
        "-"
      ],
      "token_separators": [
        ".",
        "+",
        "@",
        "-"
      ],
      "type": "string"
    }
  ],
  "name": "companies",
  "num_documents": 0,
  "symbols_to_index": [],
  "token_separators": []
}
+ curl -s 'http://localhost:8108/collections/companies/documents/import?action=create' -H 'X-TYPESENSE-API-KEY: xyz' -H 'Content-Type: text/plain' -X POST -d '{"id": "124","company_name": "Stark Industries","num_employees": 5215,"country": "USA", "email": "foo.bar+test@test-foobar.com"}
            {"id": "125","company_name": "Acme Corp","num_employees": 2133,"country": "CA", "email": "foo.bar2+test@test-foobar.com"}'
+ jq
{
  "success": true
}
{
  "success": true
}
+ curl -s http://localhost:8108/multi_search -X POST -H 'Content-Type: application/json' -H 'X-TYPESENSE-API-KEY: xyz' -d '
          {
            "searches": [
              {
                "collection": "companies",
                "q": "foo.bar+test@test-foobar.com",
                "query_by": "email"
              }
            ]
          }'
+ jq
{
  "results": [
    {
      "facet_counts": [],
      "found": 0,
      "hits": [],
      "out_of": 2,
      "page": 1,
      "request_params": {
        "collection_name": "companies",
        "first_q": "foo.bar+test@test-foobar.com",
        "per_page": 10,
        "q": "foo.bar+test@test-foobar.com"
      },
      "search_cutoff": false,
      "search_time_ms": 0
    }
  ]
}
+ docker stop typesense-repro
typesense-repro
+ docker rm typesense-repro
typesense-repro
There is no hits when running this and I'd expect a hit
f
Hey Mirko, Could you try upgrading to one of our later RC builds for v30 and only use
token_separators
instead of symbols? There are some breaking changes for v30 you should first know about, you can check them out here: https://typesense.org/docs/30.0/api/#what-s-new
m
The repro pulls like the latest rc no? NVM: I see it pulls v29. Let me test 🤔 So you're suggesting I should try this repo without the symbols_to_index? 🤔
f
Yup, try using it without the symbols to index
a
Thanks @Fanis Tharropoulos @Mirko Nitschke Using v30.0.rca33 and only tokens_separators worked
🎉 1
m
indeed! When can we expect a proper release for v30? 🤔
Or would you consider it stable enough for prod use? 🙂
f
RIght around the corner, we're gathering feedback on the changes introduced in v30. It is stable enough to be used on production, but if you're wary, you can wait a bit and we'll announce the release on #important-announcements
m
Thanks. Glad to hear it's working correctly in the latest version. I was really thinking I was doing something wrong 🙈 🙂 (and glad to know there is an active community here to jump on stuff like this) Appreciate it!
🙌 1
f
Any time 😉