Hey there, I have people records in Typesense whi...
# community-help
m
Hey there, I have people records in Typesense which is around 100M. I want to sort by available email and phone number, so I added
has_email
and
has_phone
boolean fields with
index:true
and
sort:true
. Searching now takes 15 seconds with sorting. How can I optimize this? I need to recommend records which has email and phone both available. Here is the search query:
Copy code
{
  highlight_fields: 'none',
  include_fields: 'id,organization_id,linkedin_url,email,email_status,mobile_number,title,seniority,job_start_date,first_name,last_name,name,city,state,country,country_code,country_region,organizations.id,organizations.website_url,organizations.linkedin_url,organizations.founded_year,organizations.primary_domain,organizations.name,organizations.phone',
  sort_by: 'has_email : desc, has_phone : desc',
  page: 1,
  per_page: 25,
  q: '*',
  use_cache: false
}
f
For this use-case, a better approach would be to sort based on filter conditions instead: http://localhost:8080/docs/guide/ranking-and-relevance.html#ranking-based-on-conditions You'd do
_eval([(has_email:true):3, (has_phone):2])):desc
And now you don't need to have
sort: true
on either field
m
i tried this
_eval([(has_email:true):3, (has_phone:true):2])):desc
it but it doesn't give similar results as
sort_by: 'has_email : desc, has_phone : desc'
f
If you want to sort by the document having BOTH a phone and an email, then
_eval([(has_email:true && has_phone:true):3, (has_phone):2, (has_email:true):2 ])):desc
is probably the best. It prioritizes documents that have both an email and a phone number and then still ranks documents that have at least one of the two higher than the rest
🙌 1
Your original query just found the ones that have an email and then returned them all first (sorting on booleans isn't ideal), and then the ones that had a phone and returned all of them after
m
Okay after trying above one, the result latency came down from 15 second to 8 seconds, this is good, but could it still be further optimized.? is there any performance optimization tips?.
f
Are you self hosting or are you using Typesense cloud?
m
This is self hosted cluster,
f
It shouldn't take 8 seconds. Are you just sorting by the eval clause and
q=*
?
m
Yes, this is the query
Copy code
{
  highlight_fields: 'none',
  include_fields: 'id,organization_id,linkedin_url,email,email_status,mobile_number,title,seniority,job_start_date,first_name,last_name,name,city,state,country,country_code,country_region,organizations.id,organizations.website_url,organizations.linkedin_url,organizations.founded_year,organizations.primary_domain,organizations.name,organizations.phone',
  sort_by: '_eval([(has_email:true && has_phone:true):3, (has_phone):2, (has_email:true):2 ])):desc',
  page: 1,
  per_page: 25,
  q: '*',
  use_cache: false
}
f
Why is
use_cache
set to false? Also, what's your CPU and RAM usage like?
Did you set the
sort
parameter to false for the two boolean fields?
m
both index has
sort:true
and
index: true
, there is no major production traffic. i use r7g.8xlarge instance
f
You should set them to false. What Typesense version are you using?
m
version: 27.1
f
Try updating to
v29.0rc29
m
You mean updating
sort: false
for both the indexes?
f
Yes, since you're not using them for sorting anymore
m
Okay, but still doesn't it should give better latency even though its not using sorting indexes?
f
Can you call out to the
/stats.json
and the
/metrics.json
endpoints? https://typesense.org/docs/0.20.0/api/cluster-operations.html#cluster-metrics
m
matrics.json
Copy code
{
  "system_cpu10_active_percentage": "0.00",
  "system_cpu11_active_percentage": "0.00",
  "system_cpu12_active_percentage": "0.00",
  "system_cpu13_active_percentage": "0.00",
  "system_cpu14_active_percentage": "0.00",
  "system_cpu15_active_percentage": "0.00",
  "system_cpu16_active_percentage": "0.00",
  "system_cpu17_active_percentage": "0.00",
  "system_cpu18_active_percentage": "0.00",
  "system_cpu19_active_percentage": "0.00",
  "system_cpu1_active_percentage": "100.00",
  "system_cpu20_active_percentage": "0.00",
  "system_cpu21_active_percentage": "0.00",
  "system_cpu22_active_percentage": "0.00",
  "system_cpu23_active_percentage": "0.00",
  "system_cpu24_active_percentage": "0.00",
  "system_cpu25_active_percentage": "0.00",
  "system_cpu26_active_percentage": "0.00",
  "system_cpu27_active_percentage": "0.00",
  "system_cpu28_active_percentage": "0.00",
  "system_cpu29_active_percentage": "0.00",
  "system_cpu2_active_percentage": "0.00",
  "system_cpu30_active_percentage": "0.00",
  "system_cpu31_active_percentage": "0.00",
  "system_cpu32_active_percentage": "0.00",
  "system_cpu3_active_percentage": "0.00",
  "system_cpu4_active_percentage": "0.00",
  "system_cpu5_active_percentage": "0.00",
  "system_cpu6_active_percentage": "0.00",
  "system_cpu7_active_percentage": "0.00",
  "system_cpu8_active_percentage": "0.00",
  "system_cpu9_active_percentage": "0.00",
  "system_cpu_active_percentage": "3.12",
  "system_disk_total_bytes": "527295578112",
  "system_disk_used_bytes": "200185749504",
  "system_memory_total_bytes": "265475792896",
  "system_memory_total_swap_bytes": "0",
  "system_memory_used_bytes": "129162891264",
  "system_memory_used_swap_bytes": "0",
  "system_network_received_bytes": "456099623132",
  "system_network_sent_bytes": "52271553736",
  "typesense_memory_active_bytes": "102067798016",
  "typesense_memory_allocated_bytes": "93931910896",
  "typesense_memory_fragmentation_ratio": "0.08",
  "typesense_memory_mapped_bytes": "124438773760",
  "typesense_memory_metadata_bytes": "19556841344",
  "typesense_memory_resident_bytes": "102067798016",
  "typesense_memory_retained_bytes": "2121363554304"
}
stats.json
Copy code
{
  "delete_latency_ms": 0,
  "delete_requests_per_second": 0,
  "import_latency_ms": 0,
  "import_requests_per_second": 0,
  "latency_ms": {
    "GET /health": 0,
    "GET /metrics.json": 103,
    "GET /stats.json": 0
  },
  "overloaded_requests_per_second": 0,
  "pending_write_batches": 1,
  "requests_per_second": {
    "GET /health": 0.1,
    "GET /metrics.json": 0.5,
    "GET /stats.json": 0.5,
    "POST /multi_search": 0.2
  },
  "search_latency_ms": 0,
  "search_requests_per_second": 0,
  "total_requests_per_second": 1.3,
  "write_latency_ms": 0,
  "write_requests_per_second": 0
}
f
Your cluster has 129GB worth of documents. When using
*
for searching, we have to run that eval on every document on that collection, which could be tens of millions, or even hundreds of millions. There isn't a better way to approach this currently, sadly.
m
I was wondering if it's possible to utilize multicore CPUs or additional RAM for this. I'm not an expert, so I appreciate your insights.
f
It's sadly tied to the way we execute the query on the CPU. Since we're looping on every document, we can't break this into multiple threads for that loop only
👍 1