https://typesense.org logo
Join Slack
Powered by
Good day : ) I’m having a bit of a hassle with re...
# community-help
h
Good day : ) I’m having a bit of a hassle with request times and I was wondering maybe someone can point me to the right direction. It’s not the search speed that’s the problem, but the total request time for client. Typesense says that the search itself takes anywhere between 10-200ms depending on the query, but the request itself tends to take ~2s. I’m also seeing this result when using the search directly on the TS cloud, as seen on the attached screenshot, the actual search time reported by TS was 4ms. Our node is in Cape Town, but even for South African users the times are above ~1s so it still feels wrong. For our own API which runs in Frankfurt the server response time is around 300ms and this also includes the search which in our own API is, well close to 300ms. I’m currently looking for any hints what to check / debug / try. I’m also thinking about hosting the TS cluster myself but it’s a setup chore I wouldn’t want to pick up right now. Our TS cluster ID: 
3qpv0utayifonzgkp
e
What kind of document is it? Is there maybe a lot of data in the returned records?
I've run into this when indexing the contents of books, finding that needle in a haystack can be very slow
But also, when doing embeddings, if the vectors are not excluded the payload can be very large
h
I’m not sure what considers a lot of data to you, but I wouldn’t think so no. See example record below.
The fact that it’s a book is just chance, we usually have fashion items
k
👋 Can you please tell me which collection you are querying on this cluster?
h
Of course, 
sa_production_products_v2
k
Ok in general, the total time taken end to end is: 1. Search time 2. Network round-trip time 3. Time taken to transfer the data over the wire If your collection has a lot of data or your per_page is large, no. 3 above can take some time.
To figure out if it's indeed the data transfer time, you can experiment by setting 
include_fields: id
to fetch only the ID to see if it's faster. This way we can find out what's slowing it down.
h
Hmm, I wouldn’t think any of these would take as long as 1.5s really. We’re looking at 17kb transfer, I don’t think it should take 1.5s even if the DC is in Cape Town Just tested @ screens
k
Can you please share a curl request that I can try from my end?
You can right click > copy as CURL. You can DM me.
h
Sure, I’m just using the cloud UI to do those searches tho:
Copy code
curl '<https://cloud.typesense.org/clusters/3qpv0utayifonzgkp/api/multi_search>' -X POST -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:132.0) Gecko/20100101 Firefox/132.0' -H 'Accept: application/json, text/plain, */*' -H 'Accept-Language: en-US,en;q=0.5' -H 'Accept-Encoding: gzip, deflate, br, zstd' -H 'Referer: <https://cloud.typesense.org/clusters/3qpv0utayifonzgkp/collections/sa_production_products_v2/documents/search>' -H 'Content-Type: text/plain' -H 'X-TYPESENSE-API-KEY: ehgf3scawvy6t20il158nd7oj4uqpbxk' -H 'X-CSRF-TOKEN: wFyg0RNZqHOdCxlbt6eEKtpU-PvbZKPy3N223QwaChSu8UnizzYXN8IgHShSY9qMdl8bnAuU8KJf4eU_FkBS3g' -H 'Origin: <https://cloud.typesense.org>' -H 'DNT: 1' -H 'Connection: keep-alive' -H 'Cookie: _typesense_cloud_app_session=jMFfy27osGv66HJyh5T9h5JRy6LfJ7LgK6CZrg1LjeUKMi%2F03ckSz4y6LygF5KNKDFldgC4%2Fu4hNpvMorKdOul%2BD5RdgqxUcTFpwCVgFv0Qt0c2P2VZlUI3PgCTurQMaRKkVKFdLS8FIiA0UuvvyCYdqXUAcdTUAQSu73WrE9hIfVqpWjxpUNWgOroowcdf6OISOyBt19PSz58g1178dH4tBZ55jdfY3Z12vrgeWkUU5natepgl10Z7x3VsDp2Hi%2F9rVptH%2FyzpvGjfFa47zccewVphMjA4BxmjeYPy5Py3F8NQmq59Ktd%2B4BZNCCqZhvMxDEYDqJOCLmiqOBZDExcilR75nBmtKb5ZfwWHVaakdAY9mfjqze7J2n%2Bo9oQbOyl%2FGZk2wsOPjNfz5nZsAhjk9hsTZs3M9dd%2B4Vv2cxzPBrDj6ceMzIbUg6pRaxkr2aCewhaRy0frVJCmbl85j4TtlttCKkZs72hwA654rMf5bu7WbpClUblm7IJ2%2F%2BXP%2BHvwsibLsA9%2FRfeftS3XBrrNpKYNzDM2MGLlJKERi%2FKbp0wKgTq8ygcwr%2BemQYy%2B3Dgva62BS6flzex9j9QK2s2T0uEZQQXv0prRG7NKyLjjVi428b6FgB1b8pyJztmsY8h9cuPrfk1E2Rr438ByQBOlJdhbwUhiwOl80qq7e%2Fa8XqUF%2FPdVYs7cxbYQ6N90n5QW77ViOBrwEQMbHM5wCZpInU3Q%2FEynta3GslbLk1OijUaLTjf7oHusJlAsOUi5d--%2BtdZOWckOp9DyD2V--r6Sb5iiBBWs3i%2BJ0KnCPCQ%3D%3D; _gcl_au=1.1.338719302.1730751712; _ga_XWJNP9CSY7=GS1.1.1731052021.1.1.1731053454.60.0.0; _ga=GA1.1.465754928.1731052021; __stripe_mid=086ca87f-61d1-4d8e-8a50-e65c97f637ded3fb47; __stripe_sid=99a903d0-9875-4eef-8a84-851b74028851a95595' -H 'Sec-Fetch-Dest: empty' -H 'Sec-Fetch-Mode: cors' -H 'Sec-Fetch-Site: same-origin' -H 'TE: trailers' --data-raw '{"searches":[{"query_by":"address,aiTags,description","sort_by":"_text_match:desc","highlight_full_fields":"address,aiTags,description","collection":"sa_production_products_v2","q":"red dress","page":1,"per_page":6}]}'
k
Cloud UI is not a good representative because it's being fetched from our backend.
h
2.2s from terminal with CURL
k
If you notice, in the curl above, it's calling the 
<http://cloud.typesense.org|cloud.typesense.org>
URL which hits our backend.
What we need is hitting the cluster API directly from the browser.
h
sure, let me get
give me a moment
Copy code
curl '<https://3qpv0utayifonzgkp-1.a1.typesense.net/collections/sa_production_products_v2/documents/search?preset=search_all&per_page=60&q=red+dress&page=1&sort_by=_eval(pinnedAt:true):desc,listedAt:desc,_text_match:desc&filter_by=>' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:132.0) Gecko/20100101 Firefox/132.0' -H 'Accept: application/json, text/plain, */*' -H 'Accept-Language: en-US,en;q=0.5' -H 'Accept-Encoding: gzip, deflate, br, zstd' -H 'X-TYPESENSE-API-KEY: molRt93kKF9w9CIZwCeGZyM3Fm78mPMB' -H 'DNT: 1' -H 'Connection: keep-alive' -H 'Referer: <https://www.yaga.co.za/search/red%20dress>' -H 'Cookie: _ga_8TKTHNEF5T=GS1.1.1729491358.11.0.1729491358.60.0.0; _ga=GA1.1.855395626.1716990785; OptanonConsent=isGpcEnabled=0&datestamp=Tue+Jul+09+2024+22%3A47%3A26+GMT%2B0300+(Eastern+European+Summer+Time)&version=202404.1.0&browserGpcFlag=0&isIABGlobal=false&hosts=&landingPath=NotLandingPage&groups=C0001%3A1%2CC0002%3A1%2CC0003%3A1%2CC0004%3A1&AwaitingReconsent=false&geolocation=%3B; OptanonAlertBoxClosed=2024-06-03T21:21:51.308Z; _ga_N57BRMD2J4=GS1.1.1720554446.39.0.1720554446.60.0.0; __exponea_etc__=3f9c9dbb-3b1d-42c1-9077-b1905e5c7214; _ga_ZLWNJQ3LYX=GS1.1.1730133961.10.0.1730133961.0.0.0; token=Bearer%20eyJhbGciOiJFUzUxMiIsInR5cCI6IkpXVCJ9.eyJpZCI6NTM2NDAxOSwicm9sZSI6InVzZXIiLCJhY3RpdmUiOnRydWUsImlhdCI6MTczMTA1Nzg2MSwiZXhwIjoxNzMzNjQ5ODYxfQ.AKgpPVVmPDPUfmH_11jD4eryGNKImcBwxRjK4Q0BNtTCmRrUB-pWDLEBci2QLUoJqh3eFy_Zn3CKMApT3PirD4tJAQQdFiammlNFihwRGcU6s9nQT8j7BIQHHhNWyBPeForiHUxviOsXENppOrXmA6IeV_b5UcS6GZYvGG1WW8uaxZBO; yaga_u_id=yaga_35eccf7d-97a0-4b6b-a32c-352d976bdd7c; __exponea_time2__=0.010387659072875977' -H 'Sec-Fetch-Dest: empty' -H 'Sec-Fetch-Mode: cors' -H 'Sec-Fetch-Site: same-origin' -H 'TE: trailers'
Directly against the TS node, this is an actual search from our production env
curl takes around a second
k
Are you also in Capetown?
h
I am not, but our customers are and our metrics are logging even higher request times for our users. I also let one of our employees from cape town test our prod search and he saw similar 1.5s+ times
I’m located in EU, Tallinn, Estonia
but even then, round trip to Cape Down DC should not take 1s
k
Let me look into what could be happening and get back to you.
h
Great, thanks. Random speed tests show latency to af-south-1 to be around 200 / 300ms If TS search time is 70ms, I would expect the total request time to be max 700ms for myself and around 100-200ms for people in Africa.
k
Here's how it looks like for me (from India)
Averaging 450 ms. 
/health
takes 320 ms.
h
Can I include the auth token in the URL somehow? I’ll let my ZA colleague to retry this.
This is sensible @ your request time
Alright, thanks anyway, I’ll try to conduct more tests with someone from SA to see what are they seeing.
m
👋 if it helps, that’s how it looks from NZ
Copy code
dnslookup: 0.001712
connect: 0.423362
appconnect: 1.289565
pretransfer: 1.289733
starttransfer: 1.906508
total: 2.327151
size: 30791
fells like TLS/SSL handshakes take most of the time
k
You can send the api key via get parameter ?x-typesense-api-key=X
The first request will involve TCP handshake and ssl certificate. But with keep alive in curl and browser, repeat requests should be much faster.
👍 1
h
Ah perfect
m
yep, looks better in the browser
👍 1
h
Moving the cluster between regions is not possible I suppose? So I should provision new cluster, re-populate the collections and then switch our production to use it?
k
Yes you've to recreate it
h
gotcha, thanks.
Open in Slack
PreviousNext
Powered by