Hey guys! I've upgraded my production cluster from...
# community-help
m
Hey guys! I've upgraded my production cluster from v28 to v29.rc29 to use new features, but I encountered significantly slower performance. I typically run 'control' queries to monitor performance changes, and I noticed all these queries slowed down. For example, a query involving a join between a collection of 120k records and another with 10 million records, applying complex filters, group_by, and sort operations, increased from 500ms to 950ms. Other simpler query increased from 40ms to 90ms in average. Do you have any internal benchmarks that track performance changes between versions? I ended up downgrading back to v28 due to this issue and everything is ok again. Is that a known issue?
f
Our benchmarks run every 6 hours and we keep track of the commits of the previous 30 days: https://github.com/typesense/typesense/actions/workflows/benchmark-testing.yml In v29 there have been improvements to grouping performance. Were there any writes active when you tested with the v29 rc builds?
m
Really nice to see you guys keep a benchmark running. As I was upgrading the cluster, I stopped all writes until the upgrade was finished and tested.
f
Do you have a timeframe for this? Did this happen in the past 24h?
m
yes. I've upgraded yesterday then I noticed the problem right away and downgraded to 28.
Using your benchmarks, is there anyway to check the difference between v28 and the latest v29?
I can't find the results there
f
It goes commit hash by commit hash. We have hard caps for performance regressions so they would fail if there were any problems
m
I see. I'll download a snapshot to my machine and test against both versions again. I don't want to upgrade production nodes again just to test it
I got a snapshot here. Running on my machine, my testing query jumped from 120-150ms (v28) to 220-250ms (v29.rc29). If I remove the group_by parameter, there's no difference between the versions, both versions are about 100-110ms. I removed facets and sorting and they don't make difference in the results. So, it seems the issue is on the group_by changes you guys made. Can you confirm which RC does not have the group_by changes? Here is an example of my multisearch query:
Copy code
{
  "searches": [
    {
      "collection": "v3_skus",
      "q": "*",
      "group_by": "product_id",
      "group_limit": 99,
      "include_fields": "product_id,sku_id,seller_id,sku_name,product_name",
      "facet_by": "brand_facet,category_facet",
      "filter_by": "( (seller_id:66a9981f9e6d8686ba64a3f3) || (seller_id:6710ec596dd49c5db66804a4) || (seller_id:67d345ee0fb810922965009c && attribute_business:['Company Health','Company Health Medicine','Company','Essential Home']) || (seller_id:67d345b76a00b7b0951b4a71 && attribute_business:['Company Health Medicine','Company Health','Company','Essential Home']) || (seller_id:67d344896a00b7b0951b4a6f && attribute_business:['Company Health Medicine','Company','Company Health','Essential Home']) || (seller_id:67d36ab80fb81092296500b8 && attribute_business:['Company Health Medicine','Company Health','Company','Essential Home']) || (seller_id:67d36ae40fb81092296500ba && attribute_business:['Company Health Medicine','Company','Company Health','Essential Home']) || (seller_id:670972463814f3334d650c74 && attribute_business:['Company Health']) || (seller_id:670972463814f3334d650c74 && attribute_business:['Company']) || (seller_id:670972463814f3334d650c74 && attribute_business:['Company Health Medicine']) || (seller_id:670972463814f3334d650c74 && attribute_business:['Essential Home']) ) && $v5_offers(paymentFormId:!'null' && ((seller_id:66a9981f9e6d8686ba64a3f3 && clusters:['null']) || (seller_id:6710ec596dd49c5db66804a4 && clusters:['null']) || (seller_id:67d345ee0fb810922965009c && clusters:['null']) || (seller_id:67d345b76a00b7b0951b4a71 && clusters:['null']) || (seller_id:67d344896a00b7b0951b4a6f && clusters:['null']) || (seller_id:67d36ab80fb81092296500b8 && clusters:['null']) || (seller_id:67d36ae40fb81092296500ba && clusters:['null']) || (seller_id:670972463814f3334d650c74 && clusters:['null']))) && seller_active:true && sku_active:true && category_active:true && brand_active:true && availableToShowroom:true",
      "sort_by": "_eval([(available:true):-1, (seller_id:66a9981f9e6d8686ba64a3f3):200000, (seller_id:6710ec596dd49c5db66804a4):300000, (seller_id:67d345ee0fb810922965009c):700000, (seller_id:67d345b76a00b7b0951b4a71):800000, (seller_id:67d344896a00b7b0951b4a6f):400000, (seller_id:67d36ab80fb81092296500b8):500000, (seller_id:67d36ae40fb81092296500ba):600000, (seller_id:670972463814f3334d650c74):100000]):asc,last_update:desc"
    }
  ]
}
f
Any RC build from
29.0.rc5
onwards has those changes. Group by performance was improved as a result of it. Can you provide a set of reproducible steps so we can test it ourselves and see if things can be improved on the query level?
m
@Fanis Tharropoulos I'm going to test some of those earlier RC builds to see the results here.
v28 and v29-rc3 the response time are the same. close to 100ms. v29-rc29 the response time jump to about 200ms I'll keep testing other versions here, I'll try to identify which specific RC is causing this issue for my scenario.
f
Could you first try rc28 / rc27?
m
RC5 the response time is already at ~200ms
so... it's probably the RC4
Starting the RC4 now to see...
ops.... sorry... the RC5 is the issue. Like you said. it was the RC5 where changes were made. but I'm starting the RC4 here just to make sure.
f
So, could you share a subset of the dataset that you're running and so we can debug it after launching
v29.0
so we can see what we can do in terms of refactoring? The changes implemented for grouping in
v29.0
improve RAM usage, but maybe the CPU impact on your queries was too much. You can send it through DMs if you prefer.
m
Tested on RC4 and the response time is also ~200ms so... RC4 is the RC where this issue started. I'll send you in private the whole data and the query.
The data is on your DM. Thanks for your help!
f
Hey, we just released a fix for this in
rc31
. Can you try updating to that version and see if it fixes your performance issues?
m
Niiice! I'm gonna test it right now.
perfect! now the response time is indeed much better!
🙌 1
thank you so much for listening my scenario and find a fix before the final release
❤️ 1
you guys rocks!
❤️ 1