Addressing Exceeding Query Length and High CPU Usage

Hi,

We started getting a bunch of Query string exceeds max allowed length of 4000 errors on our servers.

I do not see any queries that would result in an unbounded query to typesense on our client code. I am struggling to trace exactly what query is triggering this behaviour.

• Is this the payload body size or the URL length for GET requests?
• Is there any way of knowing exactly which request is failing - like a URL to narrow it down to a collection?
• Could this be internal traffic between the cluster?
Additional context:
• Running 0.21.0
• Servers have not been rebooted since around October 2021 and have been running fine
• This is the first time we noticed this issue, due to downtime on typesense. The CPU workloadd increased to 100% for around 10 minutes across all 3 servers. We are still fault finding on this to see if any downstream service could have created this pressure - but have not identified anything thus far.
When it does occur, it lasts for around a minute and interestingly, the query string reported seems to be increasing in size over time, so I assume whatever the query is - it is increasing in length on each retry

A sample of the log below:

<... more similar events truncated...>
E20220527 06:23:38.236594 13397 string_utils.cpp:85] Query string exceeds max allowed length of 4000. Actual length: 5657
E20220527 06:23:38.329757 13397 string_utils.cpp:85] Query string exceeds max allowed length of 4000. Actual length: 5660
E20220527 06:23:38.436268 13397 string_utils.cpp:85] Query string exceeds max allowed length of 4000. Actual length: 5663
E20220527 06:23:38.537355 13397 string_utils.cpp:85] Query string exceeds max allowed length of 4000. Actual length: 5666
E20220527 06:23:38.635035 13397 string_utils.cpp:85] Query string exceeds max allowed length of 4000. Actual length: 5669
E20220527 06:23:38.728849 13397 string_utils.cpp:85] Query string exceeds max allowed length of 4000. Actual length: 5672
E20220527 06:23:38.827121 13397 string_utils.cpp:85] Query string exceeds max allowed length of 4000. Actual length: 5675
E20220527 06:23:38.924321 13397 string_utils.cpp:85] Query string exceeds max allowed length of 4000. Actual length: 5678
E20220527 06:23:39.030211 13397 string_utils.cpp:85] Query string exceeds max allowed length of 4000. Actual length: 5681
E20220527 06:23:39.119545 13397 string_utils.cpp:85] Query string exceeds max allowed length of 4000. Actual length: 5684
E20220527 06:23:39.217941 13397 string_utils.cpp:85] Query string exceeds max allowed length of 4000. Actual length: 5687
E20220527 06:23:39.317699 13397 string_utils.cpp:85] Query string exceeds max allowed length of 4000. Actual length: 5690
E20220527 06:23:39.421378 13397 string_utils.cpp:85] Query string exceeds max allowed length of 4000. Actual length: 5693
E20220527 06:23:39.522917 13397 string_utils.cpp:85] Query string exceeds max allowed length of 4000. Actual length: 5696
E20220527 06:23:39.621501 13397 string_utils.cpp:85] Query string exceeds max allowed length of 4000. Actual length: 5699
E20220527 06:23:39.722431 13397 string_utils.cpp:85] Query string exceeds max allowed length of 4000. Actual length: 5702
E20220527 06:23:39.830339 13397 string_utils.cpp:85] Query string exceeds max allowed length of 4000. Actual length: 5705
E20220527 06:23:39.920497 13397 string_utils.cpp:85] Query string exceeds max allowed length of 4000. Actual length: 5708
E20220527 06:23:40.019423 13397 string_utils.cpp:85] Query string exceeds max allowed length of 4000. Actual length: 5711
E20220527 06:23:40.113782 13397 string_utils.cpp:85] Query string exceeds max allowed length of 4000. Actual length: 5714
E20220527 06:23:41.403084 13397 string_utils.cpp:85] Query string exceeds max allowed length of 4000. Actual length: 5716
<... more similar events truncated...>

Thanks!

You can try enabling slow request log with value of 0 : https://typesense.org/docs/0.21.0/api/cluster-operations.html#toggle-slow-request-log

This will log all queries and hopefully catch those long queries as well.

Thanks - I will enable that and hopefully get some insight into the long queries. Is there anything else I can look for regarding the high CPU usage - either in the logs or elsewhere?

Check ifconfig to see incoming IPs and spot check them to see whether they are bots or from random countries.

We currently do not have reason to suspect DDoS . One reason why is because our network level metrics appear normal during the downtime - no increase in traffic. Also our collections are not open to the public and we have a fairly small, known user base who are issued short lived tokens.

Slow query logging should also help spot long latency queries.

Thanks for the pointers, will try and gather more info using it and get to a root cause.