#community-help

Addressing Exceeding Query Length and High CPU Usage

TLDR Imtiaz reported Query string exceeds max allowed length of 4000 errors and high CPU usage. Kishore Nallan suggested enabling slow request logs to identify long queries and excessive CPU usage and checking incoming IPs for DDoS attacks.

Powered by Struct AI
May 27, 2022 (19 months ago)
Imtiaz
Photo of md5-5c86b23fcf7d52313d7a346d8cc3e86b
Imtiaz
03:14 PM
Hi,

We started getting a bunch of Query string exceeds max allowed length of 4000 errors on our servers.

I do not see any queries that would result in an unbounded query to typesense on our client code. I am struggling to trace exactly what query is triggering this behaviour.

• Is this the payload body size or the URL length for GET requests?
• Is there any way of knowing exactly which request is failing - like a URL to narrow it down to a collection?
• Could this be internal traffic between the cluster?
Additional context:
• Running 0.21.0
• Servers have not been rebooted since around October 2021 and have been running fine
• This is the first time we noticed this issue, due to downtime on typesense. The CPU workloadd increased to 100% for around 10 minutes across all 3 servers. We are still fault finding on this to see if any downstream service could have created this pressure - but have not identified anything thus far.
When it does occur, it lasts for around a minute and interestingly, the query string reported seems to be increasing in size over time, so I assume whatever the query is - it is increasing in length on each retry

A sample of the log below:

<... more similar events truncated...>
E20220527 06:23:38.236594 13397 string_utils.cpp:85] Query string exceeds max allowed length of 4000. Actual length: 5657
E20220527 06:23:38.329757 13397 string_utils.cpp:85] Query string exceeds max allowed length of 4000. Actual length: 5660
E20220527 06:23:38.436268 13397 string_utils.cpp:85] Query string exceeds max allowed length of 4000. Actual length: 5663
E20220527 06:23:38.537355 13397 string_utils.cpp:85] Query string exceeds max allowed length of 4000. Actual length: 5666
E20220527 06:23:38.635035 13397 string_utils.cpp:85] Query string exceeds max allowed length of 4000. Actual length: 5669
E20220527 06:23:38.728849 13397 string_utils.cpp:85] Query string exceeds max allowed length of 4000. Actual length: 5672
E20220527 06:23:38.827121 13397 string_utils.cpp:85] Query string exceeds max allowed length of 4000. Actual length: 5675
E20220527 06:23:38.924321 13397 string_utils.cpp:85] Query string exceeds max allowed length of 4000. Actual length: 5678
E20220527 06:23:39.030211 13397 string_utils.cpp:85] Query string exceeds max allowed length of 4000. Actual length: 5681
E20220527 06:23:39.119545 13397 string_utils.cpp:85] Query string exceeds max allowed length of 4000. Actual length: 5684
E20220527 06:23:39.217941 13397 string_utils.cpp:85] Query string exceeds max allowed length of 4000. Actual length: 5687
E20220527 06:23:39.317699 13397 string_utils.cpp:85] Query string exceeds max allowed length of 4000. Actual length: 5690
E20220527 06:23:39.421378 13397 string_utils.cpp:85] Query string exceeds max allowed length of 4000. Actual length: 5693
E20220527 06:23:39.522917 13397 string_utils.cpp:85] Query string exceeds max allowed length of 4000. Actual length: 5696
E20220527 06:23:39.621501 13397 string_utils.cpp:85] Query string exceeds max allowed length of 4000. Actual length: 5699
E20220527 06:23:39.722431 13397 string_utils.cpp:85] Query string exceeds max allowed length of 4000. Actual length: 5702
E20220527 06:23:39.830339 13397 string_utils.cpp:85] Query string exceeds max allowed length of 4000. Actual length: 5705
E20220527 06:23:39.920497 13397 string_utils.cpp:85] Query string exceeds max allowed length of 4000. Actual length: 5708
E20220527 06:23:40.019423 13397 string_utils.cpp:85] Query string exceeds max allowed length of 4000. Actual length: 5711
E20220527 06:23:40.113782 13397 string_utils.cpp:85] Query string exceeds max allowed length of 4000. Actual length: 5714
E20220527 06:23:41.403084 13397 string_utils.cpp:85] Query string exceeds max allowed length of 4000. Actual length: 5716
<... more similar events truncated...>

Thanks!
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
03:16 PM
You can try enabling slow request log with value of 0 : https://typesense.org/docs/0.21.0/api/cluster-operations.html#toggle-slow-request-log

This will log all queries and hopefully catch those long queries as well.
03:17
Kishore Nallan
03:17 PM
Such long queries are dropped by Typesense with a 400 status code though, so they are unlikely to be a source of high CPU usage unless the server itself is under some form of DDoS attack.
Imtiaz
Photo of md5-5c86b23fcf7d52313d7a346d8cc3e86b
Imtiaz
03:20 PM
Thanks - I will enable that and hopefully get some insight into the long queries. Is there anything else I can look for regarding the high CPU usage - either in the logs or elsewhere?
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
03:22 PM
Check ifconfig to see incoming IPs and spot check them to see whether they are bots or from random countries.
Imtiaz
Photo of md5-5c86b23fcf7d52313d7a346d8cc3e86b
Imtiaz
03:22 PM
We currently do not have reason to suspect DDoS . One reason why is because our network level metrics appear normal during the downtime - no increase in traffic. Also our collections are not open to the public and we have a fairly small, known user base who are issued short lived tokens.
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
03:23 PM
Slow query logging should also help spot long latency queries.
Imtiaz
Photo of md5-5c86b23fcf7d52313d7a346d8cc3e86b
Imtiaz
03:25 PM
Thanks for the pointers, will try and gather more info using it and get to a root cause.

Typesense

Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI

Indexed 3015 threads (79% resolved)

Join Our Community

Similar Threads

Addressing High CPU Usage in Typesense

Robert reported high CPU usage on Typesense, even after halting all incoming searches. Kishore Nallan suggested logging heavy queries and increasing thread count. The issue was resolved after Robert found and truncated unusually large documents in the database.

35
14mo

Large JSONL Documents Import Issue & Resolution

Suraj was having trouble loading large JSONL documents into Typesense server. After several discussions and attempts, it was discovered that the issue was due to data quality. Once the team extracted the data again, the upload process worked smoothly.

run

4

94
9mo

Troubleshooting Stalled Writes in TypeSense Instance

Robert was experiencing typesense instances getting stuck after trying to import documents. Kishore Nallan provided suggestions and added specific logs to diagnose the issue. The two identified queries causing troubles but the issues had not been fully resolved yet.

7

57
13mo

Issue Resolution and Upgrade Problems in Typesense Version 0.26rc16

Ankit reported an issue with Typesense, which was addressed by Kishore Nallan and Jason. However, Ankit experienced difficulties while trying to upgrade, with the server status showing as "Not ready or lagging" 503. The resolution remains incomplete.

1

19
2mo

Unresolved High-Volume Write Issue in Typesense

Greg experienced issues with Typesense where it became unresponsive during high-volume write operations. Jason and Kishore Nallan suggested several solutions, but the issue remained unresolved. They suspect that the problem occurs when concurrent writes are happening to the same collection.

2

43
2d