Addressing Exceeding Query Length and High CPU Usage
TLDR Imtiaz reported Query string exceeds max allowed length of 4000
errors and high CPU usage. Kishore Nallan suggested enabling slow request logs to identify long queries and excessive CPU usage and checking incoming IPs for DDoS attacks.
May 27, 2022 (19 months ago)
Imtiaz
03:14 PMWe started getting a bunch of
Query string exceeds max allowed length of 4000
errors on our servers.I do not see any queries that would result in an unbounded query to typesense on our client code. I am struggling to trace exactly what query is triggering this behaviour.
• Is this the payload body size or the URL length for GET requests?
• Is there any way of knowing exactly which request is failing - like a URL to narrow it down to a collection?
• Could this be internal traffic between the cluster?
Additional context:
• Running 0.21.0
• Servers have not been rebooted since around October 2021 and have been running fine
• This is the first time we noticed this issue, due to downtime on typesense. The CPU workloadd increased to 100% for around 10 minutes across all 3 servers. We are still fault finding on this to see if any downstream service could have created this pressure - but have not identified anything thus far.
When it does occur, it lasts for around a minute and interestingly, the query string reported seems to be increasing in size over time, so I assume whatever the query is - it is increasing in length on each retry
A sample of the log below:
<... more similar events truncated...>
E20220527 06:23:38.236594 13397 string_utils.cpp:85] Query string exceeds max allowed length of 4000. Actual length: 5657
E20220527 06:23:38.329757 13397 string_utils.cpp:85] Query string exceeds max allowed length of 4000. Actual length: 5660
E20220527 06:23:38.436268 13397 string_utils.cpp:85] Query string exceeds max allowed length of 4000. Actual length: 5663
E20220527 06:23:38.537355 13397 string_utils.cpp:85] Query string exceeds max allowed length of 4000. Actual length: 5666
E20220527 06:23:38.635035 13397 string_utils.cpp:85] Query string exceeds max allowed length of 4000. Actual length: 5669
E20220527 06:23:38.728849 13397 string_utils.cpp:85] Query string exceeds max allowed length of 4000. Actual length: 5672
E20220527 06:23:38.827121 13397 string_utils.cpp:85] Query string exceeds max allowed length of 4000. Actual length: 5675
E20220527 06:23:38.924321 13397 string_utils.cpp:85] Query string exceeds max allowed length of 4000. Actual length: 5678
E20220527 06:23:39.030211 13397 string_utils.cpp:85] Query string exceeds max allowed length of 4000. Actual length: 5681
E20220527 06:23:39.119545 13397 string_utils.cpp:85] Query string exceeds max allowed length of 4000. Actual length: 5684
E20220527 06:23:39.217941 13397 string_utils.cpp:85] Query string exceeds max allowed length of 4000. Actual length: 5687
E20220527 06:23:39.317699 13397 string_utils.cpp:85] Query string exceeds max allowed length of 4000. Actual length: 5690
E20220527 06:23:39.421378 13397 string_utils.cpp:85] Query string exceeds max allowed length of 4000. Actual length: 5693
E20220527 06:23:39.522917 13397 string_utils.cpp:85] Query string exceeds max allowed length of 4000. Actual length: 5696
E20220527 06:23:39.621501 13397 string_utils.cpp:85] Query string exceeds max allowed length of 4000. Actual length: 5699
E20220527 06:23:39.722431 13397 string_utils.cpp:85] Query string exceeds max allowed length of 4000. Actual length: 5702
E20220527 06:23:39.830339 13397 string_utils.cpp:85] Query string exceeds max allowed length of 4000. Actual length: 5705
E20220527 06:23:39.920497 13397 string_utils.cpp:85] Query string exceeds max allowed length of 4000. Actual length: 5708
E20220527 06:23:40.019423 13397 string_utils.cpp:85] Query string exceeds max allowed length of 4000. Actual length: 5711
E20220527 06:23:40.113782 13397 string_utils.cpp:85] Query string exceeds max allowed length of 4000. Actual length: 5714
E20220527 06:23:41.403084 13397 string_utils.cpp:85] Query string exceeds max allowed length of 4000. Actual length: 5716
<... more similar events truncated...>
Thanks!
Kishore Nallan
03:16 PM0
: https://typesense.org/docs/0.21.0/api/cluster-operations.html#toggle-slow-request-logThis will log all queries and hopefully catch those long queries as well.
Kishore Nallan
03:17 PMImtiaz
03:20 PMKishore Nallan
03:22 PMImtiaz
03:22 PMKishore Nallan
03:23 PMImtiaz
03:25 PMTypesense
Indexed 3015 threads (79% resolved)
Similar Threads
Addressing High CPU Usage in Typesense
Robert reported high CPU usage on Typesense, even after halting all incoming searches. Kishore Nallan suggested logging heavy queries and increasing thread count. The issue was resolved after Robert found and truncated unusually large documents in the database.
Large JSONL Documents Import Issue & Resolution
Suraj was having trouble loading large JSONL documents into Typesense server. After several discussions and attempts, it was discovered that the issue was due to data quality. Once the team extracted the data again, the upload process worked smoothly.
Troubleshooting Stalled Writes in TypeSense Instance
Robert was experiencing typesense instances getting stuck after trying to import documents. Kishore Nallan provided suggestions and added specific logs to diagnose the issue. The two identified queries causing troubles but the issues had not been fully resolved yet.
Issue Resolution and Upgrade Problems in Typesense Version 0.26rc16
Ankit reported an issue with Typesense, which was addressed by Kishore Nallan and Jason. However, Ankit experienced difficulties while trying to upgrade, with the server status showing as "Not ready or lagging" 503. The resolution remains incomplete.
Unresolved High-Volume Write Issue in Typesense
Greg experienced issues with Typesense where it became unresponsive during high-volume write operations. Jason and Kishore Nallan suggested several solutions, but the issue remained unresolved. They suspect that the problem occurs when concurrent writes are happening to the same collection.