Troubleshooting Typesense 503 Errors and Usage Queries
TLDR Kevin encountered 503s using typesense. Jason asked for logs and explained why 503s occur. They made recommendations to remedy the issue and resolved Kevin's import parameter confusion. User was asked to open a github issue for accepting booleans.
Powered by Struct AI
1
1
18
2mo
Solved
Jul 13, 2023 (2 months ago)
Kevin
Kevin
08:35 PMhello- I was using typesense without any issues just a couple hours ago. we have it deployed on digitalocean and i was able to successfully insert roughly 6m documents into my collection. all of a sudden, i started getting 503s when i hit the /health endpoint. we have not made any updates to digitalocean within that timespan. i am wondering if this is related to rebuilding the index or some other expected issue?
Jason
Jason
08:38 PMCould you post say the last 100 lines from the logs?
08:39
Jason
08:39 PMIn general, here’s why you see 503s: https://typesense.org/docs/guide/syncing-data-into-typesense.html#handling-http-503s
Kevin
Kevin
08:43 PMueue_size: 0, local_sequence: 24476859
I20230713 20:40:18.999783 26938 raft_server.h:60] Peer refresh succeeded!
E20230713 20:40:24.000121 26902 raft_server.cpp:624] 4800 queued writes > healthy read lag of 1000
E20230713 20:40:24.000198 26902 raft_server.cpp:636] 4800 queued writes > healthy write lag of 500
I20230713 20:40:29.000618 26902 raft_server.cpp:546] Term: 2, last_index index: 34920, committed_index: 34920, known_applied_index: 34920, applying_index: 0, queued_writes: 4789, pending_queue_size: 0, local_sequence: 24476859
I20230713 20:40:29.000923 26928 raft_server.h:60] Peer refresh succeeded!
E20230713 20:40:33.001021 26902 raft_server.cpp:624] 4781 queued writes > healthy read lag of 1000
E20230713 20:40:33.001092 26902 raft_server.cpp:636] 4781 queued writes > healthy write lag of 500
I20230713 20:40:39.001595 26902 raft_server.cpp:546] Term: 2, last_index index: 34920, committed_index: 34920, known_applied_index: 34920, applying_index: 0, queued_writes: 4768, pending_queue_size: 0, local_sequence: 24476859
I20230713 20:40:39.001729 26938 raft_server.h:60] Peer refresh succeeded!
E20230713 20:40:42.001917 26902 raft_server.cpp:624] 4761 queued writes > healthy read lag of 1000
E20230713 20:40:42.001986 26902 raft_server.cpp:636] 4761 queued writes > healthy write lag of 500
I20230713 20:40:42.239393 26903 batched_indexer.cpp:284] Running GC for aborted requests, req map size: 1
I20230713 20:40:49.002573 26902 raft_server.cpp:546] Term: 2, last_index index: 34920, committed_index: 34920, known_applied_index: 34920, applying_index: 0, queued_writes: 4747, pending_queue_size: 0, local_sequence: 24476859
I20230713 20:40:49.002897 26928 raft_server.h:60] Peer refresh succeeded!
E20230713 20:40:51.002811 26902 raft_server.cpp:624] 4743 queued writes > healthy read lag of 1000
E20230713 20:40:51.002892 26902 raft_server.cpp:636] 4743 queued writes > healthy write lag of 500
I20230713 20:40:59.003530 26902 raft_server.cpp:546] Term: 2, last_index index: 34920, committed_index: 34920, known_applied_index: 34920, applying_index: 0, queued_writes: 4726, pending_queue_size: 0, local_sequence: 24476859
I20230713 20:40:59.003659 26938 raft_server.h:60] Peer refresh succeeded!
E20230713 20:41:00.003693 26902 raft_server.cpp:624] 4724 queued writes > healthy read lag of 1000
E20230713 20:41:00.003772 26902 raft_server.cpp:636] 4724 queued writes > healthy write lag of 500
I20230713 20:41:09.004508 26902 raft_server.cpp:546] Term: 2, last_index index: 34920, committed_index: 34920, known_applied_index: 34920, applying_index: 0, queued_writes: 4705, pending_queue_size: 0, local_sequence: 24476859
E20230713 20:41:09.004582 26902 raft_server.cpp:624] 4705 queued writes > healthy read lag of 1000
E20230713 20:41:09.004607 26902 raft_server.cpp:636] 4705 queued writes > healthy write lag of 500
I20230713 20:41:09.004624 26928 raft_server.h:60] Peer refresh succeeded!
E20230713 20:41:18.005333 26902 raft_server.cpp:624] 4687 queued writes > healthy read lag of 1000
E20230713 20:41:18.005399 26902 raft_server.cpp:636] 4687 queued writes > healthy write lag of 500
I20230713 20:41:19.005517 26902 raft_server.cpp:546] Term: 2, last_index index: 34920, committed_index: 34920, known_applied_index: 34920, applying_index: 0, queued_writes: 4685, pending_queue_size: 0, local_sequence: 24476859
I20230713 20:41:19.005776 26938 raft_server.h:60] Peer refresh succeeded!
E20230713 20:41:27.006239 26902 raft_server.cpp:624] 4668 queued writes > healthy read lag of 1000
E20230713 20:41:27.006325 26902 raft_server.cpp:636] 4668 queued writes > healthy write lag of 500
I20230713 20:41:29.006515 26902 raft_server.cpp:546] Term: 2, last_index index: 34920, committed_index: 34920, known_applied_index: 34920, applying_index: 0, queued_writes: 4664, pending_queue_size: 0, local_sequence: 24476859
I20230713 20:41:29.006635 26928 raft_server.h:60] Peer refresh succeeded!
E20230713 20:41:36.007102 26902 raft_server.cpp:624] 4650 queued writes > healthy read lag of 1000
E20230713 20:41:36.007184 26902 raft_server.cpp:636] 4650 queued writes > healthy write lag of 500
I20230713 20:41:39.007444 26902 raft_server.cpp:546] Term: 2, last_index index: 34920, committed_index: 34920, known_applied_index: 34920, applying_index: 0, queued_writes: 4644, pending_queue_size: 0, local_sequence: 24476859
I20230713 20:41:39.007699 26938 raft_server.h:60] Peer refresh succeeded!
I20230713 20:41:43.246860 26903 batched_indexer.cpp:284] Running GC for aborted requests, req map size: 1
E20230713 20:41:45.007987 26902 raft_server.cpp:624] 4632 queued writes > healthy read lag of 1000
E20230713 20:41:45.008046 26902 raft_server.cpp:636] 4632 queued writes > healthy write lag of 500
I20230713 20:41:49.008385 26902 raft_server.cpp:546] Term: 2, last_index index: 34920, committed_index: 34920, known_applied_index: 34920, applying_index: 0, queued_writes: 4624, pending_queue_size: 0, local_sequence: 24476859
I20230713 20:41:49.008605 26928 raft_server.h:60] Peer refresh succeeded!
E20230713 20:41:54.008860 26902 raft_server.cpp:624] 4613 queued writes > healthy read lag of 1000
E20230713 20:41:54.008942 26902 raft_server.cpp:636] 4613 queued writes > healthy write lag of 500
I20230713 20:41:59.009361 26902 raft_server.cpp:546] Term: 2, last_index index: 34920, committed_index: 34920, known_applied_index: 34920, applying_index: 0, queued_writes: 4603, pending_queue_size: 0, local_sequence: 24476859
I20230713 20:41:59.009531 26938 raft_server.h:60] Peer refresh succeeded!
E20230713 20:42:03.009764 26902 raft_server.cpp:624] 4595 queued writes > healthy read lag of 1000
E20230713 20:42:03.009831 26902 raft_server.cpp:636] 4595 queued writes > healthy write lag of 500
I20230713 20:42:09.010331 26902 raft_server.cpp:546] Term: 2, last_index index: 34920, committed_index: 34920, known_applied_index: 34920, applying_index: 0, queued_writes: 4583, pending_queue_size: 0, local_sequence: 24476859
I20230713 20:42:09.010445 26928 raft_server.h:60] Peer refresh succeeded!
E20230713 20:42:12.010648 26902 raft_server.cpp:624] 4577 queued writes > healthy read lag of 1000
E20230713 20:42:12.010730 26902 raft_server.cpp:636] 4577 queued writes > healthy write lag of 500
I20230713 20:42:19.011272 26902 raft_server.cpp:546] Term: 2, last_index index: 34920, committed_index: 34920, known_applied_index: 34920, applying_index: 0, queued_writes: 4563, pending_queue_size: 0, local_sequence: 24476859
I20230713 20:42:19.011497 26938 raft_server.h:60] Peer refresh succeeded!
E20230713 20:42:21.011502 26902 raft_server.cpp:624] 4559 queued writes > healthy read lag of 1000
E20230713 20:42:21.011561 26902 raft_server.cpp:636] 4559 queued writes > healthy write lag of 500
I20230713 20:42:29.012213 26902 raft_server.cpp:546] Term: 2, last_index index: 34920, committed_index: 34920, known_applied_index: 34920, applying_index: 0, queued_writes: 4543, pending_queue_size: 0, local_sequence: 24476859
I20230713 20:42:29.012432 26928 raft_server.h:60] Peer refresh succeeded!
E20230713 20:42:30.012383 26902 raft_server.cpp:624] 4541 queued writes > healthy read lag of 1000
E20230713 20:42:30.012452 26902 raft_server.cpp:636] 4541 queued writes > healthy write lag of 500
I20230713 20:42:39.013185 26902 raft_server.cpp:546] Term: 2, last_index index: 34920, committed_index: 34920, known_applied_index: 34920, applying_index: 0, queued_writes: 4524, pending_queue_size: 0, local_sequence: 24476859
E20230713 20:42:39.013257 26902 raft_server.cpp:624] 4524 queued writes > healthy read lag of 1000
E20230713 20:42:39.013283 26902 raft_server.cpp:636] 4524 queued writes > healthy write lag of 500
I20230713 20:42:39.013476 26938 raft_server.h:60] Peer refresh succeeded!
I20230713 20:42:44.254422 26903 batched_indexer.cpp:284] Running GC for aborted requests, req map size: 108:43
Kevin
08:43 PMah i didn't know that- i assume the 6 million documents are still being written and i can't do anything until that's finished?
Jason
Jason
08:44 PMYou can change the
healthy-read-lag and health-write-lag server params to high values and restart the server08:45
Jason
08:45 PMMay I know how many CPU cores you have and if you have sufficient RAM to hold the entire dataset in RAM?
08:45
Jason
08:45 PMAlso, you want to use the import endpoint to index a large number of documents instead of the single document endpoint
David
David
08:47 PM8 vCPUs
64GB / 1200GB Disk
64GB / 1200GB Disk
08:48
David
08:48 PMThe dataset is 3 gb, so I don’t think it’s a RAM limitation
1
08:48
David
08:48 PMWhat would you recommend values wise for the read and write lag params?
Jason
Jason
08:49 PMYou could try say 15K for write lag and 10K for read lag
1
Kevin
Kevin
09:04 PMgreat, we will try that. one other thing i wanted to ask was if returnid is a parameter that works with the jsonl import in the python sdk?
here is how we are running the import:
res = ts.collections['entities'].documents.import(jsonl_file.read().encode('utf-8'), {'action': 'upsert', "return_id": True})
however i get a 400 saying: Parameter
here is how we are running the import:
res = ts.collections['entities'].documents.import(jsonl_file.read().encode('utf-8'), {'action': 'upsert', "return_id": True})
however i get a 400 saying: Parameter
return_id must be a true|false.Jason
Jason
09:09 PMHmm, could you check if
"return_id": "true" works?09:10
Jason
09:10 PMOn a related side note, could you also make sure you’ve increased your client-side timeout when instantiating the client to as high as say 60 minutes (we never want the import api call to timeout and retry)
Kevin
Kevin
09:11 PMoh that worked, thanks!
Jason
Jason
09:12 PMHmmm, could you open an issue in the typesense-python github repo mentioning this? We should accept booleans as well
Kevin
Kevin
09:14 PMsure thing
Typesense
Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI
Indexed 2779 threads (79% resolved)
Similar Threads
Troubleshooting Stalled Writes in TypeSense Instance
Robert was experiencing typesense instances getting stuck after trying to import documents. Kishore Nallan provided suggestions and added specific logs to diagnose the issue. The two identified queries causing troubles but the issues had not been fully resolved yet.
57
10mo
Large JSONL Documents Import Issue & Resolution
Suraj was having trouble loading large JSONL documents into Typesense server. After several discussions and attempts, it was discovered that the issue was due to data quality. Once the team extracted the data again, the upload process worked smoothly.
4
94
6mo
Solved
Addressing High CPU Usage in Typesense
Robert reported high CPU usage on Typesense, even after halting all incoming searches. Kishore Nallan suggested logging heavy queries and increasing thread count. The issue was resolved after Robert found and truncated unusually large documents in the database.
35
12mo
Solved
Server Startup Troubleshooting
Robert was unsure why their server was taking a long time to start up. Jason explained that the server was still indexing data into memory.
5
12mo
Solved
Resolving Server Stoppage Issues in Typesense Multi VM Cluster
gaurav faced issues with the Typesense server in a multi VM cluster, including automatic stoppage and errors. Kishore Nallan identified the lack of a quorum and suggested using three nodes. When the issue persisted, they advised running Typesense via `nohup` or `systemd` to prevent session closure from stopping the process.
31
11mo