Troubleshooting Typesense 503 Errors and Usage Queries
TLDR Kevin encountered 503s using typesense. Jason asked for logs and explained why 503s occur. They made recommendations to remedy the issue and resolved Kevin's import parameter confusion. User was asked to open a github issue for accepting booleans.
1
1
Jul 13, 2023 (2 months ago)
Kevin
08:35 PMJason
08:38 PMJason
08:39 PMKevin
08:43 PMueue_size: 0, local_sequence: 24476859
I20230713 20:40:18.999783 26938 raft_server.h:60] Peer refresh succeeded!
E20230713 20:40:24.000121 26902 raft_server.cpp:624] 4800 queued writes > healthy read lag of 1000
E20230713 20:40:24.000198 26902 raft_server.cpp:636] 4800 queued writes > healthy write lag of 500
I20230713 20:40:29.000618 26902 raft_server.cpp:546] Term: 2, last_index index: 34920, committed_index: 34920, known_applied_index: 34920, applying_index: 0, queued_writes: 4789, pending_queue_size: 0, local_sequence: 24476859
I20230713 20:40:29.000923 26928 raft_server.h:60] Peer refresh succeeded!
E20230713 20:40:33.001021 26902 raft_server.cpp:624] 4781 queued writes > healthy read lag of 1000
E20230713 20:40:33.001092 26902 raft_server.cpp:636] 4781 queued writes > healthy write lag of 500
I20230713 20:40:39.001595 26902 raft_server.cpp:546] Term: 2, last_index index: 34920, committed_index: 34920, known_applied_index: 34920, applying_index: 0, queued_writes: 4768, pending_queue_size: 0, local_sequence: 24476859
I20230713 20:40:39.001729 26938 raft_server.h:60] Peer refresh succeeded!
E20230713 20:40:42.001917 26902 raft_server.cpp:624] 4761 queued writes > healthy read lag of 1000
E20230713 20:40:42.001986 26902 raft_server.cpp:636] 4761 queued writes > healthy write lag of 500
I20230713 20:40:42.239393 26903 batched_indexer.cpp:284] Running GC for aborted requests, req map size: 1
I20230713 20:40:49.002573 26902 raft_server.cpp:546] Term: 2, last_index index: 34920, committed_index: 34920, known_applied_index: 34920, applying_index: 0, queued_writes: 4747, pending_queue_size: 0, local_sequence: 24476859
I20230713 20:40:49.002897 26928 raft_server.h:60] Peer refresh succeeded!
E20230713 20:40:51.002811 26902 raft_server.cpp:624] 4743 queued writes > healthy read lag of 1000
E20230713 20:40:51.002892 26902 raft_server.cpp:636] 4743 queued writes > healthy write lag of 500
I20230713 20:40:59.003530 26902 raft_server.cpp:546] Term: 2, last_index index: 34920, committed_index: 34920, known_applied_index: 34920, applying_index: 0, queued_writes: 4726, pending_queue_size: 0, local_sequence: 24476859
I20230713 20:40:59.003659 26938 raft_server.h:60] Peer refresh succeeded!
E20230713 20:41:00.003693 26902 raft_server.cpp:624] 4724 queued writes > healthy read lag of 1000
E20230713 20:41:00.003772 26902 raft_server.cpp:636] 4724 queued writes > healthy write lag of 500
I20230713 20:41:09.004508 26902 raft_server.cpp:546] Term: 2, last_index index: 34920, committed_index: 34920, known_applied_index: 34920, applying_index: 0, queued_writes: 4705, pending_queue_size: 0, local_sequence: 24476859
E20230713 20:41:09.004582 26902 raft_server.cpp:624] 4705 queued writes > healthy read lag of 1000
E20230713 20:41:09.004607 26902 raft_server.cpp:636] 4705 queued writes > healthy write lag of 500
I20230713 20:41:09.004624 26928 raft_server.h:60] Peer refresh succeeded!
E20230713 20:41:18.005333 26902 raft_server.cpp:624] 4687 queued writes > healthy read lag of 1000
E20230713 20:41:18.005399 26902 raft_server.cpp:636] 4687 queued writes > healthy write lag of 500
I20230713 20:41:19.005517 26902 raft_server.cpp:546] Term: 2, last_index index: 34920, committed_index: 34920, known_applied_index: 34920, applying_index: 0, queued_writes: 4685, pending_queue_size: 0, local_sequence: 24476859
I20230713 20:41:19.005776 26938 raft_server.h:60] Peer refresh succeeded!
E20230713 20:41:27.006239 26902 raft_server.cpp:624] 4668 queued writes > healthy read lag of 1000
E20230713 20:41:27.006325 26902 raft_server.cpp:636] 4668 queued writes > healthy write lag of 500
I20230713 20:41:29.006515 26902 raft_server.cpp:546] Term: 2, last_index index: 34920, committed_index: 34920, known_applied_index: 34920, applying_index: 0, queued_writes: 4664, pending_queue_size: 0, local_sequence: 24476859
I20230713 20:41:29.006635 26928 raft_server.h:60] Peer refresh succeeded!
E20230713 20:41:36.007102 26902 raft_server.cpp:624] 4650 queued writes > healthy read lag of 1000
E20230713 20:41:36.007184 26902 raft_server.cpp:636] 4650 queued writes > healthy write lag of 500
I20230713 20:41:39.007444 26902 raft_server.cpp:546] Term: 2, last_index index: 34920, committed_index: 34920, known_applied_index: 34920, applying_index: 0, queued_writes: 4644, pending_queue_size: 0, local_sequence: 24476859
I20230713 20:41:39.007699 26938 raft_server.h:60] Peer refresh succeeded!
I20230713 20:41:43.246860 26903 batched_indexer.cpp:284] Running GC for aborted requests, req map size: 1
E20230713 20:41:45.007987 26902 raft_server.cpp:624] 4632 queued writes > healthy read lag of 1000
E20230713 20:41:45.008046 26902 raft_server.cpp:636] 4632 queued writes > healthy write lag of 500
I20230713 20:41:49.008385 26902 raft_server.cpp:546] Term: 2, last_index index: 34920, committed_index: 34920, known_applied_index: 34920, applying_index: 0, queued_writes: 4624, pending_queue_size: 0, local_sequence: 24476859
I20230713 20:41:49.008605 26928 raft_server.h:60] Peer refresh succeeded!
E20230713 20:41:54.008860 26902 raft_server.cpp:624] 4613 queued writes > healthy read lag of 1000
E20230713 20:41:54.008942 26902 raft_server.cpp:636] 4613 queued writes > healthy write lag of 500
I20230713 20:41:59.009361 26902 raft_server.cpp:546] Term: 2, last_index index: 34920, committed_index: 34920, known_applied_index: 34920, applying_index: 0, queued_writes: 4603, pending_queue_size: 0, local_sequence: 24476859
I20230713 20:41:59.009531 26938 raft_server.h:60] Peer refresh succeeded!
E20230713 20:42:03.009764 26902 raft_server.cpp:624] 4595 queued writes > healthy read lag of 1000
E20230713 20:42:03.009831 26902 raft_server.cpp:636] 4595 queued writes > healthy write lag of 500
I20230713 20:42:09.010331 26902 raft_server.cpp:546] Term: 2, last_index index: 34920, committed_index: 34920, known_applied_index: 34920, applying_index: 0, queued_writes: 4583, pending_queue_size: 0, local_sequence: 24476859
I20230713 20:42:09.010445 26928 raft_server.h:60] Peer refresh succeeded!
E20230713 20:42:12.010648 26902 raft_server.cpp:624] 4577 queued writes > healthy read lag of 1000
E20230713 20:42:12.010730 26902 raft_server.cpp:636] 4577 queued writes > healthy write lag of 500
I20230713 20:42:19.011272 26902 raft_server.cpp:546] Term: 2, last_index index: 34920, committed_index: 34920, known_applied_index: 34920, applying_index: 0, queued_writes: 4563, pending_queue_size: 0, local_sequence: 24476859
I20230713 20:42:19.011497 26938 raft_server.h:60] Peer refresh succeeded!
E20230713 20:42:21.011502 26902 raft_server.cpp:624] 4559 queued writes > healthy read lag of 1000
E20230713 20:42:21.011561 26902 raft_server.cpp:636] 4559 queued writes > healthy write lag of 500
I20230713 20:42:29.012213 26902 raft_server.cpp:546] Term: 2, last_index index: 34920, committed_index: 34920, known_applied_index: 34920, applying_index: 0, queued_writes: 4543, pending_queue_size: 0, local_sequence: 24476859
I20230713 20:42:29.012432 26928 raft_server.h:60] Peer refresh succeeded!
E20230713 20:42:30.012383 26902 raft_server.cpp:624] 4541 queued writes > healthy read lag of 1000
E20230713 20:42:30.012452 26902 raft_server.cpp:636] 4541 queued writes > healthy write lag of 500
I20230713 20:42:39.013185 26902 raft_server.cpp:546] Term: 2, last_index index: 34920, committed_index: 34920, known_applied_index: 34920, applying_index: 0, queued_writes: 4524, pending_queue_size: 0, local_sequence: 24476859
E20230713 20:42:39.013257 26902 raft_server.cpp:624] 4524 queued writes > healthy read lag of 1000
E20230713 20:42:39.013283 26902 raft_server.cpp:636] 4524 queued writes > healthy write lag of 500
I20230713 20:42:39.013476 26938 raft_server.h:60] Peer refresh succeeded!
I20230713 20:42:44.254422 26903 batched_indexer.cpp:284] Running GC for aborted requests, req map size: 1
Kevin
08:43 PMJason
08:44 PMhealthy-read-lag
and health-write-lag
server params to high values and restart the serverJason
08:45 PMJason
08:45 PMDavid
08:47 PM64GB / 1200GB Disk
David
08:48 PM1
David
08:48 PMJason
08:49 PM1
Kevin
09:04 PMhere is how we are running the import:
res = ts.collections['entities'].documents.import(jsonl_file.read().encode('utf-8'), {'action': 'upsert', "return_id": True})
however i get a 400 saying: Parameter
return_id
must be a true|false.Jason
09:09 PM"return_id": "true"
works?Jason
09:10 PMKevin
09:11 PMJason
09:12 PMKevin
09:14 PMTypesense
Indexed 2779 threads (79% resolved)
Similar Threads
Troubleshooting Stalled Writes in TypeSense Instance
Robert was experiencing typesense instances getting stuck after trying to import documents. Kishore Nallan provided suggestions and added specific logs to diagnose the issue. The two identified queries causing troubles but the issues had not been fully resolved yet.
Large JSONL Documents Import Issue & Resolution
Suraj was having trouble loading large JSONL documents into Typesense server. After several discussions and attempts, it was discovered that the issue was due to data quality. Once the team extracted the data again, the upload process worked smoothly.
Addressing High CPU Usage in Typesense
Robert reported high CPU usage on Typesense, even after halting all incoming searches. Kishore Nallan suggested logging heavy queries and increasing thread count. The issue was resolved after Robert found and truncated unusually large documents in the database.
Server Startup Troubleshooting
Robert was unsure why their server was taking a long time to start up. Jason explained that the server was still indexing data into memory.
Resolving Server Stoppage Issues in Typesense Multi VM Cluster
gaurav faced issues with the Typesense server in a multi VM cluster, including automatic stoppage and errors. Kishore Nallan identified the lack of a quorum and suggested using three nodes. When the issue persisted, they advised running Typesense via `nohup` or `systemd` to prevent session closure from stopping the process.