Typesense Processes Intermittently Crash When Indexing Documents
TLDR Adrian experiences intermittent typesense process crashes when indexing 500k documents. Kishore Nallan explains that it's not a crash but a backpressure mechanism rejecting writes when ingesting too fast.

Apr 20, 2023 (5 months ago)
Adrian
06:45 PMAdrian
06:48 PMI0420 18:31:31.142714 334 external/com_github_brpc_braft/src/braft/log.cpp:114] Created new segment `/usr/share/typesense/data/state/log/log_inprogress_00000000000000001178' with fd=39
I20230420 18:31:36.829527 262 raft_server.cpp:551] Term: 10, last_index index: 1186, committed_index: 1186, known_applied_index: 1186, applying_index: 0, queued_writes: 16, pending_queue_size: 0, local_sequence: 1700546
I20230420 18:31:46.896195 262 raft_server.cpp:551] Term: 10, last_index index: 1205, committed_index: 1205, known_applied_index: 1205, applying_index: 0, queued_writes: 22, pending_queue_size: 0, local_sequence: 1713504
I0420 18:31:47.343566 334 external/com_github_brpc_braft/src/braft/log.cpp:536] close a full segment. Current first_index: 1178 last_index: 1205 raft_sync_segments: 0 will_sync: 1 path: /usr/share/typesense/data/state/log/log_00000000000000001178_00000000000000001205
I0420 18:31:47.343642 334 external/com_github_brpc_braft/src/braft/log.cpp:550] Renamed `/usr/share/typesense/data/state/log/log_inprogress_00000000000000001178' to `/usr/share/typesense/data/state/log/log_00000000000000001178_00000000000000001205'
I0420 18:31:47.344864 334 external/com_github_brpc_braft/src/braft/log.cpp:114] Created new segment `/usr/share/typesense/data/state/log/log_inprogress_00000000000000001206' with fd=41
I20230420 18:31:56.905944 262 raft_server.cpp:551] Term: 10, last_index index: 1233, committed_index: 1233, known_applied_index: 1233, applying_index: 0, queued_writes: 46, pending_queue_size: 0, local_sequence: 1722741
I0420 18:31:58.559419 332 external/com_github_brpc_braft/src/braft/log.cpp:536] close a full segment. Current first_index: 1206 last_index: 1234 raft_sync_segments: 0 will_sync: 1 path: /usr/share/typesense/data/state/log/log_00000000000000001206_00000000000000001234
I0420 18:31:58.559503 332 external/com_github_brpc_braft/src/braft/log.cpp:550] Renamed `/usr/share/typesense/data/state/log/log_inprogress_00000000000000001206' to `/usr/share/typesense/data/state/log/log_00000000000000001206_00000000000000001234'
I0420 18:31:58.560835 332 external/com_github_brpc_braft/src/braft/log.cpp:114] Created new segment `/usr/share/typesense/data/state/log/log_inprogress_00000000000000001235' with fd=29
I20230420 18:32:06.917093 262 raft_server.cpp:551] Term: 10, last_index index: 1260, committed_index: 1260, known_applied_index: 1260, applying_index: 0, queued_writes: 61, pending_queue_size: 0, local_sequence: 1734210
I0420 18:32:09.687240 332 external/com_github_brpc_braft/src/braft/log.cpp:536] close a full segment. Current first_index: 1235 last_index: 1264 raft_sync_segments: 0 will_sync: 1 path: /usr/share/typesense/data/state/log/log_00000000000000001235_00000000000000001264
I0420 18:32:09.687352 332 external/com_github_brpc_braft/src/braft/log.cpp:550] Renamed `/usr/share/typesense/data/state/log/log_inprogress_00000000000000001235' to `/usr/share/typesense/data/state/log/log_00000000000000001235_00000000000000001264'
I0420 18:32:09.688412 332 external/com_github_brpc_braft/src/braft/log.cpp:114] Created new segment `/usr/share/typesense/data/state/log/log_inprogress_00000000000000001265' with fd=47
I20230420 18:32:16.942395 262 raft_server.cpp:551] Term: 10, last_index index: 1288, committed_index: 1288, known_applied_index: 1288, applying_index: 0, queued_writes: 81, pending_queue_size: 0, local_sequence: 1744351
I0420 18:32:20.677118 332 external/com_github_brpc_braft/src/braft/log.cpp:536] close a full segment. Current first_index: 1265 last_index: 1294 raft_sync_segments: 0 will_sync: 1 path: /usr/share/typesense/data/state/log/log_00000000000000001265_00000000000000001294
I0420 18:32:20.677200 332 external/com_github_brpc_braft/src/braft/log.cpp:550] Renamed `/usr/share/typesense/data/state/log/log_inprogress_00000000000000001265' to `/usr/share/typesense/data/state/log/log_00000000000000001265_00000000000000001294'
I0420 18:32:20.678689 332 external/com_github_brpc_braft/src/braft/log.cpp:114] Created new segment `/usr/share/typesense/data/state/log/log_inprogress_00000000000000001295' with fd=49
I20230420 18:32:27.001346 262 raft_server.cpp:551] Term: 10, last_index index: 1309, committed_index: 1309, known_applied_index: 1309, applying_index: 0, queued_writes: 89, pending_queue_size: 0, local_sequence: 1754377
I20230420 18:32:31.471853 263 batched_indexer.cpp:284] Running GC for aborted requests, req map size: 13
I0420 18:32:32.015195 335 external/com_github_brpc_braft/src/braft/log.cpp:536] close a full segment. Current first_index: 1295 last_index: 1326 raft_sync_segments: 0 will_sync: 1 path: /usr/share/typesense/data/state/log/log_00000000000000001295_00000000000000001326
I0420 18:32:32.015277 335 external/com_github_brpc_braft/src/braft/log.cpp:550] Renamed `/usr/share/typesense/data/state/log/log_inprogress_00000000000000001295' to `/usr/share/typesense/data/state/log/log_00000000000000001295_00000000000000001326'
I0420 18:32:32.016613 335 external/com_github_brpc_braft/src/braft/log.cpp:114] Created new segment `/usr/share/typesense/data/state/log/log_inprogress_00000000000000001327' with fd=46
I20230420 18:32:37.014884 262 raft_server.cpp:551] Term: 10, last_index index: 1341, committed_index: 1340, known_applied_index: 1340, applying_index: 0, queued_writes: 101, pending_queue_size: 0, local_sequence: 1766181
I0420 18:32:44.592146 334 external/com_github_brpc_braft/src/braft/log.cpp:536] close a full segment. Current first_index: 1327 last_index: 1354 raft_sync_segments: 0 will_sync: 1 path: /usr/share/typesense/data/state/log/log_00000000000000001327_00000000000000001354
I0420 18:32:44.592230 334 external/com_github_brpc_braft/src/braft/log.cpp:550] Renamed `/usr/share/typesense/data/state/log/log_inprogress_00000000000000001327' to `/usr/share/typesense/data/state/log/log_00000000000000001327_00000000000000001354'
I0420 18:32:44.593456 334 external/com_github_brpc_braft/src/braft/log.cpp:114] Created new segment `/usr/share/typesense/data/state/log/log_inprogress_00000000000000001355' with fd=33
I20230420 18:32:47.030795 262 raft_server.cpp:551] Term: 10, last_index index: 1356, committed_index: 1356, known_applied_index: 1356, applying_index: 0, queued_writes: 121, pending_queue_size: 0, local_sequence: 1775332
I20230420 18:32:57.042730 262 raft_server.cpp:551] Term: 10, last_index index: 1379, committed_index: 1377, known_applied_index: 1376, applying_index: 1377, queued_writes: 122, pending_queue_size: 0, local_sequence: 1787819
I0420 18:33:01.907192 334 external/com_github_brpc_braft/src/braft/log.cpp:536] close a full segment. Current first_index: 1355 last_index: 1380 raft_sync_segments: 0 will_sync: 1 path: /usr/share/typesense/data/state/log/log_00000000000000001355_00000000000000001380
I0420 18:33:01.907278 334 external/com_github_brpc_braft/src/braft/log.cpp:550] Renamed `/usr/share/typesense/data/state/log/log_inprogress_00000000000000001355' to `/usr/share/typesense/data/state/log/log_00000000000000001355_00000000000000001380'
I0420 18:33:01.908586 334 external/com_github_brpc_braft/src/braft/log.cpp:114] Created new segment `/usr/share/typesense/data/state/log/log_inprogress_00000000000000001381' with fd=32
I20230420 18:33:07.104025 262 raft_server.cpp:551] Term: 10, last_index index: 1394, committed_index: 1394, known_applied_index: 1394, applying_index: 0, queued_writes: 141, pending_queue_size: 0, local_sequence: 1796801
I0420 18:33:14.167294 335 external/com_github_brpc_braft/src/braft/log.cpp:536] close a full segment. Current first_index: 1381 last_index: 1405 raft_sync_segments: 0 will_sync: 1 path: /usr/share/typesense/data/state/log/log_00000000000000001381_00000000000000001405
I0420 18:33:14.167418 335 external/com_github_brpc_braft/src/braft/log.cpp:550] Renamed `/usr/share/typesense/data/state/log/log_inprogress_00000000000000001381' to `/usr/share/typesense/data/state/log/log_00000000000000001381_00000000000000001405'
I0420 18:33:14.169168 335 external/com_github_brpc_braft/src/braft/log.cpp:114] Created new segment `/usr/share/typesense/data/state/log/log_inprogress_00000000000000001406' with fd=53
I20230420 18:33:17.116726 262 raft_server.cpp:551] Term: 10, last_index index: 1414, committed_index: 1414, known_applied_index: 1414, applying_index: 0, queued_writes: 150, pending_queue_size: 0, local_sequence: 1806039
I0420 18:33:25.597174 332 external/com_github_brpc_braft/src/braft/log.cpp:536] close a full segment. Current first_index: 1406 last_index: 1437 raft_sync_segments: 0 will_sync: 1 path: /usr/share/typesense/data/state/log/log_00000000000000001406_00000000000000001437
I0420 18:33:25.597271 332 external/com_github_brpc_braft/src/braft/log.cpp:550] Renamed `/usr/share/typesense/data/state/log/log_inprogress_00000000000000001406' to `/usr/share/typesense/data/state/log/log_00000000000000001406_00000000000000001437'
I0420 18:33:25.599777 332 external/com_github_brpc_braft/src/braft/log.cpp:114] Created new segment `/usr/share/typesense/data/state/log/log_inprogress_00000000000000001438' with fd=40
I20230420 18:33:27.125538 262 raft_server.cpp:551] Term: 10, last_index index: 1441, committed_index: 1441, known_applied_index: 1441, applying_index: 0, queued_writes: 166, pending_queue_size: 0, local_sequence: 1816994
I20230420 18:33:32.485888 263 batched_indexer.cpp:284] Running GC for aborted requests, req map size: 22
I20230420 18:33:37.198104 262 raft_server.cpp:551] Term: 10, last_index index: 1458, committed_index: 1457, known_applied_index: 1456, applying_index: 1457, queued_writes: 171, pending_queue_size: 0, local_sequence: 1827621
I0420 18:33:37.271946 335 external/com_github_brpc_braft/src/braft/log.cpp:536] close a full segment. Current first_index: 1438 last_index: 1459 raft_sync_segments: 0 will_sync: 1 path: /usr/share/typesense/data/state/log/log_00000000000000001438_00000000000000001459
I0420 18:33:37.272021 335 external/com_github_brpc_braft/src/braft/log.cpp:550] Renamed `/usr/share/typesense/data/state/log/log_inprogress_00000000000000001438' to `/usr/share/typesense/data/state/log/log_00000000000000001438_00000000000000001459'
I0420 18:33:37.273421 335 external/com_github_brpc_braft/src/braft/log.cpp:114] Created new segment `/usr/share/typesense/data/state/log/log_inprogress_00000000000000001460' with fd=56
I20230420 18:33:47.213721 262 raft_server.cpp:551] Term: 10, last_index index: 1477, committed_index: 1477, known_applied_index: 1477, applying_index: 0, queued_writes: 186, pending_queue_size: 0, local_sequence: 1835280
I0420 18:33:50.018077 331 external/com_github_brpc_braft/src/braft/log.cpp:536] close a full segment. Current first_index: 1460 last_index: 1486 raft_sync_segments: 0 will_sync: 1 path: /usr/share/typesense/data/state/log/log_00000000000000001460_00000000000000001486
I0420 18:33:50.018164 331 external/com_github_brpc_braft/src/braft/log.cpp:550] Renamed `/usr/share/typesense/data/state/log/log_inprogress_00000000000000001460' to `/usr/share/typesense/data/state/log/log_00000000000000001460_00000000000000001486'
I0420 18:33:50.019704 331 external/com_github_brpc_braft/src/braft/log.cpp:114] Created new segment `/usr/share/typesense/data/state/log/log_inprogress_00000000000000001487' with fd=51
I20230420 18:33:57.225459 262 raft_server.cpp:551] Term: 10, last_index index: 1506, committed_index: 1505, known_applied_index: 1505, applying_index: 0, queued_writes: 200, pending_queue_size: 0, local_sequence: 1844423
Adrian
07:24 PMI20230420 19:13:53.121011 1 typesense_server_utils.cpp:325] Starting Typesense 0.25.0.rc18
I20230420 19:13:53.121088 1 typesense_server_utils.cpp:328] Typesense is using jemalloc.
I20230420 19:13:53.130818 1 typesense_server_utils.cpp:377] Thread pool size: 64
I20230420 19:13:53.141335 1 store.h:64] Initializing DB by opening state dir: /usr/share/typesense/data/db
I20230420 19:13:53.201871 1 store.h:64] Initializing DB by opening state dir: /usr/share/typesense/data/meta
I20230420 19:13:53.231447 1 ratelimit_manager.cpp:546] Loaded 0 rate limit rules.
I20230420 19:13:53.231493 1 ratelimit_manager.cpp:547] Loaded 0 rate limit bans.
I20230420 19:13:53.231572 1 typesense_server_utils.cpp:479] Starting API service...
I20230420 19:13:53.231750 263 batched_indexer.cpp:124] Starting batch indexer with 64 threads.
I0420 19:13:53.232262 1 src/http_server.cpp:178] Typesense has started listening on port 8108
F0420 19:13:53.232325 262 external/com_github_brpc_brpc/src/butil/at_exit.cc:46] Check failed: false. Tried to RegisterCallback without an AtExitManager
F0420 19:13:53.232346 262 external/com_github_brpc_brpc/src/butil/at_exit.cc:46] Check failed: false. Tried to RegisterCallback without an AtExitManager
I0420 19:13:53.244718 262 external/com_github_brpc_brpc/src/brpc/server.cpp:1107] Server[braft::RaftStatImpl+braft::FileServiceImpl+braft::RaftServiceImpl+braft::CliServiceImpl] is serving on port=8107.
I0420 19:13:53.263092 262 external/com_github_brpc_brpc/src/brpc/server.cpp:1110] Check out in web browser.
I20230420 19:13:53.265166 262 raft_server.cpp:67] Nodes configuration: 192.168.130.16:8107:8108,192.168.158.166:8107:8108,192.168.149.207:8107:8108
I0420 19:13:53.266084 262 external/com_github_brpc_braft/src/braft/log.cpp:690] Use murmurhash32 as the checksum type of appending entries
I0420 19:13:53.273462 262 external/com_github_brpc_braft/src/braft/log.cpp:1172] log load_meta /usr/share/typesense/data/state/log/log_meta first_log_index: 906 time: 7340
I0420 19:13:53.273522 262 external/com_github_brpc_braft/src/braft/log.cpp:1014] restore closed segment, path: /usr/share/typesense/data/state/log first_index: 1078 last_index: 1116
I0420 19:13:53.273541 262 external/com_github_brpc_braft/src/braft/log.cpp:1014] restore closed segment, path: /usr/share/typesense/data/state/log first_index: 1381 last_index: 1405
I0420 19:13:53.273547 262 external/com_github_brpc_braft/src/braft/log.cpp:1014] restore closed segment, path: /usr/share/typesense/data/state/log first_index: 1327 last_index: 1354
I0420 19:13:53.273554 262 external/com_github_brpc_braft/src/braft/log.cpp:1014] restore closed segment, path: /usr/share/typesense/data/state/log first_index: 1574 last_index: 1621
I0420 19:13:53.273559 262 external/com_github_brpc_braft/src/braft/log.cpp:1014] restore closed segment, path: /usr/share/typesense/data/state/log first_index: 1867 last_index: 1918
I0420 19:13:53.273564 262 external/com_github_brpc_braft/src/braft/log.cpp:1014] restore closed segment, path: /usr/share/typesense/data/state/log first_index: 1295 last_index: 1326
I0420 19:13:53.273568 262 external/com_github_brpc_braft/src/braft/log.cpp:1014] restore closed segment, path: /usr/share/typesense/data/state/log first_index: 1672 last_index: 1715
I0420 19:13:53.273575 262 external/com_github_brpc_braft/src/braft/log.cpp:1014] restore closed segment, path: /usr/share/typesense/data/state/log first_index: 1460 last_index: 1486
I0420 19:13:53.273580 262 external/com_github_brpc_braft/src/braft/log.cpp:1014] restore closed segment, path: /usr/share/typesense/data/state/log first_index: 1235 last_index: 1264
I0420 19:13:53.273586 262 external/com_github_brpc_braft/src/braft/log.cpp:1014] restore closed segment, path: /usr/share/typesense/data/state/log first_index: 895 last_index: 922
I0420 19:13:53.273591 262 external/com_github_brpc_braft/src/braft/log.cpp:1014] restore closed segment, path: /usr/share/typesense/data/state/log first_index: 1716 last_index: 1764
I0420 19:13:53.273597 262 external/com_github_brpc_braft/src/braft/log.cpp:1014] restore closed segment, path: /usr/share/typesense/data/state/log first_index: 1521 last_index: 1573
I0420 19:13:53.273603 262 external/com_github_brpc_braft/src/braft/log.cpp:1014] restore closed segment, path: /usr/share/typesense/data/state/log first_index: 999 last_index: 1032
I0420 19:13:53.273609 262 external/com_github_brpc_braft/src/braft/log.cpp:1014] restore closed segment, path: /usr/share/typesense/data/state/log first_index: 1622 last_index: 1671
I0420 19:13:53.273615 262 external/com_github_brpc_braft/src/braft/log.cpp:1014] restore closed segment, path: /usr/share/typesense/data/state/log first_index: 1117 last_index: 1148
I0420 19:13:53.273621 262 external/com_github_brpc_braft/src/braft/log.cpp:1014] restore closed segment, path: /usr/share/typesense/data/state/log first_index: 1178 last_index: 1205
I0420 19:13:53.273626 262 external/com_github_brpc_braft/src/braft/log.cpp:1014] restore closed segment, path: /usr/share/typesense/data/state/log first_index: 1033 last_index: 1042
I0420 19:13:53.273631 262 external/com_github_brpc_braft/src/braft/log.cpp:1014] restore closed segment, path: /usr/share/typesense/data/state/log first_index: 1438 last_index: 1459
I0420 19:13:53.273643 262 external/com_github_brpc_braft/src/braft/log.cpp:1014] restore closed segment, path: /usr/share/typesense/data/state/log first_index: 1355 last_index: 1380
I0420 19:13:53.273661 262 external/com_github_brpc_braft/src/braft/log.cpp:1014] restore closed segment, path: /usr/share/typesense/data/state/log first_index: 1815 last_index: 1866
I0420 19:13:53.273668 262 external/com_github_brpc_braft/src/braft/log.cpp:1014] restore closed segment, path: /usr/share/typesense/data/state/log first_index: 1406 last_index: 1437
I0420 19:13:53.273675 262 external/com_github_brpc_braft/src/braft/log.cpp:1014] restore closed segment, path: /usr/share/typesense/data/state/log first_index: 1206 last_index: 1234
I0420 19:13:53.273680 262 external/com_github_brpc_braft/src/braft/log.cpp:1014] restore closed segment, path: /usr/share/typesense/data/state/log first_index: 963 last_index: 998
... cut some nearly identical entries here
I20230420 19:13:53.278442 263 batched_indexer.cpp:129] BatchedIndexer skip_index: -9999
... cut some nearly identical entries here
I0420 19:13:54.539883 262 external/com_github_brpc_braft/src/braft/log.cpp:1100] load closed segment, path: /usr/share/typesense/data/state/log first_index: 1765 last_index: 1814
I0420 19:13:54.615623 262 external/com_github_brpc_braft/src/braft/log.cpp:1100] load closed segment, path: /usr/share/typesense/data/state/log first_index: 1815 last_index: 1866
I0420 19:13:54.736990 262 external/com_github_brpc_braft/src/braft/log.cpp:1100] load closed segment, path: /usr/share/typesense/data/state/log first_index: 1867 last_index: 1918
I0420 19:13:54.816458 262 external/com_github_brpc_braft/src/braft/log.cpp:1112] load open segment, path: /usr/share/typesense/data/state/log first_index: 1919
I20230420 19:13:54.825958 274 raft_server.cpp:516] on_snapshot_load
I20230420 19:13:54.883880 274 store.h:299] rm /usr/share/typesense/data/db success
I20230420 19:13:54.886502 274 store.h:309] copy snapshot /usr/share/typesense/data/state/snapshot/snapshot_00000000000000000908/db_snapshot to /usr/share/typesense/data/db success
I20230420 19:13:54.886556 274 store.h:64] Initializing DB by opening state dir: /usr/share/typesense/data/db
I20230420 19:13:54.919273 274 store.h:323] DB open success!
I20230420 19:13:54.919308 274 raft_server.cpp:495] Loading collections from disk...
I0420 19:13:54.919357 274 src/collection_manager.cpp:172] CollectionManager::load()
I20230420 19:13:54.921833 274 auth_manager.cpp:34] Indexing 2 API key(s) found on disk.
I0420 19:13:54.921908 274 src/collection_manager.cpp:192] Loading upto 32 collections in parallel, 1000 documents at a time.
I0420 19:13:54.921934 274 src/collection_manager.cpp:201] Found 1 collection(s) on disk.
I0420 19:13:54.923417 354 src/collection_manager.cpp:122] Found collection testing_collection with 4 memory shards.
I0420 19:13:54.923631 354 src/collection_manager.cpp:1240] Loading collection testing_collection
I0420 19:14:26.298273 354 src/collection_manager.cpp:1346] Loaded 475136 documents from testing_collection so far.
I0420 19:14:30.273257 354 src/collection_manager.cpp:1357] Indexed 526476/526476 documents into collection testing_collection
I0420 19:14:30.273313 354 src/collection_manager.cpp:240] Loaded 1 collection(s) so far
I0420 19:14:30.285028 274 src/collection_manager.cpp:290] Loaded 1 collection(s).
I0420 19:14:30.286936 274 src/collection_manager.cpp:294] Initializing batched indexer from snapshot state...
I20230420 19:14:30.288376 274 batched_indexer.cpp:446] Restored 0 in-flight requests from snapshot.
I20230420 19:14:30.288421 274 raft_server.cpp:502] Finished loading collections from disk.
I20230420 19:14:30.288453 274 raft_server.h:278] Configuration of this group is 192.168.130.16:8107:8108,192.168.158.166:8107:8108,192.168.149.207:8107:8108
I0420 19:14:30.288577 274 external/com_github_brpc_braft/src/braft/snapshot_executor.cpp:264] node default_group:192.168.158.166:8107:8108 snapshot_load_done, last_included_index: 908 last_included_term: 8 peers: "192.168.130.16:8107:8108" peers: "192.168.158.166:8107:8108" peers: "192.168.149.207:8107:8108"
I0420 19:14:30.290172 262 external/com_github_brpc_braft/src/braft/raft_meta.cpp:521] Loaded single stable meta, path /usr/share/typesense/data/state/meta term 10 votedfor 192.168.130.16:8107:8108 time: 1351
I0420 19:14:30.290237 262 external/com_github_brpc_braft/src/braft/node.cpp:608] node default_group:192.168.158.166:8107:8108 init, term: 10 last_log_id: (index=1923,term=10) conf: 192.168.130.16:8107:8108,192.168.158.166:8107:8108,192.168.149.207:8107:8108 old_conf:
I20230420 19:14:30.290299 262 raft_server.cpp:133] Node last_index: 1923
I20230420 19:14:30.290328 262 typesense_server_utils.cpp:274] Typesense peering service is running on 192.168.158.166:8107
I20230420 19:14:30.290339 262 typesense_server_utils.cpp:275] Snapshot interval configured as: 3600s
I20230420 19:14:30.290347 262 typesense_server_utils.cpp:276] Snapshot max byte count configured as: 4194304
W0420 19:14:30.290363 262 external/com_github_brpc_brpc/src/brpc/controller.cpp:1487] SIGINT was installed with 1
I20230420 19:14:30.291049 262 raft_server.cpp:551] Term: 10, last_index index: 1923, committed_index: 0, known_applied_index: 908, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1691505
W20230420 19:14:30.291072 262 raft_server.cpp:578] Multi-node with no leader: refusing to reset peers.
E20230420 19:14:30.503494 277 raft_server.cpp:625] 1015 lagging entries > healthy read lag of 1000
E20230420 19:14:30.503545 277 raft_server.cpp:637] 1015 lagging entries > healthy write lag of 500
I20230420 19:14:30.503556 277 raft_server.h:283] Node starts following { leader_id=192.168.130.16:8107:8108, term=10, status=Follower receives message from new leader with the same term.}
I20230420 19:14:33.372130 277 raft_server.h:278] Configuration of this group is 192.168.130.16:8107:8108,192.168.158.166:8107:8108,192.168.149.207:8107:8108
I20230420 19:14:36.301144 277 raft_server.h:278] Configuration of this group is 192.168.130.16:8107:8108,192.168.158.166:8107:8108,192.168.149.207:8107:8108
Apr 21, 2023 (5 months ago)
Kishore Nallan
03:13 AMKishore Nallan
03:14 AME20230420 19:14:30.503494 277 raft_server.cpp:625] 1015 lagging entries > healthy read lag of 1000
E20230420 19:14:30.503545 277 raft_server.cpp:637] 1015 lagging entries > healthy write lag of 500
You can increase the
--healthy-read-lag
and --healthy-write-lag
configuration values or just add a small delay between 2 batches of imports.Adrian
02:28 PMAdrian
02:28 PM
Typesense
Indexed 2764 threads (79% resolved)
Similar Threads
Typesense Node Stuck in Segfault Loop After Stress Test
Adrian encountered a segfault loop when stress testing a Typesense cluster. Kishore Nallan recommended trying a newer RC build and suggested potential issues with hostname resolution.



Large JSONL Documents Import Issue & Resolution
Suraj was having trouble loading large JSONL documents into Typesense server. After several discussions and attempts, it was discovered that the issue was due to data quality. Once the team extracted the data again, the upload process worked smoothly.

Troubleshooting Stalled Writes in TypeSense Instance
Robert was experiencing typesense instances getting stuck after trying to import documents. Kishore Nallan provided suggestions and added specific logs to diagnose the issue. The two identified queries causing troubles but the issues had not been fully resolved yet.

