#community-help

Typesense Processes Intermittently Crash When Indexing Documents

TLDR Adrian experiences intermittent typesense process crashes when indexing 500k documents. Kishore Nallan explains that it's not a crash but a backpressure mechanism rejecting writes when ingesting too fast.

5mo

Join the chat

Apr 20, 2023 (5 months ago)

Photo of md5-27ff63286c7b3dcb91085f39e910c437

Adrian

06:45 PM

Hey. I'm trying to index 500k documents. I'm doing so using the Import endpoint 10k documents at a time. With action := "upsert", and batchSize := 1000. The typesense processes are intermittently crashing while I'm doing so. Is this expected? Am I trying to upload too many documents at at time? I don't see any obvious issues in the log for the typesense servers that are crashing.

06:48

Adrian

06:48 PM

This is the log right before crashing

I0420 18:31:31.142714   334 external/com_github_brpc_braft/src/braft/log.cpp:114] Created new segment `/usr/share/typesense/data/state/log/log_inprogress_00000000000000001178' with fd=39
I20230420 18:31:36.829527   262 raft_server.cpp:551] Term: 10, last_index index: 1186, committed_index: 1186, known_applied_index: 1186, applying_index: 0, queued_writes: 16, pending_queue_size: 0, local_sequence: 1700546
I20230420 18:31:46.896195   262 raft_server.cpp:551] Term: 10, last_index index: 1205, committed_index: 1205, known_applied_index: 1205, applying_index: 0, queued_writes: 22, pending_queue_size: 0, local_sequence: 1713504
I0420 18:31:47.343566   334 external/com_github_brpc_braft/src/braft/log.cpp:536] close a full segment. Current first_index: 1178 last_index: 1205 raft_sync_segments: 0 will_sync: 1 path: /usr/share/typesense/data/state/log/log_00000000000000001178_00000000000000001205
I0420 18:31:47.343642   334 external/com_github_brpc_braft/src/braft/log.cpp:550] Renamed `/usr/share/typesense/data/state/log/log_inprogress_00000000000000001178' to `/usr/share/typesense/data/state/log/log_00000000000000001178_00000000000000001205'
I0420 18:31:47.344864   334 external/com_github_brpc_braft/src/braft/log.cpp:114] Created new segment `/usr/share/typesense/data/state/log/log_inprogress_00000000000000001206' with fd=41
I20230420 18:31:56.905944   262 raft_server.cpp:551] Term: 10, last_index index: 1233, committed_index: 1233, known_applied_index: 1233, applying_index: 0, queued_writes: 46, pending_queue_size: 0, local_sequence: 1722741
I0420 18:31:58.559419   332 external/com_github_brpc_braft/src/braft/log.cpp:536] close a full segment. Current first_index: 1206 last_index: 1234 raft_sync_segments: 0 will_sync: 1 path: /usr/share/typesense/data/state/log/log_00000000000000001206_00000000000000001234
I0420 18:31:58.559503   332 external/com_github_brpc_braft/src/braft/log.cpp:550] Renamed `/usr/share/typesense/data/state/log/log_inprogress_00000000000000001206' to `/usr/share/typesense/data/state/log/log_00000000000000001206_00000000000000001234'
I0420 18:31:58.560835   332 external/com_github_brpc_braft/src/braft/log.cpp:114] Created new segment `/usr/share/typesense/data/state/log/log_inprogress_00000000000000001235' with fd=29
I20230420 18:32:06.917093   262 raft_server.cpp:551] Term: 10, last_index index: 1260, committed_index: 1260, known_applied_index: 1260, applying_index: 0, queued_writes: 61, pending_queue_size: 0, local_sequence: 1734210
I0420 18:32:09.687240   332 external/com_github_brpc_braft/src/braft/log.cpp:536] close a full segment. Current first_index: 1235 last_index: 1264 raft_sync_segments: 0 will_sync: 1 path: /usr/share/typesense/data/state/log/log_00000000000000001235_00000000000000001264
I0420 18:32:09.687352   332 external/com_github_brpc_braft/src/braft/log.cpp:550] Renamed `/usr/share/typesense/data/state/log/log_inprogress_00000000000000001235' to `/usr/share/typesense/data/state/log/log_00000000000000001235_00000000000000001264'
I0420 18:32:09.688412   332 external/com_github_brpc_braft/src/braft/log.cpp:114] Created new segment `/usr/share/typesense/data/state/log/log_inprogress_00000000000000001265' with fd=47
I20230420 18:32:16.942395   262 raft_server.cpp:551] Term: 10, last_index index: 1288, committed_index: 1288, known_applied_index: 1288, applying_index: 0, queued_writes: 81, pending_queue_size: 0, local_sequence: 1744351
I0420 18:32:20.677118   332 external/com_github_brpc_braft/src/braft/log.cpp:536] close a full segment. Current first_index: 1265 last_index: 1294 raft_sync_segments: 0 will_sync: 1 path: /usr/share/typesense/data/state/log/log_00000000000000001265_00000000000000001294
I0420 18:32:20.677200   332 external/com_github_brpc_braft/src/braft/log.cpp:550] Renamed `/usr/share/typesense/data/state/log/log_inprogress_00000000000000001265' to `/usr/share/typesense/data/state/log/log_00000000000000001265_00000000000000001294'
I0420 18:32:20.678689   332 external/com_github_brpc_braft/src/braft/log.cpp:114] Created new segment `/usr/share/typesense/data/state/log/log_inprogress_00000000000000001295' with fd=49
I20230420 18:32:27.001346   262 raft_server.cpp:551] Term: 10, last_index index: 1309, committed_index: 1309, known_applied_index: 1309, applying_index: 0, queued_writes: 89, pending_queue_size: 0, local_sequence: 1754377
I20230420 18:32:31.471853   263 batched_indexer.cpp:284] Running GC for aborted requests, req map size: 13
I0420 18:32:32.015195   335 external/com_github_brpc_braft/src/braft/log.cpp:536] close a full segment. Current first_index: 1295 last_index: 1326 raft_sync_segments: 0 will_sync: 1 path: /usr/share/typesense/data/state/log/log_00000000000000001295_00000000000000001326
I0420 18:32:32.015277   335 external/com_github_brpc_braft/src/braft/log.cpp:550] Renamed `/usr/share/typesense/data/state/log/log_inprogress_00000000000000001295' to `/usr/share/typesense/data/state/log/log_00000000000000001295_00000000000000001326'
I0420 18:32:32.016613   335 external/com_github_brpc_braft/src/braft/log.cpp:114] Created new segment `/usr/share/typesense/data/state/log/log_inprogress_00000000000000001327' with fd=46
I20230420 18:32:37.014884   262 raft_server.cpp:551] Term: 10, last_index index: 1341, committed_index: 1340, known_applied_index: 1340, applying_index: 0, queued_writes: 101, pending_queue_size: 0, local_sequence: 1766181
I0420 18:32:44.592146   334 external/com_github_brpc_braft/src/braft/log.cpp:536] close a full segment. Current first_index: 1327 last_index: 1354 raft_sync_segments: 0 will_sync: 1 path: /usr/share/typesense/data/state/log/log_00000000000000001327_00000000000000001354
I0420 18:32:44.592230   334 external/com_github_brpc_braft/src/braft/log.cpp:550] Renamed `/usr/share/typesense/data/state/log/log_inprogress_00000000000000001327' to `/usr/share/typesense/data/state/log/log_00000000000000001327_00000000000000001354'
I0420 18:32:44.593456   334 external/com_github_brpc_braft/src/braft/log.cpp:114] Created new segment `/usr/share/typesense/data/state/log/log_inprogress_00000000000000001355' with fd=33
I20230420 18:32:47.030795   262 raft_server.cpp:551] Term: 10, last_index index: 1356, committed_index: 1356, known_applied_index: 1356, applying_index: 0, queued_writes: 121, pending_queue_size: 0, local_sequence: 1775332
I20230420 18:32:57.042730   262 raft_server.cpp:551] Term: 10, last_index index: 1379, committed_index: 1377, known_applied_index: 1376, applying_index: 1377, queued_writes: 122, pending_queue_size: 0, local_sequence: 1787819
I0420 18:33:01.907192   334 external/com_github_brpc_braft/src/braft/log.cpp:536] close a full segment. Current first_index: 1355 last_index: 1380 raft_sync_segments: 0 will_sync: 1 path: /usr/share/typesense/data/state/log/log_00000000000000001355_00000000000000001380
I0420 18:33:01.907278   334 external/com_github_brpc_braft/src/braft/log.cpp:550] Renamed `/usr/share/typesense/data/state/log/log_inprogress_00000000000000001355' to `/usr/share/typesense/data/state/log/log_00000000000000001355_00000000000000001380'
I0420 18:33:01.908586   334 external/com_github_brpc_braft/src/braft/log.cpp:114] Created new segment `/usr/share/typesense/data/state/log/log_inprogress_00000000000000001381' with fd=32
I20230420 18:33:07.104025   262 raft_server.cpp:551] Term: 10, last_index index: 1394, committed_index: 1394, known_applied_index: 1394, applying_index: 0, queued_writes: 141, pending_queue_size: 0, local_sequence: 1796801
I0420 18:33:14.167294   335 external/com_github_brpc_braft/src/braft/log.cpp:536] close a full segment. Current first_index: 1381 last_index: 1405 raft_sync_segments: 0 will_sync: 1 path: /usr/share/typesense/data/state/log/log_00000000000000001381_00000000000000001405
I0420 18:33:14.167418   335 external/com_github_brpc_braft/src/braft/log.cpp:550] Renamed `/usr/share/typesense/data/state/log/log_inprogress_00000000000000001381' to `/usr/share/typesense/data/state/log/log_00000000000000001381_00000000000000001405'
I0420 18:33:14.169168   335 external/com_github_brpc_braft/src/braft/log.cpp:114] Created new segment `/usr/share/typesense/data/state/log/log_inprogress_00000000000000001406' with fd=53
I20230420 18:33:17.116726   262 raft_server.cpp:551] Term: 10, last_index index: 1414, committed_index: 1414, known_applied_index: 1414, applying_index: 0, queued_writes: 150, pending_queue_size: 0, local_sequence: 1806039
I0420 18:33:25.597174   332 external/com_github_brpc_braft/src/braft/log.cpp:536] close a full segment. Current first_index: 1406 last_index: 1437 raft_sync_segments: 0 will_sync: 1 path: /usr/share/typesense/data/state/log/log_00000000000000001406_00000000000000001437
I0420 18:33:25.597271   332 external/com_github_brpc_braft/src/braft/log.cpp:550] Renamed `/usr/share/typesense/data/state/log/log_inprogress_00000000000000001406' to `/usr/share/typesense/data/state/log/log_00000000000000001406_00000000000000001437'
I0420 18:33:25.599777   332 external/com_github_brpc_braft/src/braft/log.cpp:114] Created new segment `/usr/share/typesense/data/state/log/log_inprogress_00000000000000001438' with fd=40
I20230420 18:33:27.125538   262 raft_server.cpp:551] Term: 10, last_index index: 1441, committed_index: 1441, known_applied_index: 1441, applying_index: 0, queued_writes: 166, pending_queue_size: 0, local_sequence: 1816994
I20230420 18:33:32.485888   263 batched_indexer.cpp:284] Running GC for aborted requests, req map size: 22
I20230420 18:33:37.198104   262 raft_server.cpp:551] Term: 10, last_index index: 1458, committed_index: 1457, known_applied_index: 1456, applying_index: 1457, queued_writes: 171, pending_queue_size: 0, local_sequence: 1827621
I0420 18:33:37.271946   335 external/com_github_brpc_braft/src/braft/log.cpp:536] close a full segment. Current first_index: 1438 last_index: 1459 raft_sync_segments: 0 will_sync: 1 path: /usr/share/typesense/data/state/log/log_00000000000000001438_00000000000000001459
I0420 18:33:37.272021   335 external/com_github_brpc_braft/src/braft/log.cpp:550] Renamed `/usr/share/typesense/data/state/log/log_inprogress_00000000000000001438' to `/usr/share/typesense/data/state/log/log_00000000000000001438_00000000000000001459'
I0420 18:33:37.273421   335 external/com_github_brpc_braft/src/braft/log.cpp:114] Created new segment `/usr/share/typesense/data/state/log/log_inprogress_00000000000000001460' with fd=56
I20230420 18:33:47.213721   262 raft_server.cpp:551] Term: 10, last_index index: 1477, committed_index: 1477, known_applied_index: 1477, applying_index: 0, queued_writes: 186, pending_queue_size: 0, local_sequence: 1835280
I0420 18:33:50.018077   331 external/com_github_brpc_braft/src/braft/log.cpp:536] close a full segment. Current first_index: 1460 last_index: 1486 raft_sync_segments: 0 will_sync: 1 path: /usr/share/typesense/data/state/log/log_00000000000000001460_00000000000000001486
I0420 18:33:50.018164   331 external/com_github_brpc_braft/src/braft/log.cpp:550] Renamed `/usr/share/typesense/data/state/log/log_inprogress_00000000000000001460' to `/usr/share/typesense/data/state/log/log_00000000000000001460_00000000000000001486'
I0420 18:33:50.019704   331 external/com_github_brpc_braft/src/braft/log.cpp:114] Created new segment `/usr/share/typesense/data/state/log/log_inprogress_00000000000000001487' with fd=51
I20230420 18:33:57.225459   262 raft_server.cpp:551] Term: 10, last_index index: 1506, committed_index: 1505, known_applied_index: 1505, applying_index: 0, queued_writes: 200, pending_queue_size: 0, local_sequence: 1844423

07:24

Adrian

07:24 PM

Here is another example. This is the full log this time

I20230420 19:13:53.121011     1 typesense_server_utils.cpp:325] Starting Typesense 0.25.0.rc18
I20230420 19:13:53.121088     1 typesense_server_utils.cpp:328] Typesense is using jemalloc.
I20230420 19:13:53.130818     1 typesense_server_utils.cpp:377] Thread pool size: 64
I20230420 19:13:53.141335     1 store.h:64] Initializing DB by opening state dir: /usr/share/typesense/data/db
I20230420 19:13:53.201871     1 store.h:64] Initializing DB by opening state dir: /usr/share/typesense/data/meta
I20230420 19:13:53.231447     1 ratelimit_manager.cpp:546] Loaded 0 rate limit rules.
I20230420 19:13:53.231493     1 ratelimit_manager.cpp:547] Loaded 0 rate limit bans.
I20230420 19:13:53.231572     1 typesense_server_utils.cpp:479] Starting API service...
I20230420 19:13:53.231750   263 batched_indexer.cpp:124] Starting batch indexer with 64 threads.
I0420 19:13:53.232262     1 src/http_server.cpp:178] Typesense has started listening on port 8108
F0420 19:13:53.232325   262 external/com_github_brpc_brpc/src/butil/at_exit.cc:46] Check failed: false. Tried to RegisterCallback without an AtExitManager
F0420 19:13:53.232346   262 external/com_github_brpc_brpc/src/butil/at_exit.cc:46] Check failed: false. Tried to RegisterCallback without an AtExitManager
I0420 19:13:53.244718   262 external/com_github_brpc_brpc/src/brpc/server.cpp:1107] Server[braft::RaftStatImpl+braft::FileServiceImpl+braft::RaftServiceImpl+braft::CliServiceImpl] is serving on port=8107.
I0420 19:13:53.263092   262 external/com_github_brpc_brpc/src/brpc/server.cpp:1110] Check out  in web browser.
I20230420 19:13:53.265166   262 raft_server.cpp:67] Nodes configuration: 192.168.130.16:8107:8108,192.168.158.166:8107:8108,192.168.149.207:8107:8108
I0420 19:13:53.266084   262 external/com_github_brpc_braft/src/braft/log.cpp:690] Use murmurhash32 as the checksum type of appending entries
I0420 19:13:53.273462   262 external/com_github_brpc_braft/src/braft/log.cpp:1172] log load_meta /usr/share/typesense/data/state/log/log_meta first_log_index: 906 time: 7340
I0420 19:13:53.273522   262 external/com_github_brpc_braft/src/braft/log.cpp:1014] restore closed segment, path: /usr/share/typesense/data/state/log first_index: 1078 last_index: 1116
I0420 19:13:53.273541   262 external/com_github_brpc_braft/src/braft/log.cpp:1014] restore closed segment, path: /usr/share/typesense/data/state/log first_index: 1381 last_index: 1405
I0420 19:13:53.273547   262 external/com_github_brpc_braft/src/braft/log.cpp:1014] restore closed segment, path: /usr/share/typesense/data/state/log first_index: 1327 last_index: 1354
I0420 19:13:53.273554   262 external/com_github_brpc_braft/src/braft/log.cpp:1014] restore closed segment, path: /usr/share/typesense/data/state/log first_index: 1574 last_index: 1621
I0420 19:13:53.273559   262 external/com_github_brpc_braft/src/braft/log.cpp:1014] restore closed segment, path: /usr/share/typesense/data/state/log first_index: 1867 last_index: 1918
I0420 19:13:53.273564   262 external/com_github_brpc_braft/src/braft/log.cpp:1014] restore closed segment, path: /usr/share/typesense/data/state/log first_index: 1295 last_index: 1326
I0420 19:13:53.273568   262 external/com_github_brpc_braft/src/braft/log.cpp:1014] restore closed segment, path: /usr/share/typesense/data/state/log first_index: 1672 last_index: 1715
I0420 19:13:53.273575   262 external/com_github_brpc_braft/src/braft/log.cpp:1014] restore closed segment, path: /usr/share/typesense/data/state/log first_index: 1460 last_index: 1486
I0420 19:13:53.273580   262 external/com_github_brpc_braft/src/braft/log.cpp:1014] restore closed segment, path: /usr/share/typesense/data/state/log first_index: 1235 last_index: 1264
I0420 19:13:53.273586   262 external/com_github_brpc_braft/src/braft/log.cpp:1014] restore closed segment, path: /usr/share/typesense/data/state/log first_index: 895 last_index: 922
I0420 19:13:53.273591   262 external/com_github_brpc_braft/src/braft/log.cpp:1014] restore closed segment, path: /usr/share/typesense/data/state/log first_index: 1716 last_index: 1764
I0420 19:13:53.273597   262 external/com_github_brpc_braft/src/braft/log.cpp:1014] restore closed segment, path: /usr/share/typesense/data/state/log first_index: 1521 last_index: 1573
I0420 19:13:53.273603   262 external/com_github_brpc_braft/src/braft/log.cpp:1014] restore closed segment, path: /usr/share/typesense/data/state/log first_index: 999 last_index: 1032
I0420 19:13:53.273609   262 external/com_github_brpc_braft/src/braft/log.cpp:1014] restore closed segment, path: /usr/share/typesense/data/state/log first_index: 1622 last_index: 1671
I0420 19:13:53.273615   262 external/com_github_brpc_braft/src/braft/log.cpp:1014] restore closed segment, path: /usr/share/typesense/data/state/log first_index: 1117 last_index: 1148
I0420 19:13:53.273621   262 external/com_github_brpc_braft/src/braft/log.cpp:1014] restore closed segment, path: /usr/share/typesense/data/state/log first_index: 1178 last_index: 1205
I0420 19:13:53.273626   262 external/com_github_brpc_braft/src/braft/log.cpp:1014] restore closed segment, path: /usr/share/typesense/data/state/log first_index: 1033 last_index: 1042
I0420 19:13:53.273631   262 external/com_github_brpc_braft/src/braft/log.cpp:1014] restore closed segment, path: /usr/share/typesense/data/state/log first_index: 1438 last_index: 1459
I0420 19:13:53.273643   262 external/com_github_brpc_braft/src/braft/log.cpp:1014] restore closed segment, path: /usr/share/typesense/data/state/log first_index: 1355 last_index: 1380
I0420 19:13:53.273661   262 external/com_github_brpc_braft/src/braft/log.cpp:1014] restore closed segment, path: /usr/share/typesense/data/state/log first_index: 1815 last_index: 1866
I0420 19:13:53.273668   262 external/com_github_brpc_braft/src/braft/log.cpp:1014] restore closed segment, path: /usr/share/typesense/data/state/log first_index: 1406 last_index: 1437
I0420 19:13:53.273675   262 external/com_github_brpc_braft/src/braft/log.cpp:1014] restore closed segment, path: /usr/share/typesense/data/state/log first_index: 1206 last_index: 1234
I0420 19:13:53.273680   262 external/com_github_brpc_braft/src/braft/log.cpp:1014] restore closed segment, path: /usr/share/typesense/data/state/log first_index: 963 last_index: 998

... cut some nearly identical entries here
I20230420 19:13:53.278442   263 batched_indexer.cpp:129] BatchedIndexer skip_index: -9999
... cut some nearly identical entries here
I0420 19:13:54.539883   262 external/com_github_brpc_braft/src/braft/log.cpp:1100] load closed segment, path: /usr/share/typesense/data/state/log first_index: 1765 last_index: 1814
I0420 19:13:54.615623   262 external/com_github_brpc_braft/src/braft/log.cpp:1100] load closed segment, path: /usr/share/typesense/data/state/log first_index: 1815 last_index: 1866
I0420 19:13:54.736990   262 external/com_github_brpc_braft/src/braft/log.cpp:1100] load closed segment, path: /usr/share/typesense/data/state/log first_index: 1867 last_index: 1918
I0420 19:13:54.816458   262 external/com_github_brpc_braft/src/braft/log.cpp:1112] load open segment, path: /usr/share/typesense/data/state/log first_index: 1919
I20230420 19:13:54.825958   274 raft_server.cpp:516] on_snapshot_load
I20230420 19:13:54.883880   274 store.h:299] rm /usr/share/typesense/data/db success
I20230420 19:13:54.886502   274 store.h:309] copy snapshot /usr/share/typesense/data/state/snapshot/snapshot_00000000000000000908/db_snapshot to /usr/share/typesense/data/db success
I20230420 19:13:54.886556   274 store.h:64] Initializing DB by opening state dir: /usr/share/typesense/data/db
I20230420 19:13:54.919273   274 store.h:323] DB open success!
I20230420 19:13:54.919308   274 raft_server.cpp:495] Loading collections from disk...
I0420 19:13:54.919357   274 src/collection_manager.cpp:172] CollectionManager::load()
I20230420 19:13:54.921833   274 auth_manager.cpp:34] Indexing 2 API key(s) found on disk.
I0420 19:13:54.921908   274 src/collection_manager.cpp:192] Loading upto 32 collections in parallel, 1000 documents at a time.
I0420 19:13:54.921934   274 src/collection_manager.cpp:201] Found 1 collection(s) on disk.
I0420 19:13:54.923417   354 src/collection_manager.cpp:122] Found collection testing_collection with 4 memory shards.
I0420 19:13:54.923631   354 src/collection_manager.cpp:1240] Loading collection testing_collection
I0420 19:14:26.298273   354 src/collection_manager.cpp:1346] Loaded 475136 documents from testing_collection so far.
I0420 19:14:30.273257   354 src/collection_manager.cpp:1357] Indexed 526476/526476 documents into collection testing_collection
I0420 19:14:30.273313   354 src/collection_manager.cpp:240] Loaded 1 collection(s) so far
I0420 19:14:30.285028   274 src/collection_manager.cpp:290] Loaded 1 collection(s).
I0420 19:14:30.286936   274 src/collection_manager.cpp:294] Initializing batched indexer from snapshot state...
I20230420 19:14:30.288376   274 batched_indexer.cpp:446] Restored 0 in-flight requests from snapshot.
I20230420 19:14:30.288421   274 raft_server.cpp:502] Finished loading collections from disk.
I20230420 19:14:30.288453   274 raft_server.h:278] Configuration of this group is 192.168.130.16:8107:8108,192.168.158.166:8107:8108,192.168.149.207:8107:8108
I0420 19:14:30.288577   274 external/com_github_brpc_braft/src/braft/snapshot_executor.cpp:264] node default_group:192.168.158.166:8107:8108 snapshot_load_done, last_included_index: 908 last_included_term: 8 peers: "192.168.130.16:8107:8108" peers: "192.168.158.166:8107:8108" peers: "192.168.149.207:8107:8108"
I0420 19:14:30.290172   262 external/com_github_brpc_braft/src/braft/raft_meta.cpp:521] Loaded single stable meta, path /usr/share/typesense/data/state/meta term 10 votedfor 192.168.130.16:8107:8108 time: 1351
I0420 19:14:30.290237   262 external/com_github_brpc_braft/src/braft/node.cpp:608] node default_group:192.168.158.166:8107:8108 init, term: 10 last_log_id: (index=1923,term=10) conf: 192.168.130.16:8107:8108,192.168.158.166:8107:8108,192.168.149.207:8107:8108 old_conf:
I20230420 19:14:30.290299   262 raft_server.cpp:133] Node last_index: 1923
I20230420 19:14:30.290328   262 typesense_server_utils.cpp:274] Typesense peering service is running on 192.168.158.166:8107
I20230420 19:14:30.290339   262 typesense_server_utils.cpp:275] Snapshot interval configured as: 3600s
I20230420 19:14:30.290347   262 typesense_server_utils.cpp:276] Snapshot max byte count configured as: 4194304
W0420 19:14:30.290363   262 external/com_github_brpc_brpc/src/brpc/controller.cpp:1487] SIGINT was installed with 1
I20230420 19:14:30.291049   262 raft_server.cpp:551] Term: 10, last_index index: 1923, committed_index: 0, known_applied_index: 908, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1691505
W20230420 19:14:30.291072   262 raft_server.cpp:578] Multi-node with no leader: refusing to reset peers.
E20230420 19:14:30.503494   277 raft_server.cpp:625] 1015 lagging entries > healthy read lag of 1000
E20230420 19:14:30.503545   277 raft_server.cpp:637] 1015 lagging entries > healthy write lag of 500
I20230420 19:14:30.503556   277 raft_server.h:283] Node starts following { leader_id=192.168.130.16:8107:8108, term=10, status=Follower receives message from new leader with the same term.}
I20230420 19:14:33.372130   277 raft_server.h:278] Configuration of this group is 192.168.130.16:8107:8108,192.168.158.166:8107:8108,192.168.149.207:8107:8108
I20230420 19:14:36.301144   277 raft_server.h:278] Configuration of this group is 192.168.130.16:8107:8108,192.168.158.166:8107:8108,192.168.149.207:8107:8108

Apr 21, 2023 (5 months ago)

Photo of md5-4e872368b2b2668460205b409e95c2ea

Kishore Nallan

03:13 AM

That's not a crash. When you ingest too fast there is a backpressure mechanism to reject writes until the writes catch up. This prevents searches from being slowed down.

03:14

Kishore Nallan

03:14 AM

See this:

E20230420 19:14:30.503494   277 raft_server.cpp:625] 1015 lagging entries > healthy read lag of 1000
E20230420 19:14:30.503545   277 raft_server.cpp:637] 1015 lagging entries > healthy write lag of 500

You can increase the --healthy-read-lag and --healthy-write-lag configuration values or just add a small delay between 2 batches of imports.

Adrian

02:28 PM

thanks for the info!

02:28

Adrian

02:28 PM

turns out the pods were also running out of memory, which was causing full crashes and restarts as well. Working now that I increased memory limits

Typesense

Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI

Indexed 2764 threads (79% resolved)

Join Our Community Manage subscription

Similar Threads

Typesense Node Stuck in Segfault Loop After Stress Test

Adrian encountered a segfault loop when stress testing a Typesense cluster. Kishore Nallan recommended trying a newer RC build and suggested potential issues with hostname resolution.

5mo

Large JSONL Documents Import Issue & Resolution

Suraj was having trouble loading large JSONL documents into Typesense server. After several discussions and attempts, it was discovered that the issue was due to data quality. Once the team extracted the data again, the upload process worked smoothly.

6mo

Troubleshooting Stalled Writes in TypeSense Instance

Robert was experiencing typesense instances getting stuck after trying to import documents. Kishore Nallan provided suggestions and added specific logs to diagnose the issue. The two identified queries causing troubles but the issues had not been fully resolved yet.

10mo