Hey I m trying to index 500k documents I m doing so using th typesense #community-help

Hey. I'm trying to index 500k documents. I'm doing...

Adrian Kager

04/20/2023, 6:45 PM

Hey. I'm trying to index 500k documents. I'm doing so using the Import endpoint 10k documents at a time. With action := "upsert", and batchSize := 1000. The typesense processes are intermittently crashing while I'm doing so. Is this expected? Am I trying to upload too many documents at at time? I don't see any obvious issues in the log for the typesense servers that are crashing.

Adrian Kager

04/20/2023, 6:48 PM

This is the log right before crashing

Copy code

I0420 18:31:31.142714   334 external/com_github_brpc_braft/src/braft/log.cpp:114] Created new segment `/usr/share/typesense/data/state/log/log_inprogress_00000000000000001178' with fd=39
I20230420 18:31:36.829527   262 raft_server.cpp:551] Term: 10, last_index index: 1186, committed_index: 1186, known_applied_index: 1186, applying_index: 0, queued_writes: 16, pending_queue_size: 0, local_sequence: 1700546
I20230420 18:31:46.896195   262 raft_server.cpp:551] Term: 10, last_index index: 1205, committed_index: 1205, known_applied_index: 1205, applying_index: 0, queued_writes: 22, pending_queue_size: 0, local_sequence: 1713504
I0420 18:31:47.343566   334 external/com_github_brpc_braft/src/braft/log.cpp:536] close a full segment. Current first_index: 1178 last_index: 1205 raft_sync_segments: 0 will_sync: 1 path: /usr/share/typesense/data/state/log/log_00000000000000001178_00000000000000001205
I0420 18:31:47.343642   334 external/com_github_brpc_braft/src/braft/log.cpp:550] Renamed `/usr/share/typesense/data/state/log/log_inprogress_00000000000000001178' to `/usr/share/typesense/data/state/log/log_00000000000000001178_00000000000000001205'
I0420 18:31:47.344864   334 external/com_github_brpc_braft/src/braft/log.cpp:114] Created new segment `/usr/share/typesense/data/state/log/log_inprogress_00000000000000001206' with fd=41
I20230420 18:31:56.905944   262 raft_server.cpp:551] Term: 10, last_index index: 1233, committed_index: 1233, known_applied_index: 1233, applying_index: 0, queued_writes: 46, pending_queue_size: 0, local_sequence: 1722741
I0420 18:31:58.559419   332 external/com_github_brpc_braft/src/braft/log.cpp:536] close a full segment. Current first_index: 1206 last_index: 1234 raft_sync_segments: 0 will_sync: 1 path: /usr/share/typesense/data/state/log/log_00000000000000001206_00000000000000001234
I0420 18:31:58.559503   332 external/com_github_brpc_braft/src/braft/log.cpp:550] Renamed `/usr/share/typesense/data/state/log/log_inprogress_00000000000000001206' to `/usr/share/typesense/data/state/log/log_00000000000000001206_00000000000000001234'
I0420 18:31:58.560835   332 external/com_github_brpc_braft/src/braft/log.cpp:114] Created new segment `/usr/share/typesense/data/state/log/log_inprogress_00000000000000001235' with fd=29
I20230420 18:32:06.917093   262 raft_server.cpp:551] Term: 10, last_index index: 1260, committed_index: 1260, known_applied_index: 1260, applying_index: 0, queued_writes: 61, pending_queue_size: 0, local_sequence: 1734210
I0420 18:32:09.687240   332 external/com_github_brpc_braft/src/braft/log.cpp:536] close a full segment. Current first_index: 1235 last_index: 1264 raft_sync_segments: 0 will_sync: 1 path: /usr/share/typesense/data/state/log/log_00000000000000001235_00000000000000001264
I0420 18:32:09.687352   332 external/com_github_brpc_braft/src/braft/log.cpp:550] Renamed `/usr/share/typesense/data/state/log/log_inprogress_00000000000000001235' to `/usr/share/typesense/data/state/log/log_00000000000000001235_00000000000000001264'
I0420 18:32:09.688412   332 external/com_github_brpc_braft/src/braft/log.cpp:114] Created new segment `/usr/share/typesense/data/state/log/log_inprogress_00000000000000001265' with fd=47
I20230420 18:32:16.942395   262 raft_server.cpp:551] Term: 10, last_index index: 1288, committed_index: 1288, known_applied_index: 1288, applying_index: 0, queued_writes: 81, pending_queue_size: 0, local_sequence: 1744351
I0420 18:32:20.677118   332 external/com_github_brpc_braft/src/braft/log.cpp:536] close a full segment. Current first_index: 1265 last_index: 1294 raft_sync_segments: 0 will_sync: 1 path: /usr/share/typesense/data/state/log/log_00000000000000001265_00000000000000001294
I0420 18:32:20.677200   332 external/com_github_brpc_braft/src/braft/log.cpp:550] Renamed `/usr/share/typesense/data/state/log/log_inprogress_00000000000000001265' to `/usr/share/typesense/data/state/log/log_00000000000000001265_00000000000000001294'
I0420 18:32:20.678689   332 external/com_github_brpc_braft/src/braft/log.cpp:114] Created new segment `/usr/share/typesense/data/state/log/log_inprogress_00000000000000001295' with fd=49
I20230420 18:32:27.001346   262 raft_server.cpp:551] Term: 10, last_index index: 1309, committed_index: 1309, known_applied_index: 1309, applying_index: 0, queued_writes: 89, pending_queue_size: 0, local_sequence: 1754377
I20230420 18:32:31.471853   263 batched_indexer.cpp:284] Running GC for aborted requests, req map size: 13
I0420 18:32:32.015195   335 external/com_github_brpc_braft/src/braft/log.cpp:536] close a full segment. Current first_index: 1295 last_index: 1326 raft_sync_segments: 0 will_sync: 1 path: /usr/share/typesense/data/state/log/log_00000000000000001295_00000000000000001326
I0420 18:32:32.015277   335 external/com_github_brpc_braft/src/braft/log.cpp:550] Renamed `/usr/share/typesense/data/state/log/log_inprogress_00000000000000001295' to `/usr/share/typesense/data/state/log/log_00000000000000001295_00000000000000001326'
I0420 18:32:32.016613   335 external/com_github_brpc_braft/src/braft/log.cpp:114] Created new segment `/usr/share/typesense/data/state/log/log_inprogress_00000000000000001327' with fd=46
I20230420 18:32:37.014884   262 raft_server.cpp:551] Term: 10, last_index index: 1341, committed_index: 1340, known_applied_index: 1340, applying_index: 0, queued_writes: 101, pending_queue_size: 0, local_sequence: 1766181
I0420 18:32:44.592146   334 external/com_github_brpc_braft/src/braft/log.cpp:536] close a full segment. Current first_index: 1327 last_index: 1354 raft_sync_segments: 0 will_sync: 1 path: /usr/share/typesense/data/state/log/log_00000000000000001327_00000000000000001354
I0420 18:32:44.592230   334 external/com_github_brpc_braft/src/braft/log.cpp:550] Renamed `/usr/share/typesense/data/state/log/log_inprogress_00000000000000001327' to `/usr/share/typesense/data/state/log/log_00000000000000001327_00000000000000001354'
I0420 18:32:44.593456   334 external/com_github_brpc_braft/src/braft/log.cpp:114] Created new segment `/usr/share/typesense/data/state/log/log_inprogress_00000000000000001355' with fd=33
I20230420 18:32:47.030795   262 raft_server.cpp:551] Term: 10, last_index index: 1356, committed_index: 1356, known_applied_index: 1356, applying_index: 0, queued_writes: 121, pending_queue_size: 0, local_sequence: 1775332
I20230420 18:32:57.042730   262 raft_server.cpp:551] Term: 10, last_index index: 1379, committed_index: 1377, known_applied_index: 1376, applying_index: 1377, queued_writes: 122, pending_queue_size: 0, local_sequence: 1787819
I0420 18:33:01.907192   334 external/com_github_brpc_braft/src/braft/log.cpp:536] close a full segment. Current first_index: 1355 last_index: 1380 raft_sync_segments: 0 will_sync: 1 path: /usr/share/typesense/data/state/log/log_00000000000000001355_00000000000000001380
I0420 18:33:01.907278   334 external/com_github_brpc_braft/src/braft/log.cpp:550] Renamed `/usr/share/typesense/data/state/log/log_inprogress_00000000000000001355' to `/usr/share/typesense/data/state/log/log_00000000000000001355_00000000000000001380'
I0420 18:33:01.908586   334 external/com_github_brpc_braft/src/braft/log.cpp:114] Created new segment `/usr/share/typesense/data/state/log/log_inprogress_00000000000000001381' with fd=32
I20230420 18:33:07.104025   262 raft_server.cpp:551] Term: 10, last_index index: 1394, committed_index: 1394, known_applied_index: 1394, applying_index: 0, queued_writes: 141, pending_queue_size: 0, local_sequence: 1796801
I0420 18:33:14.167294   335 external/com_github_brpc_braft/src/braft/log.cpp:536] close a full segment. Current first_index: 1381 last_index: 1405 raft_sync_segments: 0 will_sync: 1 path: /usr/share/typesense/data/state/log/log_00000000000000001381_00000000000000001405
I0420 18:33:14.167418   335 external/com_github_brpc_braft/src/braft/log.cpp:550] Renamed `/usr/share/typesense/data/state/log/log_inprogress_00000000000000001381' to `/usr/share/typesense/data/state/log/log_00000000000000001381_00000000000000001405'
I0420 18:33:14.169168   335 external/com_github_brpc_braft/src/braft/log.cpp:114] Created new segment `/usr/share/typesense/data/state/log/log_inprogress_00000000000000001406' with fd=53
I20230420 18:33:17.116726   262 raft_server.cpp:551] Term: 10, last_index index: 1414, committed_index: 1414, known_applied_index: 1414, applying_index: 0, queued_writes: 150, pending_queue_size: 0, local_sequence: 1806039
I0420 18:33:25.597174   332 external/com_github_brpc_braft/src/braft/log.cpp:536] close a full segment. Current first_index: 1406 last_index: 1437 raft_sync_segments: 0 will_sync: 1 path: /usr/share/typesense/data/state/log/log_00000000000000001406_00000000000000001437
I0420 18:33:25.597271   332 external/com_github_brpc_braft/src/braft/log.cpp:550] Renamed `/usr/share/typesense/data/state/log/log_inprogress_00000000000000001406' to `/usr/share/typesense/data/state/log/log_00000000000000001406_00000000000000001437'
I0420 18:33:25.599777   332 external/com_github_brpc_braft/src/braft/log.cpp:114] Created new segment `/usr/share/typesense/data/state/log/log_inprogress_00000000000000001438' with fd=40
I20230420 18:33:27.125538   262 raft_server.cpp:551] Term: 10, last_index index: 1441, committed_index: 1441, known_applied_index: 1441, applying_index: 0, queued_writes: 166, pending_queue_size: 0, local_sequence: 1816994
I20230420 18:33:32.485888   263 batched_indexer.cpp:284] Running GC for aborted requests, req map size: 22
I20230420 18:33:37.198104   262 raft_server.cpp:551] Term: 10, last_index index: 1458, committed_index: 1457, known_applied_index: 1456, applying_index: 1457, queued_writes: 171, pending_queue_size: 0, local_sequence: 1827621
I0420 18:33:37.271946   335 external/com_github_brpc_braft/src/braft/log.cpp:536] close a full segment. Current first_index: 1438 last_index: 1459 raft_sync_segments: 0 will_sync: 1 path: /usr/share/typesense/data/state/log/log_00000000000000001438_00000000000000001459
I0420 18:33:37.272021   335 external/com_github_brpc_braft/src/braft/log.cpp:550] Renamed `/usr/share/typesense/data/state/log/log_inprogress_00000000000000001438' to `/usr/share/typesense/data/state/log/log_00000000000000001438_00000000000000001459'
I0420 18:33:37.273421   335 external/com_github_brpc_braft/src/braft/log.cpp:114] Created new segment `/usr/share/typesense/data/state/log/log_inprogress_00000000000000001460' with fd=56
I20230420 18:33:47.213721   262 raft_server.cpp:551] Term: 10, last_index index: 1477, committed_index: 1477, known_applied_index: 1477, applying_index: 0, queued_writes: 186, pending_queue_size: 0, local_sequence: 1835280
I0420 18:33:50.018077   331 external/com_github_brpc_braft/src/braft/log.cpp:536] close a full segment. Current first_index: 1460 last_index: 1486 raft_sync_segments: 0 will_sync: 1 path: /usr/share/typesense/data/state/log/log_00000000000000001460_00000000000000001486
I0420 18:33:50.018164   331 external/com_github_brpc_braft/src/braft/log.cpp:550] Renamed `/usr/share/typesense/data/state/log/log_inprogress_00000000000000001460' to `/usr/share/typesense/data/state/log/log_00000000000000001460_00000000000000001486'
I0420 18:33:50.019704   331 external/com_github_brpc_braft/src/braft/log.cpp:114] Created new segment `/usr/share/typesense/data/state/log/log_inprogress_00000000000000001487' with fd=51
I20230420 18:33:57.225459   262 raft_server.cpp:551] Term: 10, last_index index: 1506, committed_index: 1505, known_applied_index: 1505, applying_index: 0, queued_writes: 200, pending_queue_size: 0, local_sequence: 1844423

Adrian Kager

04/20/2023, 7:24 PM

Here is another example. This is the full log this time

Copy code

I20230420 19:13:53.121011     1 typesense_server_utils.cpp:325] Starting Typesense 0.25.0.rc18
I20230420 19:13:53.121088     1 typesense_server_utils.cpp:328] Typesense is using jemalloc.
I20230420 19:13:53.130818     1 typesense_server_utils.cpp:377] Thread pool size: 64
I20230420 19:13:53.141335     1 store.h:64] Initializing DB by opening state dir: /usr/share/typesense/data/db
I20230420 19:13:53.201871     1 store.h:64] Initializing DB by opening state dir: /usr/share/typesense/data/meta
I20230420 19:13:53.231447     1 ratelimit_manager.cpp:546] Loaded 0 rate limit rules.
I20230420 19:13:53.231493     1 ratelimit_manager.cpp:547] Loaded 0 rate limit bans.
I20230420 19:13:53.231572     1 typesense_server_utils.cpp:479] Starting API service...
I20230420 19:13:53.231750   263 batched_indexer.cpp:124] Starting batch indexer with 64 threads.
I0420 19:13:53.232262     1 src/http_server.cpp:178] Typesense has started listening on port 8108
F0420 19:13:53.232325   262 external/com_github_brpc_brpc/src/butil/at_exit.cc:46] Check failed: false. Tried to RegisterCallback without an AtExitManager
F0420 19:13:53.232346   262 external/com_github_brpc_brpc/src/butil/at_exit.cc:46] Check failed: false. Tried to RegisterCallback without an AtExitManager
I0420 19:13:53.244718   262 external/com_github_brpc_brpc/src/brpc/server.cpp:1107] Server[braft::RaftStatImpl+braft::FileServiceImpl+braft::RaftServiceImpl+braft::CliServiceImpl] is serving on port=8107.
I0420 19:13:53.263092   262 external/com_github_brpc_brpc/src/brpc/server.cpp:1110] Check out <http://typesense-1:8107> in web browser.
I20230420 19:13:53.265166   262 raft_server.cpp:67] Nodes configuration: 192.168.130.16:8107:8108,192.168.158.166:8107:8108,192.168.149.207:8107:8108
I0420 19:13:53.266084   262 external/com_github_brpc_braft/src/braft/log.cpp:690] Use murmurhash32 as the checksum type of appending entries
I0420 19:13:53.273462   262 external/com_github_brpc_braft/src/braft/log.cpp:1172] log load_meta /usr/share/typesense/data/state/log/log_meta first_log_index: 906 time: 7340
I0420 19:13:53.273522   262 external/com_github_brpc_braft/src/braft/log.cpp:1014] restore closed segment, path: /usr/share/typesense/data/state/log first_index: 1078 last_index: 1116
I0420 19:13:53.273541   262 external/com_github_brpc_braft/src/braft/log.cpp:1014] restore closed segment, path: /usr/share/typesense/data/state/log first_index: 1381 last_index: 1405
I0420 19:13:53.273547   262 external/com_github_brpc_braft/src/braft/log.cpp:1014] restore closed segment, path: /usr/share/typesense/data/state/log first_index: 1327 last_index: 1354
I0420 19:13:53.273554   262 external/com_github_brpc_braft/src/braft/log.cpp:1014] restore closed segment, path: /usr/share/typesense/data/state/log first_index: 1574 last_index: 1621
I0420 19:13:53.273559   262 external/com_github_brpc_braft/src/braft/log.cpp:1014] restore closed segment, path: /usr/share/typesense/data/state/log first_index: 1867 last_index: 1918
I0420 19:13:53.273564   262 external/com_github_brpc_braft/src/braft/log.cpp:1014] restore closed segment, path: /usr/share/typesense/data/state/log first_index: 1295 last_index: 1326
I0420 19:13:53.273568   262 external/com_github_brpc_braft/src/braft/log.cpp:1014] restore closed segment, path: /usr/share/typesense/data/state/log first_index: 1672 last_index: 1715
I0420 19:13:53.273575   262 external/com_github_brpc_braft/src/braft/log.cpp:1014] restore closed segment, path: /usr/share/typesense/data/state/log first_index: 1460 last_index: 1486
I0420 19:13:53.273580   262 external/com_github_brpc_braft/src/braft/log.cpp:1014] restore closed segment, path: /usr/share/typesense/data/state/log first_index: 1235 last_index: 1264
I0420 19:13:53.273586   262 external/com_github_brpc_braft/src/braft/log.cpp:1014] restore closed segment, path: /usr/share/typesense/data/state/log first_index: 895 last_index: 922
I0420 19:13:53.273591   262 external/com_github_brpc_braft/src/braft/log.cpp:1014] restore closed segment, path: /usr/share/typesense/data/state/log first_index: 1716 last_index: 1764
I0420 19:13:53.273597   262 external/com_github_brpc_braft/src/braft/log.cpp:1014] restore closed segment, path: /usr/share/typesense/data/state/log first_index: 1521 last_index: 1573
I0420 19:13:53.273603   262 external/com_github_brpc_braft/src/braft/log.cpp:1014] restore closed segment, path: /usr/share/typesense/data/state/log first_index: 999 last_index: 1032
I0420 19:13:53.273609   262 external/com_github_brpc_braft/src/braft/log.cpp:1014] restore closed segment, path: /usr/share/typesense/data/state/log first_index: 1622 last_index: 1671
I0420 19:13:53.273615   262 external/com_github_brpc_braft/src/braft/log.cpp:1014] restore closed segment, path: /usr/share/typesense/data/state/log first_index: 1117 last_index: 1148
I0420 19:13:53.273621   262 external/com_github_brpc_braft/src/braft/log.cpp:1014] restore closed segment, path: /usr/share/typesense/data/state/log first_index: 1178 last_index: 1205
I0420 19:13:53.273626   262 external/com_github_brpc_braft/src/braft/log.cpp:1014] restore closed segment, path: /usr/share/typesense/data/state/log first_index: 1033 last_index: 1042
I0420 19:13:53.273631   262 external/com_github_brpc_braft/src/braft/log.cpp:1014] restore closed segment, path: /usr/share/typesense/data/state/log first_index: 1438 last_index: 1459
I0420 19:13:53.273643   262 external/com_github_brpc_braft/src/braft/log.cpp:1014] restore closed segment, path: /usr/share/typesense/data/state/log first_index: 1355 last_index: 1380
I0420 19:13:53.273661   262 external/com_github_brpc_braft/src/braft/log.cpp:1014] restore closed segment, path: /usr/share/typesense/data/state/log first_index: 1815 last_index: 1866
I0420 19:13:53.273668   262 external/com_github_brpc_braft/src/braft/log.cpp:1014] restore closed segment, path: /usr/share/typesense/data/state/log first_index: 1406 last_index: 1437
I0420 19:13:53.273675   262 external/com_github_brpc_braft/src/braft/log.cpp:1014] restore closed segment, path: /usr/share/typesense/data/state/log first_index: 1206 last_index: 1234
I0420 19:13:53.273680   262 external/com_github_brpc_braft/src/braft/log.cpp:1014] restore closed segment, path: /usr/share/typesense/data/state/log first_index: 963 last_index: 998

... cut some nearly identical entries here
I20230420 19:13:53.278442   263 batched_indexer.cpp:129] BatchedIndexer skip_index: -9999
... cut some nearly identical entries here
I0420 19:13:54.539883   262 external/com_github_brpc_braft/src/braft/log.cpp:1100] load closed segment, path: /usr/share/typesense/data/state/log first_index: 1765 last_index: 1814
I0420 19:13:54.615623   262 external/com_github_brpc_braft/src/braft/log.cpp:1100] load closed segment, path: /usr/share/typesense/data/state/log first_index: 1815 last_index: 1866
I0420 19:13:54.736990   262 external/com_github_brpc_braft/src/braft/log.cpp:1100] load closed segment, path: /usr/share/typesense/data/state/log first_index: 1867 last_index: 1918
I0420 19:13:54.816458   262 external/com_github_brpc_braft/src/braft/log.cpp:1112] load open segment, path: /usr/share/typesense/data/state/log first_index: 1919
I20230420 19:13:54.825958   274 raft_server.cpp:516] on_snapshot_load
I20230420 19:13:54.883880   274 store.h:299] rm /usr/share/typesense/data/db success
I20230420 19:13:54.886502   274 store.h:309] copy snapshot /usr/share/typesense/data/state/snapshot/snapshot_00000000000000000908/db_snapshot to /usr/share/typesense/data/db success
I20230420 19:13:54.886556   274 store.h:64] Initializing DB by opening state dir: /usr/share/typesense/data/db
I20230420 19:13:54.919273   274 store.h:323] DB open success!
I20230420 19:13:54.919308   274 raft_server.cpp:495] Loading collections from disk...
I0420 19:13:54.919357   274 src/collection_manager.cpp:172] CollectionManager::load()
I20230420 19:13:54.921833   274 auth_manager.cpp:34] Indexing 2 API key(s) found on disk.
I0420 19:13:54.921908   274 src/collection_manager.cpp:192] Loading upto 32 collections in parallel, 1000 documents at a time.
I0420 19:13:54.921934   274 src/collection_manager.cpp:201] Found 1 collection(s) on disk.
I0420 19:13:54.923417   354 src/collection_manager.cpp:122] Found collection testing_collection with 4 memory shards.
I0420 19:13:54.923631   354 src/collection_manager.cpp:1240] Loading collection testing_collection
I0420 19:14:26.298273   354 src/collection_manager.cpp:1346] Loaded 475136 documents from testing_collection so far.
I0420 19:14:30.273257   354 src/collection_manager.cpp:1357] Indexed 526476/526476 documents into collection testing_collection
I0420 19:14:30.273313   354 src/collection_manager.cpp:240] Loaded 1 collection(s) so far
I0420 19:14:30.285028   274 src/collection_manager.cpp:290] Loaded 1 collection(s).
I0420 19:14:30.286936   274 src/collection_manager.cpp:294] Initializing batched indexer from snapshot state...
I20230420 19:14:30.288376   274 batched_indexer.cpp:446] Restored 0 in-flight requests from snapshot.
I20230420 19:14:30.288421   274 raft_server.cpp:502] Finished loading collections from disk.
I20230420 19:14:30.288453   274 raft_server.h:278] Configuration of this group is 192.168.130.16:8107:8108,192.168.158.166:8107:8108,192.168.149.207:8107:8108
I0420 19:14:30.288577   274 external/com_github_brpc_braft/src/braft/snapshot_executor.cpp:264] node default_group:192.168.158.166:8107:8108 snapshot_load_done, last_included_index: 908 last_included_term: 8 peers: "192.168.130.16:8107:8108" peers: "192.168.158.166:8107:8108" peers: "192.168.149.207:8107:8108"
I0420 19:14:30.290172   262 external/com_github_brpc_braft/src/braft/raft_meta.cpp:521] Loaded single stable meta, path /usr/share/typesense/data/state/meta term 10 votedfor 192.168.130.16:8107:8108 time: 1351
I0420 19:14:30.290237   262 external/com_github_brpc_braft/src/braft/node.cpp:608] node default_group:192.168.158.166:8107:8108 init, term: 10 last_log_id: (index=1923,term=10) conf: 192.168.130.16:8107:8108,192.168.158.166:8107:8108,192.168.149.207:8107:8108 old_conf:
I20230420 19:14:30.290299   262 raft_server.cpp:133] Node last_index: 1923
I20230420 19:14:30.290328   262 typesense_server_utils.cpp:274] Typesense peering service is running on 192.168.158.166:8107
I20230420 19:14:30.290339   262 typesense_server_utils.cpp:275] Snapshot interval configured as: 3600s
I20230420 19:14:30.290347   262 typesense_server_utils.cpp:276] Snapshot max byte count configured as: 4194304
W0420 19:14:30.290363   262 external/com_github_brpc_brpc/src/brpc/controller.cpp:1487] SIGINT was installed with 1
I20230420 19:14:30.291049   262 raft_server.cpp:551] Term: 10, last_index index: 1923, committed_index: 0, known_applied_index: 908, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1691505
W20230420 19:14:30.291072   262 raft_server.cpp:578] Multi-node with no leader: refusing to reset peers.
E20230420 19:14:30.503494   277 raft_server.cpp:625] 1015 lagging entries > healthy read lag of 1000
E20230420 19:14:30.503545   277 raft_server.cpp:637] 1015 lagging entries > healthy write lag of 500
I20230420 19:14:30.503556   277 raft_server.h:283] Node starts following { leader_id=192.168.130.16:8107:8108, term=10, status=Follower receives message from new leader with the same term.}
I20230420 19:14:33.372130   277 raft_server.h:278] Configuration of this group is 192.168.130.16:8107:8108,192.168.158.166:8107:8108,192.168.149.207:8107:8108
I20230420 19:14:36.301144   277 raft_server.h:278] Configuration of this group is 192.168.130.16:8107:8108,192.168.158.166:8107:8108,192.168.149.207:8107:8108

Kishore Nallan

04/21/2023, 3:13 AM

That's not a crash. When you ingest too fast there is a backpressure mechanism to reject writes until the writes catch up. This prevents searches from being slowed down.

Kishore Nallan

04/21/2023, 3:14 AM

See this:

Copy code

E20230420 19:14:30.503494   277 raft_server.cpp:625] 1015 lagging entries > healthy read lag of 1000
E20230420 19:14:30.503545   277 raft_server.cpp:637] 1015 lagging entries > healthy write lag of 500

You can increase the

--healthy-read-lag

and

--healthy-write-lag

configuration values or just add a small delay between 2 batches of imports.

Adrian Kager

04/21/2023, 2:28 PM

thanks for the info!

Adrian Kager

04/21/2023, 2:28 PM

turns out the pods were also running out of memory, which was causing full crashes and restarts as well. Working now that I increased memory limits

👍 1

Open in Slack

Previous Next