#community-help

Server Startup Troubleshooting

TLDR Robert was unsure why their server was taking a long time to start up. Jason explained that the server was still indexing data into memory.

Powered by Struct AI

1

Oct 19, 2022 (14 months ago)
Robert
Photo of md5-6384d24e1825271b2c37ad8afa24a899
Robert
06:47 PM
A server with 92M docs is taking a long time to start up and I'm not sure if there is something broken or if it's actually doing something:
W20221019 14:44:49.868903 120213 node.cpp:811] [default_group:127.0.0.1:21807:21808 ] Refusing concurrent configuration changing
E20221019 14:44:49.868935 120213 raft_server.cpp:589] Node not ready yet (known_applied_index is 0).
E20221019 14:44:49.868968 120291 raft_server.h:62] Peer refresh failed, error: Doing another configuration change
E20221019 14:44:58.869524 120213 raft_server.cpp:589] Node not ready yet (known_applied_index is 0).
I20221019 14:44:59.869647 120213 raft_server.cpp:534] Term: 6, last_index index: 976479, committed_index: 976479, known_applied_index: 0, applying_index: 271928, queued_writes: 243525, pending_queue_size: 0, local_sequence: 8653509
W20221019 14:44:59.869679 120213 node.cpp:811] [default_group:127.0.0.1:21807:21808 ] Refusing concurrent configuration changing
E20221019 14:44:59.869736 120388 raft_server.h:62] Peer refresh failed, error: Doing another configuration change
E20221019 14:45:07.870239 120213 raft_server.cpp:589] Node not ready yet (known_applied_index is 0).
I20221019 14:45:09.870419 120213 raft_server.cpp:534] Term: 6, last_index index: 976479, committed_index: 976479, known_applied_index: 0, applying_index: 275075, queued_writes: 246346, pending_queue_size: 0, local_sequence: 8754883
W20221019 14:45:09.870451 120213 node.cpp:811] [default_group:127.0.0.1:21807:21808 ] Refusing concurrent configuration changing
E20221019 14:45:09.870555 120325 raft_server.h:62] Peer refresh failed, error: Doing another configuration change
E20221019 14:45:16.870968 120213 raft_server.cpp:589] Node not ready yet (known_applied_index is 0).
I20221019 14:45:19.871181 120213 raft_server.cpp:534] Term: 6, last_index index: 976479, committed_index: 976479, known_applied_index: 0, applying_index: 282883, queued_writes: 253661, pending_queue_size: 0, local_sequence: 8910756
W20221019 14:45:19.871206 120213 node.cpp:811] [default_group:127.0.0.1:21807:21808 ] Refusing concurrent configuration changing
E20221019 14:45:19.871248 120316 raft_server.h:62] Peer refresh failed, error: Doing another configuration change
E20221019 14:45:25.871608 120213 raft_server.cpp:589] Node not ready yet (known_applied_index is 0).
I20221019 14:45:29.871889 120213 raft_server.cpp:534] Term: 6, last_index index: 976479, committed_index: 976479, known_applied_index: 0, applying_index: 286219, queued_writes: 256907, pending_queue_size: 0, local_sequence: 8941348
W20221019 14:45:29.871923 120213 node.cpp:811] [default_group:127.0.0.1:21807:21808 ] Refusing concurrent configuration changing
E20221019 14:45:29.871968 120343 raft_server.h:62] Peer refresh failed, error: Doing another configuration change
E20221019 14:45:34.872177 120213 raft_server.cpp:589] Node not ready yet (known_applied_index is 0).
I20221019 14:45:35.256573 120214 batched_indexer.cpp:250] Running GC for aborted requests, req map size: 250049
I20221019 14:45:39.872522 120213 raft_server.cpp:534] Term: 6, last_index index: 976479, committed_index: 976479, known_applied_index: 0, applying_index: 286940, queued_writes: 257606, pending_queue_size: 0, local_sequence: 8948464
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
06:48 PM
Notice how these numbers are changing between the log lines:

applying_index: 286219, queued_writes: 256907
applying_index: 286940, queued_writes: 257606

So it’s still indexing the on-disk data to memory on start up
Robert
Photo of md5-6384d24e1825271b2c37ad8afa24a899
Robert
06:49 PM
Ahh. Does it say there what number it's trying to get to? Is that the last_index? So once apply_index is 976479 then it's done loading into memory?
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
06:50 PM
Correct, that’s last_index
Robert
Photo of md5-6384d24e1825271b2c37ad8afa24a899
Robert
06:50 PM
Awesome, thanks!

1