Server Startup Troubleshooting

A server with 92M docs is taking a long time to start up and I'm not sure if there is something broken or if it's actually doing something:

W20221019 14:44:49.868903 120213 node.cpp:811] [default_group:127.0.0.1:21807:21808 ] Refusing concurrent configuration changing
E20221019 14:44:49.868935 120213 raft_server.cpp:589] Node not ready yet (known_applied_index is 0).
E20221019 14:44:49.868968 120291 raft_server.h:62] Peer refresh failed, error: Doing another configuration change
E20221019 14:44:58.869524 120213 raft_server.cpp:589] Node not ready yet (known_applied_index is 0).
I20221019 14:44:59.869647 120213 raft_server.cpp:534] Term: 6, last_index index: 976479, committed_index: 976479, known_applied_index: 0, applying_index: 271928, queued_writes: 243525, pending_queue_size: 0, local_sequence: 8653509
W20221019 14:44:59.869679 120213 node.cpp:811] [default_group:127.0.0.1:21807:21808 ] Refusing concurrent configuration changing
E20221019 14:44:59.869736 120388 raft_server.h:62] Peer refresh failed, error: Doing another configuration change
E20221019 14:45:07.870239 120213 raft_server.cpp:589] Node not ready yet (known_applied_index is 0).
I20221019 14:45:09.870419 120213 raft_server.cpp:534] Term: 6, last_index index: 976479, committed_index: 976479, known_applied_index: 0, applying_index: 275075, queued_writes: 246346, pending_queue_size: 0, local_sequence: 8754883
W20221019 14:45:09.870451 120213 node.cpp:811] [default_group:127.0.0.1:21807:21808 ] Refusing concurrent configuration changing
E20221019 14:45:09.870555 120325 raft_server.h:62] Peer refresh failed, error: Doing another configuration change
E20221019 14:45:16.870968 120213 raft_server.cpp:589] Node not ready yet (known_applied_index is 0).
I20221019 14:45:19.871181 120213 raft_server.cpp:534] Term: 6, last_index index: 976479, committed_index: 976479, known_applied_index: 0, applying_index: 282883, queued_writes: 253661, pending_queue_size: 0, local_sequence: 8910756
W20221019 14:45:19.871206 120213 node.cpp:811] [default_group:127.0.0.1:21807:21808 ] Refusing concurrent configuration changing
E20221019 14:45:19.871248 120316 raft_server.h:62] Peer refresh failed, error: Doing another configuration change
E20221019 14:45:25.871608 120213 raft_server.cpp:589] Node not ready yet (known_applied_index is 0).
I20221019 14:45:29.871889 120213 raft_server.cpp:534] Term: 6, last_index index: 976479, committed_index: 976479, known_applied_index: 0, applying_index: 286219, queued_writes: 256907, pending_queue_size: 0, local_sequence: 8941348
W20221019 14:45:29.871923 120213 node.cpp:811] [default_group:127.0.0.1:21807:21808 ] Refusing concurrent configuration changing
E20221019 14:45:29.871968 120343 raft_server.h:62] Peer refresh failed, error: Doing another configuration change
E20221019 14:45:34.872177 120213 raft_server.cpp:589] Node not ready yet (known_applied_index is 0).
I20221019 14:45:35.256573 120214 batched_indexer.cpp:250] Running GC for aborted requests, req map size: 250049
I20221019 14:45:39.872522 120213 raft_server.cpp:534] Term: 6, last_index index: 976479, committed_index: 976479, known_applied_index: 0, applying_index: 286940, queued_writes: 257606, pending_queue_size: 0, local_sequence: 8948464

Notice how these numbers are changing between the log lines:

applying_index: 286219, queued_writes: 256907
applying_index: 286940, queued_writes: 257606

So it’s still indexing the on-disk data to memory on start up

Ahh. Does it say there what number it's trying to get to? Is that the last_index? So once apply_index is 976479 then it's done loading into memory?

Correct, that’s last_index

Awesome, thanks!