Resolving Server Stoppage Issues in Typesense Multi VM Cluster

hi i was trying to run typesense in multi VM cluster and it worked as expected for a while but then it automatically stops.
i am using ssm to access VM and what i have seen typesense on docker works perfectly but typesense service such as typesense-server crashes after a while. any ideas?

also now i am again trying to set up cluster one VM is throwing 503 error.

Please share all relevant logs

Sure,
Its a 2 VM configuration. VM1 throws 503, {'ok': False} in health and VM2 200 {'ok': True}

To start server i had started using below commands

 VM1: sudo typesense-server \
>   --data-dir /home/charlie/cluster/typesense \
>   --api-key=xyz \
>   --api-address 0.0.0.0 \
>   --api-port 8108 \
>   --peering-address 10.212.22.59 \
>   --peering-port 8107 \
>   --log-dir=/home/charlie/cluster/logs \
>   --nodes=/home/charlie/cluster/nodes
Log directory is configured as: /home/charlie/cluster/logs```

VM2: sudo typesense-server \
> --data-dir /home/charlie/cluster/typesense \
> --api-key=xyz \
> --api-address 0.0.0.0 \
> --api-port 8108 \
> --peering-address 10.212.22.189 \
> --peering-port 8107 \
> --log-dir=/home/charlie/cluster/logs
Log directory is configured as: /home/charlie/cluster/logs
E20221117 09:35:39.196200 2809 raft_server.cpp:589] Node not ready yet (known_applied_index is 0).
E20221117 09:35:39.196218 2817 raft_server.h:62] Peer refresh failed, error: Doing another configuration change`` When i debug the API i am getting state 4 for VM1 and state 1` for VM2.
Also attached logs

problem is for VM1 i dont know what’s the issue its throwing 503 error. Altough i can see from debug it is working as expected.

Typesense uses Raft which requires 3 nodes for quorum.

Okay cool, altough it worked on 2 VM also.

It will work initially but no guarantee and can lead to split brain. Without an odd number, you can't get quorum.

Okay NP even i would set up 3 VM cluster

Does this happen on 2 node or 3 node configuration?

2 VM, let me check for 3 VM

Yeah, please do. And if it fails, check the logs at the time it does. Mostly with 3 nodes it just works. Unless all host IPs change at the same time (do rolling rotations if you want to change the host for some reason).

So i am facing the issue again in 3 VM. I dont think its issue of VM anymore as all 3 VM stops after some time.

I am using aws EC2 instances using SSM.
From logs i can see my typesense server starts from 13.56 till 14.16 after that it automatically stops. You can see the logs but there isnt any which can point what’s the issue.

I20221117 14:14:45.019073 21928 raft_server.cpp:534] Term: 32, last_index index: 161, committed_index: 161, known_applied_index: 161, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 759502
I20221117 14:14:45.019165 21940 raft_server.h:60] Peer refresh succeeded!
I20221117 14:14:55.020509 21928 raft_server.cpp:534] Term: 32, last_index index: 161, committed_index: 161, known_applied_index: 161, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 759502
I20221117 14:14:55.020674 21942 raft_server.h:60] Peer refresh succeeded!
I20221117 14:15:05.022079 21928 raft_server.cpp:534] Term: 32, last_index index: 161, committed_index: 161, known_applied_index: 161, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 759502
I20221117 14:15:05.022176 21951 raft_server.h:60] Peer refresh succeeded!
I20221117 14:15:15.023499 21928 raft_server.cpp:534] Term: 32, last_index index: 161, committed_index: 161, known_applied_index: 161, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 759502
I20221117 14:15:15.023589 21940 raft_server.h:60] Peer refresh succeeded!
I20221117 14:15:25.025126 21928 raft_server.cpp:534] Term: 32, last_index index: 161, committed_index: 161, known_applied_index: 161, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 759502
I20221117 14:15:25.025230 21942 raft_server.h:60] Peer refresh succeeded!
I20221117 14:15:33.873915 21929 batched_indexer.cpp:250] Running GC for aborted requests, req map size: 0
I20221117 14:15:35.026558 21928 raft_server.cpp:534] Term: 32, last_index index: 161, committed_index: 161, known_applied_index: 161, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 759502
I20221117 14:15:35.026647 21951 raft_server.h:60] Peer refresh succeeded!
I20221117 14:15:45.028023 21928 raft_server.cpp:534] Term: 32, last_index index: 161, committed_index: 161, known_applied_index: 161, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 759502
I20221117 14:15:45.028113 21940 raft_server.h:60] Peer refresh succeeded!
I20221117 14:15:55.029551 21928 raft_server.cpp:534] Term: 32, last_index index: 161, committed_index: 161, known_applied_index: 161, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 759502
I20221117 14:15:55.029634 21942 raft_server.h:60] Peer refresh succeeded!
I20221117 14:16:05.030953 21928 raft_server.cpp:534] Term: 32, last_index index: 161, committed_index: 161, known_applied_index: 161, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 759502
I20221117 14:16:05.031044 21951 raft_server.h:60] Peer refresh succeeded!
I20221117 14:16:15.032476 21928 raft_server.cpp:534] Term: 32, last_index index: 161, committed_index: 161, known_applied_index: 161, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 759502
I20221117 14:16:15.032645 21940 raft_server.h:60] Peer refresh succeeded!
I20221117 14:16:25.034013 21928 raft_server.cpp:534] Term: 32, last_index index: 161, committed_index: 161, known_applied_index: 161, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 759502
I20221117 14:16:25.034106 21942 raft_server.h:60] Peer refresh succeeded!
I20221117 14:16:34.881245 21929 batched_indexer.cpp:250] Running GC for aborted requests, req map size: 0
I20221117 14:16:35.035606 21928 raft_server.cpp:534] Term: 32, last_index index: 161, committed_index: 161, known_applied_index: 161, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 759502
I20221117 14:16:35.035696 21951 raft_server.h:60] Peer refresh succeeded!
I20221117 14:16:45.037305 21928 raft_server.cpp:534] Term: 32, last_index index: 161, committed_index: 161, known_applied_index: 161, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 759502
I20221117 14:16:45.037391 21940 raft_server.h:60] Peer refresh succeeded!

Now when i check in /var/log/messages i can see some logs pertaining to same time as typesense server.

Nov 17 13:56:14 ds-pro-search-0202 systemd-logind: New session c70 of user root.
Nov 17 14:16:53 ds-pro-search-0202 systemd-logind: Removed session c70.

Do you know anything on this?

When you mean by "it stops" does the process die?

yes

And the logs posted above were the last few lines? Nothing after that?

yes

And all 3 nodes shut down at the same time?

no its based when the VM service SSM session i started

How are you running Typesense? If you are not running the server via nohup or screen, it will shut down when your SSH session expires.

okay sure thanks. Will check that out.
Will update on this. However thanks for this will test out.

RPM does not work?

yes

What error are you getting? And what CentOS version? We've other users who use RPM and CentOS.

Sorry u misunderstood, RPM works in installing typesense, however as we were discussing previously i am having difficulity in running the service after my terminal/ session closes

Just for refrence

CentOS Linux release 7.9.2009 

:thinking_face: RPM also uses systemd. That certainly works.

In any case, the quickest fix for you now is to change your command structure to run via nohup this way:

nohup ./typesense-server <arguments> &

This will 100% ensure that process is not killed when session closes.

hi Kishore Nallan, it worked. For best use i can also recommend systemd