Hi team, we have implemented typesense in kubernetives. suddenly our typesense went down. it was showing like. I20230306 063329.445390 193 typesense_server_utils.cpp:333] raft_server.stop() I20230306 063329.445400 193 server.cpp:1100] Server[braft:RaftStatImpl+braftFileServiceImpl+braftRaftServiceImpl+braft:CliServiceImpl] is going to quit I20230306 063329.445614 193 typesense_server_utils.cpp:336] raft_server.join() I20230306 063329.445672 193 typesense_server_utils.cpp:339] Typesense peering service has quit. I20230306 063329.446014 193 typesense_server_utils.cpp:461] Shutting down batch indexer... I20230306 063329.446023 193 typesense_server_utils.cpp:464] Waiting for batch indexing thread to be done... I20230306 063329.835271 194 batched_indexer.cpp:282] Notifying batch indexer threads about shutdown... I20230306 063329.835619 194 batched_indexer.cpp:290] Batched indexer threadpool shutdown... I20230306 063329.836972 193 typesense_server_utils.cpp:467] Shutting down server_thread_pool I20230306 063329.838626 193 typesense_server_utils.cpp:471] Shutting down app_thread_pool. I20230306 063329.840443 1 typesense_server_utils.cpp:485] Typesense API service has quit. I20230306 063329.840528 1 typesense_server_utils.cpp:489] Deleting batch indexer I20230306 063329.840588 1 typesense_server_utils.cpp:493] CURL clean up I20230306 063329.840595 1 typesense_server_utils.cpp:497] Deleting server I20230306 063329.840634 1 typesense_server_utils.cpp:501] CollectionManager dispose, this might take some time... I20230306 063329.890949 1 typesense_server_utils.cpp:505] Bye. can anyone help me to fix this ?

Hi Siva, there is not enough information in that log snippet to show what happened before that.

Hi Kishore,

Cluster or a single pod?

Hi Kishore, i’ve created test infra for typesense. pods are created but the while running the typesense it was throughing this error. please have a look. I20230307 060711.883797 1 typesense_server_utils.cpp:345] Starting Typesense 0.23.0 I20230307 060711.883841 1 typesense_server_utils.cpp:348] Typesense is using jemalloc. I20230307 060711.884037 1 typesense_server_utils.cpp:397] Thread pool size: 16 I20230307 060711.886054 1 store.h:61] Initializing DB by opening state dir: /usr/share/typesense/data/db I20230307 060712.001137 1 store.h:61] Initializing DB by opening state dir: /usr/share/typesense/data/meta I20230307 060712.017688 1 typesense_server_utils.cpp:478] Starting API service... I20230307 060712.078938 1 http_server.cpp:174] Typesense has started listening on port 8108 I20230307 060712.079020 96 batched_indexer.cpp:120] Starting batch indexer with 16 threads. I20230307 060712.080508 96 batched_indexer.cpp:126] BatchedIndexer skip_index: -9999 I20230307 060712.084529 95 server.cpp:1045] Server[braft:RaftStatImpl+braftFileServiceImpl+braftRaftServiceImpl+braft:CliServiceImpl] is serving on port=8107. I20230307 060712.084543 95 server.cpp:1048] Check out http://typesense-test-0:8107 in web browser. E20230307 060712.099704 95 raft_server.cpp:166] Unable to resolve host: typesense-0.ts-test.typesense.svc.cluster.local E20230307 060712.111562 95 raft_server.cpp:166] Unable to resolve host: typesense-1.ts-test.typesense.svc.cluster.local E20230307 060712.188621 95 raft_server.cpp:166] Unable to resolve host: typesense-2.ts-test.typesense.svc.cluster.local E20230307 060712.188663 95 configuration.cpp:43] Fail to parse typesense-0.ts-test.typesense.svc.cluster.local81078108 E20230307 060712.188717 95 raft_server.cpp:51] Failed to parse nodes configuration:

typesense-0.ts-test.typesense.svc.cluster.local:8107:8108,typesense-1.ts-test.typesense.svc.cluster.local:8107:8108,typesense-2.ts-test.typesense.svc.cluster.local:8107:8108

-- will retry shortly... E20230307 060742.210857 95 raft_server.cpp:166] Unable to resolve host: typesense-0.ts-test.typesense.svc.cluster.local E20230307 060742.222651 95 raft_server.cpp:166] Unable to resolve host: typesense-1.ts-test.typesense.svc.cluster.local E20230307 060742.232034 95 raft_server.cpp:166] Unable to resolve host: typesense-2.ts-test.typesense.svc.cluster.local E20230307 060742.232065 95 configuration.cpp:43] Fail to parse typesense-0.ts-test.typesense.svc.cluster.local81078108 E20230307 060742.232070 95 raft_server.cpp:51] Failed to parse nodes configuration:

typesense-0.ts-test.typesense.svc.cluster.local:8107:8108,typesense-1.ts-test.typesense.svc.cluster.local:8107:8108,typesense-2.ts-test.typesense.svc.cluster.local:8107:8108

-- will retry shortly...

Sorry we are not well versed with Kubernetes so I'm not able to help here.

@siva.sunkara kubernetes deployment needs some fine tuning