#community-help

Typesense Issues in Kubernetes Environment

TLDR siva.sunkara experienced Typesense issues in Kubernetes and shared logs. Kishore Nallan recommended stand-alone nodes and shared a Github issue thread. Sergio suggested fine-tuning Kubernetes deployment and linked to a working setup.

Powered by Struct AI
Mar 06, 2023 (7 months ago)
siva.sunkara
Photo of md5-a2785b9d22ba23f3627d4bd877e95e7c
siva.sunkara
06:51 AM
Hi team, we have implemented typesense in kubernetives. suddenly our typesense went down. it was showing like.

I20230306 06:33:29.445390 193 typesense_server_utils.cpp:333] raft_server.stop()
I20230306 06:33:29.445400 193 server.cpp:1100] Server[braft::RaftStatImpl+braft::FileServiceImpl+braft::RaftServiceImpl+braft::CliServiceImpl] is going to quit
I20230306 06:33:29.445614 193 typesense_server_utils.cpp:336] raft_server.join()
I20230306 06:33:29.445672 193 typesense_server_utils.cpp:339] Typesense peering service has quit.
I20230306 06:33:29.446014 193 typesense_server_utils.cpp:461] Shutting down batch indexer...
I20230306 06:33:29.446023 193 typesense_server_utils.cpp:464] Waiting for batch indexing thread to be done...
I20230306 06:33:29.835271 194 batched_indexer.cpp:282] Notifying batch indexer threads about shutdown...
I20230306 06:33:29.835619 194 batched_indexer.cpp:290] Batched indexer threadpool shutdown...
I20230306 06:33:29.836972 193 typesense_server_utils.cpp:467] Shutting down server_thread_pool
I20230306 06:33:29.838626 193 typesense_server_utils.cpp:471] Shutting down app_thread_pool.
I20230306 06:33:29.840443 1 typesense_server_utils.cpp:485] Typesense API service has quit.
I20230306 06:33:29.840528 1 typesense_server_utils.cpp:489] Deleting batch indexer
I20230306 06:33:29.840588 1 typesense_server_utils.cpp:493] CURL clean up
I20230306 06:33:29.840595 1 typesense_server_utils.cpp:497] Deleting server
I20230306 06:33:29.840634 1 typesense_server_utils.cpp:501] CollectionManager dispose, this might take some time...
I20230306 06:33:29.890949 1 typesense_server_utils.cpp:505] Bye.


can anyone help me to fix this ?
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
07:34 AM
Hi Siva, there is not enough information in that log snippet to show what happened before that.
siva.sunkara
Photo of md5-a2785b9d22ba23f3627d4bd877e95e7c
siva.sunkara
09:06 AM
Hi Kishore,
09:08
siva.sunkara
09:08 AM
haven’t done anything it was automatically went down that’s way i’m understanding what happend. i’m sharing you complete error in the file. please have a look.
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
09:09 AM
Cluster or a single pod?
09:11
Kishore Nallan
09:11 AM
You are probably running into the issue discussed here: https://github.com/typesense/typesense/issues/465#issuecomment-1173536082

My recommendation is to run Typesense on stand-alone nodes. Don't use Kubernetes if you want to do clustering.
Mar 07, 2023 (7 months ago)
siva.sunkara
Photo of md5-a2785b9d22ba23f3627d4bd877e95e7c
siva.sunkara
06:10 AM
Hi Kishore,

i’ve created test infra for typesense. pods are created but the while running the typesense it was throughing this error. please have a look.

I20230307 06:07:11.883797 1 typesense_server_utils.cpp:345] Starting Typesense 0.23.0
I20230307 06:07:11.883841 1 typesense_server_utils.cpp:348] Typesense is using jemalloc.
I20230307 06:07:11.884037 1 typesense_server_utils.cpp:397] Thread pool size: 16
I20230307 06:07:11.886054 1 store.h:61] Initializing DB by opening state dir: /usr/share/typesense/data/db
I20230307 06:07:12.001137 1 store.h:61] Initializing DB by opening state dir: /usr/share/typesense/data/meta
I20230307 06:07:12.017688 1 typesense_server_utils.cpp:478] Starting API service...
I20230307 06:07:12.078938 1 http_server.cpp:174] Typesense has started listening on port 8108
I20230307 06:07:12.079020 96 batched_indexer.cpp:120] Starting batch indexer with 16 threads.
I20230307 06:07:12.080508 96 batched_indexer.cpp:126] BatchedIndexer skip_index: -9999
I20230307 06:07:12.084529 95 server.cpp:1045] Server[braft::RaftStatImpl+braft::FileServiceImpl+braft::RaftServiceImpl+braft::CliServiceImpl] is serving on port=8107.
I20230307 06:07:12.084543 95 server.cpp:1048] Check out http://typesense-test-0:8107 in web browser.
E20230307 06:07:12.099704 95 raft_server.cpp:166] Unable to resolve host: typesense-0.ts-test.typesense.svc.cluster.local
E20230307 06:07:12.111562 95 raft_server.cpp:166] Unable to resolve host: typesense-1.ts-test.typesense.svc.cluster.local
E20230307 06:07:12.188621 95 raft_server.cpp:166] Unable to resolve host: typesense-2.ts-test.typesense.svc.cluster.local
E20230307 06:07:12.188663 95 configuration.cpp:43] Fail to parse typesense-0.ts-test.typesense.svc.cluster.local:8107:8108
E20230307 06:07:12.188717 95 raft_server.cpp:51] Failed to parse nodes configuration: typesense-0.ts-test.typesense.svc.cluster.local:8107:8108,typesense-1.ts-test.typesense.svc.cluster.local:8107:8108,typesense-2.ts-test.typesense.svc.cluster.local:8107:8108 -- will retry shortly...
E20230307 06:07:42.210857 95 raft_server.cpp:166] Unable to resolve host: typesense-0.ts-test.typesense.svc.cluster.local
E20230307 06:07:42.222651 95 raft_server.cpp:166] Unable to resolve host: typesense-1.ts-test.typesense.svc.cluster.local
E20230307 06:07:42.232034 95 raft_server.cpp:166] Unable to resolve host: typesense-2.ts-test.typesense.svc.cluster.local
E20230307 06:07:42.232065 95 configuration.cpp:43] Fail to parse typesense-0.ts-test.typesense.svc.cluster.local:8107:8108
E20230307 06:07:42.232070 95 raft_server.cpp:51] Failed to parse nodes configuration: typesense-0.ts-test.typesense.svc.cluster.local:8107:8108,typesense-1.ts-test.typesense.svc.cluster.local:8107:8108,typesense-2.ts-test.typesense.svc.cluster.local:8107:8108 -- will retry shortly...
06:12
siva.sunkara
06:12 AM
And i’m using this nodelist in cm

data:
nodes: “typesense-0.ts-test.typesense.svc.cluster.local:8107:8108,typesense-1.ts-test.typesense.svc.cluster.local:8107:8108,typesense-2.ts-test.typesense.svc.cluster.local:8107:8108”
06:25
siva.sunkara
06:25 AM
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
08:44 AM
Sorry we are not well versed with Kubernetes so I'm not able to help here.
Mar 09, 2023 (7 months ago)
Sergio
Photo of md5-19856b8e92142bdd0747d7a3706736c8
Sergio
04:34 PM
siva.sunkara kubernetes deployment needs some fine tuning
04:35
Sergio
04:35 PM
The Github issue thread that Kishore Nallan shared above is a great source for running it smoothly in kuberentes.
04:35
Sergio
04:35 PM
I have not open source our setup yet, but it goes similar to this one
https://github.com/Alasano/typesense-k8s-node-resolver/blob/main/typesense.yml