Typesense Issues in Kubernetes Environment
TLDR siva.sunkara experienced Typesense issues in Kubernetes and shared logs. Kishore Nallan recommended stand-alone nodes and shared a Github issue thread. Sergio suggested fine-tuning Kubernetes deployment and linked to a working setup.
Powered by Struct AI
13
7mo
Mar 06, 2023 (7 months ago)
siva.sunkara
siva.sunkara
06:51 AMHi team, we have implemented typesense in kubernetives. suddenly our typesense went down. it was showing like.
I20230306 06:33:29.445390 193 typesense_server_utils.cpp:333] raft_server.stop()
I20230306 06:33:29.445400 193 server.cpp:1100] Server[braft::RaftStatImpl+braft::FileServiceImpl+braft::RaftServiceImpl+braft::CliServiceImpl] is going to quit
I20230306 06:33:29.445614 193 typesense_server_utils.cpp:336] raft_server.join()
I20230306 06:33:29.445672 193 typesense_server_utils.cpp:339] Typesense peering service has quit.
I20230306 06:33:29.446014 193 typesense_server_utils.cpp:461] Shutting down batch indexer...
I20230306 06:33:29.446023 193 typesense_server_utils.cpp:464] Waiting for batch indexing thread to be done...
I20230306 06:33:29.835271 194 batched_indexer.cpp:282] Notifying batch indexer threads about shutdown...
I20230306 06:33:29.835619 194 batched_indexer.cpp:290] Batched indexer threadpool shutdown...
I20230306 06:33:29.836972 193 typesense_server_utils.cpp:467] Shutting down server_thread_pool
I20230306 06:33:29.838626 193 typesense_server_utils.cpp:471] Shutting down app_thread_pool.
I20230306 06:33:29.840443 1 typesense_server_utils.cpp:485] Typesense API service has quit.
I20230306 06:33:29.840528 1 typesense_server_utils.cpp:489] Deleting batch indexer
I20230306 06:33:29.840588 1 typesense_server_utils.cpp:493] CURL clean up
I20230306 06:33:29.840595 1 typesense_server_utils.cpp:497] Deleting server
I20230306 06:33:29.840634 1 typesense_server_utils.cpp:501] CollectionManager dispose, this might take some time...
I20230306 06:33:29.890949 1 typesense_server_utils.cpp:505] Bye.
can anyone help me to fix this ?
I20230306 06:33:29.445390 193 typesense_server_utils.cpp:333] raft_server.stop()
I20230306 06:33:29.445400 193 server.cpp:1100] Server[braft::RaftStatImpl+braft::FileServiceImpl+braft::RaftServiceImpl+braft::CliServiceImpl] is going to quit
I20230306 06:33:29.445614 193 typesense_server_utils.cpp:336] raft_server.join()
I20230306 06:33:29.445672 193 typesense_server_utils.cpp:339] Typesense peering service has quit.
I20230306 06:33:29.446014 193 typesense_server_utils.cpp:461] Shutting down batch indexer...
I20230306 06:33:29.446023 193 typesense_server_utils.cpp:464] Waiting for batch indexing thread to be done...
I20230306 06:33:29.835271 194 batched_indexer.cpp:282] Notifying batch indexer threads about shutdown...
I20230306 06:33:29.835619 194 batched_indexer.cpp:290] Batched indexer threadpool shutdown...
I20230306 06:33:29.836972 193 typesense_server_utils.cpp:467] Shutting down server_thread_pool
I20230306 06:33:29.838626 193 typesense_server_utils.cpp:471] Shutting down app_thread_pool.
I20230306 06:33:29.840443 1 typesense_server_utils.cpp:485] Typesense API service has quit.
I20230306 06:33:29.840528 1 typesense_server_utils.cpp:489] Deleting batch indexer
I20230306 06:33:29.840588 1 typesense_server_utils.cpp:493] CURL clean up
I20230306 06:33:29.840595 1 typesense_server_utils.cpp:497] Deleting server
I20230306 06:33:29.840634 1 typesense_server_utils.cpp:501] CollectionManager dispose, this might take some time...
I20230306 06:33:29.890949 1 typesense_server_utils.cpp:505] Bye.
can anyone help me to fix this ?
Kishore Nallan
Kishore Nallan
07:34 AMHi Siva, there is not enough information in that log snippet to show what happened before that.
siva.sunkara
siva.sunkara
09:06 AMHi Kishore,
09:08
siva.sunkara
09:08 AMhaven’t done anything it was automatically went down that’s way i’m understanding what happend. i’m sharing you complete error in the file. please have a look.
Kishore Nallan
Kishore Nallan
09:09 AMCluster or a single pod?
09:11
Kishore Nallan
09:11 AMYou are probably running into the issue discussed here: https://github.com/typesense/typesense/issues/465#issuecomment-1173536082
My recommendation is to run Typesense on stand-alone nodes. Don't use Kubernetes if you want to do clustering.
My recommendation is to run Typesense on stand-alone nodes. Don't use Kubernetes if you want to do clustering.
Mar 07, 2023 (7 months ago)
siva.sunkara
siva.sunkara
06:10 AMHi Kishore,
i’ve created test infra for typesense. pods are created but the while running the typesense it was throughing this error. please have a look.
I20230307 06:07:11.883797 1 typesense_server_utils.cpp:345] Starting Typesense 0.23.0
I20230307 06:07:11.883841 1 typesense_server_utils.cpp:348] Typesense is using jemalloc.
I20230307 06:07:11.884037 1 typesense_server_utils.cpp:397] Thread pool size: 16
I20230307 06:07:11.886054 1 store.h:61] Initializing DB by opening state dir: /usr/share/typesense/data/db
I20230307 06:07:12.001137 1 store.h:61] Initializing DB by opening state dir: /usr/share/typesense/data/meta
I20230307 06:07:12.017688 1 typesense_server_utils.cpp:478] Starting API service...
I20230307 06:07:12.078938 1 http_server.cpp:174] Typesense has started listening on port 8108
I20230307 06:07:12.079020 96 batched_indexer.cpp:120] Starting batch indexer with 16 threads.
I20230307 06:07:12.080508 96 batched_indexer.cpp:126] BatchedIndexer skip_index: -9999
I20230307 06:07:12.084529 95 server.cpp:1045] Server[braft::RaftStatImpl+braft::FileServiceImpl+braft::RaftServiceImpl+braft::CliServiceImpl] is serving on port=8107.
I20230307 06:07:12.084543 95 server.cpp:1048] Check out http://typesense-test-0:8107 in web browser.
E20230307 06:07:12.099704 95 raft_server.cpp:166] Unable to resolve host: typesense-0.ts-test.typesense.svc.cluster.local
E20230307 06:07:12.111562 95 raft_server.cpp:166] Unable to resolve host: typesense-1.ts-test.typesense.svc.cluster.local
E20230307 06:07:12.188621 95 raft_server.cpp:166] Unable to resolve host: typesense-2.ts-test.typesense.svc.cluster.local
E20230307 06:07:12.188663 95 configuration.cpp:43] Fail to parse typesense-0.ts-test.typesense.svc.cluster.local:8107:8108
E20230307 06:07:12.188717 95 raft_server.cpp:51] Failed to parse nodes configuration:
E20230307 06:07:42.210857 95 raft_server.cpp:166] Unable to resolve host: typesense-0.ts-test.typesense.svc.cluster.local
E20230307 06:07:42.222651 95 raft_server.cpp:166] Unable to resolve host: typesense-1.ts-test.typesense.svc.cluster.local
E20230307 06:07:42.232034 95 raft_server.cpp:166] Unable to resolve host: typesense-2.ts-test.typesense.svc.cluster.local
E20230307 06:07:42.232065 95 configuration.cpp:43] Fail to parse typesense-0.ts-test.typesense.svc.cluster.local:8107:8108
E20230307 06:07:42.232070 95 raft_server.cpp:51] Failed to parse nodes configuration:
i’ve created test infra for typesense. pods are created but the while running the typesense it was throughing this error. please have a look.
I20230307 06:07:11.883797 1 typesense_server_utils.cpp:345] Starting Typesense 0.23.0
I20230307 06:07:11.883841 1 typesense_server_utils.cpp:348] Typesense is using jemalloc.
I20230307 06:07:11.884037 1 typesense_server_utils.cpp:397] Thread pool size: 16
I20230307 06:07:11.886054 1 store.h:61] Initializing DB by opening state dir: /usr/share/typesense/data/db
I20230307 06:07:12.001137 1 store.h:61] Initializing DB by opening state dir: /usr/share/typesense/data/meta
I20230307 06:07:12.017688 1 typesense_server_utils.cpp:478] Starting API service...
I20230307 06:07:12.078938 1 http_server.cpp:174] Typesense has started listening on port 8108
I20230307 06:07:12.079020 96 batched_indexer.cpp:120] Starting batch indexer with 16 threads.
I20230307 06:07:12.080508 96 batched_indexer.cpp:126] BatchedIndexer skip_index: -9999
I20230307 06:07:12.084529 95 server.cpp:1045] Server[braft::RaftStatImpl+braft::FileServiceImpl+braft::RaftServiceImpl+braft::CliServiceImpl] is serving on port=8107.
I20230307 06:07:12.084543 95 server.cpp:1048] Check out http://typesense-test-0:8107 in web browser.
E20230307 06:07:12.099704 95 raft_server.cpp:166] Unable to resolve host: typesense-0.ts-test.typesense.svc.cluster.local
E20230307 06:07:12.111562 95 raft_server.cpp:166] Unable to resolve host: typesense-1.ts-test.typesense.svc.cluster.local
E20230307 06:07:12.188621 95 raft_server.cpp:166] Unable to resolve host: typesense-2.ts-test.typesense.svc.cluster.local
E20230307 06:07:12.188663 95 configuration.cpp:43] Fail to parse typesense-0.ts-test.typesense.svc.cluster.local:8107:8108
E20230307 06:07:12.188717 95 raft_server.cpp:51] Failed to parse nodes configuration:
typesense-0.ts-test.typesense.svc.cluster.local:8107:8108,typesense-1.ts-test.typesense.svc.cluster.local:8107:8108,typesense-2.ts-test.typesense.svc.cluster.local:8107:8108 -- will retry shortly...E20230307 06:07:42.210857 95 raft_server.cpp:166] Unable to resolve host: typesense-0.ts-test.typesense.svc.cluster.local
E20230307 06:07:42.222651 95 raft_server.cpp:166] Unable to resolve host: typesense-1.ts-test.typesense.svc.cluster.local
E20230307 06:07:42.232034 95 raft_server.cpp:166] Unable to resolve host: typesense-2.ts-test.typesense.svc.cluster.local
E20230307 06:07:42.232065 95 configuration.cpp:43] Fail to parse typesense-0.ts-test.typesense.svc.cluster.local:8107:8108
E20230307 06:07:42.232070 95 raft_server.cpp:51] Failed to parse nodes configuration:
typesense-0.ts-test.typesense.svc.cluster.local:8107:8108,typesense-1.ts-test.typesense.svc.cluster.local:8107:8108,typesense-2.ts-test.typesense.svc.cluster.local:8107:8108 -- will retry shortly...06:12
siva.sunkara
06:12 AMAnd i’m using this nodelist in cm
data:
nodes: “typesense-0.ts-test.typesense.svc.cluster.local:8107:8108,typesense-1.ts-test.typesense.svc.cluster.local:8107:8108,typesense-2.ts-test.typesense.svc.cluster.local:8107:8108”
data:
nodes: “typesense-0.ts-test.typesense.svc.cluster.local:8107:8108,typesense-1.ts-test.typesense.svc.cluster.local:8107:8108,typesense-2.ts-test.typesense.svc.cluster.local:8107:8108”
06:25
siva.sunkara
06:25 AMi’m following this code on kubernetives.
https://raw.githubusercontent.com/typesense/typesense-kubernetes/master/base/install.yaml
https://raw.githubusercontent.com/typesense/typesense-kubernetes/master/base/install.yaml
Kishore Nallan
Kishore Nallan
08:44 AMSorry we are not well versed with Kubernetes so I'm not able to help here.
Mar 09, 2023 (7 months ago)
Sergio
Sergio
04:34 PM siva.sunkara kubernetes deployment needs some fine tuning
04:35
Sergio
04:35 PMThe Github issue thread that Kishore Nallan shared above is a great source for running it smoothly in kuberentes.
04:35
Sergio
04:35 PMI have not open source our setup yet, but it goes similar to this one
https://github.com/Alasano/typesense-k8s-node-resolver/blob/main/typesense.yml
https://github.com/Alasano/typesense-k8s-node-resolver/blob/main/typesense.yml
Typesense
Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI
Indexed 2779 threads (79% resolved)
Similar Threads
Issues deploying Typesense to AWS EKS
Pavan had issues when deploying Typesense to AWS EKS. Kishore Nallan suggested deployment to plain EC2 instances and provided the API key information. Eventually, Pavan resolved the issue with Helm.
11
15mo
Solved
Typesense Node Stuck in Segfault Loop After Stress Test
Adrian encountered a segfault loop when stress testing a Typesense cluster. Kishore Nallan recommended trying a newer RC build and suggested potential issues with hostname resolution.
31
5mo
"Resolving Startup Issue with Typesense Single Replica Set in Local `K8s` Setup"
Vishal encountered issues starting a Typesense single replica set due to directory permission issues. Kishore Nallan suggested checking if the volume was accessible for writes. Vishal resolved the problem by adjusting permissions on the host.
10
20mo
Solved
Resolving Server Stoppage Issues in Typesense Multi VM Cluster
gaurav faced issues with the Typesense server in a multi VM cluster, including automatic stoppage and errors. Kishore Nallan identified the lack of a quorum and suggested using three nodes. When the issue persisted, they advised running Typesense via `nohup` or `systemd` to prevent session closure from stopping the process.
31
11mo
Addressing High CPU Usage in Typesense
Robert reported high CPU usage on Typesense, even after halting all incoming searches. Kishore Nallan suggested logging heavy queries and increasing thread count. The issue was resolved after Robert found and truncated unusually large documents in the database.
35
12mo
Solved