Typesense Issues in Kubernetes Environment
TLDR siva.sunkara experienced Typesense issues in Kubernetes and shared logs. Kishore Nallan recommended stand-alone nodes and shared a Github issue thread. Sergio suggested fine-tuning Kubernetes deployment and linked to a working setup.
Mar 06, 2023 (7 months ago)
siva.sunkara
06:51 AMI20230306 06:33:29.445390 193 typesense_server_utils.cpp:333] raft_server.stop()
I20230306 06:33:29.445400 193 server.cpp:1100] Server[braft::RaftStatImpl+braft::FileServiceImpl+braft::RaftServiceImpl+braft::CliServiceImpl] is going to quit
I20230306 06:33:29.445614 193 typesense_server_utils.cpp:336] raft_server.join()
I20230306 06:33:29.445672 193 typesense_server_utils.cpp:339] Typesense peering service has quit.
I20230306 06:33:29.446014 193 typesense_server_utils.cpp:461] Shutting down batch indexer...
I20230306 06:33:29.446023 193 typesense_server_utils.cpp:464] Waiting for batch indexing thread to be done...
I20230306 06:33:29.835271 194 batched_indexer.cpp:282] Notifying batch indexer threads about shutdown...
I20230306 06:33:29.835619 194 batched_indexer.cpp:290] Batched indexer threadpool shutdown...
I20230306 06:33:29.836972 193 typesense_server_utils.cpp:467] Shutting down server_thread_pool
I20230306 06:33:29.838626 193 typesense_server_utils.cpp:471] Shutting down app_thread_pool.
I20230306 06:33:29.840443 1 typesense_server_utils.cpp:485] Typesense API service has quit.
I20230306 06:33:29.840528 1 typesense_server_utils.cpp:489] Deleting batch indexer
I20230306 06:33:29.840588 1 typesense_server_utils.cpp:493] CURL clean up
I20230306 06:33:29.840595 1 typesense_server_utils.cpp:497] Deleting server
I20230306 06:33:29.840634 1 typesense_server_utils.cpp:501] CollectionManager dispose, this might take some time...
I20230306 06:33:29.890949 1 typesense_server_utils.cpp:505] Bye.
can anyone help me to fix this ?
Kishore Nallan
07:34 AMsiva.sunkara
09:06 AMsiva.sunkara
09:08 AMKishore Nallan
09:09 AMKishore Nallan
09:11 AMMy recommendation is to run Typesense on stand-alone nodes. Don't use Kubernetes if you want to do clustering.
Mar 07, 2023 (7 months ago)
siva.sunkara
06:10 AMi’ve created test infra for typesense. pods are created but the while running the typesense it was throughing this error. please have a look.
I20230307 06:07:11.883797 1 typesense_server_utils.cpp:345] Starting Typesense 0.23.0
I20230307 06:07:11.883841 1 typesense_server_utils.cpp:348] Typesense is using jemalloc.
I20230307 06:07:11.884037 1 typesense_server_utils.cpp:397] Thread pool size: 16
I20230307 06:07:11.886054 1 store.h:61] Initializing DB by opening state dir: /usr/share/typesense/data/db
I20230307 06:07:12.001137 1 store.h:61] Initializing DB by opening state dir: /usr/share/typesense/data/meta
I20230307 06:07:12.017688 1 typesense_server_utils.cpp:478] Starting API service...
I20230307 06:07:12.078938 1 http_server.cpp:174] Typesense has started listening on port 8108
I20230307 06:07:12.079020 96 batched_indexer.cpp:120] Starting batch indexer with 16 threads.
I20230307 06:07:12.080508 96 batched_indexer.cpp:126] BatchedIndexer skip_index: -9999
I20230307 06:07:12.084529 95 server.cpp:1045] Server[braft::RaftStatImpl+braft::FileServiceImpl+braft::RaftServiceImpl+braft::CliServiceImpl] is serving on port=8107.
I20230307 06:07:12.084543 95 server.cpp:1048] Check out http://typesense-test-0:8107 in web browser.
E20230307 06:07:12.099704 95 raft_server.cpp:166] Unable to resolve host: typesense-0.ts-test.typesense.svc.cluster.local
E20230307 06:07:12.111562 95 raft_server.cpp:166] Unable to resolve host: typesense-1.ts-test.typesense.svc.cluster.local
E20230307 06:07:12.188621 95 raft_server.cpp:166] Unable to resolve host: typesense-2.ts-test.typesense.svc.cluster.local
E20230307 06:07:12.188663 95 configuration.cpp:43] Fail to parse typesense-0.ts-test.typesense.svc.cluster.local:8107:8108
E20230307 06:07:12.188717 95 raft_server.cpp:51] Failed to parse nodes configuration:
typesense-0.ts-test.typesense.svc.cluster.local:8107:8108,typesense-1.ts-test.typesense.svc.cluster.local:8107:8108,typesense-2.ts-test.typesense.svc.cluster.local:8107:8108
-- will retry shortly...E20230307 06:07:42.210857 95 raft_server.cpp:166] Unable to resolve host: typesense-0.ts-test.typesense.svc.cluster.local
E20230307 06:07:42.222651 95 raft_server.cpp:166] Unable to resolve host: typesense-1.ts-test.typesense.svc.cluster.local
E20230307 06:07:42.232034 95 raft_server.cpp:166] Unable to resolve host: typesense-2.ts-test.typesense.svc.cluster.local
E20230307 06:07:42.232065 95 configuration.cpp:43] Fail to parse typesense-0.ts-test.typesense.svc.cluster.local:8107:8108
E20230307 06:07:42.232070 95 raft_server.cpp:51] Failed to parse nodes configuration:
typesense-0.ts-test.typesense.svc.cluster.local:8107:8108,typesense-1.ts-test.typesense.svc.cluster.local:8107:8108,typesense-2.ts-test.typesense.svc.cluster.local:8107:8108
-- will retry shortly...siva.sunkara
06:12 AMdata:
nodes: “typesense-0.ts-test.typesense.svc.cluster.local:8107:8108,typesense-1.ts-test.typesense.svc.cluster.local:8107:8108,typesense-2.ts-test.typesense.svc.cluster.local:8107:8108”
siva.sunkara
06:25 AMhttps://raw.githubusercontent.com/typesense/typesense-kubernetes/master/base/install.yaml
Kishore Nallan
08:44 AMMar 09, 2023 (7 months ago)
Sergio
04:34 PMSergio
04:35 PMSergio
04:35 PMhttps://github.com/Alasano/typesense-k8s-node-resolver/blob/main/typesense.yml
Typesense
Indexed 2779 threads (79% resolved)
Similar Threads
Issues deploying Typesense to AWS EKS
Pavan had issues when deploying Typesense to AWS EKS. Kishore Nallan suggested deployment to plain EC2 instances and provided the API key information. Eventually, Pavan resolved the issue with Helm.
Typesense Node Stuck in Segfault Loop After Stress Test
Adrian encountered a segfault loop when stress testing a Typesense cluster. Kishore Nallan recommended trying a newer RC build and suggested potential issues with hostname resolution.
"Resolving Startup Issue with Typesense Single Replica Set in Local `K8s` Setup"
Vishal encountered issues starting a Typesense single replica set due to directory permission issues. Kishore Nallan suggested checking if the volume was accessible for writes. Vishal resolved the problem by adjusting permissions on the host.
Resolving Server Stoppage Issues in Typesense Multi VM Cluster
gaurav faced issues with the Typesense server in a multi VM cluster, including automatic stoppage and errors. Kishore Nallan identified the lack of a quorum and suggested using three nodes. When the issue persisted, they advised running Typesense via `nohup` or `systemd` to prevent session closure from stopping the process.
Addressing High CPU Usage in Typesense
Robert reported high CPU usage on Typesense, even after halting all incoming searches. Kishore Nallan suggested logging heavy queries and increasing thread count. The issue was resolved after Robert found and truncated unusually large documents in the database.