#community-help

Troubleshooting Typesense Issues in Kubectl v1.26

TLDR Arnob and Rubai face issues with Typesense erasing data. Jason and Kishore Nallan suggest HA setup and persistent volume usage. Sai recommends the Typesense Kubernetes operator but Arnob experiences new errors.

Powered by Struct AI

1

Jul 30, 2023 (4 months ago)
Arnob
Photo of md5-b1aea561b79f980e6a366345edafe139
Arnob
08:26 AM
I had a problem with typesense. I use typesense in kubectl v1.26. After some days automatically all the collection delete also data. is there any solution of this problem
Jul 31, 2023 (4 months ago)
Rubai
Photo of md5-89fb99de3bf7e23767aaf9108a5636ad
Rubai
01:35 PM
I faced the same problem Jason
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
04:08 PM
This is likely because all Typesense nodes were reset by k8s
04:09
Jason
04:09 PM
You want to make sure that at most only one node in a 3-node Typesense cluster is rotated at a given time.
Rubai
Photo of md5-89fb99de3bf7e23767aaf9108a5636ad
Rubai
07:53 PM
can you please elaborate 3-node typesense cluster means , we have only one docker file for typesense deployment in AWS
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
08:31 PM
In a prod environment, you want to setup Typesense in HA mode with multiple nodes: https://typesense.org/docs/guide/high-availability.html
Aug 01, 2023 (4 months ago)
Rubai
Photo of md5-89fb99de3bf7e23767aaf9108a5636ad
Rubai
07:14 AM
why it's reset after some time ? also when it's reset ?
09:09
Rubai
09:09 AM
will this be solved if we use a docker volume instead of a local folder for typesense-data Jason Kishore Nallan
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
09:12 AM
You should be using a persistent volume for the data directory, aren't you not?
Rubai
Photo of md5-89fb99de3bf7e23767aaf9108a5636ad
Rubai
09:12 AM
no we are not currently
09:13
Rubai
09:13 AM
can we use a database instead of a folder?
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
09:13 AM
As long as you use a persistent disk volume like ebs you will be fine.
09:14
Kishore Nallan
09:14 AM
No
Arnob
Photo of md5-b1aea561b79f980e6a366345edafe139
Arnob
10:32 AM
if you save data at pvc, it will not save the data. After some day those are also deleted.
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
10:33 AM
That should not happen. Many people use this setup so it's probably an issue with your configuration.
Arnob
Photo of md5-b1aea561b79f980e6a366345edafe139
Arnob
12:15 PM
ok got it. if you share any reference configuration it will be helpful for me. I don't find any configuration in k8s.
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
12:19 PM
You can try this: https://github.com/sai3010/Typesense-Kubernetes-Operator

cc Sai who maintains it can help if you run into any issues.
Sai
Photo of md5-6c1f32ee79d1166e54e3f17345b8d814
Sai
12:22 PM
Arnob Please try that operator, it should solve your issue with managing typesense , but you will have to create a storage class manually and assign it with retain policy else the data will be lost. Happy to help if you need anything.
Arnob
Photo of md5-b1aea561b79f980e6a366345edafe139
Arnob
12:28 PM
Kishore Nallan and Sai thank you for share this resource. I will try and give u the feedback.

1

Aug 02, 2023 (4 months ago)
Arnob
Photo of md5-b1aea561b79f980e6a366345edafe139
Arnob
09:13 AM
Sai hello, after apply the typesense Kubernetes operator it giving some error.
The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/kopf/_cogs/aiokits/aiotasks.py", line 108, in guard
    await coro
  File "/usr/local/lib/python3.9/site-packages/kopf/_core/engines/peering.py", line 176, in keepalive
    await touch(
  File "/usr/local/lib/python3.9/site-packages/kopf/_core/engines/peering.py", line 221, in touch
    rsp = await patching.patch_obj(
  File "/usr/local/lib/python3.9/site-packages/kopf/_cogs/clients/patching.py", line 47, in patch_obj
    patched_body = await api.patch(
  File "/usr/local/lib/python3.9/site-packages/kopf/_cogs/clients/api.py", line 155, in patch
    response = await request(
  File "/usr/local/lib/python3.9/site-packages/kopf/_cogs/clients/auth.py", line 45, in wrapper
    return await fn(*args, **kwargs, context=context)
  File "/usr/local/lib/python3.9/site-packages/kopf/_cogs/clients/api.py", line 85, in request
    await errors.check_response(response)  # but do not parse it!
  File "/usr/local/lib/python3.9/site-packages/kopf/_cogs/clients/errors.py", line 150, in check_response
    raise cls(payload, status=response.status) from e
kopf._cogs.clients.errors.APIForbiddenError: (' "default" is forbidden: User "system:serviceaccount:typesense-rnd-v4:typesense-operator-sa" cannot patch resource "clusterkopfpeerings" in API group "" at the cluster scope', {'kind': 'Status', 'apiVersion': 'v1', 'metadata': {}, 'status': 'Failure', 'message': ' "default" is forbidden: User "system:serviceaccount:typesense-rnd-v4:typesense-operator-sa" cannot patch resource "clusterkopfpeerings" in API group "" at the cluster scope', 'reason': 'Forbidden', 'details': {'name': 'default', 'group': '', 'kind': 'clusterkopfpeerings'}, 'code': 403})
09:20
Arnob
09:20 AM
After apply the operator-config.yaml it delete deployment.apps/typesense-operator also cant port-forward get typesense service.
Image 1 for After apply the `operator-config.yaml`  it delete `deployment.apps/typesense-operator`  also cant port-forward get typesense service.
Sai
Photo of md5-6c1f32ee79d1166e54e3f17345b8d814
Sai
09:21 AM
Can you share your operator config file ? Please DM me
Arnob
Photo of md5-b1aea561b79f980e6a366345edafe139
Arnob
09:22 AM
ok

Typesense

Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI

Indexed 3005 threads (79% resolved)

Join Our Community

Similar Threads

Issues deploying Typesense to AWS EKS

Pavan had issues when deploying Typesense to AWS EKS. Kishore Nallan suggested deployment to plain EC2 instances and provided the API key information. Eventually, Pavan resolved the issue with Helm.

1

11
17mo
Solved

Issues with Typesense and k8s Snapshot Restoration

Arnob experienced data loss and errors with Typesense in k8s. Kishore Nallan explained corruption could be from premature pod termination. To resolve, Kishore Nallan suggested deleting the data directory on the malfunctioning pod for automatic restoration from the leader.

9
4mo
Solved

Typesense Issues in Kubernetes Environment

siva.sunkara experienced Typesense issues in Kubernetes and shared logs. Kishore Nallan recommended stand-alone nodes and shared a Github issue thread. Sergio suggested fine-tuning Kubernetes deployment and linked to a working setup.

13
9mo

Typesense Error in Unit Testing for Python

Mehdi was encountering errors while running unit tests for Typesense functionalities in Python. Through discussion with Kishore Nallan, they recognized resource constraints might be causing the issue during JSONL file import. As suggested by Kishore Nallan, adding a short sleep after the import resolved the problem.

17
23mo
Solved

"Resolving Startup Issue with Typesense Single Replica Set in Local `K8s` Setup"

Vishal encountered issues starting a Typesense single replica set due to directory permission issues. Kishore Nallan suggested checking if the volume was accessible for writes. Vishal resolved the problem by adjusting permissions on the host.

2

10
22mo
Solved