#community-help

Typesense Instance Issues and Solutions on K8s

TLDR Richard encountered issues with a typesense instance, including lag, unresponsiveness, and shut downs. Jason provided potential causes and solution, including adjusting configuration settings and clearing data directory. This resolved Richard's issues.

14mo

Join the chat

Oct 18, 2022 (14 months ago)

Photo of md5-9b63610ff6be18110bf1729efc8251b3

Richard

11:06 PM

Hey all: deployed a typesense instance (no HA/peers) on k8s and I may have done too many writes at once…? I’ve stopped all write requests but it seems like the instance is stuck (restarting the process doesn’t help). The logs just seem to complain about lags in reads/writes.

Is there a way to clear the log and reset the node?

Photo of md5-8813087cccc512313602b6d9f9ece19f

Jason

11:13 PM

A read / write lag usually indicates CPU saturation

11:14

Jason

11:14 PM

You can configure the thresholds when Typesense will return a 503 using these parameters: --healthy-read-lag and --healthy-write-lag

https://typesense.org/docs/0.23.1/api/server-configuration.html

Richard

11:14 PM

interesting: it also shuts down after a few mins (and hence it never finishes catching up?).

Jason

11:14 PM

If you want to just clear all the data and reset the node, you want to stop the process, and then clear the contents of the data dir

11:15

Jason

11:15 PM

Could you check journalctl, may be RAM is also running out and OS OOM-Reaper is killing the process?

Richard

11:17 PM

Oh you know what: it’s probably because I had a probe on k8s that killed it after a while, doh.

11:17

Richard

11:17 PM

since it was returning 503s to the probe

Jason

11:17 PM

Ah makes sense

Richard

11:19 PM

So if I want to throttle writes, but not reads, i would set a lower healthy write lag but a higher read lag?

Jason

11:19 PM

Correct

Richard

11:20 PM

amazing, thanks for the speedy responses 👍 appreciate it!

Typesense

Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI

Indexed 3015 threads (79% resolved)

Join Our Community

Similar Threads

Stability Issues with Self-Hosted Typesense Server on AWS ECS

Ryan reports stability issues with a self-hosted typesense server, whilst Jason suggests various potential factors to consider, including configuration, CPU allocation, modifying the healthcheck, and testing on EC2 instances. However, no definitive solution is reached.

4mo

Addressing Typesense Server Issues and Optimization Needs

Robert had an issue with a 'stuck' typesense server. Jason and Kishore Nallan gave advice on handling writes, configuration for high search volumes, and running multiple typesense instances. They also recommended monitoring CPU usage and updating the server version for bug fixes.

14mo

Explanation of Typesense's server parameters `--healthy-read-lag` and `--healthy-write-lag`.

Eric asked about the significance of specific server configuration parameters in Typesense. Jason provided a detailed explanation of how these parameters work during data writes and reads.

25mo

Discussion About Typesense Nodes Not Synchronizing Correctly

Erick experienced an issue where documents weren't updated properly in a Typesense instance running on 3 nodes. Upon requesting debug logs and configs, Jason identified that these nodes weren't part of the same cluster. They couldn't resolve the nodes' failure to connect issue and recommended a fresh installation.

23mo

Understanding Typesense Throttling Parameters

Nelson queried about Typesense parameters `--healthy-read-lag` and `--healthy-write-lag`. Jason explained that these control widespread write operations and ensure read prioritization. They further elaborated on interpreting logs and network request thresholds.

27mo