Hi, I'm running self-hosted Typesense v30.1 and...
# community-help
g
Hi, I'm running self-hosted Typesense v30.1 and consistently getting Rejecting write: running out of resource type: OUT_OF_MEMORY errors. Server specs: ┌────────────────────────────────┬───────────┐ │ Metric │ Value │ ├────────────────────────────────┼───────────┤ │ Collections │ 2,273 │ ├────────────────────────────────┼───────────┤ │ Documents │ 2,515,905 │ ├────────────────────────────────┼───────────┤ │ Server RAM │ 30 GB │ ├────────────────────────────────┼───────────┤ │ Used RAM │ 23 GB │ ├────────────────────────────────┼───────────┤ │ Swap │ 119 GB │ ├────────────────────────────────┼───────────┤ │ Used Swap │ 13 GB │ ├────────────────────────────────┼───────────┤ │ TS data size on disk │ 33 GB │ ├────────────────────────────────┼───────────┤ │ Max fields per collection │ 973 │ ├────────────────────────────────┼───────────┤ │ Min fields per collection │ 3 │ ├────────────────────────────────┼───────────┤ │ CPU cores │ 16 │ └────────────────────────────────┴───────────┘ This is a 3-node raft cluster. Each node has the same specs. I've tried consolidating all data into a single collection, but write latency jumps from ~3 seconds to ~60 seconds per document. With separate collections (2,273), write performance is acceptable but I hit the OOM error constantly. What's the recommended approach to resolve this? Is there a way to tune memory usage with this many collections, or is 30 GB simply insufficient for ~2,300 collections? Thanks in advance.
a
@Georgi Nachev You'll want to profile your cluster with metrics, so you can know how your disk is behaving. With constant writes and deletes, rocksdb accumulates stale writes into disk. The solution for that is a db compaction + snapshot. But to know wether your case is simply the big amount of data or stale writes pilling up, you need to profile typesense disk usage (exposed in metrics.json and stats.json)