Hi everyone, We've self-hosted Typesense (3 node c...
# community-help
a
Hi everyone, We've self-hosted Typesense (3 node cluster). Everything works fine but we've been facing one major issue with storage. Every few weeks / month the storage gets full on one of the nodes and the entire cluster just goes down. We went from 150gb to 500gb and still facing this issue, does anyone know why this happens? We have 32gb of memory and 500gb of storage right now, and one node reached 100% storage and the entire cluster went down. (I'll be increasing storage to 1tb for now)
f
Can you use something akin to ncdu on the VPS to see what's taking up so much space? Unless you're saving a lot in storage and not RAM (setting many fields as unindexed), then 500gb of internal storage should be more than enough for 32GB worth of RAM
a
I dropped the data dir as we're trying to get the node back online. However, on other nodes, its the snapshot that is taking up most of the storage. Also, I think we have many unindexed fields, but the storage space taken should be around 100-150gb
f
Something else other than Typesense may be taking up storage there, could you take a look to verify my assumption ?
a
I removed Typesense data dir and 99% storage is freed up on that cluster, and there is nothing else in that directory. (Its a separate disk that we've attached to the VM, we only use it for Typesense data dir)
Also, why does the entire cluster go down when storage is full on one of the nodes?
f
Did you measure the data dirs size beforehand?
Also, why does the entire cluster go down when storage is full on one of the nodes?
Are processes running on the same machine? That shouldn't happen. Each other node should be available for querying.
a
Unfortunately no, not on the node which was full. However, after removing data dir, it was 99% empty. And on other nodes, whatever storage is used up, is used by Typsense data dir.
Are processes running on the same machine? That shouldn't happen. Each other node should be available for querying.
No, they're all running on different VMs
f
Can you refer to the documentation and see if there are any differences between your implementation and the guide? https://typesense.org/docs/guide/high-availability.html#high-availability-when-self-hosting-typesense
👀 1
a
Yes I think we have same implementation -- I've confirmed this in past with this, https://typesense.org/docs/guide/high-availability.html#verifying-cluster-formation Here is config file, on each node its same except for peering-address, which is different on each node
Copy code
[server]
api-key = xxx
data-dir = /data/typesense
log-dir = /var/log/typesense
api-port = 443
peering-address = 10.0.1.4
; FYI this is 10.0.1.5 and 10.0.1.6 on other 2 nodes
peering-port = 8107
nodes = /etc/typesense/nodes
enable-cors = true
api-address = 0.0.0.0
enable-search-analytics = true
analytics-dir = /data/typesense-analytics-data
analytics-flush-interval = 300
ssl-certificate = /path/to/ssl/fullchain.pem
ssl-certificate-key = /path/to/ssl/privkey.pem
ssl-refresh-interval-seconds = 43200
nodes file, which is same on each node
Copy code
10.0.1.4:8107:443,10.0.1.6:8107:443,10.0.1.5:8107:443
f
CC: @Kishore Nallan
k
If you do high volume writes on your index, you have to do periodic compaction: https://typesense.org/docs/28.0/api/cluster-operations.html#compacting-the-on-disk-database
This API call has to be run on every node on your cluster. It take a few hours for disk to be fully reclaimed and the disk usage could increase slightly during compaction before it goes down.
Alternatively you can decrease the db compaction interval via the
--db-compaction-interval
server side flag. Default is
604,800 (7 days)
👍 1
a
That makes sense, thank you!
Are there any recommendations on how frequently we should run this?
Also, sorry for repeating again but do we know why the entire cluster goes down when storage gets full on one of the nodes?
k
Entire cluster should not go down. I'm not sure why that happens. But I recommend keeping a watch on your disk size and trim it to keep it under 80% usage.
👍 1
a
Got it, I'll confirm that our setup is correct, just in case.
Thank you for your help, Fanis and Kishore, appreciate it! 🙌
🙌 2