#community-help

Disk Space Issue with Typesense Machine

TLDR John reported increasing disk space usage on their Typesense machine due to old indexes. Kishore Nallan suggested triggering a manual compaction and offered a command for it in the recent 0.24 RC build. John will test this solution next week.

10mo

Join the chat

Dec 07, 2022 (10 months ago)

Photo of md5-21545f1facb7836c149bc4c70752bd2b

John

08:21 AM

We’re running low on disk space on our Typesense machine and it seems like it’s because it’s keeping some old indexes or something (.sst files), because the collection hasn’t changed much but the disk usage keeps increasing. Any idea what this could be?

Photo of md5-4e872368b2b2668460205b409e95c2ea

Kishore Nallan

08:22 AM

What version are you using?

John

08:22 AM

0.24.0.rcn34

Kishore Nallan

08:23 AM

Do you have regular updates?

08:24

Kishore Nallan

08:24 AM

And what happens when you restart Typesense? Does the disk size get released?

John

08:27 AM

We create a new collection, repoint the alias, then delete the new collection. This happens once per hour.

No, it doesn’t seem to get released

Kishore Nallan

08:31 AM

Oh I see, the compaction does not happen that often. We might have to expose an API to trigger that more aggressively

John

08:39 AM

How often does compaction happen?

Kishore Nallan

08:45 AM

Rocksdb has various levels and they get merged as each level crosses some predefined thresholds in size. We don't really change those defaults too much. I think there is also a global compaction that runs every 7 days but I need to check that.

08:47

Kishore Nallan

08:47 AM

How many SST files are there in db dir and what's their average size?

John

09:42 AM

~280 files
• ~80 files at 65M each, from Nov 22-Nov 29
• No SST files for a few days, then ~40 at 65M each, from Dec 3-Dec 4.
• Finally about 120 at 20M each, from Dec 5 to now.

Dec 08, 2022 (10 months ago)

Kishore Nallan

08:57 AM

I will see if we can add an operations API to trigger manual compaction of rocksdb. How much data do you write on a full refresh?

08:58

Kishore Nallan

08:58 AM

Btw, if you are creating snapshots explicitly by calling the snapshot API, then those files will be hard linked to the DB SST files and will prevent files from getting purged.

John

09:02 AM

About 500 MB data, and quite a lot of facets so the resulting indices are probably pretty large. We’re not creating snapshots explicitly.

Kishore Nallan

09:03 AM

Only raw data are stored on disk. So disk size will be more or less same as ingested data (some duplication happens due to buffering etc but indices are only in-memory)

09:04

Kishore Nallan

09:04 AM

It's surprising that the storage is growing unbounded this way. We have many customers with similar data size / usage pattern without any issues.

John

09:15 AM

It’s not really a big deal for us as long as the disk space is released at some point. In practice it just means that we have to make the disk a bit larger to accommodate. I’ll let you know if it continues to be an issue! Thanks.

Kishore Nallan

03:52 PM

John Can you please check the size of the db/LOG file?

John

04:03 PM

218M

Kishore Nallan

04:08 PM

Ok then it's not that. Just realized that we need to set up proper rotation for that as well.

04:09

Kishore Nallan

04:09 PM

I will have that and an end-point for DB compaction in a couple of days in the next RC build.

Dec 22, 2022 (10 months ago)

Kishore Nallan

06:52 AM

John

In recent 0.24 RC builds (e.g. 0.24.0.rcn43) we have a way for you to trigger a compaction manually like this:

curl -X POST '' --header 'X-TYPESENSE-API-KEY: abcd'

This will restore the disk space immediately. Can you please try that out?

John

08:22 PM

Hey, I’m unfortunately not able to try that out for a few days, but hopefully next week.

Typesense

Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI

Indexed 2779 threads (79% resolved)

Join Our Community Manage subscription

Similar Threads

Slow, High CPU Write Operations After Collection Drop in Typesense

Himank discussed an issue in Typesense where deleting and recreating a collection led to slow write operations and high CPU usage. Kishore Nallan suggested using an alias to avoid this issue. Numerous tests and debugging was conducted as pboros contributed with local testing. Kishore Nallan aimed to start implementing a range delete and full db compaction after deletion to potentially solve the issue.

232

14mo

Trouble with Typesense Memory Usage when Restarting Docker Container

Blend reports increased memory usage when restarting Docker with Typesense, information shared with Jason and Kishore Nallan. Potential data edge case identified as potential cause, although resolution undetermined.

10mo

Upsert Script Issues and Missing Spaces in Typesense

Alex is experiencing issues with missing spaces in Typesense, specifically appearing more frequently during upserts. The missing spaces persist despite changing versions and implementing a workaround. Despite detailed troubleshooting with Kishore Nallan, the issue remains unresolved. The problem appears to be random and unpredictable. Further testing and dataset sharing were suggested for resolution.

19mo

Querying Issues with Newly Created Collection

User Dima reported issues with missed search results in a new collection he created. User Kishore Nallan offered solutions but the problem persists. Dima cannot share the data, which makes it harder to resolve the issue.

2mo

Resolving Typesense Cloud Cluster Issue with Cron Job

Issei reported a problem with an unhealthy Typesense Cloud cluster. With the particular help of Jason and Kishore Nallan, they discovered that a problematic cron job was responsible. A solution, using a different endpoint for data export, was agreed on and implemented.

29mo