#community-help

Increased Memory Usage After Implementing `pinned_hits` in Typesense

TLDR John experienced a significant increase in memory usage after using pinned_hits at query time in Typesense. Kishore Nallan suggested that the increase was due to the filtering of the dataset but the issue remains unresolved.

13mo

Join the chat

Nov 19, 2022 (13 months ago)

Photo of md5-21545f1facb7836c149bc4c70752bd2b

John

04:40 PM

Our Typesense instance was using ~900 MB of memory consistently for weeks, and recently we started using pinned_hits at query time and see rapidly increasing memory usage. First an almost immediate jump from 900 MB to 1.6 GB and then a slow but consistent increase from 1.6 GB to now 3.3 GB since Friday. Is this expected behaviour? Looks a bit concerning

04:41

John

04:41 PM

{
    "system_cpu1_active_percentage": "0.00",
    "system_cpu2_active_percentage": "0.00",
    "system_cpu_active_percentage": "0.00",
    "system_disk_total_bytes": "105089261568",
    "system_disk_used_bytes": "1497489408",
    "system_memory_total_bytes": "8344117248",
    "system_memory_used_bytes": "4312408064",
    "system_network_received_bytes": "71278807601",
    "system_network_sent_bytes": "11252234532",
    "typesense_memory_active_bytes": "3264450560",
    "typesense_memory_allocated_bytes": "3173333928",
    "typesense_memory_fragmentation_ratio": "0.03",
    "typesense_memory_mapped_bytes": "3366674432",
    "typesense_memory_metadata_bytes": "65228800",
    "typesense_memory_resident_bytes": "3264450560",
    "typesense_memory_retained_bytes": "1692180480"
}

Nov 20, 2022 (13 months ago)

Photo of md5-4e872368b2b2668460205b409e95c2ea

Kishore Nallan

02:27 AM

Pinned hits should not cause a memory increase. Are you able to reproduce in another environment or on your local machine? Also how many requests per second are you seeing?

John

08:33 AM

10-20 requests per second. Haven’t tried to reproduce, suspect it’ll be hard but I’ll try 🙂

Nov 21, 2022 (13 months ago)

John

10:26 AM

I couldn’t really reproduce it locally, but it continues to happen in production even after restarting the instance. I’ll continue to try to reproduce, but if you have any tips or thoughts that’s very welcome

It started happening after we added pinned_hits at query time (quite many), together with filter_curated_hits and some fairly large filters (only get documents where their group ID is not in list)

Kishore Nallan

11:16 AM

Some of the filtering could require run time memory for working dataset. But pinning doesn't incur any memory penalty.

John

11:49 AM

But would you expect filtering to result in monotonically increasing memory usage?

Kishore Nallan

11:53 AM

No it will cause a higher memory baseline but it won't keep increasing.

Typesense

Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI

Indexed 3005 threads (79% resolved)

Join Our Community

Similar Threads

Crash and Recovery Issues with Node Reindexing

Greg encountered issues with node health during reindexing, with service unresponsive and recovery taking significant time. Jason and Kishore Nallan suggested it might be a case of high volume writes and not a crash. Problem wasn't fully resolved after attempted solutions and data sharing for further debugging.

yesterday

Troubleshooting Stalled Writes in TypeSense Instance

Robert was experiencing typesense instances getting stuck after trying to import documents. Kishore Nallan provided suggestions and added specific logs to diagnose the issue. The two identified queries causing troubles but the issues had not been fully resolved yet.

12mo

Slow, High CPU Write Operations After Collection Drop in Typesense

Himank discussed an issue in Typesense where deleting and recreating a collection led to slow write operations and high CPU usage. Kishore Nallan suggested using an alias to avoid this issue. Numerous tests and debugging was conducted as pboros contributed with local testing. Kishore Nallan aimed to start implementing a range delete and full db compaction after deletion to potentially solve the issue.

232

17mo

Addressing High CPU Usage in Typesense

Robert reported high CPU usage on Typesense, even after halting all incoming searches. Kishore Nallan suggested logging heavy queries and increasing thread count. The issue was resolved after Robert found and truncated unusually large documents in the database.

14mo

Resolving Typesense Result Issue in Document Collection Queries

Mike was encountering errors when searching for specific query in their Typesense document collection. Jason suggested it may be due to the `drop_tokens_threshold` setting. There was a misunderstanding but after further explanation from Jason, Mike understood and decided to continue the conversation via email.

21mo