Increased Memory Usage After Implementing `pinned_hits` in Typesense

TLDR John experienced a significant increase in memory usage after using pinned_hits at query time in Typesense. Kishore Nallan suggested that the increase was due to the filtering of the dataset but the issue remains unresolved.

Powered by Struct AI
Nov 19, 2022 (13 months ago)
Photo of md5-21545f1facb7836c149bc4c70752bd2b
04:40 PM
Our Typesense instance was using ~900 MB of memory consistently for weeks, and recently we started using pinned_hits at query time and see rapidly increasing memory usage. First an almost immediate jump from 900 MB to 1.6 GB and then a slow but consistent increase from 1.6 GB to now 3.3 GB since Friday. Is this expected behaviour? Looks a bit concerning
04:41 PM
    "system_cpu1_active_percentage": "0.00",
    "system_cpu2_active_percentage": "0.00",
    "system_cpu_active_percentage": "0.00",
    "system_disk_total_bytes": "105089261568",
    "system_disk_used_bytes": "1497489408",
    "system_memory_total_bytes": "8344117248",
    "system_memory_used_bytes": "4312408064",
    "system_network_received_bytes": "71278807601",
    "system_network_sent_bytes": "11252234532",
    "typesense_memory_active_bytes": "3264450560",
    "typesense_memory_allocated_bytes": "3173333928",
    "typesense_memory_fragmentation_ratio": "0.03",
    "typesense_memory_mapped_bytes": "3366674432",
    "typesense_memory_metadata_bytes": "65228800",
    "typesense_memory_resident_bytes": "3264450560",
    "typesense_memory_retained_bytes": "1692180480"
Nov 20, 2022 (13 months ago)
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
02:27 AM
Pinned hits should not cause a memory increase. Are you able to reproduce in another environment or on your local machine? Also how many requests per second are you seeing?
Photo of md5-21545f1facb7836c149bc4c70752bd2b
08:33 AM
10-20 requests per second. Haven’t tried to reproduce, suspect it’ll be hard but I’ll try 🙂
Nov 21, 2022 (13 months ago)
Photo of md5-21545f1facb7836c149bc4c70752bd2b
10:26 AM
I couldn’t really reproduce it locally, but it continues to happen in production even after restarting the instance. I’ll continue to try to reproduce, but if you have any tips or thoughts that’s very welcome

It started happening after we added pinned_hits at query time (quite many), together with filter_curated_hits and some fairly large filters (only get documents where their group ID is not in list)
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
11:16 AM
Some of the filtering could require run time memory for working dataset. But pinning doesn't incur any memory penalty.
Photo of md5-21545f1facb7836c149bc4c70752bd2b
11:49 AM
But would you expect filtering to result in monotonically increasing memory usage?
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
11:53 AM
No it will cause a higher memory baseline but it won't keep increasing.


Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI

Indexed 3005 threads (79% resolved)

Join Our Community

Similar Threads

Crash and Recovery Issues with Node Reindexing

Greg encountered issues with node health during reindexing, with service unresponsive and recovery taking significant time. Jason and Kishore Nallan suggested it might be a case of high volume writes and not a crash. Problem wasn't fully resolved after attempted solutions and data sharing for further debugging.



Troubleshooting Stalled Writes in TypeSense Instance

Robert was experiencing typesense instances getting stuck after trying to import documents. Kishore Nallan provided suggestions and added specific logs to diagnose the issue. The two identified queries causing troubles but the issues had not been fully resolved yet.



Slow, High CPU Write Operations After Collection Drop in Typesense

Himank discussed an issue in Typesense where deleting and recreating a collection led to slow write operations and high CPU usage. Kishore Nallan suggested using an alias to avoid this issue. Numerous tests and debugging was conducted as pboros contributed with local testing. Kishore Nallan aimed to start implementing a range delete and full db compaction after deletion to potentially solve the issue.



Addressing High CPU Usage in Typesense

Robert reported high CPU usage on Typesense, even after halting all incoming searches. Kishore Nallan suggested logging heavy queries and increasing thread count. The issue was resolved after Robert found and truncated unusually large documents in the database.


Resolving Typesense Result Issue in Document Collection Queries

Mike was encountering errors when searching for specific query in their Typesense document collection. Jason suggested it may be due to the `drop_tokens_threshold` setting. There was a misunderstanding but after further explanation from Jason, Mike understood and decided to continue the conversation via email.