#community-help

Increased Memory Usage After Implementing `pinned_hits` in Typesense

TLDR John experienced a significant increase in memory usage after using pinned_hits at query time in Typesense. Kishore Nallan suggested that the increase was due to the filtering of the dataset but the issue remains unresolved.

11mo

Join the chat

Nov 19, 2022 (11 months ago)

Photo of md5-21545f1facb7836c149bc4c70752bd2b

John

04:40 PM

Our Typesense instance was using ~900 MB of memory consistently for weeks, and recently we started using pinned_hits at query time and see rapidly increasing memory usage. First an almost immediate jump from 900 MB to 1.6 GB and then a slow but consistent increase from 1.6 GB to now 3.3 GB since Friday. Is this expected behaviour? Looks a bit concerning

04:41

John

04:41 PM

{
    "system_cpu1_active_percentage": "0.00",
    "system_cpu2_active_percentage": "0.00",
    "system_cpu_active_percentage": "0.00",
    "system_disk_total_bytes": "105089261568",
    "system_disk_used_bytes": "1497489408",
    "system_memory_total_bytes": "8344117248",
    "system_memory_used_bytes": "4312408064",
    "system_network_received_bytes": "71278807601",
    "system_network_sent_bytes": "11252234532",
    "typesense_memory_active_bytes": "3264450560",
    "typesense_memory_allocated_bytes": "3173333928",
    "typesense_memory_fragmentation_ratio": "0.03",
    "typesense_memory_mapped_bytes": "3366674432",
    "typesense_memory_metadata_bytes": "65228800",
    "typesense_memory_resident_bytes": "3264450560",
    "typesense_memory_retained_bytes": "1692180480"
}

Nov 20, 2022 (11 months ago)

Photo of md5-4e872368b2b2668460205b409e95c2ea

Kishore Nallan

02:27 AM

Pinned hits should not cause a memory increase. Are you able to reproduce in another environment or on your local machine? Also how many requests per second are you seeing?

John

08:33 AM

10-20 requests per second. Haven’t tried to reproduce, suspect it’ll be hard but I’ll try 🙂

Nov 21, 2022 (11 months ago)

John

10:26 AM

I couldn’t really reproduce it locally, but it continues to happen in production even after restarting the instance. I’ll continue to try to reproduce, but if you have any tips or thoughts that’s very welcome

It started happening after we added pinned_hits at query time (quite many), together with filter_curated_hits and some fairly large filters (only get documents where their group ID is not in list)

Kishore Nallan

11:16 AM

Some of the filtering could require run time memory for working dataset. But pinning doesn't incur any memory penalty.

John

11:49 AM

But would you expect filtering to result in monotonically increasing memory usage?

Kishore Nallan

11:53 AM

No it will cause a higher memory baseline but it won't keep increasing.

Typesense

Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI

Indexed 2786 threads (79% resolved)

Join Our Community Manage subscription

Similar Threads

Troubleshooting Stalled Writes in TypeSense Instance

Robert was experiencing typesense instances getting stuck after trying to import documents. Kishore Nallan provided suggestions and added specific logs to diagnose the issue. The two identified queries causing troubles but the issues had not been fully resolved yet.

10mo

Slow, High CPU Write Operations After Collection Drop in Typesense

Himank discussed an issue in Typesense where deleting and recreating a collection led to slow write operations and high CPU usage. Kishore Nallan suggested using an alias to avoid this issue. Numerous tests and debugging was conducted as pboros contributed with local testing. Kishore Nallan aimed to start implementing a range delete and full db compaction after deletion to potentially solve the issue.

232

15mo

Addressing High CPU Usage in Typesense

Robert reported high CPU usage on Typesense, even after halting all incoming searches. Kishore Nallan suggested logging heavy queries and increasing thread count. The issue was resolved after Robert found and truncated unusually large documents in the database.

12mo

Resolving Typesense Result Issue in Document Collection Queries

Mike was encountering errors when searching for specific query in their Typesense document collection. Jason suggested it may be due to the `drop_tokens_threshold` setting. There was a misunderstanding but after further explanation from Jason, Mike understood and decided to continue the conversation via email.

19mo

Memory Leak Issue in Typesense Server v0.21.0

Jim was experiencing a potential memory leak while load testing an API. Jason and Kishore Nallan suggest it may be a known issue in v0.22.0 RC builds which is resolved in later versions. Jim identifies a specific search filter escalating the issue. Kishore Nallan confirms a fix in the 0.22 RC builds.

25mo