Hi everyone, while trying to load test an API that my team built, I'm running into an issue that looks like it could be a memory leak, and was curious about the community's thoughts. It appears that Typesense's memory footprint on our server is increasing with every search we execute, and never seems to go back down again. Eventually, it gets to the point where the server becomes unresponsive and needs to be rebooted. Has anyone ever seen this sort of behavior?

@Jim Murtha we did have a memory leak in one of the earlier RC builds of v0.22.0. Could you try this again on 0.22.0.rcs19?

I believe our server is running v0.21.0 (is there is an easy way to double check this though?), so would that issue still apply?

Hmm, no the memory leak I’m referring to was only in an early RC build of v0.22.0. So this is something different. You can verify which version you’re running by doing a GET /debug request

GET /debug tells me it is indeed 0.21.0

Oh ok, could you check if the memory increases beyond 10MB over longer periods of times? Depending on your dataset size a 10MB variation could be within the realm of expected variation…

Sure. I expect that it will go beyond 10MB based on the behavior I saw in some earlier tests, but those were before I was actively monitoring memory, so I will try it out

Could you increase the concurrency or requests per minute to see if that causes memory consumption to increase more steeply?

Increasing it to 10 searches per second saw it increase by 8MB in only one minute. Unfortunately, I don't have time to test any more right now, but I'll do more in the morning and share the results. Thanks for your help so far

We are running hundreds of clusters on 0.21 in production and I haven't seen any memory leaks. The builds are also tested with Valgrind for memory leaks. However I will be happy to take a look if it's possible for you to share a sample dataset and query on which you observe this behaviour.

I believe I've isolated what is causing the memory issue in my searches. I ran 10 searches per second for 10 minutes and saw a 94MB increase in memory usage. I removed one filter which was on all of my searches, and memory usage only increased by a couple MB, and most importantly went back down again after a minute or two.

Thats great, can you please post a sample collection schema here? I can try with it. Thanks!

In my API's real world usage, this filter will probably be on most, but not all, searches. I can probably accomplish this in other, less memory intensive way

@Jim Murtha It looks like we have fixed this on the 0.22 RC builds.

The Linux binary

Can you please try with this build: https://dl.typesense.org/releases/0.22.0.rcs22/typesense-server-0.22.0.rcs22-linux-amd64.tar.gz

I'll need to regroup with my team about trying any other builds. Thanks for your help though.

Thank you. I went through the change log and found a Valgrind warning we addressed specific to the use of the NOT operator, so I am confident that the 0.22 RC builds are free of this.

@Kishore Nallan when do you think this built will be officially out for use?

There are already customers using these RC builds. We have been identifying a few smaller last mile issues to close. These are some rough edges / usability issues in the newer features in this release. It is pretty stable otherwise.