This message was deleted.
# community-help
s
This message was deleted.
k
We don't change the defaults, so it should
60
. Generally some swap is used if the node had high memory usage at some point. Generally this is not a problem because only infrequent items are paged out.
j
It's possible to change this value ?
k
It's not. We've not found this to affect search performance in general so far.
j
Yes, but then we have a problem. Because when we run a purge command that goes through the entire typesense index, and when this data is moved to swap by the linux kernel, there's an accumulation of latency that results in timeouts... Adjusting the swapiness value can solve this problem, if not increase server size. But in the event of high volumes, the server will become far too large for the actual use of Typesense...
k
Can you please DM me your cluster ID?
✅ 1
j
We have recently doubled the resources of this cluster because of this problem. However, this is not a viable long-term solution when our data volume increases, as costs will then become problematic in relation to actual/real usage.
k
The node is using 3.62/3.75 GB of memory so that's cutting very close. That's the reason some 500M of memory has been swapped out. We won't be able to change the swappiness value because if we made this too aggressive, the OS will simply terminate the process when memory limit is near and the behavior becomes unpredictable.
j
I'm sorry I wrong, I just sent you the right cluster ID.
k
This second cluster has no swap usage.
Can you show me where you are seeing swap used?
j
Yes, because we've recently increased the size of the cluster/node. However, when we had this latency problem, it had half as much RAM. RAM usage was then ~4 GB/8 GB + ~650 MB in Swap, whereas there's clearly enough RAM left over to keep everything in and avoid latency problems when processing all indexes. Today, to compensate this problem, we had to switch to a node with 16 GB of RAM, even though we don't even use 4 GB of data in memory. This is where it gets crazy and reduce swapiness to 60 -> 1 (for example) allows us to be LARGELY satisfied with an 8 GB instance
k
The swap usage does not kick in unless the server reached the 8G threshold earlier. Did it?
So once the swap happens, we find that the OS manages the release of swap back to memory even if memory is available (this is ofc controlled by the swappiness value). However swappiness also affects the initial swap behavior when memory starts running out.
j
With swapiness=60, the linux kernel will move unused data from RAM to this swap. The latency problem then arises when accessing this previously unused data. You don't need to reach 8 GB RAM usage for this to happen, because that's how swapiness works. Lowering the swapiness value delays this process (and 1 disables it completely).