Hi , i am running typesense on eks with 3 node clu...
# community-helpm
Manav Kothari
11/30/2024, 4:02 AMHi , i am running typesense on eks with 3 node cluster i see different memory usage for each pods any idea why this might be happening
Copy code
NAME CPU(cores) MEMORY(bytes)
typesense-0 5m 47082Mi
typesense-1 4m 105839Mi
typesense-2 4m 85150MiManav Kothari
11/30/2024, 4:04 AMi see this logs on each pods
Copy code
20241130 04:00:55.227258 454 raft_server.cpp:706] Term: 12, pending_queue: 0, last_index: 686477, committed: 686477, known_applied: 686477, applying: 0, pending_writes: 0, queued_writes: 0, local_sequence: 1532529754
I20241130 04:01:04.699625 455 batched_indexer.cpp:428] Running GC for aborted requests, req map size: 0
I20241130 04:01:05.231527 454 raft_server.cpp:706] Term: 12, pending_queue: 0, last_index: 686477, committed: 686477, known_applied: 686477, applying: 0, pending_writes: 0, queued_writes: 0, local_sequence: 1532529754
I20241130 04:01:15.235275 454 raft_server.cpp:706] Term: 12, pending_queue: 0, last_index: 686477, committed: 686477, known_applied: 686477, applying: 0, pending_writes: 0, queued_writes: 0, local_sequence: 1532529754Manav Kothari
11/30/2024, 4:15 AMThere is no traffic right now on this.
k
Kishore Nallan
12/01/2024, 2:56 AMCan you hit the
/metrics.jsontypesense_memory_active_bytesm
Manav Kothari
12/02/2024, 5:54 AM Copy code
Memory
active : 69.8 GB
allocated : 50.9 GB
fragmentation : 0.27
mapped : 71.9 GB
metadata : 1.66 GB
resident : 69.8 GB
retained : 126 GB Copy code
Memory
active : 53.5 GB
allocated : 50.9 GB
fragmentation : 0.05
mapped : 55.2 GB
metadata : 1.45 GB
resident : 53.5 GB
retained : 114 GB Copy code
Memory
active : 31.9 GB
allocated : 30.5 GB
fragmentation : 0.04
mapped : 33.1 GB
metadata : 1.01 GB
resident : 31.9 GB
retained : 99.7 GBManav Kothari
12/02/2024, 5:55 AMFor each node this is the values
Manav Kothari
12/02/2024, 5:59 AMis this issue releted to compaction?
k
Kishore Nallan
12/02/2024, 6:11 AMHave you verified that all nodes actually have the same record count?
Kishore Nallan
12/02/2024, 6:12 AMOne node has high fragmentation, which indicates memory "holes" that are present in the memory pages allocated by the application.
Kishore Nallan
12/02/2024, 6:12 AMIn general, it's a bit tricky to measure actual memory usage when you do extensive writes and deletes, because the memory allocator we (jemalloc) tends to reserve memory and will not release it back to OS in anticipation of reusing those memory blocks.
m
Manav Kothari
12/02/2024, 6:14 AMbut there no writes and deletes operation happening right now, is there any way to free this memory.
Manav Kothari
12/02/2024, 6:15 AMyes i am getting same record count on fetch on hitting multiple times.
k
Kishore Nallan
12/02/2024, 6:16 AMYou can restart the pod with high fragmentation, that tends to help with fragmentation because bulk loading during restart is more efficient.
👀 1
m
Manav Kothari
12/02/2024, 6:17 AMwill it lead to down time? if i do one pod restart at a time.
k
Kishore Nallan
12/02/2024, 6:38 AMNo, 3 node cluster can survive a pod going down
👍 1