Trouble with Typesense Memory Usage when Restarting Docker Container
TLDR Blend reports increased memory usage when restarting Docker with Typesense, information shared with Jason and Kishore Nallan. Potential data edge case identified as potential cause, although resolution undetermined.
6
1
1
1
Dec 20, 2022 (12 months ago)
Blend
04:29 PMI am running Typesense (
typesense:0.24.0.rcn41
) with Docker. I run the container, index about 2GB of documents, and everything seems fine. Memory usage is at around 1.7GB.If I stop the container and start it again (without destroying the data collection in disk), the collections in disk seem to be detected/loaded but memory usage never stops increasing (it got up to 16GB before I stopped the container).
Am I doing something wrong?
(90% of our data is in nested fields, that is why I do not use a stable release. Also, I tried indexing 5% of our data and this problem did not happen)
1
Jason
05:43 PMBlend
05:54 PMversion: '3.3'
services:
typesense:
ports:
- '${TYPESENSE_PORT}:${TYPESENSE_PORT}'
volumes:
- ./data:/data
command: '--data-dir /data --api-key=${TYPESENSE_API_KEY}'
image: 'typesense/typesense:0.24.0.rcn41'
The first time I start the container, I use
docker-compose up
, then docker-compose stop
and docker-compose start
Jason
05:56 PM/GET metrics.json
, just after you start and then after you stop and start?Blend
06:00 PM1
Dec 21, 2022 (12 months ago)
Kishore Nallan
01:43 AMI wonder if there is an edge case with nested fields that manifest only on a restart. If you are able to share this dataset with us privately (we will promptly destroy it after debugging), we could also take a detailed look at what is happening. Once you have finished indexing, you can zip the data directory and share it with us.
Blend
11:40 AMBefore posting about it here, the issue happened 4/4 times that I restarted the container.
I will keep checking when I deploy typesense in our prod environment, and also ask my team manager about sharing the data directory with you.
Kishore Nallan
11:41 AMBlend
11:44 AMKishore Nallan
11:45 AM1
Kishore Nallan
11:46 AMBlend
11:47 AMKishore Nallan
11:48 AMa) Clear data directory
b) Reindex the data
c) Trigger a snapshot via the manual snapshot API (https://typesense.org/docs/0.23.1/api/cluster-operations.html#create-snapshot-for-backups)
d) Restart container
1
Kishore Nallan
11:48 AMBlend
11:55 AM{ {
"system_cpu1_active_percentage": "0.00", "system_cpu1_active_percentage": "81.82",
"system_cpu2_active_percentage": "0.00", "system_cpu2_active_percentage": "100.00",
"system_cpu3_active_percentage": "9.09", "system_cpu3_active_percentage": "100.00",
"system_cpu4_active_percentage": "0.00", "system_cpu4_active_percentage": "90.00",
"system_cpu5_active_percentage": "10.00", "system_cpu5_active_percentage": "100.00",
"system_cpu6_active_percentage": "0.00", "system_cpu6_active_percentage": "100.00",
"system_cpu7_active_percentage": "0.00", "system_cpu7_active_percentage": "100.00",
"system_cpu8_active_percentage": "0.00", "system_cpu8_active_percentage": "75.00",
"system_cpu_active_percentage": "1.28", "system_cpu_active_percentage": "92.50",
"system_disk_total_bytes": "123816591360", "system_disk_total_bytes": "123816591360",
"system_disk_used_bytes": "80937127936", "system_disk_used_bytes": "83946774528",
"system_memory_total_bytes": "16575455232", "system_memory_total_bytes": "16575455232",
"system_memory_used_bytes": "6468882432", "system_memory_used_bytes": "7615643648",
"system_network_received_bytes": "22211", "system_network_received_bytes": "533235",
"system_network_sent_bytes": "15146", "system_network_sent_bytes": "403554",
"typesense_memory_active_bytes": "36155392", "typesense_memory_active_bytes": "713400320",
"typesense_memory_allocated_bytes": "33771824", "typesense_memory_allocated_bytes": "696897696",
"typesense_memory_fragmentation_ratio": "0.07", "typesense_memory_fragmentation_ratio": "0.02",
"typesense_memory_mapped_bytes": "116314112", "typesense_memory_mapped_bytes": "966184960",
"typesense_memory_metadata_bytes": "10287312", "typesense_memory_metadata_bytes": "21854176",
"typesense_memory_resident_bytes": "36155392", "typesense_memory_resident_bytes": "713400320",
"typesense_memory_retained_bytes": "57225216" "typesense_memory_retained_bytes": "631844864"
} }
Kishore Nallan
12:00 PMKishore Nallan
12:00 PM1
Blend
12:44 PMcurl "" -X POST \
-H "Content-Type: application/json" \
-H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}"
to create a snapshot.
The response was
{"success":true}
, but I do not see any file/directory being created in the specified path. Doing ls /tmp | grep typesense
shows nothing. Is this a bad sign?Kishore Nallan
12:58 PM1
Kishore Nallan
12:59 PMBlend
01:06 PMKishore Nallan
01:07 PM./data
(local data dir) is mounted as /data
inside the container and this is the path used as data-dir. This is correct.Blend
01:08 PMKishore Nallan
01:08 PMBlend
01:19 PMBlend
01:19 PMBlend
01:23 PM{
"system_cpu1_active_percentage": "72.73",
"system_cpu2_active_percentage": "40.00",
"system_cpu3_active_percentage": "100.00",
"system_cpu4_active_percentage": "40.00",
"system_cpu5_active_percentage": "37.50",
"system_cpu6_active_percentage": "100.00",
"system_cpu7_active_percentage": "44.44",
"system_cpu8_active_percentage": "60.00",
"system_cpu_active_percentage": "62.03",
"system_disk_total_bytes": "123816591360",
"system_disk_used_bytes": "82421317632",
"system_memory_total_bytes": "16575455232",
"system_memory_used_bytes": "16246272000",
"system_network_received_bytes": "4167076",
"system_network_sent_bytes": "4260233",
"typesense_memory_active_bytes": "10272575488",
"typesense_memory_allocated_bytes": "10254819992",
"typesense_memory_fragmentation_ratio": "0.00",
"typesense_memory_mapped_bytes": "10580803584",
"typesense_memory_metadata_bytes": "173259808",
"typesense_memory_resident_bytes": "10272575488",
"typesense_memory_retained_bytes": "1700118528"
}
Kishore Nallan
01:24 PMBlend
01:27 PMKishore Nallan
01:27 PMOn demand snapshot succeeded!
4 times in the log though. Did you trigger it multiple times?Kishore Nallan
01:27 PMBlend
01:30 PMOh interesting. So if I want to estimate the disk usage of our data, I should wait for a while after the indexing?
Kishore Nallan
01:30 PMI20221221 13:09:06.579972 1 typesense_server_utils.cpp:524] Bye.
And the final stop just 2 mins later:
I20221221 13:11:02.677739 1 typesense_server_utils.cpp:48] Stopping Typesense server...
Did the memory grow to 10GB+ within that short time?
Blend
01:32 PMI20221221 13:09:22.098304 1 typesense_server_utils.cpp:355] Starting Typesense 0.24.0.rcn41
But yes, the memory increases very quickly
Kishore Nallan
01:33 PMReclaiming of disk space is a bit of a complex topic. In a recent build (not available in rcn41 that you are using here), I"ve introduced an option to "compact" the DB manually. That will be useful for heavy write use cases.
1
Kishore Nallan
01:34 PM1
Blend
01:36 PMKishore Nallan
01:36 PMKishore Nallan
01:37 PMBlend
01:40 PMKishore Nallan
01:51 PM--snapshot-interval-seconds
to a very high value so that snapshotting never happens. Since the issue only happens post snapshotting, this should ensure that even Docker restarts, it will work. This is a theory, but I'm reasonably confident about it.Container shouldn't fail by itself unless it crashes due to some reason.
1
Kishore Nallan
01:52 PMKishore Nallan
04:10 PMBlend
04:41 PMDec 22, 2022 (12 months ago)
Kishore Nallan
08:40 AMTypesense
Indexed 3011 threads (79% resolved)
Similar Threads
Crash and Recovery Issues with Node Reindexing
Greg encountered issues with node health during reindexing, with service unresponsive and recovery taking significant time. Jason and Kishore Nallan suggested it might be a case of high volume writes and not a crash. Problem wasn't fully resolved after attempted solutions and data sharing for further debugging.
Troubleshooting Typesense Snapshot Errors in AWS Docker Container
Arthur experienced recurring Typesense snapshotting errors in an AWS Docker container. Kishore Nallan diagnosed two different issues: a known Google log issue to be fixed in the next release and an unknown file opening error, and guided Arthur on creating GitHub issues for both.
Large JSONL Documents Import Issue & Resolution
Suraj was having trouble loading large JSONL documents into Typesense server. After several discussions and attempts, it was discovered that the issue was due to data quality. Once the team extracted the data again, the upload process worked smoothly.
Troubleshooting Typesense Server Error on Docker
vikram was facing an error with Typesense Server Docker container and loss of data on restart. Kishore Nallan guided to avoid mounting tmp directory from localhost and explained stopping the Docker container.
Using Typesense in Docker Container – Importing JSONL File
Hakim faced 'Empty reply from server' error when importing a JSONL file into a Typesense-Docker container. Kishore Nallan and Jason suggested checking the docker logs and increasing default RAM and CPU allocation. Finally, Hakim successfully indexed the documents using a PHP script.