Hello everyone! Just want to share a bit of feedback. Today, I was experimenting and created several indices. It seems that this consumes a lot of memory, causing the containers to be killed by the OOM (I had really tight limits). Now, the services are restarting and trying to slowly load all collections into memory at once.

I20240713 20:36:22.014803   326 collection_manager.cpp:303] Loading symlink production_search to production_search_2024-07-13_13-24-53
I20240713 20:36:22.014829   326 collection_manager.cpp:303] Loading symlink production_search_backup to production_search_2024-07-13_03-00-00
I20240713 20:36:22.014851   326 collection_manager.cpp:303] Loading symlink production_search_update to production_search_2024-07-13_13-24-53
I20240713 20:36:22.014864   326 collection_manager.cpp:303] Loading symlink stage_search to stage_search_2024-07-13_14-39-54
I20240713 20:36:22.014880   326 collection_manager.cpp:303] Loading symlink stage_search_backup to stage_search_2024-07-13_13-12-14
I20240713 20:36:22.014899   326 collection_manager.cpp:303] Loading symlink stage_search_update to stage_search_2024-07-13_19-13-11
I20240713 20:36:22.014915   326 collection_manager.cpp:309] Loading upto 32 collections in parallel, 1000 documents at a time.
I20240713 20:36:22.015705   326 collection_manager.cpp:318] Found 10 collection(s) on disk.
I20240713 20:36:22.018808   364 collection_manager.cpp:160] Found collection production_search_2024-07-12_03-00-00 with 4 memory shards.
I20240713 20:36:22.018813   363 collection_manager.cpp:160] Found collection production_search_2024-07-11_03-00-00 with 4 memory shards.
I20240713 20:36:22.019196   363 collection_manager.cpp:2222] Loading collection production_search_2024-07-11_03-00-00
I20240713 20:36:22.019215   364 collection_manager.cpp:2222] Loading collection production_search_2024-07-12_03-00-00
I20240713 20:36:22.019265   365 collection_manager.cpp:160] Found collection production_search_2024-07-13_03-00-00 with 4 memory shards.
I20240713 20:36:22.019354   366 collection_manager.cpp:160] Found collection production_search_2024-07-13_13-24-53 with 4 memory shards.
I20240713 20:36:22.019368   367 collection_manager.cpp:160] Found collection stage_search_2024-07-11_03-06-53 with 4 memory shards.
I20240713 20:36:22.019541   365 collection_manager.cpp:2222] Loading collection production_search_2024-07-13_03-00-00
I20240713 20:36:22.019745   366 collection_manager.cpp:2222] Loading collection production_search_2024-07-13_13-24-53
I20240713 20:36:22.019834   367 collection_manager.cpp:2222] Loading collection stage_search_2024-07-11_03-06-53
I20240713 20:36:22.020844   368 collection_manager.cpp:160] Found collection stage_search_2024-07-12_03-07-05 with 4 memory shards.
I20240713 20:36:22.021181   368 collection_manager.cpp:2222] Loading collection stage_search_2024-07-12_03-07-05
I20240713 20:36:22.021255   370 collection_manager.cpp:160] Found collection stage_search_2024-07-13_13-12-14 with 4 memory shards.
I20240713 20:36:22.021270   369 collection_manager.cpp:160] Found collection stage_search_2024-07-13_03-04-44 with 4 memory shards.
I20240713 20:36:22.021551   372 collection_manager.cpp:160] Found collection stage_search_2024-07-13_19-13-11 with 4 memory shards.
I20240713 20:36:22.021593   369 collection_manager.cpp:2222] Loading collection stage_search_2024-07-13_03-04-44
I20240713 20:36:22.021870   372 collection_manager.cpp:2222] Loading collection stage_search_2024-07-13_19-13-11
I20240713 20:36:22.022135   371 collection_manager.cpp:160] Found collection stage_search_2024-07-13_14-39-54 with 4 memory shards.
I20240713 20:36:22.022636   371 collection_manager.cpp:2222] Loading collection stage_search_2024-07-13_14-39-54
I20240713 20:36:22.021559   370 collection_manager.cpp:2222] Loading collection stage_search_2024-07-13_13-12-14
I20240713 20:37:22.344321   269 batched_indexer.cpp:422] Running GC for aborted requests, req map size: 0
I20240713 20:38:23.368489   269 batched_indexer.cpp:422] Running GC for aborted requests, req map size: 0
I20240713 20:38:33.652344   369 collection_manager.cpp:2329] Loaded 16384 documents from stage_search_2024-07-13_03-04-44 so far.
I20240713 20:38:34.191502   368 collection_manager.cpp:2329] Loaded 16384 documents from stage_search_2024-07-12_03-07-05 so far.
I20240713 20:38:35.213320   364 collection_manager.cpp:2329] Loaded 16384 documents from production_search_2024-07-12_03-00-00 so far.
I20240713 20:38:35.297617   365 collection_manager.cpp:2329] Loaded 16384 documents from production_search_2024-07-13_03-00-00 so far.
I20240713 20:38:35.515483   363 collection_manager.cpp:2329] Loaded 16384 documents from production_search_2024-07-11_03-00-00 so far.
I20240713 20:38:35.563650   367 collection_manager.cpp:2329] Loaded 16384 documents from stage_search_2024-07-11_03-06-53 so far.
I20240713 20:38:35.635562   371 collection_manager.cpp:2329] Loaded 16384 documents from stage_search_2024-07-13_14-39-54 so far.
I20240713 20:38:35.668578   366 collection_manager.cpp:2329] Loaded 16384 documents from production_search_2024-07-13_13-24-53 so far.
I20240713 20:38:36.251996   370 collection_manager.cpp:2329] Loaded 16384 documents from stage_search_2024-07-13_13-12-14 so far.
I20240713 20:38:36.652113   372 collection_manager.cpp:2329] Loaded 16384 documents from stage_search_2024-07-13_19-13-11 so far.
I20240713 20:39:24.389688   269 batched_indexer.cpp:422] Running GC for aborted requests, req map size: 0
I20240713 20:40:25.404048   269 batched_indexer.cpp:422] Running GC for aborted requests, req map size: 0
I20240713 20:40:44.777034   369 collection_manager.cpp:2329] Loaded 32768 documents from stage_search_2024-07-13_03-04-44 so far.
I20240713 20:40:46.040321   368 collection_manager.cpp:2329] Loaded 32768 documents from stage_search_2024-07-12_03-07-05 so far.
I20240713 20:40:46.434362   365 collection_manager.cpp:2329] Loaded 32768 documents from production_search_2024-07-13_03-00-00 so far.
I20240713 20:40:46.439900   363 collection_manager.cpp:2329] Loaded 32768 documents from production_search_2024-07-11_03-00-00 so far.
I20240713 20:40:46.505545   364 collection_manager.cpp:2329] Loaded 32768 documents from production_search_2024-07-12_03-00-00 so far.
I20240713 20:40:46.976591   367 collection_manager.cpp:2329] Loaded 32768 documents from stage_search_2024-07-11_03-06-53 so far.
I20240713 20:40:47.209162   372 collection_manager.cpp:2329] Loaded 32768 documents from stage_search_2024-07-13_19-13-11 so far.
I20240713 20:40:47.272269   371 collection_manager.cpp:2329] Loaded 32768 documents from stage_search_2024-07-13_14-39-54 so far.
I20240713 20:40:48.100869   366 collection_manager.cpp:2329] Loaded 32768 documents from production_search_2024-07-13_13-24-53 so far.
I20240713 20:40:48.192099   370 collection_manager.cpp:2329] Loaded 32768 documents from stage_search_2024-07-13_13-12-14 so far.

And I can't remove the unnecessary collections.

curl -H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}" -k -X DELETE "<https://c3-typesense.ops.songsterr.com:8110/collections/stage_search_2024-07-13_19-13-11>"
{ "message": "Not Ready or Lagging"}

So for about an hour it is not operational. For comparison: creating and filling a single collection takes only two minutes. In this situation OOM was my fault - I should have removed all test collections and have better limits. But sometimes outages happen by other reasons and such long recovery doesn't look reliable for production use of self-hosted version. Is it possible in future versions to see reads on partial collections during startup instead of Not Ready or Lagging for an hour?