👋 Hi there! I wonder if anyone has encountered something similar and maybe has some pointers. We’ve been running multiple typesense deployments on Kubernetes (AWS). They are all deployed as StatefulSets, 3 pods each with EBS volume attached. It works perfectly in most cases. However for one cluster which has a bit more data then the others we notice very long boot times after restart. It looks like for about 6-10 minutes it loads data from disk and then it starts to process
queued writes
. And this sometimes can take over 40 minutes. Initially when data is loaded from disk we see increased disk IO and CPU usage, however when typesense starts to process
queued writes
CPU and disk activity is much lower. I wonder if anything can be done to speed this process up? When it happens we’re getting
503 - { "message": "Not Ready or Lagging"}
as this queue can start at 6000.