Hey! I'm wondering what is the advice on handling ...
# community-help
m
Hey! I'm wondering what is the advice on handling snapshotting. 1. Is snapshotting recommended vs catching up from the leader? 2. Are snapshots only for schema fields that are stored? 3. If we want snapshots fully working, do we need to store every field? This seems to slow down query speed significantly? 4. We have attempted setting every field to be stored and snapshot recovery seems to not be progressing even with plenty of RAM, disk and CPU available. Do we have any ideas why this could be? For context, we are self hosting in Kubernetes. We were seeing the following logs for every collection for a certain schema, this was fixed by setting every field to be stored.
Copy code
collection_manager.cpp:2112] Indexed 0/602 documents into collection tw_products_1772498224985
Typesense would then never get the documents from the leader node either, so would be ready and serving no documents. Any help or advice would be greatly appreciated. Thanks all!
👀 1
a
Hey @Matthew Simpson 1. You'll want to do both. Catching up from the leader means that the node is actually processing the writes itself, so if the raft log is big, this can take a long time. Snapshot will be simplier, since the node can just load it. Also, if the raft log is too big, Typesense will automatically transfer the snapshot and initiate from it. 2. Snapshots are an exact copy in a point of time of your node. It stores everything. 3 and 4. are about the issue you are seeing.
store: false
means you are not storing the field in disk (rocksdb). When you use
store:false
in a non-optional field, the validation fails. If you want to speed up search, since you are not searching/faceting/filtering in the fields, you'll want to set
index: false
. This tells Typesense to not load the fields from disk -> RAM. Are you monitoring disk and ram in your nodes? You'll want to check if your disk usage is increasing right after the node start, which is a sign of the snapshot being transfered. If your snapshot is big and/or connection is slow, this can take some time.
m
Thanks for the reply Alan! Do we know why we might not successfully Index the documents from snapshot when no fields were marked as
store: true
as shown in the log line above where we index 0 of the documents? I believe we need indexes on a lot of the fields due to being searched on. We can see the disk usage jump to roughly the same amount as other nodes. I believe it is pulling the snapshot correctly but failing to index them now. RAM usage is jumping up to roughly match other nodes too. This looks healthy to me For every one of our larger collections we see the following log but with no follow up
Copy code
I20260303 15:25:41.620805   640 collection_manager.cpp:1936] Loading collection pt_products_1772542254685
Our snapshots are just under 2GB in total
a
Do we know why we might not successfully Index the documents from snapshot when no fields were marked as
store: true
as shown in the log line above where we index 0 of the documents?
Because of what I mentioned above. They must be store: true (which is also the default, if you don't explicitely mark it) if the field is non-optional. You mentioned that marking with
store: true
fixes it right? Possibly because your previous schema had them as false.
m
Ahh, yes, that makes sense thank you. All of our fields are non-optional (and most need indexing) so we are now storing them all but we aren't ever managing to successfully recover from the snapshot. For 2GB snapshot, would the following per pod be sufficient • RAM 20GB • CPU 8 • Disk 20GB Do you have any other ideas on why it might not be processing the snapshot indexes?
a
Is the RAM slowly increasing after the disk finishes increasing? Also, is that log line the last ever printed? I20260303 152541.620805 640 collection_manager.cpp:1936] Loading collection pt_products_1772542254685
Also, how big is that collection?
And which verison are you running?
m
Ram stays stable. After that log there are only GC logs. The collection is roughly 600 documents, totalling around 180MB. Version 30.1 We have just had a breakthrough. We changed
--num-collections-parallel-load
to 1 from the preset and it loaded all collections within a minute. Have you seen this cause problems on Kubernetes before?
a
Hmmmmmm 1. ram stays stable -> means that its not indexing at all 2. 600 documents is nothing. Should load in seconds 3. Pretty recent version... This seems like a kubernets related issue, specially since the parallel load fixed it. We don't have much experience with kubernetes environments though, and it's a bit difficult to debug non-cloud environments (since we don't have a whole bunch of telemetry in place)
m
With the caveat that pretty much all fields in my schema are non-optional and need to be indexed, is there any recommended way to speed up query time?
a
With query time do you mean the response time for searches? The indexing or no part is more for resource management than search response times. Could you share example of search cURLs you make?