How can I know that indexing completed on a datase...
# community-helpt
Thomas
03/02/2022, 11:44 AMHow can I know that indexing completed on a dataset?
k
Kishore Nallan
03/02/2022, 11:45 AMWhen the endpoint response arrives, indexing is done. All endpoints are synchronous.
t
Thomas
03/02/2022, 11:46 AMIf we restart the instance, it take a long time before it's usable, like 30 minutes. It re-index on reboot?
k
Kishore Nallan
03/02/2022, 11:47 AMYes, only raw documents are stored on disk and indexing happens in memory on restart.
t
Thomas
03/02/2022, 11:48 AMAlright, what's the bottleneck on that? CPU or Disk speed?
k
Kishore Nallan
03/02/2022, 11:48 AMCPU. The latest 0.23 RC builds are faster in this respect.
t
Thomas
03/02/2022, 11:51 AMHow much faster is 0.23 RC?
k
Kishore Nallan
03/02/2022, 11:51 AMDepends on dataset. Primary work is around numerical fields.
t
Thomas
03/02/2022, 11:52 AMDataset is majority text
k
Kishore Nallan
03/02/2022, 11:52 AMWe recommend running a 3 node configuration so rotationse can be done without a single point of failure.
Kishore Nallan
03/02/2022, 11:53 AMThere might be a few other things we can still do to optimize text fields.
t
Thomas
03/02/2022, 11:59 AMYeah we're starting with 3 nodes in a cluster
Thomas
03/02/2022, 11:59 AMWhat's the optimal size in terms of keys?
Thomas
03/02/2022, 12:00 PMWe have 60 datapoints that need to filter on
Thomas
03/02/2022, 12:11 PM@Kishore Nallan How often do the API break, could rolling upgrades on a cluster be a problem?
k
Kishore Nallan
03/02/2022, 12:11 PMEvery node stores all the data so nodes help in increasing throughput acorss many users.
Kishore Nallan
03/02/2022, 12:12 PMWe've successfully done 5 versions so far on Typesense cloud across hundreds of deployments. We take care about backward compatibility.
Kishore Nallan
03/02/2022, 12:14 PMWe store nothing but documents on disk so not much problem with upgrades.
t
Thomas
03/02/2022, 12:18 PMSuperb 🙂
Thomas
03/02/2022, 12:18 PMThank you
k
Kishore Nallan
03/02/2022, 12:18 PM👍
t
Thomas
03/02/2022, 12:47 PM@Kishore Nallan It's not possible to dump, periodically the RAM that's the index, to disk, so it doesn't need to be re-indexed on reboot?
💡 2
k
Kishore Nallan
03/02/2022, 2:37 PMYou could try doing this via CRIU: https://criu.org/Main_Page
👍 1
t
Thomas
03/02/2022, 2:48 PMYeah that's how we currently do it with KVM, but this doesn't help if there's a hardware issue