Hi Guys, i am importing 3 million documents into a...
# community-helpd
Digamber Pradhan
03/09/2023, 5:42 PMHi Guys, i am importing 3 million documents into a collection.
I’m doing chunked bulk index at a time - it’s taking me 7 secs per 40 records.
With some quick maths - it’s going to take me 145 hours 😢
Is there a more performative way i can index these posts ?
j
Jason Bosco
03/09/2023, 6:14 PMThat sounds odd. Is this on Typesense Cloud? If so, can you share the cluster ID?
d
Digamber Pradhan
03/09/2023, 6:21 PMI have the node id - is the cluster id the same ?
j
Jason Bosco
03/09/2023, 6:36 PMYup it’s the same
d
Digamber Pradhan
03/09/2023, 6:37 PMHere you gp bt0k9i561mfvgzp4p
j
Jason Bosco
03/09/2023, 7:42 PMHmm, metrics looks fine on this cluster
Jason Bosco
03/09/2023, 7:42 PMCould you confirm that you’re using the bulk import endpoint?
Jason Bosco
03/09/2023, 7:43 PMIf so, could you try replicating the issue with curl?
d
Digamber Pradhan
03/09/2023, 7:55 PMI can indeed confirm - that it’s the bulk import endpoint
j
Jason Bosco
03/09/2023, 7:56 PMOk if you can send me a JSONL file (via email) and a curl command to replicate the issue, I can take a closer look
d
Digamber Pradhan
03/09/2023, 7:57 PMAh - ok it’s a bit late today - i will email it to you tomorrow, thanks for the help
👍 1