#community-help

Issues in Transferring Data from Firestore to Typesense

TLDR Ramees encountered issues in Google Cloud Function Log for data transfer from firestore to typesense. Jason helped troubleshoot, updated the cluster, increased the connection timeout, and resolved the issue.

Powered by Struct AI
+11
32
26mo
Solved
Join the chat
Jul 23, 2021 (26 months ago)
Ramees
Photo of md5-ddfa968eca4266b70a36c83bdcdc7b94
Ramees
04:09 AM
Hi, I am getting an error at Google Cloud Function Log while transferring data from firestore to typesense.
Request #1627012857493: Request to Node 0 failed due to "ECONNABORTED timeout of 10000ms exceeded"
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
04:12 AM
Could you share a screenshot of your extension configuration, excluding the api key?
Ramees
Photo of md5-ddfa968eca4266b70a36c83bdcdc7b94
Ramees
04:17 AM
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
04:20 AM
Could you DM me the Typesense Host field?
04:27
Jason
04:27 AM
May I know the number of records you have in your Firestore collection?
04:27
Jason
04:27 AM
And the rough size of each record?
Ramees
Photo of md5-ddfa968eca4266b70a36c83bdcdc7b94
Ramees
04:29 AM
I have only 7 documents in the collection
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
04:29 AM
The issue is that the cluster has run out of memory, for the amount of data indexed so far. And once RAM runs out things get unpredictable.
Ramees
Photo of md5-ddfa968eca4266b70a36c83bdcdc7b94
Ramees
04:30 AM
I am just starting so. for the mean time i kept the ram to .5gb
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
04:32 AM
Oh hmmm, 7 documents shouldn't fill up memory so much...
04:33
Jason
04:33 AM
Looking deeper
Ramees
Photo of md5-ddfa968eca4266b70a36c83bdcdc7b94
Ramees
04:34 AM
by the way this is my cluster config
04:34
Ramees
04:34 AM
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
04:37 AM
I've restarted the node which has addressed memory consumption and stabilized the node. But I'd like to see if we can replicate this issue.
04:37
Jason
04:37 AM
Could you try inserting another doc into Firestore?
04:40
Jason
04:40 AM
Also, was the last document you added to Firestore particularly large (like say 500KB for eg)?
Ramees
Photo of md5-ddfa968eca4266b70a36c83bdcdc7b94
Ramees
04:43 AM
still having the same error.
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
04:44 AM
Can I upgrade your cluster to v0.21.0? We addressed one related issue in there
Ramees
Photo of md5-ddfa968eca4266b70a36c83bdcdc7b94
Ramees
04:45 AM
ok
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
04:46 AM
I think I'm going to have to also increase the write timeout in the extension, that might be part of the issue
Ramees
Photo of md5-ddfa968eca4266b70a36c83bdcdc7b94
Ramees
04:46 AM
regarding the size. I have around 20 fileds with each lessthan 500 character length. should be around 1 - 2 kb
+11
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
04:46 AM
It's basically retrying multiple times, before the previous request is done and that's clogging things up
Ramees
Photo of md5-ddfa968eca4266b70a36c83bdcdc7b94
Ramees
04:47 AM
ah ok.
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
04:54 AM
Ok, I've updated the cluster to v0.21.0 and also published a new version of the extension that increases the connection timeout.

Could you visit the "Extensions" section of your Firebase project, and click on Upgrade next to the firestore-typesense-search extension? v0.2.4 is the version with the increased timeout
Ramees
Photo of md5-ddfa968eca4266b70a36c83bdcdc7b94
Ramees
04:54 AM
Ok, I will do
05:19
Ramees
05:19 AM
It is working now. but got new problem.
05:20
Ramees
05:20 AM
05:24
Ramees
05:24 AM
If I change the field to geopoint, I got another error
05:24
Ramees
05:24 AM
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
05:32 AM
Could you make the elements of the location array integers?
Ramees
Photo of md5-ddfa968eca4266b70a36c83bdcdc7b94
Ramees
05:32 AM
Ok I will
05:34
Ramees
05:34 AM
thank you. it is working
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
05:41 AM
Awesome!