https://typesense.org logo
Join Slack
Powered by
Hey guys, I'm trying to get an `export` of a fairl...
# community-help
m
Hey guys, I'm trying to get an 
export
of a fairly big index and it overwhelms the Typesense server . Wondering if export can be done in batches like import. Tried feeding it 
batch_size
but it seems to not read it. Any ideas ?
k
👋 This is fixed in the most recent 0.25 RC build (0.25.0.rc44)
m
I'm using Typesense Python library. Is 0.25 released yet ?
k
I'm talking about the server version. We are in the last stretch before release of 0.25
✅ 1
You can use the rc build, it's stable
m
Thank you, Kishore .
👍 1
Hey Kishore, I am trying the RC44 build and I can confirm it still falls over when exporting documents. Which branch is it on GH ? I can take a look to see the integration.
k
I identified one more issue related to this and fixed it again. Can you try on RC45? I've tested this on a large 12GB dataset
And when you mean by fall over, what exactly happens? The issue I fixed is limiting the memory used during export. Previously it was using a huge buffer so it ate up a lot of memory.
m
Thanks for the update. I tried RC45 but it still not works. So what happens is that the connection gets closed. I'm going to attach screenshots of the error for you below : P.S. I'm using Docker.
k
What's your client side timeout?
After how long does this happen? Can you also post the code snippet for export? I will try it out.
m
My client-side timeout is 120000.
The error happens after about 10 minutes.
Copy code
indexed_pages = client.collections['pages'].documents.export({'batch_size': 10})
I am running this on a table of 16.41 Gb, btw.
k
I tested exporting a 11G dataset on this build the other day. I ran using curl which has no timeouts. Can you try once via curl and without using the batch size option?
m
Hi Kishore, The curl for export seems to be working fine.
k
Then I think the issue is with the python client.
Open in Slack
PreviousNext
Powered by