#community-help

Issues with Large Index Export via Typesense Server

TLDR Mojan struggled with a large index export on Typesense server. Kishore Nallan informed of the issue's fix in an upcoming server version. The issue stemmed from the Python client, not the server.

2mo

Join the chat

Jul 06, 2023 (3 months ago)

Photo of md5-813f8f14e16940e952eb3fe25d0c4523

Mojan

03:33 AM

Hey guys,
I'm trying to get an export of a fairly big index and it overwhelms the Typesense server . Wondering if export can be done in batches like import. Tried feeding it batch_size but it seems to not read it. Any ideas ?

Photo of md5-4e872368b2b2668460205b409e95c2ea

Kishore Nallan

03:34 AM

👋 This is fixed in the most recent 0.25 RC build (0.25.0.rc44)

Mojan

03:38 AM

I'm using Typesense Python library. Is 0.25 released yet ?

Kishore Nallan

03:39 AM

I'm talking about the server version. We are in the last stretch before release of 0.25

03:39

Kishore Nallan

03:39 AM

You can use the rc build, it's stable

Mojan

03:39 AM

Thank you, Kishore .

07:00

Mojan

07:00 PM

Hey Kishore,

I am trying the RC44 build and I can confirm it still falls over when exporting documents.
Which branch is it on GH ? I can take a look to see the integration.

Kishore Nallan

10:46 PM

I identified one more issue related to this and fixed it again. Can you try on RC45? I've tested this on a large 12GB dataset

10:48

Kishore Nallan

10:48 PM

And when you mean by fall over, what exactly happens? The issue I fixed is limiting the memory used during export. Previously it was using a huge buffer so it ate up a lot of memory.

Jul 07, 2023 (3 months ago)

Mojan

02:56 AM

Thanks for the update. I tried RC45 but it still not works. So what happens is that the connection gets closed. I'm going to attach screenshots of the error for you below :
P.S. I'm using Docker.

Image 1 for Thanks for the update. I tried RC45 but it still not works. So what happens is that the connection gets closed. I'm going to attach screenshots of the error for you below :
P.S. I'm using Docker.

Image 2 for Thanks for the update. I tried RC45 but it still not works. So what happens is that the connection gets closed. I'm going to attach screenshots of the error for you below :
P.S. I'm using Docker.

Kishore Nallan

02:58 AM

What's your client side timeout?

03:02

Kishore Nallan

03:02 AM

After how long does this happen? Can you also post the code snippet for export? I will try it out.

Mojan

06:03 PM

My client-side timeout is 120000.

06:04

Mojan

06:04 PM

The error happens after about 10 minutes.

 indexed_pages = client.collections['pages'].documents.export({'batch_size': 10})

06:10

Mojan

06:10 PM

I am running this on a table of 16.41 Gb, btw.

Jul 08, 2023 (3 months ago)

Kishore Nallan

12:47 AM

I tested exporting a 11G dataset on this build the other day. I ran using curl which has no timeouts. Can you try once via curl and without using the batch size option?

Jul 10, 2023 (2 months ago)

Mojan

07:36 PM

Hi Kishore,

The curl for export seems to be working fine.

Jul 11, 2023 (2 months ago)

Kishore Nallan

12:58 AM

Then I think the issue is with the python client.

Typesense

Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI

Indexed 2779 threads (79% resolved)

Join Our Community Manage subscription

Similar Threads

Troubleshooting Write Timeouts in Typesense with Large CSVs

Agustin had issues with Typesense getting write timeouts while loading large CSV files. Kishore Nallan suggested chunking data or converting to JSONL before loading. Through troubleshooting, they identified a possible network problem at AWS and found a workaround.

29mo

Resolving JSONL File Import Issues in Python

Jon struggles importing a large JSONL file using Python, encountering decode errors and size restrictions. Kishore Nallan instructs to use curl for imports under 10GB, and references an update to the Python client which could more capably handle large imports.

9mo

Issues with Importing Typesense Collection to Different Server

Kevin had problems migrating a Typesense collection between Docusaurus sites on different machines. Jason advised them on JSONL format, handling server hosting, and creating a collection schema before importing documents, leading to successful import.

1mo

Bulk Import 50MB JSON Files Error - Timeout and Solutions

madhweep encounters an error while bulk importing JSON files. Kishore Nallan provided help, but the issue persists. Jason intervenes and after troubleshooting, they concluded the cluster had run out of memory causing the issue. The problem was resolved by using a cluster with sufficient memory. Daniel also experienced a similar issue, resolved by increasing the timeout.

24mo

Resolving Typesense Documents Import Error

Aljosa experienced an error while using typesense `documents().import()`, related to handling of large document arrays. Jason clarified that batch_size controls server-side batching, not client-side. He advised splitting arrays to address the issue and committed to elaborating its functionality in the docs. Aljosa proposed amending the TypeScript types to accommodate batch_size in the import options.

24mo