#community-help

Estimating JSON Weight in .documents().export() Method

TLDR gab experiences crashes in a docker container during export and suspects a memory issue. Kishore Nallan suggests trying on 0.24.0 docker, but gab later discovers it was another task crashing the container.

10mo

Join the chat

Feb 07, 2023 (10 months ago)

Photo of md5-8be2a24edf7a95c9c74abce4b1130c3e

gab

11:35 AM

Hi,
is there a way to estimate the weight of the JSON returned by the .documents().export() method ?
I'm running it in a docker container that is crashing doing the export. I cannot catch and log anything so I suspect a memory issue. I'm running a container with 8Go of allocated memory and wonder if the export would not exceed it.

Photo of md5-4e872368b2b2668460205b409e95c2ea

Kishore Nallan

11:41 AM

We recently made an optimization for export that uses too much memory. There is already a fix that will be going out in a 0.24.1 patch release this week.

11:42

Kishore Nallan

11:42 AM

I can share a fixed temporary build if you can tell me how you are running Typesense.

gab

11:46 AM

I have an instance in typesense cloud running v0.23.0.rc27
I'm using Typesense package v1.4.5 for executing the export
I haven' checked yet what would be breaking going to 0.24 :thinking_face:

Kishore Nallan

11:47 AM

Oh then this issue is not related to that change.

11:48

Kishore Nallan

11:48 AM

Can you try on 0.24.0 docker first? Since you already use it locally.

11:48

Kishore Nallan

11:48 AM

Or are you saying that you are running the export script inside Docker?

gab

11:49 AM

yes I'm running the export script inside Docker running in Google cloud run

11:51

gab

11:51 AM

But you might be right, I should try somehow to do the export locally targeting my remote instance

Feb 08, 2023 (10 months ago)

gab

08:38 AM

I found that was not related at all from the weight of this request but it was because of another asynchronous task that was silently crashing the container during the export.
thanks for the help

Typesense

Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI

Indexed 3005 threads (79% resolved)

Join Our Community

Similar Threads

Trouble with Typesense Memory Usage when Restarting Docker Container

Blend reports increased memory usage when restarting Docker with Typesense, information shared with Jason and Kishore Nallan. Potential data edge case identified as potential cause, although resolution undetermined.

12mo

Using Typesense in Docker Container – Importing JSONL File

Hakim faced 'Empty reply from server' error when importing a JSONL file into a Typesense-Docker container. Kishore Nallan and Jason suggested checking the docker logs and increasing default RAM and CPU allocation. Finally, Hakim successfully indexed the documents using a PHP script.

16mo

Troubleshooting Document Import Issue with Typesense on Docker

Harrison experienced an issue with Typesense getting stuck when importing JSONL documents. Harrison offered to provide a copy of the docker volume to aid troubleshooting, but failed to retrieve it. Kishore Nallan recommended observing the local sequence number in the logs.

19mo

Troubleshooting Typesense Document Import Error

Christopher had trouble importing 2.1M documents into Typesense due to memory errors. Jason clarified the system requirements, explaining the correlation between RAM and dataset size, and ways to tackle the issue. They both also discussed database-like query options.

10mo

Issues with Importing Typesense Collection to Different Server

Kevin had problems migrating a Typesense collection between Docusaurus sites on different machines. Jason advised them on JSONL format, handling server hosting, and creating a collection schema before importing documents, leading to successful import.

3mo