Estimating JSON Weight in .documents().export() Method
TLDR gab experiences crashes in a docker container during export and suspects a memory issue. Kishore Nallan suggests trying on 0.24.0 docker, but gab later discovers it was another task crashing the container.
Feb 07, 2023 (10 months ago)
is there a way to estimate the weight of the JSON returned by the .documents().export() method ?
I'm running it in a docker container that is crashing doing the export. I cannot catch and log anything so I suspect a memory issue. I'm running a container with 8Go of allocated memory and wonder if the export would not exceed it.
Kishore Nallan11:41 AM
Kishore Nallan11:42 AM
I'm using Typesense package v1.4.5 for executing the export
I haven' checked yet what would be breaking going to 0.24 :thinking_face:
Kishore Nallan11:47 AM
Kishore Nallan11:48 AM
Kishore Nallan11:48 AM
Feb 08, 2023 (10 months ago)
thanks for the help
Indexed 3005 threads (79% resolved)
Trouble with Typesense Memory Usage when Restarting Docker Container
Blend reports increased memory usage when restarting Docker with Typesense, information shared with Jason and Kishore Nallan. Potential data edge case identified as potential cause, although resolution undetermined.
Using Typesense in Docker Container – Importing JSONL File
Hakim faced 'Empty reply from server' error when importing a JSONL file into a Typesense-Docker container. Kishore Nallan and Jason suggested checking the docker logs and increasing default RAM and CPU allocation. Finally, Hakim successfully indexed the documents using a PHP script.
Troubleshooting Document Import Issue with Typesense on Docker
Harrison experienced an issue with Typesense getting stuck when importing JSONL documents. Harrison offered to provide a copy of the docker volume to aid troubleshooting, but failed to retrieve it. Kishore Nallan recommended observing the local sequence number in the logs.
Troubleshooting Typesense Document Import Error
Christopher had trouble importing 2.1M documents into Typesense due to memory errors. Jason clarified the system requirements, explaining the correlation between RAM and dataset size, and ways to tackle the issue. They both also discussed database-like query options.
Issues with Importing Typesense Collection to Different Server
Kevin had problems migrating a Typesense collection between Docusaurus sites on different machines. Jason advised them on JSONL format, handling server hosting, and creating a collection schema before importing documents, leading to successful import.