#community-help

Resolving JSONL File Import Issues in Python

TLDR Jon struggles importing a large JSONL file using Python, encountering decode errors and size restrictions. Kishore Nallan instructs to use curl for imports under 10GB, and references an update to the Python client which could more capably handle large imports.

9mo

Join the chat

Jan 08, 2023 (9 months ago)

Photo of md5-be53735a2b0297bb542711c1d2ecea45

Jon

04:50 PM

Hi, I am trying to import a large jsonl file. i believe the format of the jsonl file is correct. i am getting the following json decode error after a period of time:

Jan 09, 2023 (9 months ago)

Photo of md5-4e872368b2b2668460205b409e95c2ea

Kishore Nallan

01:06 AM

Have you tried importing the file directly via curl to see if the same error happens?

Jon

01:54 AM

Is there documentation on that? I thought the docs say that using import is most efficient.

01:55

Jon

01:55 AM

Should I use the API and insert one document at a time or use CURL - but where is the documentation

02:10

Jon

02:10 AM

Using CURL I get : request entity is too large

02:10

Jon

02:10 AM

These are 27GB files in Jason’s

Kishore Nallan

04:20 AM

Import is the most efficient way. Are you using the official Python client? That also might be running into some memory limitations for the large file. You have to try splitting up the file into smaller chunks.

04:32

Kishore Nallan

04:32 AM

Someone recently also contributed a way to batch import via the use of Iterable here: https://github.com/typesense/typesense-python/pull/22

This is available in 0.15.0 version of the Python client that I've just published.

Jan 16, 2023 (9 months ago)

Jon

05:57 PM

Ok thanks. Will look into this

Jan 17, 2023 (9 months ago)

Jon

03:19 PM

i don't see why i should have to split things. i am using the python client. it seems like it's a bug in your code and makes this database non-actionable

Kishore Nallan

03:24 PM

The new API added to the python client above allows you to just pass a file object. You can consume pretty large imports that way.

03:25

Kishore Nallan

03:25 PM

We index JSONL files that are several GBs in size.

Jon

03:25 PM

what do you mean new API? i am using the python client. is there something else i need to be doing.

03:25

Jon

03:25 PM

i am using jsonl

Kishore Nallan

03:29 PM

The earlier error message you posted here seems like an issue with parsing of the JSON, either the request or the response.

03:29

Kishore Nallan

03:29 PM

https://typesense-community.slack.com/archives/C01P749MET0/p1673196662855139

03:32

Kishore Nallan

03:32 PM

Typesense server also has a maximum POST body size of 10 GB.

03:34

Kishore Nallan

03:34 PM

The Python client needs to hold the file content in-memory while calling the API. This might not work for large datasets. But curl will work fine as long as POST data is less than 10 GB. So if your total dataset size is 28 GB, you will need to split into 3 files.

Typesense

Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI

Indexed 2779 threads (79% resolved)

Join Our Community Manage subscription

Similar Threads

Issues with Importing Typesense Collection to Different Server

Kevin had problems migrating a Typesense collection between Docusaurus sites on different machines. Jason advised them on JSONL format, handling server hosting, and creating a collection schema before importing documents, leading to successful import.

1mo

Troubleshooting Write Timeouts in Typesense with Large CSVs

Agustin had issues with Typesense getting write timeouts while loading large CSV files. Kishore Nallan suggested chunking data or converting to JSONL before loading. Through troubleshooting, they identified a possible network problem at AWS and found a workaround.

30mo

Bulk Import 50MB JSON Files Error - Timeout and Solutions

madhweep encounters an error while bulk importing JSON files. Kishore Nallan provided help, but the issue persists. Jason intervenes and after troubleshooting, they concluded the cluster had run out of memory causing the issue. The problem was resolved by using a cluster with sufficient memory. Daniel also experienced a similar issue, resolved by increasing the timeout.

24mo

Troubleshooting Typesense Document Import Error

Christopher had trouble importing 2.1M documents into Typesense due to memory errors. Jason clarified the system requirements, explaining the correlation between RAM and dataset size, and ways to tackle the issue. They both also discussed database-like query options.

8mo

Discussion on Document Inserting Speed and Process

David inquired about document insertion speed, and Jason provided reference values and recommended sending more documents per API call. Both David and Chetan acknowledged the suggestions, with David stating to report back on their experience.

2mo