#community-help

Using Typesense in Docker Container – Importing JSONL File

TLDR Hakim faced 'Empty reply from server' error when importing a JSONL file into a Typesense-Docker container. Kishore Nallan and Jason suggested checking the docker logs and increasing default RAM and CPU allocation. Finally, Hakim successfully indexed the documents using a PHP script.

Powered by Struct AI
9
14mo
Solved
Join the chat
Aug 26, 2022 (14 months ago)
Hakim
Photo of md5-a7605b093448e06efb0a1c150153cad6
Hakim
12:11 PM
hello,
I'm using Typesense in a docker container. and I wanted to import a jsonl file (5M lines)
i used that curl call curl -H "X-TYPESENSE-API-KEY: xyz" -X POST --data-binary @insee_current_insee.jsonl "<http://localhost:8108/collections/companies/document>
s/import?action=create"
but after around 20 minutes I got this error : curl: (52) Empty reply from server
any idea what the problem might be ?
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
12:16 PM
Hi, can you check the docker container logs
Hakim
Photo of md5-a7605b093448e06efb0a1c150153cad6
Hakim
12:23 PM
yes sure
those are the last lines of docker logs
I20220826 10:31:38.936184   101 raft_server.cpp:534] Term: 5, last_index index: 6277, committed_index: 6277, known_applied_index: 6277, applying_index: 0, queued_writes: 1493, pending_queue_size: 0, local_seq
uence: 24922                                                                                                                                                                                                    
E20220826 10:31:38.970467   101 raft_server.cpp:612] 1493 queued writes > healthy read lag of 1000                                                                                                              
I20220826 10:31:38.970467   125 raft_server.h:60] Peer refresh succeeded!                                                                                                                                       
E20220826 10:31:38.986366   101 raft_server.cpp:624] 1493 queued writes > healthy write lag of 500
12:40
Hakim
12:40 PM
the docker got killed. I run another one, and now I get { "message": "Not Ready or Lagging"}
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
03:55 PM
Not Read or Lagging happens when there is a big backlog of writes and there isn’t enough CPU capacity to handle the writes. So Typesense applies backpressure to prevent further saturation
03:55
Jason
03:55 PM
re: container crashing, could you make sure you’ve increased the default RAM and CPU allocation that Docker gives containers?
Aug 29, 2022 (14 months ago)
Hakim
Photo of md5-a7605b093448e06efb0a1c150153cad6
Hakim
08:04 AM
i can't, the server has limited resources. I manager to index my 5M documents with a php script though
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
06:49 PM
Typesense is an in-memory data store, so you’d need sufficient RAM (2x-3x the size of your dataset) and at least 2vCPUs for it to work reliably.
Aug 30, 2022 (14 months ago)
Hakim
Photo of md5-a7605b093448e06efb0a1c150153cad6
Hakim
06:49 AM
Ok thank you for the information