#community-help

Resolving Typesense Server Upload Issues Due to Insufficient Memory

TLDR Daniel had issues uploading data to a Typesense server. After recreating the server and encountering the same issue, Jason identified the problem as insufficient memory capacity.

Powered by Struct AI
Nov 16, 2021 (26 months ago)
Daniel
Photo of md5-3e862430ae06b87120e3640d3f9f8061
Daniel
03:00 AM
I'm using the correct key

./typesense-server --data-dir=/tmp/typesense-data --api-key=samplekey95

client = typesense.Client({
  'nodes': [{ 'host': 'localhost', 'port': '8108', 'protocol': 'http' }],
  'api_key': 'samplekey95'
});
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
06:08 AM
Are you able to access the API via curl?
Daniel
Photo of md5-3e862430ae06b87120e3640d3f9f8061
Daniel
05:59 PM
Didn't try
06:00
Daniel
06:00 PM
I deleted the server and created a new one
06:00
Daniel
06:00 PM
Now it works, currently waiting for 7GB of 44 million rows to be uploaded
06:01
Daniel
06:01 PM
Sadly, it just got Killed by the OS
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
06:03 PM
It sounds like you might be running out of memory...
Daniel
Photo of md5-3e862430ae06b87120e3640d3f9f8061
Daniel
06:03 PM
06:03
Daniel
06:03 PM
Yeah, it has 8GB of RAM only
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
06:03 PM
You'd usually need 2x-3x RAM compared to the size of your dataset, since Typesense is an in-memory store: https://typesense.org/docs/guide/system-requirements.html#choosing-ram
Daniel
Photo of md5-3e862430ae06b87120e3640d3f9f8061
Daniel
06:05 PM
Thanks
06:06
Daniel
06:06 PM
I will try with another dataset of 2.6GB
06:06
Daniel
06:06 PM
Not sure if it will hold. but I'll try

Typesense

Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI

Indexed 3015 threads (79% resolved)

Join Our Community

Similar Threads

Troubleshooting Typesense Document Import Error

Christopher had trouble importing 2.1M documents into Typesense due to memory errors. Jason clarified the system requirements, explaining the correlation between RAM and dataset size, and ways to tackle the issue. They both also discussed database-like query options.

3

30
11mo

Problem with Loading Records on Typesense Server

Rafael had an issue with Typesense server loop during startup after loading 8M of 15M records. Kishore Nallan identified the issue as memory shortage and advised Rafael to allocate more server memory.

20
17mo

Importing data to Typesense Cloud with JS client

Zaiste attempted to import data to Typesense Cloud but encountered an error. Kishore Nallan explained the error was due to server side backpressure and suggested a curl approach for the import.

4

13
25mo

Bulk Import 50MB JSON Files Error - Timeout and Solutions

madhweep encounters an error while bulk importing JSON files. Kishore Nallan provided help, but the issue persists. Jason intervenes and after troubleshooting, they concluded the cluster had run out of memory causing the issue. The problem was resolved by using a cluster with sufficient memory. Daniel also experienced a similar issue, resolved by increasing the timeout.

4

21
26mo

Troubleshooting Write Timeouts in Typesense with Large CSVs

Agustin had issues with Typesense getting write timeouts while loading large CSV files. Kishore Nallan suggested chunking data or converting to JSONL before loading. Through troubleshooting, they identified a possible network problem at AWS and found a workaround.

2

59
32mo