Resolving Out of Memory Issue with Cluster
TLDR Ed encountered an OUT_OF_MEMORY error while using typesense. Jason suggested increasing the cluster's RAM, accounting for the OS, which resolved the issue.
Sep 16, 2023 (2 weeks ago)
Ed
08:57 PM"message": "Rejecting write: running out of resource type: OUT_OF_MEMORY"}
{'num_deleted': 0}
Traceback (most recent call last):
File "/Users/edondurguti/learning/tke_jobboard/typesense_upload.py", line 170, in <module>
upload_and_delete_typesense()
File "/Users/edondurguti/learning/tke_jobboard/typesense_upload.py", line 151, in upload_and_delete_typesense
upsert_docs, ids = import_and_upsert_documents(
File "/Users/edondurguti/learning/tke_jobboard/typesense_upload.py", line 83, in import_and_upsert_documents
upsert = client.collections[collection_name].documents.import_(
File "/Users/edondurguti/learning/tke_jobboard/.venv/lib/python3.10/site-packages/typesense/documents.py", line 94, in import_
api_response = (self._endpoint_path('import'), documents, params, as_json=False)
File "/Users/edondurguti/learning/tke_jobboard/.venv/lib/python3.10/site-packages/typesense/api_call.py", line 153, in post
return self.make_request(, endpoint, as_json,
File "/Users/edondurguti/learning/tke_jobboard/.venv/lib/python3.10/site-packages/typesense/api_call.py", line 116, in make_request
raise ApiCall.get_exception(r.status_code)(r.status_code, error_message)
typesense.exceptions.ObjectUnprocessable: [Errno 422] Rejecting write: running out of resource type: OUT_OF_MEMORY 
Jason
08:58 PMEd
08:59 PMEd
08:59 PMEd
09:00 PM
Ed
09:01 PMdef upload_and_delete_typesense():
client = create_client()
# new_collection_name = "amso_zh"
for lang in fe_languages:
new_collection_name = f"tke_{lang}"
upsert_docs, ids = import_and_upsert_documents(
client, f"algolia_jobs-{lang}.json", new_collection_name
)
deleted = delete_document(client, new_collection_name, ids)
pprint(deleted)Ed
09:02 PM
Ed
09:06 PMEd
09:08 PMEd
09:21 PMdef import_and_upsert_documents(client, json_file, collection_name):
# Open and read the JSON data from the file
with open(json_file) as f:
json_data = json.load(f)
# get all 'id' fields
ids = [
doc["data"]["idClient"] for doc in json_data
] # we are doing this here since we already opened the file
# Convert list of JSON objects to JSONL format
jsonl_data = "\n".join(json.dumps(doc) for doc in json_data)
# Import documents into the collection
upsert = client.collections[collection_name].documents.import_(
jsonl_data.encode("utf-8"), {"action": "upsert"}
)
return upsert, idsJason
10:39 PMEd
10:50 PMJason
10:51 PMEd
10:57 PMEd
10:57 PM
Typesense
Indexed 2779 threads (79% resolved)
Similar Threads
Typesense Cloud Cluster Out of Memory Issue
Herman experienced an OUT_OF_MEMORY error after updating their Typesense schema. Jason suggested upgrading to 1GB RAM to accommodate the indexed data.
Typesense Cluster Memory Upgrade
Sruli had a memory issue in their Typesense cluster. Jason suggested upgrading to resolve the issue and helped estimate the necessary RAM upgrade. The solution was successful and the cluster is functioning properly.
Memory Usage Error with Typesense Cloud Service
Jeff.Cheng asked about an out-of-memory error with Typesense Cloud service. Kishore Nallan clarified how total memory usage is calculated, leading Jeff.Cheng to consider upgrading memory resources.
Error Resolution: OUT_OF_DISK in Typesense Cloud
Mischa had a OUT_OF_DISK error. Jason suggested upgrading the cluster's RAM to increase disk space. Mischa has requested an upgrade.
Issues with Typesense API: RAM Upgrade and Error 422
Laura encountered Error 422 while submitting products to typesense. Jason suggested checking the response body, upgrading RAM, and checking for schema issues. Error resolved after upgrading RAM.