Imputing Documents with Null Array Data Types in Typesense
TLDR Vishal encountered a "Null" data issue when inserting documents with NaN values in string[] or object[]. Jason recommended using version 0.25.0.rc35
and checking the JSON lines conversion. Issue resolved.
Jun 09, 2023 (4 months ago)
Vishal
10:26 PMJason
10:28 PM0.25.0.rc35
?Vishal
10:29 PMJason
10:30 PMVishal
10:31 PMJason
10:33 PM0.25.0.rc35
, since it has a couple more fixesVishal
10:34 PMVishal
10:47 PMVishal
10:48 PMJason
10:48 PMVishal
10:49 PMVishal
10:49 PMVishal
10:50 PMJason
10:51 PMnull
in the JSON document, it should not error outVishal
10:54 PMJun 12, 2023 (3 months ago)
Vishal
02:23 AM#Write pandas dataframe to jsonlines format in local path
with open('test.jsonl', "w") as f:
f.write(df.to_json(orient='records', lines=True))
#Read from jsonlines file and bulk import to typesense server
with open('test.jsonl', 'r', encoding='utf8') as file:
documents = jsonlines.Reader(file)
print(client.collections['collection_name'].documents.import_(documents, {'batch_size':1000, 'action': 'create', 'dirty_values': "coerce_or_drop"}))
Vishal
02:23 AMTypesense
Indexed 2779 threads (79% resolved)
Similar Threads
Resolving Error with Incorrect Field Type during Schema Update
Samuel encountered an error updating the schema for an existing collection. Kishore Nallan suggested specifying a concrete type, and later found inconsistent data within the collection that caused the error.
Cold Start Problem with Dynamic Collections
Adrian reported cold start issues with dynamic collections. Jason suggested using wildcard `*` for query_by parameters, upgrading to `0.25.0.rc34`, and clarified conventions. Adrian's issues were resolved but they reported a limitation that will potentially be addressed.
Issues with Importing Typesense Collection to Different Server
Kevin had problems migrating a Typesense collection between Docusaurus sites on different machines. Jason advised them on JSONL format, handling server hosting, and creating a collection schema before importing documents, leading to successful import.
Troubleshooting Invalid Field Error in Firestore Document Indexing
Darren receives an error when indexing Firestore documents with empty array in "_grades" field. Jason suggests submitting a bug report and manually setting the schema. The user still experiences issues. Kishore Nallan reproduces the bug, but suggests a solution might exist with an explicit 'string[]' type definition. Further investigation is needed.
Troubleshooting Typesense 400 Error with Null Fields
sonu was encountering a 400 error due to null fields during Typesense indexing. Jason suggested upgrading to a recent version, removing null fields, and marking potentially null fields as `optional: true`. gab shared a similar issue and workaround.