Resolving JSON Parsing Error in Import Function Implementation
TLDR Harpreet experienced a JSON parsing failure while implementing an import function. Kishore Nallan suggested double escaping characters in the JSON objects. After testing and discussion, both agreed that the error resulted from not handling JSON-encoded strings manually. Harpreet decided to update the test cases accordingly.
1
Jun 07, 2021 (30 months ago)
Harpreet
01:58 PM<http://localhost:8108/collections/companies/documents/import?action=create>
returns
{"code":400,"document":"{\"id\": \"124\",\"company_name\": \"Stark Industries\",\"num_employees\": \"5215\",\"country\": \"USA\"}","error":"Field `num_employees` must be an int32.","success":false}
for the request body
{"id": "124","company_name": "Stark Industries","num_employees": "5215","country": "USA"}
which is fine except it is not valid JSON. I'm just implementing import function and the code is supposed to throw ImportError but if fails before that due to parsing failure. It should return something like this:
{
"code": 400,
"document": {
"id": "124",
"company_name": "Stark Industries",
"num_employees": "5215",
"country": "USA"
},
"error": "Field `num_employees` must be an int32.",
"success": false
}
Kishore Nallan
02:01 PMHarpreet
02:02 PMKishore Nallan
02:03 PM{"code":400,"document":"{\"id\": \"124\",\"company_name\": \"Stark Industries\",\"num_employees\": \"5215\",\"country\": \"USA\"}","error":"Field `num_employees` must be an int32.","success":false}
is valid JSON.
Kishore Nallan
02:04 PMHarpreet
02:09 PMHarpreet
02:09 PMFormatException: Unexpected character (at character 27)
{"code":400,"document":"{"id": "124","company_name": "Stark Industries","nu...
Harpreet
04:33 PMTried doing this is JS too:
const resultsInJSONLFormat = '{"code":400,"document":"{\"id\": \"124\",\"company_name\": \"Stark Industries\",\"num_employees\": \"5215\",\"country\": \"USA\"}","error":"Field `num_employees` must be an int32.","success":false}'
const resultsInJSONFormat = resultsInJSONLFormat.split('\n').map(r => JSON.parse((r)))
console.log(resultsInJSONFormat)
which results in a similar error:
SyntaxError: JSON.parse: expected ',' or '}' after property value in object at line 1 column 27 of the JSON data
Harpreet
04:36 PM\
should be escaped as well in the nested objectJun 08, 2021 (30 months ago)
Kishore Nallan
03:58 AMvar str = '{"code":400,"error":"Field `num_employees` must be an int32.","success":false, "document": "{\\"id\\": 100}"}'
JSON.parse(str)
Kishore Nallan
03:58 AMvar str = '{"code":400,"error":"Field `num_employees` must be an int32.","success":false, "document": "{\"id\": 100}"}'
JSON.parse(str)
Uncaught SyntaxError: Unexpected token i in JSON at position 94
Harpreet
04:47 AMHarpreet
04:49 AMKishore Nallan
06:11 AMHarpreet
06:26 AMKishore Nallan
10:16 AMvar fs = require('fs');
var file_path = '/tmp/obj.json';
var str = '{"age":1}';
var obj = {};
obj['person'] = str;
fs.writeFileSync(file_path, JSON.stringify(obj));
var objstr = fs.readFileSync(file_path, {encoding:'utf8', flag:'r'});
var obj2 = JSON.parse(objstr)
console.log(obj2);
Kishore Nallan
10:17 AM$ cat /tmp/obj.json
{"person":"{\"age\":1}"}%
Kishore Nallan
10:20 AMimport json
f = open('/tmp/obj.json', 'r')
str = f.read()
obj = json.loads(str)
Contents of
/tmp/obj.json
% cat /tmp/obj.json
{"person":"{\"age\":1}"}
1
Harpreet
12:31 PMimport 'dart:convert';
void main() {
final str = '{"age": 1}',
obj = {'person': str},
encoded = json.encode(obj);
print('original: $obj');
print('encoded object: $encoded');
print('decoded object: ${json.decode(encoded)}');
final myEncoded = '{"person":"{\"age\": 1}"}';
print(json.decode(myEncoded));
}
The first three
print
work fine and output:original: {person: {"age": 1}}
encoded object: {"person":"{\"age\": 1}"}
decoded object: {person: {"age": 1}}
The last
print
causes the following exception:Unhandled exception:
FormatException: Unexpected character (at character 14)
{"person":"{"age": 1}"}
^
Seems like I cannot pass in the encoded string manually. Talk about being fair :3
Harpreet
12:31 PMKishore Nallan
12:33 PMHarpreet
12:33 PM"
Harpreet
12:34 PMKishore Nallan
12:34 PMHarpreet
12:36 PMKishore Nallan
12:36 PMTypesense
Indexed 2779 threads (79% resolved)
Similar Threads
Large JSONL Documents Import Issue & Resolution
Suraj was having trouble loading large JSONL documents into Typesense server. After several discussions and attempts, it was discovered that the issue was due to data quality. Once the team extracted the data again, the upload process worked smoothly.
Resolving Node.js Limitation in Loading Data to Cloud Cluster
Ethan was having trouble loading data into a cloud cluster due to a Node.js error. Jason identified the issue and suggested reading the file in a streaming fashion in chunks.
Issues with Importing Typesense Collection to Different Server
Kevin had problems migrating a Typesense collection between Docusaurus sites on different machines. Jason advised them on JSONL format, handling server hosting, and creating a collection schema before importing documents, leading to successful import.
Resolving Special Character Search Errors
suraj had issues searching for data containing special characters. Kishore Nallan resolved the issue by advising suraj to remove the parameters for 'preSegmentedQuery' and 'tokenSeparators'.
Discussing TypeSense Setup & Documentation Improvements
Harrison faced challenges with TypeSense setup and documentation. Jason clarified reasons behind the method used for handling bulk imports and explained the HTTP response. They acknowledged Harrison's points and agreed about improving the documentation.