#contributions

Resolving JSON Parsing Error in Import Function Implementation

TLDR Harpreet experienced a JSON parsing failure while implementing an import function. Kishore Nallan suggested double escaping characters in the JSON objects. After testing and discussion, both agreed that the error resulted from not handling JSON-encoded strings manually. Harpreet decided to update the test cases accordingly.

Powered by Struct AI

1

26
30mo
Solved
Join the chat
Jun 07, 2021 (30 months ago)
Harpreet
Photo of md5-745d880d794220d9f0fb9ade17c6b861
Harpreet
01:58 PM
The import endpoint <http://localhost:8108/collections/companies/documents/import?action=create>
returns
{"code":400,"document":"{\"id\": \"124\",\"company_name\": \"Stark Industries\",\"num_employees\": \"5215\",\"country\": \"USA\"}","error":"Field `num_employees` must be an int32.","success":false}

for the request body
{"id": "124","company_name": "Stark Industries","num_employees": "5215","country": "USA"}

which is fine except it is not valid JSON. I'm just implementing import function and the code is supposed to throw ImportError but if fails before that due to parsing failure. It should return something like this:

{
  "code": 400,
  "document": {
    "id": "124",
    "company_name": "Stark Industries",
    "num_employees": "5215",
    "country": "USA"
  },
  "error": "Field `num_employees` must be an int32.",
  "success": false
}

Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
02:01 PM
Import end point returns JSON lines as response, where each JSON line records the result of the import operation for the corresponding import record. (i.e. line 1 of response matches line 1 of request and so on)
Harpreet
Photo of md5-745d880d794220d9f0fb9ade17c6b861
Harpreet
02:02 PM
Yep but the response is not valid JSON
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
02:03 PM
I don't follow you.
{"code":400,"document":"{\"id\": \"124\",\"company_name\": \"Stark Industries\",\"num_employees\": \"5215\",\"country\": \"USA\"}","error":"Field `num_employees` must be an int32.","success":false}

is valid JSON.
02:04
Kishore Nallan
02:04 PM
I checked the validity here: https://jsonformatter.curiousconcept.com/
Harpreet
Photo of md5-745d880d794220d9f0fb9ade17c6b861
Harpreet
02:09 PM
Yep just checked it myself, it's valid json
02:09
Harpreet
02:09 PM
Strange that the json decoder I'm using fails to parse it
FormatException: Unexpected character (at character 27)
{"code":400,"document":"{"id": "124","company_name": "Stark Industries","nu...
04:33
Harpreet
04:33 PM
Jason Could you help me understand this?
Tried doing this is JS too:
const resultsInJSONLFormat = '{"code":400,"document":"{\"id\": \"124\",\"company_name\": \"Stark Industries\",\"num_employees\": \"5215\",\"country\": \"USA\"}","error":"Field `num_employees` must be an int32.","success":false}'

const resultsInJSONFormat = resultsInJSONLFormat.split('\n').map(r => JSON.parse((r)))

console.log(resultsInJSONFormat)

which results in a similar error:
SyntaxError: JSON.parse: expected ',' or '}' after property value in object at line 1 column 27 of the JSON data
04:36
Harpreet
04:36 PM
Okay found the mistake the \ should be escaped as well in the nested object
Jun 08, 2021 (30 months ago)
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
03:58 AM
Harpreet Double escaping seems to work for me. E.g.

var str = '{"code":400,"error":"Field `num_employees` must be an int32.","success":false, "document": "{\\"id\\": 100}"}'

JSON.parse(str)
03:58
Kishore Nallan
03:58 AM
However, the single backslash version throws that error when parsed:

var str = '{"code":400,"error":"Field `num_employees` must be an int32.","success":false, "document": "{\"id\": 100}"}'

JSON.parse(str)
Uncaught SyntaxError: Unexpected token i in JSON at position 94
Harpreet
Photo of md5-745d880d794220d9f0fb9ade17c6b861
Harpreet
04:47 AM
Kishore Nallan Yes. I believe you'll be updating the server response?
04:49
Harpreet
04:49 AM
I got this response from the latest docker image.
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
06:11 AM
Yes, I will fix the server response. Are you saying that the JS client is broken at the moment because of this issue? i.e. when the import has bad data, the JS client fails with JSON parsing error?
Harpreet
Photo of md5-745d880d794220d9f0fb9ade17c6b861
Harpreet
06:26 AM
nah I was implementing a unit test for import function in dart client and found this error. I'll just update my test case to have response with double escaped quotes and assume that's how the response will be in the future.
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
10:16 AM
Harpreet Actually, I think the current behaviour of single backslash is correct. Try this on Node:

var fs = require('fs');
var file_path = '/tmp/obj.json';

var str = '{"age":1}';
var obj = {};
obj['person'] = str;

fs.writeFileSync(file_path, JSON.stringify(obj));

var objstr = fs.readFileSync(file_path, {encoding:'utf8', flag:'r'});
var obj2 = JSON.parse(objstr)

console.log(obj2);
10:17
Kishore Nallan
10:17 AM
The generated file looks like this:

$ cat /tmp/obj.json
      {"person":"{\"age\":1}"}%
10:20
Kishore Nallan
10:20 AM
Likewise on Python:

import json
f = open('/tmp/obj.json', 'r')
str = f.read()
obj = json.loads(str)

Contents of /tmp/obj.json

% cat /tmp/obj.json
{"person":"{\"age\":1}"}

1

Harpreet
Photo of md5-745d880d794220d9f0fb9ade17c6b861
Harpreet
12:31 PM
Now I understand why an exception was being raised.
import 'dart:convert';

void main() {
  final str = '{"age": 1}',
        obj = {'person': str},
        encoded = json.encode(obj);
        
  print('original: $obj');
  print('encoded object: $encoded');
  print('decoded object: ${json.decode(encoded)}');

  final myEncoded = '{"person":"{\"age\": 1}"}';
  print(json.decode(myEncoded));
}

The first three print work fine and output:
original: {person: {"age": 1}}
encoded object: {"person":"{\"age\": 1}"}
decoded object: {person: {"age": 1}}

The last print causes the following exception:
Unhandled exception:
FormatException: Unexpected character (at character 14)
{"person":"{"age": 1}"}
             ^

Seems like I cannot pass in the encoded string manually. Talk about being fair :3
12:31
Harpreet
12:31 PM
Same with JavaScript as well
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
12:33 PM
Oh wait, scratch that.
Harpreet
Photo of md5-745d880d794220d9f0fb9ade17c6b861
Harpreet
12:33 PM
the single quotes is just syntactic sugar iirc, when the vm deals with the strings it converts them to "
12:34
Harpreet
12:34 PM
I think something similar is happening in js
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
12:34 PM
I'm not familiar with Dart, but do you understand what's happening or do you need any further help here?
Harpreet
Photo of md5-745d880d794220d9f0fb9ade17c6b861
Harpreet
12:36 PM
Nope. I'll handle it. Thank you for your time 🙂
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
12:36 PM
👍