#community-help

Resolving Typesense 0.22.2 Syntax Error and Import Issues

TLDR John faced issues with Typesense 0.22.2 code syntax and import functionality. Jason advised to reformat the JSONL records into single lines, increase ulimit, and convert dates to int64 format manually in code. A typo in the code was also corrected.

Powered by Struct AI

7

1

Apr 05, 2022 (21 months ago)
John
Photo of md5-f63b27ab4e62d7f1c7de93f1da7516f7
John
07:08 PM
Write some code that is causing me problems, Typesense 0.22.2
I have raised a ticket #568
but main issue is I don’t know which records are supposed to be the “problem”
E20220405 18:49:30.707577 15581 collection.cpp:77] JSON error: [json.exception.parse_error.101] parse error at line 1, column 8: syntax error while parsing value - unexpected ‘:’; expected end of input
The errors all say line 1
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
07:09 PM
Could you share just the first line of your JSONL file?
07:09
Jason
07:09 PM
head -1 allrec2.tmp
John
Photo of md5-f63b27ab4e62d7f1c7de93f1da7516f7
John
07:09 PM
{
“published”: “2021-12-08T19:26:53-05:00",
“updated”: “2021-12-08T19:26:53-05:00",
“title”: “La Florentine Panettone”,
“vendor”: “La Florentine”,
“id”: “bc7bf87d3ee7de79f767dc0c6b1f4f827eb304ce”,
“price”: “10.00",
“currency”: “USD”,
“link”: “https://torronecandy.com/products/la-florentine-panettone”,
“tag”: “BrandLa Florentine Holiday Specialties new items Panettone TypePanettone “,
“image”: “https://cdn.shopify.com/s/files/1/0234/7301/products/La-Florentine-panettone-17-6.jpg?v=1639009499”,
“summary”: “Vendor La Florentine Type Cakes Dessert Bars Price 1000 La Florentine Panettone is packed in Italy and made with an original recipe t
hat yields a light yellow center adorned with candied fruits Product of Italy 176 oz Best if heated before serving”
},
{
“published”: “2021-12-03T17:07:09-05:00",
“updated”: “2021-12-03T17:07:09-05:00",
“title”: “Italy Lanyard”,
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
07:10 PM
The entire record should be in a single line...
07:10
Jason
07:10 PM
What does head -1 allrec2.tmp return?
John
Photo of md5-f63b27ab4e62d7f1c7de93f1da7516f7
John
07:10 PM
Oh, ok
07:11
John
07:11 PM
I can do that wont take me long jsonl (line) eek

1

07:11
John
07:11 PM
Presume no comma separator?
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
07:11 PM
Correct

1

John
Photo of md5-f63b27ab4e62d7f1c7de93f1da7516f7
John
07:21 PM
Thanks loading now :)

1

07:40
John
07:40 PM
Getting
07:40
John
07:40 PM
W20220405 19:39:41.395893 16410 default_variables.cpp:335] Fail to open /proc/self/fd: Too many open files [24]
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
07:41 PM

1

John
Photo of md5-f63b27ab4e62d7f1c7de93f1da7516f7
John
07:42 PM
ulimit -n 8192
07:42
John
07:42 PM
just doing that already thank

1

07:46
John
07:46 PM
Ah need to convert my dates to int64 (on it)
07:47
John
07:47 PM
Mmm file didnt import
07:47
John
07:47 PM
Traceback (most recent call last):
File “type.py”, line 23, in <module>
client.collections[‘hoybuyer’].documents.import(jsonl_file.read().encode(‘utf-8’), {‘batch_size’: 50000})
File “/home/ubuntu/.local/lib/python3.8/site-packages/typesense/documents.py”, line 71, in import

api_response = self.api_call.post(self._endpoint_path(‘import’), documents, params, as_json=False)
File “/home/ubuntu/.local/lib/python3.8/site-packages/typesense/api_call.py”, line 143, in post
return self.make_request(requests.post, endpoint, as_json,
File “/home/ubuntu/.local/lib/python3.8/site-packages/typesense/api_call.py”, line 115, in make_request
raise ApiCall.get_exception(r.status_code)(r.status_code, error_message)
typesense.exceptions.ObjectNotFound: [Errno 404] Not Found
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
07:48 PM
Did you create the collection first?
John
Photo of md5-f63b27ab4e62d7f1c7de93f1da7516f7
John
07:48 PM
schema = {
“name”: “hotbuyer”,
“fields”: [
{“name”: “.*“, “type”: “auto” }
]
}
07:48
John
07:48 PM
Maybe pre define layout?
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
07:48 PM
This should work... Could you share the full code snippet you're using?
John
Photo of md5-f63b27ab4e62d7f1c7de93f1da7516f7
John
07:49 PM
I think I need to increase the timeout I got on the server
07:49
John
07:49 PM
E20220405 19:46:23.849108 16947 raft_server.cpp:601] 4153 queued writes > healthy read lag of 1000
E20220405 19:46:23.849153 16947 raft_server.cpp:613] 4153 queued writes > healthy write lag of 500
07:49
John
07:49 PM
client = typesense.Client({
‘nodes’: [{
‘host’: ‘localhost’, # For Typesense Cloud use xxx.a1.typesense.net
‘port’: ‘8108’, # For Typesense Cloud use 443
‘protocol’: ‘http’ # For Typesense Cloud use https
}],
‘apikey’: ‘abc’,
‘connection_timeout_seconds’: 3000
})

schema = {
“name”: “hotbuyer”,
“fields”: [
{“name”: “.*“, “type”: “auto” }
]
}

client.collections.create(schema)

with open(‘allrec2.jsonl’) as jsonl_file:
client.collections[‘hoybuyer’].documents.import
(jsonl_file.read().encode(‘utf-8’), {‘batch_size’: 50000})
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
07:51 PM
There's a typo in hoybuyer in the import call

1

John
Photo of md5-f63b27ab4e62d7f1c7de93f1da7516f7
John
07:52 PM
Ah thanks

1

07:54
John
07:54 PM
I presume I have to convert my dates into int64 in code (i.e python) and there is no feature to do this for me?
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
07:54 PM
Yup, that's correct

1