Resolving Typesense 0.22.2 Syntax Error and Import Issues
TLDR John faced issues with Typesense 0.22.2 code syntax and import functionality. Jason advised to reformat the JSONL records into single lines, increase ulimit, and convert dates to int64 format manually in code. A typo in the code was also corrected.
Powered by Struct AI
7
1
30
21mo
Apr 05, 2022 (21 months ago)
John
John
07:08 PMWrite some code that is causing me problems, Typesense 0.22.2
I have raised a ticket #568
but main issue is I don’t know which records are supposed to be the “problem”
E20220405 18:49:30.707577 15581 collection.cpp:77] JSON error: [json.exception.parse_error.101] parse error at line 1, column 8: syntax error while parsing value - unexpected ‘:’; expected end of input
The errors all say line 1
I have raised a ticket #568
but main issue is I don’t know which records are supposed to be the “problem”
E20220405 18:49:30.707577 15581 collection.cpp:77] JSON error: [json.exception.parse_error.101] parse error at line 1, column 8: syntax error while parsing value - unexpected ‘:’; expected end of input
The errors all say line 1
Jason
Jason
07:09 PMCould you share just the first line of your JSONL file?
07:09
Jason
07:09 PMhead -1 allrec2.tmpJohn
John
07:09 PM{
“published”: “2021-12-08T19:26:53-05:00",
“updated”: “2021-12-08T19:26:53-05:00",
“title”: “La Florentine Panettone”,
“vendor”: “La Florentine”,
“id”: “bc7bf87d3ee7de79f767dc0c6b1f4f827eb304ce”,
“price”: “10.00",
“currency”: “USD”,
“link”: “https://torronecandy.com/products/la-florentine-panettone”,
“tag”: “BrandLa Florentine Holiday Specialties new items Panettone TypePanettone “,
“image”: “https://cdn.shopify.com/s/files/1/0234/7301/products/La-Florentine-panettone-17-6.jpg?v=1639009499”,
“summary”: “Vendor La Florentine Type Cakes Dessert Bars Price 1000 La Florentine Panettone is packed in Italy and made with an original recipe t
hat yields a light yellow center adorned with candied fruits Product of Italy 176 oz Best if heated before serving”
},
{
“published”: “2021-12-03T17:07:09-05:00",
“updated”: “2021-12-03T17:07:09-05:00",
“title”: “Italy Lanyard”,
“published”: “2021-12-08T19:26:53-05:00",
“updated”: “2021-12-08T19:26:53-05:00",
“title”: “La Florentine Panettone”,
“vendor”: “La Florentine”,
“id”: “bc7bf87d3ee7de79f767dc0c6b1f4f827eb304ce”,
“price”: “10.00",
“currency”: “USD”,
“link”: “https://torronecandy.com/products/la-florentine-panettone”,
“tag”: “BrandLa Florentine Holiday Specialties new items Panettone TypePanettone “,
“image”: “https://cdn.shopify.com/s/files/1/0234/7301/products/La-Florentine-panettone-17-6.jpg?v=1639009499”,
“summary”: “Vendor La Florentine Type Cakes Dessert Bars Price 1000 La Florentine Panettone is packed in Italy and made with an original recipe t
hat yields a light yellow center adorned with candied fruits Product of Italy 176 oz Best if heated before serving”
},
{
“published”: “2021-12-03T17:07:09-05:00",
“updated”: “2021-12-03T17:07:09-05:00",
“title”: “Italy Lanyard”,
Jason
Jason
07:10 PMThe entire record should be in a single line...
07:10
Jason
07:10 PMWhat does
head -1 allrec2.tmp return?John
John
07:10 PMOh, ok
07:11
John
07:11 PMI can do that wont take me long jsonl (line) eek
1
07:11
John
07:11 PMPresume no comma separator?
Jason
Jason
07:11 PMCorrect
1
John
John
07:21 PMThanks loading now :)
1
07:40
John
07:40 PMGetting
07:40
John
07:40 PMW20220405 19:39:41.395893 16410 default_variables.cpp:335] Fail to open /proc/self/fd: Too many open files [24]
Jason
John
John
07:42 PMulimit -n 8192
07:42
John
07:42 PMjust doing that already thank
1
07:46
John
07:46 PMAh need to convert my dates to int64 (on it)
07:47
John
07:47 PMMmm file didnt import
07:47
John
07:47 PMTraceback (most recent call last):
File “type.py”, line 23, in <module>
client.collections[‘hoybuyer’].documents.import(jsonl_file.read().encode(‘utf-8’), {‘batch_size’: 50000})
File “/home/ubuntu/.local/lib/python3.8/site-packages/typesense/documents.py”, line 71, in import
api_response = self.api_call.post(self._endpoint_path(‘import’), documents, params, as_json=False)
File “/home/ubuntu/.local/lib/python3.8/site-packages/typesense/api_call.py”, line 143, in post
return self.make_request(requests.post, endpoint, as_json,
File “/home/ubuntu/.local/lib/python3.8/site-packages/typesense/api_call.py”, line 115, in make_request
raise ApiCall.get_exception(r.status_code)(r.status_code, error_message)
typesense.exceptions.ObjectNotFound: [Errno 404] Not Found
File “type.py”, line 23, in <module>
client.collections[‘hoybuyer’].documents.import(jsonl_file.read().encode(‘utf-8’), {‘batch_size’: 50000})
File “/home/ubuntu/.local/lib/python3.8/site-packages/typesense/documents.py”, line 71, in import
api_response = self.api_call.post(self._endpoint_path(‘import’), documents, params, as_json=False)
File “/home/ubuntu/.local/lib/python3.8/site-packages/typesense/api_call.py”, line 143, in post
return self.make_request(requests.post, endpoint, as_json,
File “/home/ubuntu/.local/lib/python3.8/site-packages/typesense/api_call.py”, line 115, in make_request
raise ApiCall.get_exception(r.status_code)(r.status_code, error_message)
typesense.exceptions.ObjectNotFound: [Errno 404] Not Found
Jason
Jason
07:48 PMDid you create the collection first?
John
John
07:48 PMschema = {
“name”: “hotbuyer”,
“fields”: [
{“name”: “.*“, “type”: “auto” }
]
}
“name”: “hotbuyer”,
“fields”: [
{“name”: “.*“, “type”: “auto” }
]
}
07:48
John
07:48 PMMaybe pre define layout?
Jason
Jason
07:48 PMThis should work... Could you share the full code snippet you're using?
John
John
07:49 PMI think I need to increase the timeout I got on the server
07:49
John
07:49 PME20220405 19:46:23.849108 16947 raft_server.cpp:601] 4153 queued writes > healthy read lag of 1000
E20220405 19:46:23.849153 16947 raft_server.cpp:613] 4153 queued writes > healthy write lag of 500
E20220405 19:46:23.849153 16947 raft_server.cpp:613] 4153 queued writes > healthy write lag of 500
07:49
John
07:49 PMclient = typesense.Client({
‘nodes’: [{
‘host’: ‘localhost’, # For Typesense Cloud use xxx.a1.typesense.net
‘port’: ‘8108’, # For Typesense Cloud use 443
‘protocol’: ‘http’ # For Typesense Cloud use https
}],
‘apikey’: ‘abc’,
‘connection_timeout_seconds’: 3000
})
schema = {
“name”: “hotbuyer”,
“fields”: [
{“name”: “.*“, “type”: “auto” }
]
}
client.collections.create(schema)
with open(‘allrec2.jsonl’) as jsonl_file:
client.collections[‘hoybuyer’].documents.import(jsonl_file.read().encode(‘utf-8’), {‘batch_size’: 50000})
‘nodes’: [{
‘host’: ‘localhost’, # For Typesense Cloud use xxx.a1.typesense.net
‘port’: ‘8108’, # For Typesense Cloud use 443
‘protocol’: ‘http’ # For Typesense Cloud use https
}],
‘apikey’: ‘abc’,
‘connection_timeout_seconds’: 3000
})
schema = {
“name”: “hotbuyer”,
“fields”: [
{“name”: “.*“, “type”: “auto” }
]
}
client.collections.create(schema)
with open(‘allrec2.jsonl’) as jsonl_file:
client.collections[‘hoybuyer’].documents.import(jsonl_file.read().encode(‘utf-8’), {‘batch_size’: 50000})
Jason
Jason
07:51 PMThere's a typo in
hoybuyer in the import call1
John
John
07:52 PMAh thanks
1
07:54
John
07:54 PMI presume I have to convert my dates into int64 in code (i.e python) and there is no feature to do this for me?
Jason
Jason
07:54 PMYup, that's correct
1
Typesense
Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI
Indexed 3015 threads (79% resolved)
Similar Threads
Understanding the Difference Between Query by and Filter by
satish asked the difference between Query by and filter by. Kishore Nallan explained that Query by supports fuzzy searching while filter by is a precise match restriction.
2
21mo
Overrides/Curation Query and Filtering Docs
Jameshwart sought for filtering assistance, and Kishore Nallan suggested checking the overrides/curation section in the docs.
4
7mo
Full Text Search Across Long Books: Chunking vs. Single Document
Epi asked about providing FTS for long books. Kishore Nallan suggested breaking the books into chunks for better performance and query results.
2
13mo
Collating Search Results from Different Collections.
Patrick inquired about collating search results on different collections based on text_match value. Jason confirmed it's possible, but must be done client-side.
3
18mo
Using Joins Without Filtering Results in Document Retrieval
Ahmed inquired whether joins could be used without filtering results, but needs to provide the schema of collections and a sample query as suggested by Harpreet.
2
4mo