#community-help

Resolve Connection Error on Records Upsert

TLDR Jainil faced consistent connection errors while upserting records. Jason identified an OOM issue, suggesting a capacity upgrade. Auto-scaling was discussed and the upgrade implementation process, which was in progress, was clarified.

Powered by Struct AI
white_check_mark1
11
1mo
Solved
Join the chat
Aug 21, 2023 (1 month ago)
Jainil
Photo of md5-bea1e4531665b4608c47d00462429d3a
Jainil
08:31 PM
I'm getting this pretty consistently when attempting to import/upsert records (full trace in thread) - Any thoughts?
ConnectionError: ('Connection aborted.', BadStatusLine('HTTP/1.1 0 \r\n'))
08:31
Jainil
08:31 PM
/app/huxley/core/typesense/typesense.py in import_records(collection_id, records, action)
   40   resp = collection.documents.import_(
   41     records,
---> 42     {"action": action},
   43   )
   44   failures = list(filter(lambda x: x["success"] is False, resp))

/usr/local/lib/python3.7/site-packages/typesense/documents.py in import_(self, documents, params, batch_size)
   76 
   77         docs_import = '\n'.join(document_strs)
---> 78         api_response = (self._endpoint_path('import'), docs_import, params, as_json=False)
   79         res_obj_strs = api_response.split('\n')
   80 

/usr/local/lib/python3.7/site-packages/typesense/api_call.py in post(self, endpoint, body, params, as_json)
  144     return self.make_request(, endpoint, as_json,
  145                 params=params, data=body,
--> 146                 timeout=self.config.connection_timeout_seconds)
  147 
  148   def put(self, endpoint, body, params=None):

/usr/local/lib/python3.7/site-packages/typesense/api_call.py in make_request(self, fn, endpoint, as_json, **kwargs)
  128 
  129     logger.debug('No retries left. Raising last exception: {}'.format(last_exception))
--> 130     raise last_exception
  131 
  132   def set_node_healthcheck(self, node, is_healthy):

/usr/local/lib/python3.7/site-packages/typesense/api_call.py in make_request(self, fn, endpoint, as_json, **kwargs)
   99           kwargs['data'] = json.dumps(kwargs['data'])
  100 
--> 101         r = fn(url, headers={ApiCall.API_KEY_HEADER_NAME: self.config.api_key}, **kwargs)
  102 
  103         # Treat any status code > 0 and < 500 to be an indication that node is healthy

/usr/local/lib/python3.7/site-packages/requests/api.py in post(url, data, json, **kwargs)
  113   """
  114 
--> 115   return request("post", url, data=data, json=json, **kwargs)
  116 
  117 

/usr/local/lib/python3.7/site-packages/requests/api.py in request(method, url, **kwargs)
   57   # cases, and look like a memory leak in others.
   58   with sessions.Session() as session:
---> 59     return session.request(method=method, url=url, **kwargs)
   60 
   61 

/usr/local/lib/python3.7/site-packages/requests/sessions.py in request(self, method, url, params, data, headers, cookies, files, auth, timeout, allow_redirects, proxies, hooks, stream, verify, cert, json)
  585     }
  586     send_kwargs.update(settings)
--> 587     resp = self.send(prep, **send_kwargs)
  588 
  589     return resp

/usr/local/lib/python3.7/site-packages/requests/sessions.py in send(self, request, **kwargs)
  699 
  700     # Send the request
--> 701     r = adapter.send(request, **kwargs)
  702 
  703     # Total elapsed time of the request (approximately)

/usr/local/lib/python3.7/site-packages/requests/adapters.py in send(self, request, stream, timeout, verify, cert, proxies)
  545 
  546     except (ProtocolError, OSError) as err:
--> 547       raise ConnectionError(err, request=request)
  548 
  549     except MaxRetryError as e:

ConnectionError: ('Connection aborted.', BadStatusLine('HTTP/1.1 0 \r\n'))
08:31
Jainil
08:31 PM
Clusters are on 0.24.1, started seeing this 2 days ago
08:36
Jainil
08:36 PM
Happy to DM a cluster id for an easier debugging/investigation
08:58
Jainil
08:58 PM
I've scheduled an instance and capacity upgrade to see if that resolves it. Only seems to happen on my prod instance 😬
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
09:16 PM
I do see an OOM issue on one of the nodes - so that would have contributed to it
09:16
Jason
09:16 PM
Let me know if you continue seeing it after the upgrade completes
Jainil
Photo of md5-bea1e4531665b4608c47d00462429d3a
Jainil
09:21 PM
Cool, thanks for that visibility!
We have auto upgrade capacity enabled, any idea why it wasn't handled automatically?
Also anything we can do to get the upgrade sped up? (i see its in progress) The issues affecting all of our production servers and causing an influx of bug reports 😅
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
09:23 PM
auto-scaling only kicks-in when recommended thresholds are consistently crossed for 12 hours… In this case it was a brief spike, which is why it didn’t kick-in
09:23
Jason
09:23 PM
Yeah, it’s already in progress and the reindexing time on new higher capacity nodes is dependent on the size of your dataset…
white_check_mark1
Jainil
Photo of md5-bea1e4531665b4608c47d00462429d3a
Jainil
09:23 PM
Makes sense! Thanks.