#community-help

Airbyte Timeout Issue with Typesense Sync

TLDR Jamshid experienced timeouts when syncing MySQL records with Airbyte. Jason suggested it's related to an Airbyte issue, leading Jamshid to create a custom Typesense connector to resolve the problem.

5mo

Join the chat

Jun 16, 2023 (6 months ago)

Photo of md5-317ed6510eb7587c9e9243fb6ebc4e87

Jamshid

06:53 PM

We have a table on MySQL that has around 18K records. We are syncing that to typesense through Airbyte. In that table, there are multiple json fields and when all are selected to sync, Airbyte constantly fails. In the logs it says timeout occurred on the destination side.

Our typesense cluster is 0.5 GB RAM, 2 vCPUs, Regular SSD, Single Node, Single Region.

I was wondering if these timeouts are because of our typesense resources.

Any thoughts?

Photo of md5-8813087cccc512313602b6d9f9ece19f

Jason

07:50 PM

It’s most likely related to this: https://github.com/airbytehq/airbyte/issues/24097

07:50

Jason

07:50 PM

The default timeout in the python client (which is what AirByte uses) is low, and has to be increased

07:51

Jason

07:51 PM

I would recommend following up with AirByte about that issue

Jamshid

10:08 PM

Thanks. That makes sense. Since you shared the link, there is some comments there.

Jun 20, 2023 (5 months ago)

Jamshid

10:14 PM

Okay I made my custom typesense connector and started using it. Now I don’t get timeouts https://hub.docker.com/r/jamshidhashimi/typesense

Typesense

Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI

Indexed 3015 threads (79% resolved)

Join Our Community

Similar Threads

Airbyte MySQL Connection Timeout with Typesense

Jamshid reported a connection timeout issue using Airbyte for MySQL and typesense. Jason suggested it could be due to a low client-side timeout in Airbyte.

6mo

Airbyte Connector for Typesense Integration and Issue Resolution

Cirdes shares the creation of an Airbyte connector for Typesense, later experiencing timeout issues with large-scale data synchronization. Jason provides solutions for adjustment of timeout thresholds and batch sizes, successfully resolving the issue.

14mo

Syncing records issue from BigQuery to Typesense using Airbyte

Jamshid had a problem syncing records from BigQuery to Typesense via Airbyte, with only a partial number of records syncing. Jason suggested checking AirByte logs for Typesense API responses. They found an issue with BigQuery's connector on Airbyte handling repeated fields and concluded to consider building their own sync script to mitigate the situation.

2mo