#community-help

Setting Up Typesense in GCP with HA and Ingesting Data

TLDR Rengarajan inquired about setting up Typesense with HA in GCP and using other data sources for ingestion besides mongo or firebase. Joshua and Kishore Nallan provided information and offered to further discuss the matter over a call.

23mo

Join the chat

Oct 14, 2021 (23 months ago)

Photo of md5-ad10683a7a254c4da7196e861a710185

Rengarajan

02:21 PM

Good Morning - We are looking at setting up Typesense in GCP with HA. The source of the data would be a DWH and an OLTP - and we would write APIs that would leverage Typesense APIs to continuously ingest data. If we setup HA by ourselves, would we need to ingest data into two zones if we want to implement HA by ourselves as we are not able to find any replication across zones?

We also see documentation about MongoDB integration with Typesense and Firebase with Typesense. Assuming that these would be available for any document based ingestion with the APIs provided, can we leverage the managed services in GCP?

Photo of md5-fc2280639a535856f901b06b4928137a

Joshua

03:14 PM

i believe typesense does the replication itself as long as you have 3+ typesense nodes...they just all need to be able to reach each other over the availability zones

for mongo, as long as the managed mongo supports change streams that are accessible to your typesense nodes, it should be fine: https://typesense.org/docs/guide/mongodb-full-text-search.html#step-2-start-a-mongodb-replica-set

Photo of md5-4e872368b2b2668460205b409e95c2ea

Kishore Nallan

03:15 PM

👋 Rengarajan

> If we setup HA by ourselves, would we need to ingest data into two zones if we want to implement HA by ourselves as we are not able to find any replication across zones?
You will just set up a cluster whose nodes that are across regions. You don't have to do anything special from Typesense perspective. There might be additional latency but Typesense will handle it.

> Assuming that these would be available for any document based ingestion with the APIs provided, can we leverage the managed services in GCP?
I don't quite understand this question. Can you please elaborate? Are you using Mongo as the primary data store?

Rengarajan

03:28 PM

Kishore Nallan thanks for the quick response. My second question is about the managed services and whether we could ingest data from sources other than mongo or firebase such as SAP HANA or Amazon Redshift (through a middleware)

03:31

Rengarajan

03:31 PM

Joshua thanks will check that out. And no we won't be using mongo, but have found ways to generate files that can be continuously ingested

Kishore Nallan

03:33 PM

Rengarajan Given the intricacies/options involved in data ingestion, maybe easier to discuss this over a call? Depending on your timezone either Jason or I would be happy to do a quick call.

Rengarajan

03:36 PM

Sure thanks Kishore Nallan . I am available in IST. Would sometime tomorrow be fine - or would Monday be ok? I can send an invite or can jump on a call

Kishore Nallan

03:38 PM

Tomorrow is fine Rengarajan Here's my calendar: https://calendly.com/kishorenc/30min

Feel free to pick a time that suites.

Rengarajan

03:42 PM

Thanks Kishore Nallan will setup one soon.

Kishore Nallan

05:12 PM

Rengarajan ~~I've a scheduling conflict that just came up in the evening tomorrow. I will be away from 5 pm to 8.30 pm. You free any time between noon till 5 pm?~~

I've DMed you

Typesense

Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI

Indexed 2764 threads (79% resolved)

Join Our Community Manage subscription

Similar Threads

Optimizing Typesense Implementation for Large Collections

Oskar faced performance issues with his document collection in Typesense due to filter additions. Jason suggested trying a newer Typesense build and potentially partitioning the data into country-wise collections. They also discussed reducing network latency with CDN solutions.

8mo

Handling Kinesis Stream Event Batching with Typesense

Dui had questions about how to handle Kinesis stream events with Typesense. Kishore Nallan suggested using upsert mode for creation/update and differentiating with logical deletion. After various discussions including identifying and resolving a bug, they finalized to introduce an `emplace` action in Typesense v0.23.

20mo

Understanding Typesense with Java and MongoDB

Vasudev needed help implementing a Java project with Typesense and MongoDB. Kishore Nallan guided them in designing the project, explaining different methods, providing templates, and responding to error issues. At last, Vasudev successfully implemented the project and expressed gratitude.

14mo

Database Management and Personalized Search Queries Discussion

Alexander posed several database and search-related questions. Mac suggested considering GraphQL, considering performance and high availability needs, and leveraging Cloudflare workers. Kishore Nallan provided insights on the capabilities of Typesense and discussed the idea of search personalization using machine learning.

16mo

Issue with Typesense Go-Client and Upserting Documents

Anton experiences issues with upserting documents and data getting lost when using Typesense Go-Client. Jason suggests troubleshooting via a call, details of which are not available in the thread.

30mo