#community-help

Setting Up Typesense in GCP with HA and Ingesting Data

TLDR Rengarajan inquired about setting up Typesense with HA in GCP and using other data sources for ingestion besides mongo or firebase. Joshua and Kishore Nallan provided information and offered to further discuss the matter over a call.

Powered by Struct AI
+12
heavy_plus_sign2
Oct 14, 2021 (23 months ago)
Rengarajan
Photo of md5-ad10683a7a254c4da7196e861a710185
Rengarajan
02:21 PM
Good Morning - We are looking at setting up Typesense in GCP with HA. The source of the data would be a DWH and an OLTP - and we would write APIs that would leverage Typesense APIs to continuously ingest data. If we setup HA by ourselves, would we need to ingest data into two zones if we want to implement HA by ourselves as we are not able to find any replication across zones?

We also see documentation about MongoDB integration with Typesense and Firebase with Typesense. Assuming that these would be available for any document based ingestion with the APIs provided, can we leverage the managed services in GCP?
Joshua
Photo of md5-fc2280639a535856f901b06b4928137a
Joshua
03:14 PM
i believe typesense does the replication itself as long as you have 3+ typesense nodes...they just all need to be able to reach each other over the availability zones

for mongo, as long as the managed mongo supports change streams that are accessible to your typesense nodes, it should be fine: https://typesense.org/docs/guide/mongodb-full-text-search.html#step-2-start-a-mongodb-replica-set
heavy_plus_sign2
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
03:15 PM
👋 Rengarajan

> If we setup HA by ourselves, would we need to ingest data into two zones if we want to implement HA by ourselves as we are not able to find any replication across zones?
You will just set up a cluster whose nodes that are across regions. You don't have to do anything special from Typesense perspective. There might be additional latency but Typesense will handle it.

> Assuming that these would be available for any document based ingestion with the APIs provided, can we leverage the managed services in GCP?
I don't quite understand this question. Can you please elaborate? Are you using Mongo as the primary data store?
Rengarajan
Photo of md5-ad10683a7a254c4da7196e861a710185
Rengarajan
03:28 PM
Kishore Nallan thanks for the quick response. My second question is about the managed services and whether we could ingest data from sources other than mongo or firebase such as SAP HANA or Amazon Redshift (through a middleware)
03:31
Rengarajan
03:31 PM
Joshua thanks will check that out. And no we won't be using mongo, but have found ways to generate files that can be continuously ingested
+11
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
03:33 PM
Rengarajan Given the intricacies/options involved in data ingestion, maybe easier to discuss this over a call? Depending on your timezone either Jason or I would be happy to do a quick call.
Rengarajan
Photo of md5-ad10683a7a254c4da7196e861a710185
Rengarajan
03:36 PM
Sure thanks Kishore Nallan . I am available in IST. Would sometime tomorrow be fine - or would Monday be ok? I can send an invite or can jump on a call
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
03:38 PM
Tomorrow is fine Rengarajan Here's my calendar: https://calendly.com/kishorenc/30min

Feel free to pick a time that suites.
Rengarajan
Photo of md5-ad10683a7a254c4da7196e861a710185
Rengarajan
03:42 PM
Thanks Kishore Nallan will setup one soon.
+11
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
05:12 PM
Rengarajan I've a scheduling conflict that just came up in the evening tomorrow. I will be away from 5 pm to 8.30 pm. You free any time between noon till 5 pm?

I've DMed you