#important-announcements

Discussion on Firebase Extension for Typesense

TLDR Jason announced the release of a Firebase Extension for Typesense, leading to a discussion with Gabe about capabilities and scalability concerns. Other users, John and A, simply expressed their admiration.

Powered by Struct AI

10

4

1

1

Jun 16, 2021 (32 months ago)
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
07:01 PM
Hey @channel, I'm excited to announce the availability of our new Firebase Extension for Typesense:

https://github.com/typesense/firestore-typesense-search

It syncs data from your Firestore database to Typesense automatically on changes. If you're a Firebase user, I'd love to get your feedback.

10

4

07:02
Jason
07:02 PM
Here's our announcement on Twitter if you want to spread the word 🙂

https://twitter.com/typesense/status/1405237388591304705
Gabe
Photo of md5-7aff1bf99393eb318e36513504a16e85
Gabe
07:06 PM
Nice! I probably won't use this because we don't sync our exact firestore data to typesense, but do some processing on it beforehand, but cool to see nonetheless.

A couple of questions:
• Because Firestore is an object store that allows having nested data, how do you handle users wanting to search nested data? IIRC typsense does not support this (yet)
• How does this scale...because firestore/firebase functions are "infinitely" scalable & typesense is not, I forsee issues arising from too much attempted indexing activity at once.
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
07:20 PM
Great questions!

> Because Firestore is an object store that allows having nested data, how do you handle users wanting to search nested data? IIRC typsense does not support this (yet)
The current MVP version does not handle nested data in Firestore, because Typesense server doesn't support it yet. We've been discussing adding support, especially for Firebase users, based on demand for this...

> How does this scale...because firestore/firebase functions are "infinitely" scalable & typesense is not, I forsee issues arising from too much attempted indexing activity at once.
Scalability would be a function of the Typesense cluster configuration. High-throughput reads & writes would require larger number of CPU cores to handle the load.

Another thing I've been thinking about is may be adding a buffering mechanism on the Firestore side that can accumulate changes and bulk insert changes into Typesense, which should help accommodate higher throughput writes on lower number of CPU cores as well.

Question for you: is the pre-processing you're doing only to handle nested fields, or do you do additional transformations?
Gabe
Photo of md5-7aff1bf99393eb318e36513504a16e85
Gabe
08:40 PM
nested fields + attaching userIds for filtering that aren't present in the firestore documents.
08:41
Gabe
08:41 PM
another possibility for the scalability issue is you can set up the functions to retry when they throw/get an error response from typesense.
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
08:47 PM
Ooh interesting, thanks for that
08:48
Jason
08:48 PM
> nested fields + attaching userIds for filtering that aren't present in the firestore documents.
Is the userID stored in a different collection that you "join" on the fly before indexing to Typesense currently?
Jun 17, 2021 (32 months ago)
Gabe
Photo of md5-7aff1bf99393eb318e36513504a16e85
Gabe
06:58 PM
essentially. We actually have an intermediate firestore collection that contains this data so we can make use of atomicity in firestore & then we propagate the data in it to typesense using something similar to your extension with a filter that only does so if at least one user is associated with the document.
06:59
Gabe
06:59 PM
(some of our data is proactively ingested but has no associated users so doesn't need to be searchable/in typesense)
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
09:40 PM
So if Typesense server supported nested fields, you'd still need a way to specify a filter on the Firestore collection, to indicate what subset of those records are synced to Typesense via the extension?
Jun 18, 2021 (32 months ago)
John
Photo of md5-7a0ab48aa8979a59e1d8c3919797c1f8
John
12:55 AM
This is huge!!! Congratulations!!!

1

Jun 21, 2021 (31 months ago)
Gabe
Photo of md5-7aff1bf99393eb318e36513504a16e85
Gabe
08:08 PM
Jason yup, but we may have a unique set up. I wouldn't go solving for our use case unless you see others with the same needs.

1

Jul 09, 2021 (31 months ago)
A
Photo of md5-98c72c3023867be0346b48ae4cb22001
A
06:29 AM
This is amazing.

Typesense

Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI

Indexed 3015 threads (79% resolved)

Join Our Community

Similar Threads

Nested Objects Issue in Firebase & Typesense Integration

Shaun encountered issues with nested objects being flattened in Typesense. Jason found the root issue and provided a solution involving updating the Firebase extension to 1.0.3.

7

57
6mo

Configuring Typesense Extension for Nested Fields in Firebase Collection

David had issues indexing nested fields using the Typesense extension. Jason offered advice on specifying schemas and prioritizing specific fields, but there remained unresolved challenges with the schema.

4

14
7mo

Handling Order of Firestore Events for Synchronization with Typesense.

Ross ran into an issue with Firestore events triggering out of order, causing synchronization inconsistency between Firestore and Typesense. With advice and input from Jason and Kishore Nallan, they implemented a debouncing solution using redis, ensuring that the latest Firestore data is synced to Typesense accurately.

3

41
19mo

Troubleshooting 400 Error When Upgrading Typesense Firestore Extension

Orion experienced a `400` error after updating the Typesense Firestore extension, causing issues with cloud functions. They traced the issue back to a data type conflict in their Typesense collection schema after updating. With help from Jason and Kishore Nallan, they resolved the issue by recreating the collection.

5

96
14mo

Resolving Typesense Cloud and Firebase Timestamp Error

Eoin faced an issue with Firebase timestamp when setting up Typesense Cloud. Jason suggested converting timestamps to Unix and sync them as int64 fields in Typesense.

18
25mo