Discussion on Firebase Extension for Typesense
TLDR Jason announced the release of a Firebase Extension for Typesense, leading to a discussion with Gabe about capabilities and scalability concerns. Other users, John and A, simply expressed their admiration.
10
4
1
1
Jun 16, 2021 (32 months ago)
Jason
07:01 PMhttps://github.com/typesense/firestore-typesense-search
It syncs data from your Firestore database to Typesense automatically on changes. If you're a Firebase user, I'd love to get your feedback.
10
4
Jason
07:02 PMhttps://twitter.com/typesense/status/1405237388591304705
Gabe
07:06 PMA couple of questions:
• Because Firestore is an object store that allows having nested data, how do you handle users wanting to search nested data? IIRC typsense does not support this (yet)
• How does this scale...because firestore/firebase functions are "infinitely" scalable & typesense is not, I forsee issues arising from too much attempted indexing activity at once.
Jason
07:20 PM> Because Firestore is an object store that allows having nested data, how do you handle users wanting to search nested data? IIRC typsense does not support this (yet)
The current MVP version does not handle nested data in Firestore, because Typesense server doesn't support it yet. We've been discussing adding support, especially for Firebase users, based on demand for this...
> How does this scale...because firestore/firebase functions are "infinitely" scalable & typesense is not, I forsee issues arising from too much attempted indexing activity at once.
Scalability would be a function of the Typesense cluster configuration. High-throughput reads & writes would require larger number of CPU cores to handle the load.
Another thing I've been thinking about is may be adding a buffering mechanism on the Firestore side that can accumulate changes and bulk insert changes into Typesense, which should help accommodate higher throughput writes on lower number of CPU cores as well.
Question for you: is the pre-processing you're doing only to handle nested fields, or do you do additional transformations?
Gabe
08:40 PMGabe
08:41 PMJason
08:47 PMJason
08:48 PMIs the userID stored in a different collection that you "join" on the fly before indexing to Typesense currently?
Jun 17, 2021 (32 months ago)
Gabe
06:58 PMGabe
06:59 PMJason
09:40 PMJun 18, 2021 (32 months ago)
John
12:55 AM1
Jun 21, 2021 (31 months ago)
Gabe
08:08 PM1
Jul 09, 2021 (31 months ago)
A
06:29 AM
Typesense
Indexed 3015 threads (79% resolved)
Similar Threads
Nested Objects Issue in Firebase & Typesense Integration
Shaun encountered issues with nested objects being flattened in Typesense. Jason found the root issue and provided a solution involving updating the Firebase extension to 1.0.3.
Configuring Typesense Extension for Nested Fields in Firebase Collection
David had issues indexing nested fields using the Typesense extension. Jason offered advice on specifying schemas and prioritizing specific fields, but there remained unresolved challenges with the schema.
Handling Order of Firestore Events for Synchronization with Typesense.
Ross ran into an issue with Firestore events triggering out of order, causing synchronization inconsistency between Firestore and Typesense. With advice and input from Jason and Kishore Nallan, they implemented a debouncing solution using redis, ensuring that the latest Firestore data is synced to Typesense accurately.
Troubleshooting 400 Error When Upgrading Typesense Firestore Extension
Orion experienced a `400` error after updating the Typesense Firestore extension, causing issues with cloud functions. They traced the issue back to a data type conflict in their Typesense collection schema after updating. With help from Jason and Kishore Nallan, they resolved the issue by recreating the collection.
Resolving Typesense Cloud and Firebase Timestamp Error
Eoin faced an issue with Firebase timestamp when setting up Typesense Cloud. Jason suggested converting timestamps to Unix and sync them as int64 fields in Typesense.