Managing Typesense Across Large Datasets
TLDR Phillip inquired about best practices with Typesense for large datasets. Ross shared personal practices, and Jason confirmed Typesense isn't designed to be a primary datastore. Patrick shared a related discussion link. Viji mentioned using DynamoDB Streams as an option.
3
1
Aug 15, 2022 (14 months ago)
Phillip
04:22 PM1. When people use typesense for large datasets, do they usually keep the dataset in typesense and in a relational database and somehow keep them in sync? Is there a design pattern or best practice for this? Or is it more normal to keep everything in only one place?
2. If the data is only in typesense, what is the best way to go about relating a thing in your database to a thing in typesense in a many to many fashion? Normally a link/join table would be used. What is the best way to do this without that/is that somehow still the correct way?
Ross
05:10 PM2. while we don't do this since i believe the recommended approach is not to rely on Typesense as your primary datastore -- i have seen people use multiple Typesense queries to perform "join-like" operations between multiple collections if needed
Viji
06:03 PM
Jason
06:18 PMJason
06:19 PMJason
06:22 PM3
Aug 16, 2022 (14 months ago)
Patrick
03:28 PM1
Viji
08:55 PMUsing DynamoDB Streams may also be an option for us
https://typesense.org/docs/guide/dynamodb-full-text-search.html#step-1-create-typesense-cluster
Typesense
Indexed 2776 threads (79% resolved)
Similar Threads
Troubleshooting Typesense Document Import Error
Christopher had trouble importing 2.1M documents into Typesense due to memory errors. Jason clarified the system requirements, explaining the correlation between RAM and dataset size, and ways to tackle the issue. They both also discussed database-like query options.
Efficient Dataset Creation and Maintenance in Typesense with React
sonu asked about Typesense dataset management in a React project. Nick shared their solution using a postgres database, cron, and the import endpoint. Kishore Nallan suggested updating Typesense whenever MySQL data changes and explained primary databases' role. The dataset file would reside with Typesense, not affecting the front-end load time.
Understanding Typesense's Functionality and Usage as a Search Engine
Tim asked about Typesense's function as a DB storage. Jason clarified Typesense functions primarily as a search engine and is not meant to be a primary data store. They also discussed data management strategies using Typesense. Joe provided further insight and recommended resources.
Handling Order of Firestore Events for Synchronization with Typesense.
Ross ran into an issue with Firestore events triggering out of order, causing synchronization inconsistency between Firestore and Typesense. With advice and input from Jason and Kishore Nallan, they implemented a debouncing solution using redis, ensuring that the latest Firestore data is synced to Typesense accurately.
Understanding Typesense with Java and MongoDB
Vasudev needed help implementing a Java project with Typesense and MongoDB. Kishore Nallan guided them in designing the project, explaining different methods, providing templates, and responding to error issues. At last, Vasudev successfully implemented the project and expressed gratitude.