Hello, I’m currently designing the data structure ...
# community-help
a
Hello, I’m currently designing the data structure for a new project and would appreciate some feedback. Context: We’re building a solution for libraries. There is a large catalog of books (millions), where each book is treated as a product. Each library holds its own subset of these books, along with stock indicating how many copies are available for rental. There needs to be a central Typesense Cluster. Search requirements: • On the general website, users should be able to search across all books. • When searching within a specific library, results should be limited to books that are available for rental in that library. Current approach: I have 1 cluster with an index for products(books) and one for instances(actual stock info and shelf info, these can be joined using the reference to products) a collection with library info like opening hours To support filtering by library, I’m considering adding an
IntArray
field to each book document. This array would contain the IDs of the libraries where the book is available. Then, filtering by library would simply involve filtering on this field. I chose an
IntArray
over a
StringArray
(e.g. library names) for performance reasons. Scale: • ~700 libraries • ~4 million books Question: Does this approach make sense from a performance and scalability perspective in Typesense? Or would you recommend a different data modeling strategy?
a
Hi @Antoon Van Dessel Your approach makes sense, since it avoids the overhead of a JOIN at search time, which matters at 4M documents. I wouldn't recommend a different strategy for this use case. The main trade-off is RAM. Each value in the array adds a few bytes of index overhead per document, so with 4M books and up to 700 libraries per book, memory usage can grow. At your current scale (~700 libraries) this should be manageable, but it's something to watch if the number of libraries grows significantly. A few implementation tips: 1. Use scoped search API keys (docs) with an embedded filter like
filter_by: library_ids:=42
for per-library search. For searching all books, use an unscoped key. 2. Filtering on many library IDs at once (
library_ids:=[41, 42, 43, ...]
) adds latency as the list grows - keep multi-value filters short when possible. 3. Avoid
range_index
on this field (not needed for exact matching, adds overhead). 4. Avoid faceting on this field if possible (adds memory overhead). Once you have this set up and start running some searches, let us know and we can review your full configuration.
a
Nice! Will let you know our findings thanks!
@Alan Martini I was wondering if we were to use the Join and not the proposed solution here. Do you believe it could be possible? Or would the overhead be to big? I know and understand this is a big estimation game now without knowing all the details, but I was just wondering.
a
Yes, it is possible. It would be more RAM efficient, but less performant on searches. How much less perfromant will really depend on the shape and amount of your data, as well as what kind of search you are doing. Also, the setup would be more complicated (as with any JOINed tables). But it would be more scalable.