Hi! I have formed a dataset of 120GB of podcast feeds, 2M podcasts, ~50M episodes and am trying to make them searchable. For anyone who listens to podcasts, the search experience in this space is usually very bad and slow, while it needs to be very high quality to be able to discover new content. Now I am trying to find out what exactly uses the most resources in Typesense to optimize this. What factors are mostly responsible for RAM usage? Here some of my ideas: • Does field name size make a difference? • Does size variability within the same field make a difference? • Can I reduce the size by throwing out stopwords and fillers from descriptions without loosing search quality? I think this could be something interesting in general. If this stuff is not that benchmarked I could help run some tests.

👋 Field name won't make any difference. Size variability also won't matter. The main memory consumption would stem from the actual text index where we need to store the document and positions where each word appears in the corpus (called the inverted index). By removing stop words you can certainly save memory.

Thank you for your response. Is that something worth benchmarking? From the speed of your response I guess you know this stuff fully already 🙂

Yes ofc I write Typesense 😄

Is this use case require a Vector Search . I have seen Spotify blog post which talks about solving this using the same https://engineering.atspotify.com/2022/03/introducing-natural-language-search-for-podcast-episodes/

Sure, I was just curious if that's an opportunity to get involved

My suggestion is to benchmark 1M records first and then use that to develop a sense of the memory usage.

Here is step by step tutorial to do the same -

@satish venkatakrishnan That looks very interesting

My understanding is Typesense dont support vector search yet . But Jason told thats the area of interest

With vector search you will probably get really good semantically good search results, but it will not support type ahead / instant searching. So it depends on what kind of UX you are looking for.

Maybe two different searches can be used for each purpose

@Kishore Nallan Yes you are right. But Podcast search generally involve a Topic name and we want to get all the episodes /podcast related to the same. Again me being the user I want that 🙂

Yup yup, certainly instant search is certainly not great for everything.

So that said, When can we get Vector search in Typesense 🙂

@Kishore Nallan I am planing to use it though. Type ahead is great! I want it to feel snappy

Relevant part from that article above:

Although Dense Retrieval / Natural Language Search has very interesting properties, it often fails to perform as well as traditional IR methods on exact term matching (and is also more expensive to run on all queries). That’s why we decided to make our Natural Language Search an additional source rather than just replace our other retrieval sources (including our Elasticsearch cluster).

That is actually very relevant

That's a very tricky problem. You need either enough search volume to identify popular records or find a proxy for the podcast popularity: maybe your primary source has number of stars / rating?

Sure I can calculate some value of popularity, maybe even from the primary source. But I still have to mix it in with relevancy don't I?

Given 2 documents both match all query tokens, the simplest start is to rank the records by the popularity score. Once you have enough real world search and click data, eventually you can build a ML mode for predicting this more broadly.

@Alexander Zierhut Is this data available publically where we can work together on solving this problem

@satish venkatakrishnan I'll write you a pm