I was just looking through some examples on full-t...
# community-help
d
I was just looking through some examples on full-text search on the docs site - eg https://typesense.org/docs/guide/mongodb-full-text-search.html#full-example and so on. Looking at the code, this didn’t seem to be searching on contents, just attributes…? I’m wondering how best to approach implementing full text search over collections of markdown documents, some of which can be several hundred KB. How are you all approaching this issue? Chunking..?
I’m considering something like the following: • chunk by block of some sort - maybe split by heading? paragraph? • create an embedding for each block, store embeddings with ref to which doc it came from • create embedding on search query, then vector search over chunks is this overcomplicating it?
y
The guide you point to doesn't mention embeddings, which are usually not involved in full text search. In full text search I'm not sure about the point of chunking contents. For semantic search there is definitely interest in adopting a good chuncking strategy, but it is very dependent on the use-case and the nature of your documents
f
We have a guide specifically for semantic search here: https://typesense.org/docs/guide/semantic-search.html#use-case As far as markdown documents go, you can break it down based on headings and their contents, it's what the documentation scraper does as well
d
thanks, will check these out