Implementing Stemming, Lemmatization, Stopwords with Typesense
TLDR Carlo asked about implementing stemming, lemmatization, stopwords with Typesense. Kishore Nallan suggested the Porter stemmer and mentioned stopwords is under development. Gustavo suggested using GPT-3.5-Turbo.
Kishore Nallan
Thu, 13 Jul 2023 08:13:36 UTCPorter stemmer is the most popular stemming library used. You have to stem the values during indexing and also stem the queries before sending to Typesense. However, I suspect that most people don't use stemmers because prefix searching & typo correction is usually enough to handle plurals etc.
Kishore Nallan
Thu, 13 Jul 2023 08:13:57 UTCStopwords is under development.
Carlo
Thu, 13 Jul 2023 08:31:12 UTCthnx!
Gustavo
Thu, 13 Jul 2023 12:30:14 UTCYou can also use GPT-3.5-Turbo to do that as well as add synonyms, labels, categories, etc.
Typesense
Indexed 3051 threads
Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI
View in App
Carlo
Thu, 13 Jul 2023 06:10:14 UTCI've seen a ticket that stekming, lemmatization, stopwords aren't currently supported by typesense. Has anyone succesfully implemented that before it reaches typesense, or know a good workaround?