Hi all! I noticed stopwords are currently removed ...
# community-help
v
Hi all! I noticed stopwords are currently removed only from the query, not from the documents we score against. Is there a way to apply a similar stopword-handling (or equivalent normalization) on the document side? The issue I’m trying to solve: users often omit common words like “the” in their queries, but documents containing them seem to get scored lower even when they’re the most relevant match. Any workaround or recommended approach?
f
You'd have to preprocess the data to remove the stopwords beforehand. Stopwords only apply during search time. Maybe have a
normalized_<field name>
field on your dataset alongside the main one
v
thanks! I was hoping to avoid that - the current way its done for the queries where the stopwords can be updated and you can choose the stopwords to use at query time - these are quite powerful features which i had hoped to use but for the document
f
Could you post a reproducible example like this one so we can test it ourselves?
👍 1