In recent v28 RC builds, you can try setting

rerank_hybrid_matches: true

search parameter. When enabled, it'll compute text_match_score for records found with vector search only and vice versa. This might improve overall quality of hybrid search.