Most embedding models only support token lengths o...
# important-announcements
k
Most embedding models only support token lengths of 512 because more than that, the meaning of the embedding gets diluted. For e.g. you have only a few hundred dimensions to encode the semantic meaning of the data. So to encode large text you have to split them up.