Is there any configuration I'm missing if reference sort

"sort_by": "$JoinedCollectionName(field_name:asc)"

is not working? The field_name is indexed correctly, if I query the referenced collection, the sorting works there.