Yes.

querying the collection with the reference field, i don't need it

Unless you do want to filter the results based on some condition then you can specify

filter_by

as well.