#community-help

Troubleshooting Typesense API Sorting Error in Item Collections

TLDR Nikhil encountered a sorting error when using Typesense API due to an optional reference field in the items collection schema. Harpreet and Jason helped debug and identified a bug with sorting in general for 2 fields with the first one being the join field, promising a fix in a forthcoming build.

Join the chat

Nov 09, 2023 (3 weeks ago)

Photo of md5-07cb09a4ed26d997ee7f71d5f2a017e0

Nikhil

02:13 PM

I am using v0.26.0.rc29 in typesense cloud to be able to sort items by category name with the help of join.
When I make the following API call:-

/collections/items/documents/search?q=*&per_page=100&include_fields=$categories(description),description&sort_by=$categories(description:asc),description:asc

I get the error:-

{
    "message": "Could not find a reference for doc 996"
}

We have custom ids in our documents so I am not able to retrieve 996.
Questions:-
1. How do I figure out which one is doc 996 when custom id is specified in our documents?
2. Is there a way for me to ignore a missing reference?

08:55

Nikhil

08:55 PM

Jason Kishore Nallan I have a demo coming up soon where I am hoping to show this feature. Any chance you can let me know how to find the doc 996 so that I can fix any referencing issue? Thanks!

Photo of md5-8813087cccc512313602b6d9f9ece19f

Jason

08:58 PM

doc 996 when custom id is specified in our documents

Not sure I understand this... but if you specify an id field in the docs, this is what that ID refers to. If you don't set one, it is automatically generated by Typesense.

If you didn't set an id field explicitly, you could do something like GET /collections/<name>/documents/<id> (where id would be 996 in your case) to get the document that that ID refers to

08:58

Jason

08:58 PM

> Is there a way for me to ignore a missing reference?
There is no way to do this at the moment

Nikhil

09:01 PM

Thanks Jason. I am still not able to find the document. In the next sync, I got error for doc 577 and I can't find it

09:02

Nikhil

09:02 PM

Also in my case id looks like this:-

Jason

09:04 PM

Hmm is that last screenshot from the categories collection?

Nikhil

09:04 PM

No that is from items collection

09:05

Nikhil

09:05 PM

I am getting the same error in the other collection too

Jason

09:05 PM

Could you share your full collection schema for both items and categories?

Nikhil

09:05 PM

For items, the schema is quite big. Should I share the join + id part?

Jason

09:06 PM

Yeah just that part would be sufficient

Nikhil

09:07 PM

const itemsCollectionSchemaFields: CollectionCreateSchema["fields"] = [
    { name: "id", type: "string", index: true, optional: false, sort: true, infix: true, facet: true },
    { name: "_id", type: "string", index: true, optional: false, sort: false, infix: false },
    { name: "commodityId", type: "string", index: true, optional: true, sort: false, infix: false, facet: true, reference: "commodities.id" }
];

09:07

Nikhil

09:07 PM

const commoditiesCollectionCreateSchema: CollectionCreateSchema["fields"] = [
    { name: "id", type: "string", index: true, optional: false, sort: false, infix: false }, // commodityId
    { name: "description", type: "string", index: true, optional: false, sort: true, infix: false },
    { name: "active", type: "bool", index: true, optional: false, sort: false, infix: false }, // useful for deleting all documents
];

Jason

09:15 PM

In the search query you shared above you have $categories... could you share that collection's schema?

Nikhil

09:16 PM

I had changed that query for clarity. Actually the collection name is commodities

Jason

09:17 PM

Could you put together a minimal but complete example using curl requests like this that replicates the issue?

Nikhil

09:17 PM

Do you think referencing id is the issue? Should I just use another field in the commodities collection?

09:18

Nikhil

09:18 PM

I will prepare a repro but just wanted to take a shot before a meeting coming up soon

Jason

09:19 PM

Yeah that could be an issue, since id is a special field... If that's the issue, we can try addressing that

09:19

Jason

09:19 PM

Worth a shot, but I can't tell for sure if that's the issue yet

Nikhil

09:19 PM

Ok I will check. Thanks!

10:08

Nikhil

10:08 PM

Jason It still didn't work even though I changed the field to commodityId

Image 1 for <@4L6c7> It still didn't work even though I changed the field to `commodityId`

10:10

Nikhil

10:10 PM

Jason Is there no way for me to retrieve this document? It would help me create the repro if I knew what data is causing the failure. Thanks!

Jason

11:06 PM

Hmmm, I'm not sure... Could you try replicating the issue with a similar schema (at least a subset that includes the joins) and similar documents to what you have?

11:06

Jason

11:06 PM

CC: Harpreet

Nov 10, 2023 (3 weeks ago)

Nikhil

09:06 AM

Harpreet Is there some way for me to debug this?

Photo of md5-745d880d794220d9f0fb9ade17c6b861

Harpreet

09:15 AM

Sorry, I didn't get the earlier ping. Looking into this now.

09:38

Harpreet

09:38 AM

This issue is because the reference field is optional

{ name: "commodityId", type: "string", index: true, optional: true, sort: false, infix: false, facet: true, reference: "commodities.id" }

and when we try to compute the sort score of a document that doesn't have a reference on the basis of $commodities(description:asc), it returns this error.

Nikhil

09:41 AM

Thanks Harpreet. I will fix this and try again

10:11

Nikhil

10:11 AM

Harpreet To make upsert work with import, do I need to specify my own id field? I had removed id from schema and put commodityId instead. But I am concerned that next import would duplicate the values

Harpreet

10:16 AM

Since you didn't have id field in the docs, it would indeed duplicate the values.

10:18

Harpreet

10:18 AM

You'll have to match the id that typesense generated for your doc to update a particular doc.

Nikhil

12:21 PM

Harpreet If both the collections have description field, can there be an issue with sorting by both the props? In this case I am trying to sort by commodity description and then item description. However items are not getting sorted even though commodities are

Harpreet

12:23 PM

I don't think having common field name would cause such issue.

12:25

Harpreet

12:25 PM

Can you share an example?

Nikhil

12:27 PM

See this for e.g.

12:28

Nikhil

12:28 PM

Item name "Coffee" should have come before "Hot Choc"

Harpreet

12:32 PM

Okay I'll try to reproduce this.

Nikhil

12:52 PM

Thanks Harpreet. Is there any workaround you can suggest?

Harpreet

12:53 PM

Cannot think of anything at the moment. I'll have to debug this to understand what's happening.

Nikhil

12:56 PM

Ok we have a production app where I am hoping to deploy this change due to it being escalated quite a bit. Please let me know once you have debugged this. I feel otherwise the typesense build seems quite stable

01:02

Nikhil

01:02 PM

Harpreet I checked that this sorting is broken in general for 2 fields with first one being join field. Same name is not the issue

Nov 14, 2023 (2 weeks ago)

Harpreet

09:32 AM

Hey Nikhil I've fixed the sorting bug. It will be available in the next rc build.

Nikhil

10:40 AM

Thanks Harpreet! Looking forward to trying that out

Typesense

Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI

Indexed 3015 threads (79% resolved)

Join Our Community

Similar Threads

Issue with Query Hangs Identifying Different Collection Field Names

Kanak experiences query hangs when collection field names differ. Ahmed had an issue adding documents to collections, which was self-resolved. Kishore Nallan resolved Kanak's issue with the updated build.

4mo

Cold Start Problem with Dynamic Collections

Adrian reported cold start issues with dynamic collections. Jason suggested using wildcard `*` for query_by parameters, upgrading to `0.25.0.rc34`, and clarified conventions. Adrian's issues were resolved but they reported a limitation that will potentially be addressed.

6mo

Methods for Fetching, Querying, and Modifying Collections in Typesense

Bill inquired about performing OR queries, querying empty arrays and modifying collections in Typesense. Kishore Nallan explained the current limitations and provided workarounds and recommendations for each case. The conversation also touched upon the usage of cache in Typesense and the workings of the _eval function.

10mo

Resolving Issues with Reference Field in v0.26.0.rc11

Shaun reported problems applying the reference field in `v0.26.0.rc11`. The issue made the tested field disappear from UI, and an attempt to fix it resulted in an error. Harpreet suggested re-indexing data and promised to consider improving the feature in future updates. Harpreet is currently working on a related feature - sorting by reference field.

3mo

Threading Problem During Multiple Collection Creation and Batch Insertion in Typesense

Johan has a problem with creating multiple collections and batch-inserting documents into Typesense, which is returning results from different collections. Kishore Nallan helps troubleshoot the issue and suggests a potential local race condition, which is fixed in a later build.

17mo