#community-help

Troubleshooting Typesense API Sorting Error in Item Collections

TLDR Nikhil encountered a sorting error when using Typesense API due to an optional reference field in the items collection schema. Harpreet and Jason helped debug and identified a bug with sorting in general for 2 fields with the first one being the join field, promising a fix in a forthcoming build.

Powered by Struct AI

2

Nov 09, 2023 (3 weeks ago)
Nikhil
Photo of md5-07cb09a4ed26d997ee7f71d5f2a017e0
Nikhil
02:13 PM
I am using v0.26.0.rc29 in typesense cloud to be able to sort items by category name with the help of join.
When I make the following API call:-
/collections/items/documents/search?q=*&per_page=100&include_fields=$categories(description),description&sort_by=$categories(description:asc),description:asc

I get the error:-
{
    "message": "Could not find a reference for doc 996"
}

We have custom ids in our documents so I am not able to retrieve 996.
Questions:-
1. How do I figure out which one is doc 996 when custom id is specified in our documents?
2. Is there a way for me to ignore a missing reference?
08:55
Nikhil
08:55 PM
Jason Kishore Nallan I have a demo coming up soon where I am hoping to show this feature. Any chance you can let me know how to find the doc 996 so that I can fix any referencing issue? Thanks!
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
08:58 PM
doc 996 when custom id is specified in our documents
Not sure I understand this... but if you specify an id field in the docs, this is what that ID refers to. If you don't set one, it is automatically generated by Typesense.

If you didn't set an id field explicitly, you could do something like GET /collections/<name>/documents/<id> (where id would be 996 in your case) to get the document that that ID refers to
08:58
Jason
08:58 PM
&gt; Is there a way for me to ignore a missing reference?
There is no way to do this at the moment
Nikhil
Photo of md5-07cb09a4ed26d997ee7f71d5f2a017e0
Nikhil
09:01 PM
Thanks Jason. I am still not able to find the document. In the next sync, I got error for doc 577 and I can't find it
Image 1 for Thanks Jason. I am still not able to find the document. In the next sync, I got error for doc 577 and I can't find it
09:02
Nikhil
09:02 PM
Also in my case id looks like this:-
Image 1 for Also in my case `id` looks like this:-
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
09:04 PM
Hmm is that last screenshot from the categories collection?
Nikhil
Photo of md5-07cb09a4ed26d997ee7f71d5f2a017e0
Nikhil
09:04 PM
No that is from items collection
09:05
Nikhil
09:05 PM
I am getting the same error in the other collection too
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
09:05 PM
Could you share your full collection schema for both items and categories?
Nikhil
Photo of md5-07cb09a4ed26d997ee7f71d5f2a017e0
Nikhil
09:05 PM
For items, the schema is quite big. Should I share the join + id part?
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
09:06 PM
Yeah just that part would be sufficient
Nikhil
Photo of md5-07cb09a4ed26d997ee7f71d5f2a017e0
Nikhil
09:07 PM
const itemsCollectionSchemaFields: CollectionCreateSchema["fields"] = [
    { name: "id", type: "string", index: true, optional: false, sort: true, infix: true, facet: true },
    { name: "_id", type: "string", index: true, optional: false, sort: false, infix: false },
    { name: "commodityId", type: "string", index: true, optional: true, sort: false, infix: false, facet: true, reference: "commodities.id" }
];
09:07
Nikhil
09:07 PM
const commoditiesCollectionCreateSchema: CollectionCreateSchema["fields"] = [
    { name: "id", type: "string", index: true, optional: false, sort: false, infix: false }, // commodityId
    { name: "description", type: "string", index: true, optional: false, sort: true, infix: false },
    { name: "active", type: "bool", index: true, optional: false, sort: false, infix: false }, // useful for deleting all documents
];
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
09:15 PM
In the search query you shared above you have $categories... could you share that collection's schema?
Nikhil
Photo of md5-07cb09a4ed26d997ee7f71d5f2a017e0
Nikhil
09:16 PM
I had changed that query for clarity. Actually the collection name is commodities
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
09:17 PM
Could you put together a minimal but complete example using curl requests like this that replicates the issue?
Nikhil
Photo of md5-07cb09a4ed26d997ee7f71d5f2a017e0
Nikhil
09:17 PM
Do you think referencing id is the issue? Should I just use another field in the commodities collection?
09:18
Nikhil
09:18 PM
I will prepare a repro but just wanted to take a shot before a meeting coming up soon
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
09:19 PM
Yeah that could be an issue, since id is a special field... If that's the issue, we can try addressing that
09:19
Jason
09:19 PM
Worth a shot, but I can't tell for sure if that's the issue yet
Nikhil
Photo of md5-07cb09a4ed26d997ee7f71d5f2a017e0
Nikhil
09:19 PM
Ok I will check. Thanks!
10:08
Nikhil
10:08 PM
Jason It still didn't work even though I changed the field to commodityId
Image 1 for <@4L6c7> It still didn't work even though I changed the field to `commodityId`
10:10
Nikhil
10:10 PM
Jason Is there no way for me to retrieve this document? It would help me create the repro if I knew what data is causing the failure. Thanks!
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
11:06 PM
Hmmm, I'm not sure... Could you try replicating the issue with a similar schema (at least a subset that includes the joins) and similar documents to what you have?
11:06
Jason
11:06 PM
CC: Harpreet
Nov 10, 2023 (3 weeks ago)
Nikhil
Photo of md5-07cb09a4ed26d997ee7f71d5f2a017e0
Nikhil
09:06 AM
Harpreet Is there some way for me to debug this?
Harpreet
Photo of md5-745d880d794220d9f0fb9ade17c6b861
Harpreet
09:15 AM
Sorry, I didn't get the earlier ping. Looking into this now.

1

09:38
Harpreet
09:38 AM
This issue is because the reference field is optional
{ name: "commodityId", type: "string", index: true, optional: true, sort: false, infix: false, facet: true, reference: "commodities.id" }

and when we try to compute the sort score of a document that doesn't have a reference on the basis of $commodities(description:asc), it returns this error.
Nikhil
Photo of md5-07cb09a4ed26d997ee7f71d5f2a017e0
Nikhil
09:41 AM
Thanks Harpreet. I will fix this and try again
10:11
Nikhil
10:11 AM
Harpreet To make upsert work with import, do I need to specify my own id field? I had removed id from schema and put commodityId instead. But I am concerned that next import would duplicate the values
Harpreet
Photo of md5-745d880d794220d9f0fb9ade17c6b861
Harpreet
10:16 AM
Since you didn't have id field in the docs, it would indeed duplicate the values.
10:18
Harpreet
10:18 AM
You'll have to match the id that typesense generated for your doc to update a particular doc.
Nikhil
Photo of md5-07cb09a4ed26d997ee7f71d5f2a017e0
Nikhil
12:21 PM
Harpreet If both the collections have description field, can there be an issue with sorting by both the props? In this case I am trying to sort by commodity description and then item description. However items are not getting sorted even though commodities are
Harpreet
Photo of md5-745d880d794220d9f0fb9ade17c6b861
Harpreet
12:23 PM
I don't think having common field name would cause such issue.
12:25
Harpreet
12:25 PM
Can you share an example?
Nikhil
Photo of md5-07cb09a4ed26d997ee7f71d5f2a017e0
Nikhil
12:27 PM
See this for e.g.
Image 1 for See this for e.g.
12:28
Nikhil
12:28 PM
Item name "Coffee" should have come before "Hot Choc"
Harpreet
Photo of md5-745d880d794220d9f0fb9ade17c6b861
Harpreet
12:32 PM
Okay I'll try to reproduce this.
Nikhil
Photo of md5-07cb09a4ed26d997ee7f71d5f2a017e0
Nikhil
12:52 PM
Thanks Harpreet. Is there any workaround you can suggest?
Harpreet
Photo of md5-745d880d794220d9f0fb9ade17c6b861
Harpreet
12:53 PM
Cannot think of anything at the moment. I'll have to debug this to understand what's happening.
Nikhil
Photo of md5-07cb09a4ed26d997ee7f71d5f2a017e0
Nikhil
12:56 PM
Ok we have a production app where I am hoping to deploy this change due to it being escalated quite a bit. Please let me know once you have debugged this. I feel otherwise the typesense build seems quite stable
01:02
Nikhil
01:02 PM
Harpreet I checked that this sorting is broken in general for 2 fields with first one being join field. Same name is not the issue

1

Nov 14, 2023 (2 weeks ago)
Harpreet
Photo of md5-745d880d794220d9f0fb9ade17c6b861
Harpreet
09:32 AM
Hey Nikhil I've fixed the sorting bug. It will be available in the next rc build.
Nikhil
Photo of md5-07cb09a4ed26d997ee7f71d5f2a017e0
Nikhil
10:40 AM
Thanks Harpreet! Looking forward to trying that out

Typesense

Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI

Indexed 3015 threads (79% resolved)

Join Our Community

Similar Threads

Issue with Query Hangs Identifying Different Collection Field Names

Kanak experiences query hangs when collection field names differ. Ahmed had an issue adding documents to collections, which was self-resolved. Kishore Nallan resolved Kanak's issue with the updated build.

5
4mo

Cold Start Problem with Dynamic Collections

Adrian reported cold start issues with dynamic collections. Jason suggested using wildcard `*` for query_by parameters, upgrading to `0.25.0.rc34`, and clarified conventions. Adrian's issues were resolved but they reported a limitation that will potentially be addressed.

6

39
6mo

Methods for Fetching, Querying, and Modifying Collections in Typesense

Bill inquired about performing OR queries, querying empty arrays and modifying collections in Typesense. Kishore Nallan explained the current limitations and provided workarounds and recommendations for each case. The conversation also touched upon the usage of cache in Typesense and the workings of the _eval function.

5

52
10mo

Resolving Issues with Reference Field in v0.26.0.rc11

Shaun reported problems applying the reference field in `v0.26.0.rc11`. The issue made the tested field disappear from UI, and an attempt to fix it resulted in an error. Harpreet suggested re-indexing data and promised to consider improving the feature in future updates. Harpreet is currently working on a related feature - sorting by reference field.

4

22
3mo

Threading Problem During Multiple Collection Creation and Batch Insertion in Typesense

Johan has a problem with creating multiple collections and batch-inserting documents into Typesense, which is returning results from different collections. Kishore Nallan helps troubleshoot the issue and suggests a potential local race condition, which is fixed in a later build.

35
17mo