#community-help

Updating Bulk Records and Resolving Typsense Issues.

TLDR Greg inquired about updating bulk records. Jason proposed using the documents/import endpoint for bulk updating, identified issues with Typesense, and provided solutions. Greg appreciated the assistance. Conversation related to the procedure was shared with Viktor.

Powered by Struct AI

5

1

1

1

Nov 17, 2022 (13 months ago)
Greg
Photo of md5-11c0f771a29e2aa8d72ae9544cc39017
Greg
10:59 PM
What is the best way to handle updating lots of records, say 15k? Querying records is limited to 250 results. I can create a process to loop through all the pages to get and update each page. Is there a better solution?
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
11:00 PM
You want to query your primary datastore for the list of document IDs to update, and then use the documents/import endpoint with action=emplace to do a bulk update
Greg
Photo of md5-11c0f771a29e2aa8d72ae9544cc39017
Greg
11:02 PM
But the query is limited to 250 results, right?
11:02
Greg
11:02 PM
So I have to do multiple queries to get them all.
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
11:03 PM
You want to query your primary datastore not Typesense
Greg
Photo of md5-11c0f771a29e2aa8d72ae9544cc39017
Greg
11:03 PM
Can I update 15k records with one update request?
11:04
Greg
11:04 PM
We aren’t currently saving the Typesense ID to our database.
11:04
Greg
11:04 PM
Never needed them
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
11:04 PM
> Can I update 15k records with one update request?
Yup, you just want to make sure your client-side timeout is high as say 30 minutes, so it never timesout
11:05
Jason
11:05 PM
> We aren’t currently saving the Typesense ID to our database.
You can have Typesense use the ID you use in your database for each record, by sending the database ID in a field called id
11:05
Jason
11:05 PM
You can then use that same field to reference the same document in Typesense and send further updates
Greg
Photo of md5-11c0f771a29e2aa8d72ae9544cc39017
Greg
11:05 PM
It is one DB ID for each record.
11:06
Greg
11:06 PM
We are chunking a book into small parts.
11:06
Greg
11:06 PM
So we can do full text on a book
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
11:06 PM
And you have one DB record for each book?
Greg
Photo of md5-11c0f771a29e2aa8d72ae9544cc39017
Greg
11:07 PM
Yes. One book has a single DB record but thousands of typescript records
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
11:07 PM
Ah, I see what you mean…
11:07
Jason
11:07 PM
The updates you want to do are for all Typesense records belonging to a single book?
Greg
Photo of md5-11c0f771a29e2aa8d72ae9544cc39017
Greg
11:08 PM
Yes
11:08
Greg
11:08 PM
Like title update
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
11:10 PM
You can use the documents/export endpoint with a filter_by book_id parameter to just export the books you want, along with include_fields=id, to only export the Typesense IDs for each record
11:10
Jason
11:10 PM
This documents/export endpoint doesn’t have any limits on number of documents returned
Greg
Photo of md5-11c0f771a29e2aa8d72ae9544cc39017
Greg
11:10 PM
Ah. That makes a lot of sense and much better than making 60+ API requests
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
11:11 PM
Yeah!
11:11
Jason
11:11 PM
Just about to write a docs article about how to keep data in Typesense in sync, will include this use case
Greg
Photo of md5-11c0f771a29e2aa8d72ae9544cc39017
Greg
11:11 PM
Thank you!

1

11:19
Greg
11:19 PM
Sorry. One last question. I added the include_fields parameter but the result is returning everything.
11:19
Greg
11:19 PM
This is my code

const currentDocuments = await typesenseClient
  .collections(collectionId)
  .documents()
  .export({
    filter_by: `mediaId:=${mediaId}`,
    include_fields: ['id'],
  });
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
11:20 PM
It should be include_fields: 'id' (comma separated string instead of an array)
Greg
Photo of md5-11c0f771a29e2aa8d72ae9544cc39017
Greg
11:21 PM
Ah, thank you!

1

11:38
Greg
11:38 PM
This one might be a bug 😕. I am getting an error like this with the new nested object functionality:
{
  code: 404,
  document: '{"id":"2448","title":"AaA Prepare Ye a Highway"}',
  error: 'Field `contributors` not found.',
  success: false
}

Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
11:39 PM
Could you confirm you’re using action=emplace as a parameter to the import endpoint?
Greg
Photo of md5-11c0f771a29e2aa8d72ae9544cc39017
Greg
11:39 PM
This is the contributors schema:
{
      "facet": false,
      "index": true,
      "infix": false,
      "locale": "",
      "name": "contributors",
      "nested": true,
      "nested_array": 2,
      "optional": false,
      "sort": false,
      "type": "object"
    }
11:39
Greg
11:39 PM
No, I was doing update
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
11:40 PM
Oh wait, update should work :thinking_face:
Greg
Photo of md5-11c0f771a29e2aa8d72ae9544cc39017
Greg
11:40 PM
I just changed to emplace. Same result
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
11:41 PM
May I know which version of Typesense this is on?
Greg
Photo of md5-11c0f771a29e2aa8d72ae9544cc39017
Greg
11:43 PM
Where do I see that? I had you upgrade my instances on Nov 4
11:45
Greg
11:45 PM
This is the ID: i6tg5nj09vokaqysp
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
11:48 PM
You’re currently on 0.24.0.rcn27… Can we try upgrading you to the latest RC? If the issue still persists there, we can take a closer look
Greg
Photo of md5-11c0f771a29e2aa8d72ae9544cc39017
Greg
11:49 PM
For sure
11:49
Greg
11:49 PM
I have 3 instances. This one is my dev environment.
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
11:49 PM
Ah ok, let’s try on dev and then we can upgrade the others once we’ve identified/fixed the issue

1

11:51
Jason
11:51 PM
Btw, heads-up about this change we had to do in the highlight response structure in this version we’re upgrading you to: https://github.com/typesense/typesense/issues/227#issuecomment-1315627904
11:52
Jason
11:52 PM
The upgrade is done. You’re now running 0.24.0.rcn34. Could you try now?
Greg
Photo of md5-11c0f771a29e2aa8d72ae9544cc39017
Greg
11:53 PM
Yes. And thanks for the heads up.
11:53
Greg
11:53 PM
Same result

1

11:56
Greg
11:56 PM
It does work if I update the contributors field
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
11:57 PM
Could you use this script as your base and modify it with your particular schema, a few sample records and then use the import curl command with action=update to show a minimal reproduceable example?
11:58
Jason
11:58 PM
We can then debug it from there
Greg
Photo of md5-11c0f771a29e2aa8d72ae9544cc39017
Greg
11:58 PM
Sure thing. You are also welcome to use my dev instance if it helps.
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
11:59 PM
That would be easier yeah! Could you then just share the JSON you’re using to update?
Nov 18, 2022 (13 months ago)
Greg
Photo of md5-11c0f771a29e2aa8d72ae9544cc39017
Greg
12:02 AM
This is a small sample of the update records:
[
  { "id": "2352", "title": "AaA Prepare Ye a Highway" },
  { "id": "2353", "title": "AaA Prepare Ye a Highway" },
  { "id": "2354", "title": "AaA Prepare Ye a Highway" },
  { "id": "2355", "title": "AaA Prepare Ye a Highway" },
  { "id": "2356", "title": "AaA Prepare Ye a Highway" },
  { "id": "2357", "title": "AaA Prepare Ye a Highway" },
  { "id": "2358", "title": "AaA Prepare Ye a Highway" },
  { "id": "2359", "title": "AaA Prepare Ye a Highway" },
  { "id": "2360", "title": "AaA Prepare Ye a Highway" },
  { "id": "2361", "title": "AaA Prepare Ye a Highway" }
]

Is that enough?
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
12:02 AM
Yup, perfect
12:02
Jason
12:02 AM
Will keep you posted
Greg
Photo of md5-11c0f771a29e2aa8d72ae9544cc39017
Greg
12:02 AM
Need anything else from me?
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
12:03 AM
Not for now, thank you!
Greg
Photo of md5-11c0f771a29e2aa8d72ae9544cc39017
Greg
12:03 AM
Thanks for the help! You are awesome.

1

Nov 20, 2022 (13 months ago)
Viktor
Photo of md5-972da58c82de3b38862220702e852eda
Viktor
11:51 AM
> Just about to write a docs article about how to keep data in Typesense in sync, will include this use case
Would love to read the post

1

Nov 22, 2022 (13 months ago)
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
12:03 AM

1

Nov 23, 2022 (13 months ago)
Viktor
Photo of md5-972da58c82de3b38862220702e852eda
Viktor
05:34 PM
Thanks!

1