#community-help

Updating Bulk Records and Resolving Typsense Issues.

TLDR Greg inquired about updating bulk records. Jason proposed using the documents/import endpoint for bulk updating, identified issues with Typesense, and provided solutions. Greg appreciated the assistance. Conversation related to the procedure was shared with Viktor.

13mo

Join the chat

Nov 17, 2022 (13 months ago)

Photo of md5-11c0f771a29e2aa8d72ae9544cc39017

Greg

10:59 PM

What is the best way to handle updating lots of records, say 15k? Querying records is limited to 250 results. I can create a process to loop through all the pages to get and update each page. Is there a better solution?

Photo of md5-8813087cccc512313602b6d9f9ece19f

Jason

11:00 PM

You want to query your primary datastore for the list of document IDs to update, and then use the documents/import endpoint with action=emplace to do a bulk update

Greg

11:02 PM

But the query is limited to 250 results, right?

11:02

Greg

11:02 PM

So I have to do multiple queries to get them all.

Jason

11:03 PM

You want to query your primary datastore not Typesense

Greg

11:03 PM

Can I update 15k records with one update request?

11:04

Greg

11:04 PM

We aren’t currently saving the Typesense ID to our database.

11:04

Greg

11:04 PM

Never needed them

Jason

11:04 PM

> Can I update 15k records with one update request?
Yup, you just want to make sure your client-side timeout is high as say 30 minutes, so it never timesout

11:05

Jason

11:05 PM

> We aren’t currently saving the Typesense ID to our database.
You can have Typesense use the ID you use in your database for each record, by sending the database ID in a field called id

11:05

Jason

11:05 PM

You can then use that same field to reference the same document in Typesense and send further updates

Greg

11:05 PM

It is one DB ID for each record.

11:06

Greg

11:06 PM

We are chunking a book into small parts.

11:06

Greg

11:06 PM

So we can do full text on a book

Jason

11:06 PM

And you have one DB record for each book?

Greg

11:07 PM

Yes. One book has a single DB record but thousands of typescript records

Jason

11:07 PM

Ah, I see what you mean…

11:07

Jason

11:07 PM

The updates you want to do are for all Typesense records belonging to a single book?

Greg

11:08 PM

Yes

11:08

Greg

11:08 PM

Like title update

Jason

11:10 PM

You can use the documents/export endpoint with a filter_by book_id parameter to just export the books you want, along with include_fields=id, to only export the Typesense IDs for each record

11:10

Jason

11:10 PM

This documents/export endpoint doesn’t have any limits on number of documents returned

11:10

Jason

11:10 PM

https://typesense.org/docs/0.23.1/api/documents.html#export-documents

Greg

11:10 PM

Ah. That makes a lot of sense and much better than making 60+ API requests

Jason

11:11 PM

Yeah!

11:11

Jason

11:11 PM

Just about to write a docs article about how to keep data in Typesense in sync, will include this use case

Greg

11:11 PM

Thank you!

11:19

Greg

11:19 PM

Sorry. One last question. I added the include_fields parameter but the result is returning everything.

11:19

Greg

11:19 PM

This is my code

const currentDocuments = await typesenseClient
  .collections(collectionId)
  .documents()
  .export({
    filter_by: `mediaId:=${mediaId}`,
    include_fields: ['id'],
  });

Jason

11:20 PM

It should be include_fields: 'id' (comma separated string instead of an array)

Greg

11:21 PM

Ah, thank you!

11:38

Greg

11:38 PM

This one might be a bug 😕. I am getting an error like this with the new nested object functionality:

{
  code: 404,
  document: '{"id":"2448","title":"AaA Prepare Ye a Highway"}',
  error: 'Field `contributors` not found.',
  success: false
}

Jason

11:39 PM

Could you confirm you’re using action=emplace as a parameter to the import endpoint?

Greg

11:39 PM

This is the contributors schema:

{
      "facet": false,
      "index": true,
      "infix": false,
      "locale": "",
      "name": "contributors",
      "nested": true,
      "nested_array": 2,
      "optional": false,
      "sort": false,
      "type": "object"
    }

11:39

Greg

11:39 PM

No, I was doing update

Jason

11:40 PM

Oh wait, update should work :thinking_face:

Greg

11:40 PM

I just changed to emplace. Same result

Jason

11:41 PM

May I know which version of Typesense this is on?

Greg

11:43 PM

Where do I see that? I had you upgrade my instances on Nov 4

11:45

Greg

11:45 PM

This is the ID: i6tg5nj09vokaqysp

Jason

11:48 PM

You’re currently on 0.24.0.rcn27… Can we try upgrading you to the latest RC? If the issue still persists there, we can take a closer look

Greg

11:49 PM

For sure

11:49

Greg

11:49 PM

I have 3 instances. This one is my dev environment.

Jason

11:49 PM

Ah ok, let’s try on dev and then we can upgrade the others once we’ve identified/fixed the issue

11:51

Jason

11:51 PM

Btw, heads-up about this change we had to do in the highlight response structure in this version we’re upgrading you to: https://github.com/typesense/typesense/issues/227#issuecomment-1315627904

11:52

Jason

11:52 PM

The upgrade is done. You’re now running 0.24.0.rcn34. Could you try now?

Greg

11:53 PM

Yes. And thanks for the heads up.

11:53

Greg

11:53 PM

Same result

11:56

Greg

11:56 PM

It does work if I update the contributors field

Jason

11:57 PM

Could you use this script as your base and modify it with your particular schema, a few sample records and then use the import curl command with action=update to show a minimal reproduceable example?

11:58

Jason

11:58 PM

We can then debug it from there

Greg

11:58 PM

Sure thing. You are also welcome to use my dev instance if it helps.

Jason

11:59 PM

That would be easier yeah! Could you then just share the JSON you’re using to update?

11:59

Jason

11:59 PM

The call that threw this error: https://typesense-community.slack.com/archives/C01P749MET0/p1668728311911059?thread_ts=1668725981.045459&cid=C01P749MET0

Nov 18, 2022 (13 months ago)

Greg

12:02 AM

This is a small sample of the update records:

[
  { "id": "2352", "title": "AaA Prepare Ye a Highway" },
  { "id": "2353", "title": "AaA Prepare Ye a Highway" },
  { "id": "2354", "title": "AaA Prepare Ye a Highway" },
  { "id": "2355", "title": "AaA Prepare Ye a Highway" },
  { "id": "2356", "title": "AaA Prepare Ye a Highway" },
  { "id": "2357", "title": "AaA Prepare Ye a Highway" },
  { "id": "2358", "title": "AaA Prepare Ye a Highway" },
  { "id": "2359", "title": "AaA Prepare Ye a Highway" },
  { "id": "2360", "title": "AaA Prepare Ye a Highway" },
  { "id": "2361", "title": "AaA Prepare Ye a Highway" }
]

Is that enough?

Jason

12:02 AM

Yup, perfect

12:02

Jason

12:02 AM

Will keep you posted

Greg

12:02 AM

Need anything else from me?

Jason

12:03 AM

Not for now, thank you!

Greg

12:03 AM

Thanks for the help! You are awesome.

Nov 20, 2022 (13 months ago)

Photo of md5-972da58c82de3b38862220702e852eda

Viktor

11:51 AM

> Just about to write a docs article about how to keep data in Typesense in sync, will include this use case
Would love to read the post

Nov 22, 2022 (13 months ago)

Jason

12:03 AM

Here’s the article I was referring to: https://typesense.org/docs/guide/syncing-data-into-typesense.html

Nov 23, 2022 (13 months ago)

Viktor

05:34 PM

Thanks!

Typesense

Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI

Indexed 3015 threads (79% resolved)

Join Our Community

Similar Threads

Collating Search Results from Different Collections.

Patrick inquired about collating search results on different collections based on text_match value. Jason confirmed it's possible, but must be done client-side.

18mo

Collection-Level Scoring in Typesense Multi-Search

Mile questioned the possibility of collection-level scoring in Typesense. Jason advised adjusting scores client-side, and requested Mile to create a GitHub issue to assess community interest.

1mo

Merging and Ordering Multi-Search Results

Jack asked how to merge and order search results from different collections using the 'score' value. Kishore Nallan confirmed its use.

2mo

Understanding Rank Fusion Score in Typesense's Multi-Search

Ayush had queries about comparing `rank_fusion_score` across collections in Typesense's hybrid search. They were explained and clarified by Kishore Nallan on how the score works and how it could be manipulated for their use-case.

3mo

Reranking Search Results from Different Sources

Viktor seeks advice on reranking search results. Kishore Nallan suggests hybrid search and custom ranking algorithms. John recommends Metarank as a potential solution.

8mo