#community-help

Adding Data to Existing Collection with Typesense

TLDR Rotfisch wants to add new data to an existing collection in Typesense. Jainil suggested using action=create for importing documents, while Jason explained that merging collections is not possible without modifying the docsearch-scraper.

8mo

Join the chat

Apr 08, 2023 (8 months ago)

Photo of md5-4fdfd972be680409429627667fd6b8bf

Rotfisch

05:02 PM

I am trying to add data to an existing collection with typesense. My first approach was to use docscraper to write new data to an existing one. This failed because the index already existed. Then I created a new collection, exported it and tried to import it into my existing collection. This didn't work because the id of the entries already exist. With the action mode upsert the entries would be overwritten.
What is the best way to add new data to a collection?

Photo of md5-bea1e4531665b4608c47d00462429d3a

Jainil

05:37 PM

You should be able to use action=create, when importing your documents. That will create entries for ID's that don't exist, and fail for ID's that do exist.

Apr 09, 2023 (8 months ago)

Photo of md5-8813087cccc512313602b6d9f9ece19f

Jason

01:47 PM

If you’re using docsearch scraper, then that takes care of creating the collection for you. It creates a new collection on each run, then deletes the old collection, and updates the collection alias to point to the new collection

Apr 11, 2023 (8 months ago)

Rotfisch

07:21 AM

Thank you for your answers. The point is that I want to add new documents to my old collection. When I create a new index with the documents to be added and export them as a json file, they are created with new ids. Importing this with acton=create it will always leads to an error when imported into my old collection because the index numbers already exist. Docsearch scraper maps to my alias, but the old collection is deleted, so the collection contains only the new documents I wanted to have together with the old documents in one collection. The old documents are lost.

Jason

01:14 PM

It sounds like you’re trying to manually add documents into the collection created by docsearch-scraper. This is not possible to do safely with the way the scraper works by default. So you would have to fork it, and make changes to it, so it doesn’t delete the old collection…

Typesense

Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI

Indexed 3011 threads (79% resolved)

Join Our Community

Similar Threads

Crawler Deleting Old Collection and Creating New Name

James faced issues with Typesense as the crawler changed collection names, breaking their production website. Jason suggested changing "index_name" in their config file to their desired name and explained the reason behind the name combination differences.

10mo

API Key Permissions for Typesense Docsearch Scraper

JP asked about configuring API key actions for reindexing a site using Typesense. Kishore Nallan clarified the required permission. Then, JP and Jason discussed specific permission configurations, and JP made a documentation update to illustrate their findings.

11mo

Adding and Updating Collections in Typesense Server

sonu asked about methods to add and update collections in a Typesense server. Jason advised that documents can be added individually or in bulk, and updates can be batched using the import function, even with a small React app with a MySQL database.

32mo