#community-help

Adding Data to Existing Collection with Typesense

TLDR Rotfisch wants to add new data to an existing collection in Typesense. Jainil suggested using action=create for importing documents, while Jason explained that merging collections is not possible without modifying the docsearch-scraper.

Powered by Struct AI
Apr 08, 2023 (8 months ago)
Rotfisch
Photo of md5-4fdfd972be680409429627667fd6b8bf
Rotfisch
05:02 PM
I am trying to add data to an existing collection with typesense. My first approach was to use docscraper to write new data to an existing one. This failed because the index already existed. Then I created a new collection, exported it and tried to import it into my existing collection. This didn't work because the id of the entries already exist. With the action mode upsert the entries would be overwritten.
What is the best way to add new data to a collection?
Jainil
Photo of md5-bea1e4531665b4608c47d00462429d3a
Jainil
05:37 PM
You should be able to use action=create, when importing your documents. That will create entries for ID's that don't exist, and fail for ID's that do exist.
Apr 09, 2023 (8 months ago)
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
01:47 PM
If you’re using docsearch scraper, then that takes care of creating the collection for you. It creates a new collection on each run, then deletes the old collection, and updates the collection alias to point to the new collection
Apr 11, 2023 (8 months ago)
Rotfisch
Photo of md5-4fdfd972be680409429627667fd6b8bf
Rotfisch
07:21 AM
Thank you for your answers. The point is that I want to add new documents to my old collection. When I create a new index with the documents to be added and export them as a json file, they are created with new ids. Importing this with acton=create it will always leads to an error when imported into my old collection because the index numbers already exist. Docsearch scraper maps to my alias, but the old collection is deleted, so the collection contains only the new documents I wanted to have together with the old documents in one collection. The old documents are lost.
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
01:14 PM
It sounds like you’re trying to manually add documents into the collection created by docsearch-scraper. This is not possible to do safely with the way the scraper works by default. So you would have to fork it, and make changes to it, so it doesn’t delete the old collection…