#community-help

Ensuring Uniqueness During Records Creation in Typesense

TLDR Krish inquired about ensuring uniqueness during record creation in Typesense. Kishore Nallan explained that uniqueness can be maintained using the 'id' field in documents, and when creating documents, the system automatically assigns a unique id if none is provided.

9mo

Join the chat

Dec 14, 2022 (9 months ago)

Photo of md5-7dff5651efcba5c9cdfac93918dc7eeb

Krish

08:20 AM

Another question: while creating records how do I ensure uniqueness? I tried looking up your doc, but didn't find anything. Does Typesense check for dup documents while creation or should the client implement it?

Photo of md5-4e872368b2b2668460205b409e95c2ea

Kishore Nallan

11:04 AM

The id field in the document is used as the identifier by convention. It can be used to ensure uniqueness.

Krish

11:31 AM

So, are you saying that it would error out if there is an id collision during a create() call?

Kishore Nallan

11:31 AM

Yes

Krish

11:32 AM

Which also means, the client will have to supply a custom id for the create calls based on some algo, right?

Kishore Nallan

11:33 AM

If no id is provided then id is autoassigned in running sequence. If an id is provided we will use that.

Krish

11:34 AM

got it, essentially if no id is provided we'd never run into an id collision during a create() call.

Kishore Nallan

11:34 AM

Correct. During create call, if you pass the return_id=true flag then the created id is returned (in 0.24 RC)

Krish

11:35 AM

ahh, interesting... ok I on 0.23, not sure if its there in it, But it doesn't make sense for the client to keep track of ids, right?

11:37

Krish

11:37 AM

I was thinking of generating a number of type int32 using crc32 hashing mechanism on the doc. Would you recommend anything else?

Kishore Nallan

11:38 AM

id must be a string. You can just hash the document to produce a SHA and use that as ID.

Krish

11:38 AM

wouldn't sha be compute heavy?

Kishore Nallan

11:39 AM

Depends, but there are other non-cryptographic hashes available which might be faster.

Krish

11:40 AM

could you recommend one? One you thing would be faster?

Kishore Nallan

11:40 AM

murmurhash or xxhash

Krish

11:40 AM

All this while I thought ID has to be a number. Glad its a string

Kishore Nallan

11:41 AM

The Typesense assigned id happens to be an integer simply because we have a running sequence to choose from that won't duplicate but it can be anything.

Krish

11:41 AM

awesome ,,,

11:42

Krish

11:42 AM

Thank you Kishore. I think I have what I need to proceed.

Typesense

Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI

Indexed 2764 threads (79% resolved)

Join Our Community Manage subscription

Similar Threads

Understanding Document ID Fields and Rectifying Duplicate Document Error

John queried about understanding document ID fields and fixed a duplication issue with guidance from Jason. They discovered a bug preventing document deletion due to a URL encodable character, with John opening a GitHub issue for it.

17mo

Issue with Typesense Schema and Ruby Client

Mateo faces issues while creating a schema and using Ruby client for Typesense. Jason suggests using a new field instead of 'id' and provides assistance for Ruby client errors.

7mo

Handling Kinesis Stream Event Batching with Typesense

Dui had questions about how to handle Kinesis stream events with Typesense. Kishore Nallan suggested using upsert mode for creation/update and differentiating with logical deletion. After various discussions including identifying and resolving a bug, they finalized to introduce an `emplace` action in Typesense v0.23.

20mo

Troubleshooting Create Index Issue on Typesense Instance

John was having issues creating new indices on a Typesense instance which Jason suggested could be due to old bugs or heavy search queries monopolizing CPU. After a restart, the issue was resolved. Jason also recommended upgrading the Typesense version.

Threading Problem During Multiple Collection Creation and Batch Insertion in Typesense

Johan has a problem with creating multiple collections and batch-inserting documents into Typesense, which is returning results from different collections. Kishore Nallan helps troubleshoot the issue and suggests a potential local race condition, which is fixed in a later build.

14mo