Updating Typesense Indexes using Cron Jobs and MySql
TLDR sonu was unsure how to track deleted ids for batch updates to Typesense. Kishore Nallan proposed a detailed system involving MySql data and also referenced the Typesense API documentation. sonu understood the suggestion.
Jun 12, 2021 (30 months ago)
should i use a file and save ids to track it or just alter the table to have something like is_deleted_typesense to track deleted ids from typesense.... any suggestions ?
Kishore Nallan07:14 AM
1. Each row in your mysql table has
2. When a record is deleted the
is_deletedis set to
updated_atalso should be updated.
3. When a record is updated, similarly, the
updated_atfield should also be updated.
4. When the Typesense sync cron job runs, it will query for the records that need to be synced with this query:
SELECT * FROM table WHERE updated_at < NOW() AND id > $prev_id ORDER BY updated_at ASC, id ASC
$prev_idvalue is the largest record ID processed in every cron -- this should be stored after every cron and is referenced for the next cron.
6. We use
updated_at < NOW()to prevent picking up records currently being written to.
7. To prune your database of deleted records, you can run another weekly cron that just deletes all the records which have the
is_deletedflag set but is older than a few days old (to prevent interfering with Typesense cron which runs more often)
Kishore Nallan07:19 AM
is_deletedflag to decide whether you should be deleting the record from Typesense or updating Typesense. The update and delete batch operations must be separated since upserts and deletes cannot be batched together in the same API call.
Kishore Nallan07:20 AM
Jun 13, 2021 (30 months ago)
Indexed 2786 threads (79% resolved)
Handling Kinesis Stream Event Batching with Typesense
Dui had questions about how to handle Kinesis stream events with Typesense. Kishore Nallan suggested using upsert mode for creation/update and differentiating with logical deletion. After various discussions including identifying and resolving a bug, they finalized to introduce an `emplace` action in Typesense v0.23.
Discussing Data Changes and Typesense Update Methods
sonu asked about the efficient way to manage database updates in Typesense. Kishore Nallan suggested a cron-based approach or to queue updates, along with idea of 'is_deleted' flag to handle deletion.
Issue with Typesense Schema and Ruby Client
Mateo faces issues while creating a schema and using Ruby client for Typesense. Jason suggests using a new field instead of 'id' and provides assistance for Ruby client errors.