@Kishore Nallan i was implementing the cron job for batch update to typesense but adding boolean  field of is_deleted in mysql won't do it,  as i won't change boolean value of it after deleting from typesense should i use a file and save ids to track it or just alter the table to have something like is_deleted_typesense to track deleted ids from typesense.... any suggestions ?

Here is what I had in mind: 1. Each row in your mysql table has

updated_at

and

is_deleted

2. When a record is deleted the

is_deleted

is set to

true

and the

updated_at

also should be updated. 3. When a record is updated, similarly, the

updated_at

field should also be updated. 4. When the Typesense sync cron job runs, it will query for the records that need to be synced with this query:

SELECT * FROM table WHERE updated_at < NOW() AND id > $prev_id ORDER BY updated_at ASC, id ASC

5. The

$prev_id

value is the largest record ID processed in every cron -- this should be stored after every cron and is referenced for the next cron. 6. We use

updated_at < NOW()

to prevent picking up records currently being written to. 7. To prune your database of deleted records, you can run another weekly cron that just deletes all the records which have the

is_deleted

flag set but is older than a few days old (to prevent interfering with Typesense cron which runs more often)

got it