Hi team, We're on v29rc14 and have encountered the data corruption issue again. This was previously brought up in the 3 threads below, but starting a new thread here since this was expected to be resolved with v29. Could you please help us with any advice on the notes below. And more importantly, any insight on your side as to the cause and why v29 isn't helping? We have determined a reliable way to detect the issue is occurring, and something of a workaround when it occurs. Here are those details: 1. This is happening today on our "ktgh072..." cluster for many documents (including documentId 4598733). a. You can find the document by querying the ID, but not by searching for it using other fields as filter_by parameters. b. We have ~65k of these on that cluster. c. To assist in troubleshooting, we have left that cluster as is. We spun up a new cluster to use for production, where we will follow the mitigation steps noted below. But for as long as it would be helpful to your team, we can leave this static cluster in place. 2. To detect these: a. We do a collection export (of just the ID to keep it smaller and faster) and count the results b. We query the collection and check the "out_of" property c. The difference seems to indicate the documents that have become corrupted and are not searchable via normal filter_by 3. To resolve this issue, we found 2 approaches: a. We can drop the collection and rebuild it. This is a little more cumbersome because we have to recreate the collection and update our aliases. b. We can truncate the collection, and reload it. This is our current approach. 4. We've resorted to checking the counts every hour, so we can know when the problem occurs and decided about triggering our mitigation plan. a. Our current load process doesn't support loading a new collection and swapping aliases. We are considering taking on the work to mitigate the issue without downtime. 5. Now that we know how to check for the issue and are monitoring, we'll let you know if we see any pattern in the records that are affected or how quickly it builds up. Thread 1 Thread 2 Thread 3

Could you a curl request of the query you're using in Step 2b?

@Jason Bosco Does help in your troubleshooting for us to keep the old cluster up? The one that has corrupt records?

Oh sorry, I mentioned the step number wrong. Could you share the curl request with the filters in step 1a. That will help us replicate the same issue, and see what's happening

These are the curl requests:

curl --location '<https://hostname.typesense.net/collections/products_prod/documents/search?q=*&filter_by=productId%3A%3D4598733>'

And:

curl --location '<https://hostname.typesense.net/collections/products_prod/documents/search?q=*&filter_by=id%3A%3D4598733>'

BUT, I notice that now it is working again in this case. Yesterday, we switched to a new cluster, but we forgot to update the server responsible for the incremental updates. So, yesterday, during the whole day, that server was working on this collection. That situation, or something else, could have caused the reported inconsistency to be fixed.

Got it, we're looking into this. CC: @Kishore Nallan

Ok the problem is that there is a

manufacturer.mfrNumbers

field in the schema that is a nested string array and has a type

string[]

. But some documents on disk have that field as null:

"manufacturer": [
    {
      "id": 10544,
      "mfrAliases": null,
      "mfrNumber": "TSCG15R-16R",
      "mfrNumberClean": "TSCG15R16R",
      "mfrNumbers": null,
      "name": "Nordent Manufacturing, Inc."
    }
  ],

There are exactly 211 documents like this, which are not indexed in-memory and those are the exact ones that are not accessible via filtering. I need to now check why we allow this import in the first place. Are you able to insert the same payload successfully on a new collection?

We're checking the 3 ways we touch documents (bulk import with upsert, single document creation, single document update) to see if we can reproduce. We actually have 2 versions of each approach, because we have 2 joined collections and we use the asyn_reference=true. So we could do all 3 operations with and without related documents existing.