Issue with Field Indexing and Multiple Data Types
TLDR Raymond encountered an issue where a field seemed to be indexed twice and hence couldn't be deleted. Jason advised upgrading to a patch version, but the problem remained. Kishore Nallan suspected a race condition and an issue with conflicting data types. An effective solution wasn't achieved.
2
1
1
1
Aug 10, 2022 (14 months ago)
Raymond
06:46 PMI can't seem to delete either them. I've also attached the response for the delete operation
Jason
06:53 PMNo, this shouldn’t be possible. Unless there’s a bug. May I know what the left side vs right side content is in your diff?
> I’ve also attached the response for the delete operation
Hmm, I vaguely remember us fixing an issue related to auto fields and dropping in v0.23.1. Could you try this after upgrading to that patch version?
Jason
06:54 PMJason
06:54 PM1
Raymond
06:54 PMJason
06:55 PM1
Raymond
06:55 PMBoth are existing fields in the schema, both represent date fields from GitHub but some webhooks send it as string and some as int64, ideally it should not create a new one, right?
Jason
06:56 PMJason
06:56 PMRaymond
06:57 PMJason
06:58 PMRaymond
06:59 PMJason
07:40 PM1
Jason
07:41 PMRaymond
07:43 PMJason
07:47 PMdata.repository.created_at
. Could you share the curl command you’re using along with the output?Raymond
07:47 PMJason
07:48 PMRaymond
07:50 PM{
"message": "Schema change is incompatible with the type of documents already stored in this collection. Existing data for field `data.repository.pushed_at` cannot be coerced into a string."
}
Jason
07:51 PMRaymond
07:52 PMJason
07:55 PMRaymond
07:59 PMAug 11, 2022 (14 months ago)
Kishore Nallan
03:48 AM1. Somehow
data.repository.created_at
has ended up with two different string
and int64
types in the schema.2. There is also another issue of
data.repository.pushed_at
having a different type on-disk compared to the schema which prevents schema modification.I will look into why these happened.
Kishore Nallan
04:13 AMRaymond
10:41 AMThat might have caused the issue with the
date string
to int64
conversionKishore Nallan
10:42 AMRaymond
10:43 AMdata.repository.pushed_at
having a different type on-disk compared to the schema which prevents schema modification.For this case I sincerely don't know what to do about it because I already dropped the field.
Raymond
10:43 AMThis is the nodejs code https://github.com/frain-dev/indexer/blob/master/index.js
Kishore Nallan
10:44 AMRaymond
10:46 AMRaymond
10:46 AMRaymond
10:46 AMKishore Nallan
10:47 AM1
Kishore Nallan
10:50 AMSo any schema operation is going to fail.
Raymond
10:52 AMKishore Nallan
10:57 AMRaymond
10:58 AMKishore Nallan
10:59 AMRaymond
10:59 AMKishore Nallan
11:00 AMKishore Nallan
11:00 AMRaymond
11:01 AMKishore Nallan
11:02 AMRaymond
11:02 AMKishore Nallan
11:22 AM1
Kishore Nallan
12:56 PMRaymond
12:57 PMKishore Nallan
01:05 PMAug 12, 2022 (14 months ago)
Raymond
08:12 AMKishore Nallan
08:34 AMTypesense
Indexed 2786 threads (79% resolved)
Similar Threads
Handling Kinesis Stream Event Batching with Typesense
Dui had questions about how to handle Kinesis stream events with Typesense. Kishore Nallan suggested using upsert mode for creation/update and differentiating with logical deletion. After various discussions including identifying and resolving a bug, they finalized to introduce an `emplace` action in Typesense v0.23.
Troubleshooting 400 Error When Upgrading Typesense Firestore Extension
Orion experienced a `400` error after updating the Typesense Firestore extension, causing issues with cloud functions. They traced the issue back to a data type conflict in their Typesense collection schema after updating. With help from Jason and Kishore Nallan, they resolved the issue by recreating the collection.
Threading Problem During Multiple Collection Creation and Batch Insertion in Typesense
Johan has a problem with creating multiple collections and batch-inserting documents into Typesense, which is returning results from different collections. Kishore Nallan helps troubleshoot the issue and suggests a potential local race condition, which is fixed in a later build.
Large JSONL Documents Import Issue & Resolution
Suraj was having trouble loading large JSONL documents into Typesense server. After several discussions and attempts, it was discovered that the issue was due to data quality. Once the team extracted the data again, the upload process worked smoothly.
Cold Start Problem with Dynamic Collections
Adrian reported cold start issues with dynamic collections. Jason suggested using wildcard `*` for query_by parameters, upgrading to `0.25.0.rc34`, and clarified conventions. Adrian's issues were resolved but they reported a limitation that will potentially be addressed.