#community-help

Resolving Error with Incorrect Field Type during Schema Update

TLDR Samuel encountered an error updating the schema for an existing collection. Kishore Nallan suggested specifying a concrete type, and later found inconsistent data within the collection that caused the error.

Powered by Struct AI

5

2

Mar 01, 2023 (9 months ago)
Samuel
Photo of md5-7f4ba9a2fc0e9dd5200c8ff347f1ea6e
Samuel
10:28 AM
Hello, It is possible to find source of Request failed with HTTP code 400 | Server said: Field ** has an incorrect type. error when updating schema on already existing collection? I have tried to export data and check for any corrupted fields but wasn't able to find anything wrong
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
10:34 AM
Might be some edge case because field name ** sounds weird. Are you able to share the data directory with me via DM so I can take a look? You can just zip it and share if it's something you are comfortable sharing.
Samuel
Photo of md5-7f4ba9a2fc0e9dd5200c8ff347f1ea6e
Samuel
10:35 AM
I'm running on cloud typesense and field name is not ** I just recorded it for purpose of sending this message original error is Error: e: Request failed with HTTP code 400 | Server said: Field properties.WORK.assignee.value has an incorrect type.
10:36
Samuel
10:36 AM
where properties.WORK.assignee.value can be null | [] | string[]
10:36
Samuel
10:36 AM
{
  "fields": [
    {
      "facet": true,
      "index": true,
      "infix": false,
      "locale": "",
      "name": "properties.WORK.assignee.value",
      "optional": true,
      "sort": false,
      "type": "auto"
    }
  ]
}
10:36
Samuel
10:36 AM
This is schema update I'm trying to apply
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
10:40 AM
There was an issue that I fixed a couple of weeks back that involved a nested value being null. This might be related to it. Are any of the existing records null on-disk (for this field)?
Samuel
Photo of md5-7f4ba9a2fc0e9dd5200c8ff347f1ea6e
Samuel
10:41 AM
Yes there are some
10:41
Samuel
10:41 AM
Cluster has Typesense v0.24.0 version might be related ?
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
10:42 AM
Hold on let check if that could be an issue.
10:42
Kishore Nallan
10:42 AM
*that == null values

1

1

10:52
Kishore Nallan
10:52 AM
Hmm I am not able to reproduce that on a simple collection. Mind if I take a look at your collection? Can you DM me the cluster ID and collection name.

1

11:19
Kishore Nallan
11:19 AM
Ah, I just realized that the type cannot be auto --> it must be a concrete type.
11:19
Kishore Nallan
11:19 AM
While the type can be auto during schema declaration, for schema changes, it needs to be a concrete type.
11:22
Kishore Nallan
11:22 AM
Try using string[] --> it should work!
Samuel
Photo of md5-7f4ba9a2fc0e9dd5200c8ff347f1ea6e
Samuel
12:08 PM
Thank you, works perfectly :))

1

Mar 17, 2023 (9 months ago)
Samuel
Photo of md5-7f4ba9a2fc0e9dd5200c8ff347f1ea6e
Samuel
08:20 AM
I have this problem again on same cluster 😞
08:21
Samuel
08:21 AM
Image 1 for
08:21
Samuel
08:21 AM
Image 1 for
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
08:22 AM
Do you already have records with that value but with a different type?
Samuel
Photo of md5-7f4ba9a2fc0e9dd5200c8ff347f1ea6e
Samuel
08:24 AM
It worked perfectly until yesterday but something has happened and our schema got "reseted" without any action from us. After that I cannot update schema back to support this field. I believe that if this field configuration was present every insert of document with invalid data would get rejected. Or is there different behaviour?
08:24
Samuel
08:24 AM
I can download data again a check the types there if needed
09:04
Samuel
09:04 AM
Didn't find any inconsistency in our data
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
09:30 AM
Ok let me check. Give me an hour or so.

1

11:25
Kishore Nallan
11:25 AM
Can you please post the update schema you tried (in the screenshot) as text here?
Samuel
Photo of md5-7f4ba9a2fc0e9dd5200c8ff347f1ea6e
Samuel
11:26 AM
sure
11:27
Samuel
11:27 AM
{
  "fields": [
    {
      "facet": true,
      "index": true,
      "infix": false,
      "locale": "",
      "name": "properties.WORK.assignee.value",
      "optional": true,
      "sort": false,
      "type": "string[]"
    }
  ]
}
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
11:34 AM
Thanks, debugging now. Will keep you posted shortly.

1

11:40
Kishore Nallan
11:40 AM
properties.WORK.assignee.value seems to be an array of objects?
Samuel
Photo of md5-7f4ba9a2fc0e9dd5200c8ff347f1ea6e
Samuel
11:40 AM
No it's always array of string
11:41
Samuel
11:41 AM
[{
  "lastUpdateBy": "KOBQAiHm8eTYmdYDSJtHfnxOBIv1",
  "updatedAt": {
    "_nanoseconds": 583000000,
    "_seconds": 1676508480
  },
  "value": [
    "KOBQAiHm8eTYmdYDSJtHfnxOBIv1"
  ]
}
,{
  "lastUpdateBy": "AJSUE9y9RtZw5BY7AIqafyPHhgT2",
  "updatedAt": {
    "_nanoseconds": 415000000,
    "_seconds": 1675938146
  },
  "value": [
    "AJSUE9y9RtZw5BY7AIqafyPHhgT2"
  ]
}]
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
11:41 AM
See here I found one with an array of objects:

"properties":{
      "WORK":{
         "assignee":{
            "lastUpdateBy":"CWzdjyOShMg4bHeGO4NkJzigLnE2",
            "updatedAt":{
               "_nanoseconds":660000000,
               "_seconds":1678966234
            },
            "value":[
               {
                  "displayName":"Social media manager",
                  "id":"Social media manager"
               }
            ]
         },
Samuel
Photo of md5-7f4ba9a2fc0e9dd5200c8ff347f1ea6e
Samuel
11:41 AM
oh wow
11:42
Samuel
11:42 AM
I'm sorry, I did all I was able via jq '.properties.WORK.assignee | select(.properties...' but didn't found that object
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
11:42 AM
Sorry for the trouble here -- we really should be surfacing the id of the bad document along with the error message
11:43
Kishore Nallan
11:43 AM
Check doc id NR4FCzUQgqjQGtP6Tzzy
11:49
Kishore Nallan
11:49 AM
The other thing I just realized we are unable to support right now is to add a new object[] field. This is a limitation that we have to address.
Samuel
Photo of md5-7f4ba9a2fc0e9dd5200c8ff347f1ea6e
Samuel
11:49 AM
Thank you for your help! I found the docs with invalid data and sorry I took your time
11:50
Samuel
11:50 AM
just one question, If I will fix those bad data, and I will update the schema, then I will try to insert document with incos. data will it get rejected ?
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
11:50 AM
No worry. I truly consider every question an area for us to improve on.
Samuel
Photo of md5-7f4ba9a2fc0e9dd5200c8ff347f1ea6e
Samuel
11:52 AM
for example I will have set a correct schema type now to string[] and I will do the document.insert() which will contain invalid data. Will it get rejected ?
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
11:53 AM
I don't get your question. If you fix that document and retry the schema update you have to target primitive types like string [] yes
11:54
Kishore Nallan
11:54 AM
Once you have a schema change done, subsequent docs inserted have to conform

1

Typesense

Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI

Indexed 3015 threads (79% resolved)

Join Our Community

Similar Threads

Issue with Field Indexing and Multiple Data Types

Raymond encountered an issue where a field seemed to be indexed twice and hence couldn't be deleted. Jason advised upgrading to a patch version, but the problem remained. Kishore Nallan suspected a race condition and an issue with conflicting data types. An effective solution wasn't achieved.

5

51
17mo

Issues with Schema Creation and Nested Fields

Sean encountered a problem with schema creation involving auto nested fields. Kishore Nallan suggested checking the API response for errors and adding problematic fields to the schema as optional. Sean confirmed the advice.

1

9
2mo

Troubleshooting Invalid Field Error in Firestore Document Indexing

Darren receives an error when indexing Firestore documents with empty array in "_grades" field. Jason suggests submitting a bug report and manually setting the schema. The user still experiences issues. Kishore Nallan reproduces the bug, but suggests a solution might exist with an explicit 'string[]' type definition. Further investigation is needed.

1

22
25mo

Handling Kinesis Stream Event Batching with Typesense

Dui had questions about how to handle Kinesis stream events with Typesense. Kishore Nallan suggested using upsert mode for creation/update and differentiating with logical deletion. After various discussions including identifying and resolving a bug, they finalized to introduce an `emplace` action in Typesense v0.23.

8

91
24mo

Cold Start Problem with Dynamic Collections

Adrian reported cold start issues with dynamic collections. Jason suggested using wildcard `*` for query_by parameters, upgrading to `0.25.0.rc34`, and clarified conventions. Adrian's issues were resolved but they reported a limitation that will potentially be addressed.

6

39
6mo