#community-help

Adding new field to a collection issue in Typesense

TLDR Ankit receives a 'request malformed' error while adding a new field to a Typesense collection. Kishore Nallan explains the issue is due to schema validation. They advise creating a Github issue for supporting coercion. After Ankit reproduces the issue and shares details, Kishore Nallan acknowledges the issue and promises to investigate.

Powered by Struct AI

1

18
2mo
Solved
Join the chat
Sep 07, 2023 (2 months ago)
Ankit
Photo of md5-d9ca032e3941589aafa8433269974f96
Ankit
10:41 AM
Hey all, quick question, I just wanted to add a new field to a collection and I get this error
typesense.exceptions.RequestMalformed: [Errno 400] Schema change is incompatible with the type of documents already stored in this collection. Existing data for field `XXX` cannot be coerced into an array of string.

The XXX field in the schema is an object field that isn't indexed and just to show some display values and when adding to it we use coerce_or_drop. Why is another field being coerced when adding a non-related field? What would be the correct way to add a new field without caring about the schema within this object field.
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
10:42 AM
Doesn't XXX already exist in the schema?
Ankit
Photo of md5-d9ca032e3941589aafa8433269974f96
Ankit
10:43 AM
Yeah, "XXX" already exists, I was a completely new field.
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
10:44 AM
During an alter we do a full schema validation so if there any inconsistencies with any field then that can show up as an error.
Ankit
Photo of md5-d9ca032e3941589aafa8433269974f96
Ankit
10:44 AM
Ah, okay. Any way to avoid that as of now? Like coerce_or_drop is there for the upsert.
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
10:45 AM
Alter doesn't support coercion coercion yet.
10:45
Kishore Nallan
10:45 AM
You have to fix the data to conform to schema first by reindexing. Or just create new collection with updated schema and index into it.
Ankit
Photo of md5-d9ca032e3941589aafa8433269974f96
Ankit
10:46 AM
Got it. Thanks!
Any plans or a ticket already in place to add a feature to avoid schema validation while doing schema updates on all fields or fields to exclude?
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
10:49 AM
I think the correct fix would be to support coercion. Would you be able to create a small example that showcases the problem and then create a Github issue with that? We do have alter improvements planned in near future and we will prioritize this as part of that.
Ankit
Photo of md5-d9ca032e3941589aafa8433269974f96
Ankit
10:50 AM
Yeah, sure. Will create the issue.
11:00
Ankit
11:00 AM
https://github.com/typesense/typesense/issues/1211

Let me know if you need additional details.
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
11:10 AM
One thing I don't understand is if XXX is defined as object how come it's an array on some records.
Ankit
Photo of md5-d9ca032e3941589aafa8433269974f96
Ankit
11:27 AM
XXX.A.B is an array of strings.
In our schema, XXX is a display object, so we didn't care about the schema much if data is available there we display it. So the keys within the object may be strings, arrays or objects.
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
11:29 AM
It'll be great if you can reproduce this issue on a test collection so you can post the exact schema, sample data and the sequence of api calls that led to this error.
Ankit
Photo of md5-d9ca032e3941589aafa8433269974f96
Ankit
11:30 AM
Okay. Will try and reproduce it with data as this came up in our collections with 4m+ records and we are still identifying the bad data and how that got it.
Once identified I'll update the issue.
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
11:30 AM
Thanks
Ankit
Photo of md5-d9ca032e3941589aafa8433269974f96
Ankit
05:04 PM
Updated the ticket.
Sep 08, 2023 (2 months ago)
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
12:17 AM
Thank you 🙏 the x.a is an array in one nested entry but a plain string in another. While this is handled by ingestion but not by validation. Will look into it.

1

Typesense

Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI

Indexed 3011 threads (79% resolved)

Join Our Community

Similar Threads

Threading Problem During Multiple Collection Creation and Batch Insertion in Typesense

Johan has a problem with creating multiple collections and batch-inserting documents into Typesense, which is returning results from different collections. Kishore Nallan helps troubleshoot the issue and suggests a potential local race condition, which is fixed in a later build.

35
17mo
Solved

Issue with Field Indexing and Multiple Data Types

Raymond encountered an issue where a field seemed to be indexed twice and hence couldn't be deleted. Jason advised upgrading to a patch version, but the problem remained. Kishore Nallan suspected a race condition and an issue with conflicting data types. An effective solution wasn't achieved.

5

51
16mo

Issues with Schema Creation and Nested Fields

Sean encountered a problem with schema creation involving auto nested fields. Kishore Nallan suggested checking the API response for errors and adding problematic fields to the schema as optional. Sean confirmed the advice.

1

9
2mo
Solved

Methods for Fetching, Querying, and Modifying Collections in Typesense

Bill inquired about performing OR queries, querying empty arrays and modifying collections in Typesense. Kishore Nallan explained the current limitations and provided workarounds and recommendations for each case. The conversation also touched upon the usage of cache in Typesense and the workings of the _eval function.

5

52
10mo
Solved

Resolving Error with Incorrect Field Type during Schema Update

Samuel encountered an error updating the schema for an existing collection. Kishore Nallan suggested specifying a concrete type, and later found inconsistent data within the collection that caused the error.

7

43
9mo
Solved