#community-help

Resolving Auto-Schema and Configuration Issues in Typesense

TLDR Narayan was struggling with auto-schema and configuration issues in Typesense. With the help of Kishore Nallan, they understood and solved the problems by adding 'optional' to all nested fields. They will find a way to handle 'None'.

Powered by Struct AI

1

Sep 22, 2023 (2 months ago)
Narayan
Photo of md5-6441fd8d08a3a44e184f7cba8ddb1d72
Narayan
03:05 AM
Is there a way to disable auto-schema? It introduces additional fields in my base schema for nested objects fields and then expects them to be or array-type. This prevents me to add those objects to typesense. Also, the object type field is also set to not be indexed. If I specify a schema I would expect that auto schema does not kick in😅
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
03:08 AM
In Typesense a field can have only one type. So the types of children of object fields are stored in the schema and validated. If you define a person field as a type of object and then index an object with a person.name string field, this information is added to the schema so that we can ensure that future person.name values are also string.
Narayan
Photo of md5-6441fd8d08a3a44e184f7cba8ddb1d72
Narayan
05:03 AM
Thanks. I re-read the docs and I think I understand now. I ran into some other issues with the data I was importing and the schema expectations.
05:05
Narayan
05:05 AM
Can you help me with how I should configure the schema for this case:

Schema
{
    "name": "Company",
    "enable_nested_fields": True,
    "fields": [
        {
            "name": "id",
            "type": "string"
        },
        {
            "name": "persons",
            "type": "object[]",
            "index": False,
            "optional": True,
        },
        {
            "name": "persons.id",
            "type": "int64[]"
        },
        {
            "name": "persons.addresses.zip",
            "type": "string[]"
        }
    ]
}

When I try to import the following docs typesense complain:
[
    {
        "id": "131415",
        "persons": []
    },
    {
        "id": "789",
        "persons": [
            {
                "id": 1,
                "addresses": [{"zip": None}]
            },
            {
                "id": 2,
                "addresses": [{"zip": "11112"}]
            },
            {
                "id": 3,
                "addresses": [{"zip": "11113"}]
            }
        ]
    },{
        "id": "1011112",
        "persons": [
            {
                "id": 4,
                "addresses": [{"zip": "11111"}]
            },
            {
                "id": 5,
                "addresses": [{"zip": "11112"}]
            },
            {
                "id": 6,
                "addresses": [{"zip": "11115"}]
            }
        ]
    }
]

The errors:
[{'code': 400, 'document': '{"id": "131415", "persons": []}', 'error': 'Field `persons.addresses.zip` has been declared in the schema, but is not found in the document.', 'success': False}]
05:06
Narayan
05:06 AM
I’ve also tried to mark the fields persons.id and persons.addresses.zip as optional but it still complains for the first two docs. Only when I provide all the necessary fields and none of them are None , like in the 3rd case, it works.
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
05:08 AM
Let me try running that, but is there a reason for defining those child fields?
Narayan
Photo of md5-6441fd8d08a3a44e184f7cba8ddb1d72
Narayan
05:10 AM
this is just a toy example. but I need to index these fields for faster search/filtering based on these properties.
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
05:10 AM
The issue is this empty doc:

 {
        "id": "131415",
        "persons": []
    },
Narayan
Photo of md5-6441fd8d08a3a44e184f7cba8ddb1d72
Narayan
05:12 AM
but I’ve marked persons as optional. Should I instead just use {"id":131415} when there are no persons?
05:12
Narayan
05:12 AM
btw, {"id":131415} this too does not work and still raises the same error
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
05:13 AM
There is also:

"addresses": [{"zip": None}]
Narayan
Photo of md5-6441fd8d08a3a44e184f7cba8ddb1d72
Narayan
05:13 AM
yep, how should I configure the schema so that these cases can be still be added to typesense?
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
05:13 AM
You have defined persons as optional but have explicit child reference which is not marked as optional, so the explicit child field paths get prirority.
05:14
Kishore Nallan
05:14 AM
Add "optional": True, to the child fields
05:15
Kishore Nallan
05:15 AM
And avoid having None
Narayan
Photo of md5-6441fd8d08a3a44e184f7cba8ddb1d72
Narayan
05:16 AM
let me give that a try.
05:21
Narayan
05:21 AM
adding optional to all nested fields helped. thanks! I’ll figure out how to deal with None

1

Typesense

Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI

Indexed 3015 threads (79% resolved)

Join Our Community