Issues with Schema Creation and Nested Fields

I'm running into a strange issue when I create a schema with auto nested fields.
My code is set to index batches of 1000 documents at a time using the batchsize field
```client.collections[collName].documents.import(records, {'action': 'upsert', 'batch_size': 1000})

If I run the script for a minute and stop it after 50,000 documents i see 50,000 records using the retrieve function.
When I add `enable_nested_fields: True` to the schema, create a new index and process the same 50,000 records I see a fraction of those listed in the retrieve call ; `'num_documents': 3385` Does my schema look correct?

my_schema = {
"name": collName,
"enable_nested_fields": True,
"fields": [
{
'name' : 'title',
'type' : 'string',
'optional': True
},
{
'name' : 'description',
'type' : 'string',
'optional': True
},
{
'name' : 'product_type',
'type' : 'string',
'facet' : True,
'optional': True
},
{
'name' : 'vendor',
'type' : 'string',
'facet' : True,
'optional': True
},
{
'name' : 'sid',
'type' : 'int64',
'optional': True
},
{
'name' : 'tags',
'type' : 'string[]',
'facet' : True,
'optional': True
},
{
'name' : 'url',
'type' : 'string',
'index': False,
'optional': True
},
{
'name' : 'handle',
'type' : 'string',
'index': False,
'optional': True
},
{
'name' : 'last-updated',
'type' : 'int32',
'optional': True
},
{
'name' : 'missing',
'type' : 'bool',
'optional': True
},
{
"name" : "categories",
"type" : "string[]",
"optional": True
},
{"name": ".*", "type": "auto" }
]
}
client.collections.create(my_schema)```

Check the api response. If some docs are not indexed due to some issues, the errors will be present in the api response

Thank you for this. I was looking in the typsesense logs for errors and forgot to check the import response.

Error is error":"Field variants.option3 must be an array of string.", Looking at the actual documents I do see that some of them have variants.option3 as null. I don't need to index null values as listed on this typesense article.

What is the best practice in this scenario? Drop null fields before inserting or turn them into an empty string?

Add variants.option3 to schema and set it as optional field

Sounds good. I was planning on typing out the full schema anyways.

Interesting question! I'm not sure can you try it out? I presume it must work.

Doesn't appear to work. Schema is

my_schema = {
    "name": collName,
    "enable_nested_fields": True,
    "fields": [
        
        {"name": ".*", "type": "auto", "optional": True }
    ]
}

That's ok though i needed to type it out anyways for faceting.

Thanks for confirming.