#community-help

Troubleshooting Invalid Field Error in Firestore Document Indexing

TLDR Darren receives an error when indexing Firestore documents with empty array in "_grades" field. Jason suggests submitting a bug report and manually setting the schema. The user still experiences issues. Kishore Nallan reproduces the bug, but suggests a solution might exist with an explicit 'string[]' type definition. Further investigation is needed.

Powered by Struct AI

1

Dec 22, 2021 (25 months ago)
Darren
Photo of md5-fa83de4db3e1444aecaf7a587e35eb16
Darren
05:15 PM
Can anyone help with an error I don't understand:

The document (from Firestore) I'm trying to index looks like this:
{
>    _location: null,
>    description: null,
>    label: 'Kindergarten',
>    type: 'grade',
>    meta: { created: { at: null, by: null }, updated: { at: null, by: null } },
>    _grades: [],
>    _people: [],
>    tenant: 'one',
>    collection: 'groups',
>    id: 'grade-kindergarten-all'
}

The collection schema is:

{
    "created_at": 1640193019,
    "default_sorting_field": "",
    "fields": [
        {
            "facet": false,
            "index": true,
            "name": ".*",
            "optional": true,
            "type": "auto"
        },
        {
            "facet": false,
            "index": true,
            "name": "label",
            "optional": true,
            "type": "string"
        }
    ],
    "name": "groups",
    "num_documents": 0,
    "symbols_to_index": [],
    "token_separators": []

... and the error is:

>  #push RequestMalformed [Error]: Request failed with HTTP code 400 | Server said: Type of field `_grades` is invalid.

So seems like the _grades field (which has avalue of an empty array [] ) is causing a problem ... is this expected?
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
05:16 PM
Could you share the exact error you see?
Darren
Photo of md5-fa83de4db3e1444aecaf7a587e35eb16
Darren
05:18 PM
Sorry, I posted early ... I was updating when you posted. Thanks Jason
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
05:20 PM
Hmm, I wonder if there are documents where grades is not an array, so docs with a different data type are being rejected. Could you do a GET /collections and share the schema output from there? That should show what grades's auto-detected type
Darren
Photo of md5-fa83de4db3e1444aecaf7a587e35eb16
Darren
05:21 PM
I already posted that
05:21
Darren
05:21 PM
See above.
"fields": [
        {
            "facet": false,
            "index": true,
            "name": ".*",
            "optional": true,
            "type": "auto"
        },
        {
            "facet": false,
            "index": true,
            "name": "label",
            "optional": true,
            "type": "string"
        }
    ],
05:22
Darren
05:22 PM
FYI the collection is currently empty
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
05:22 PM
Oh I see
05:23
Jason
05:23 PM
This sounds like a bug then... We should definitely allow empty arrays to be sent through. Could you open a Github issue describing this?
Darren
Photo of md5-fa83de4db3e1444aecaf7a587e35eb16
Darren
05:23 PM
GET /collections/groups

{
    "facet_counts": [],
    "found": 0,
    "hits": [],
    "out_of": 0,
    "page": 1,
    "request_params": {
        "collection_name": "groups",
        "per_page": 10,
        "q": "*"
    },
    "search_cutoff": false,
    "search_time_ms": 0
}
05:23
Darren
05:23 PM
Yeah
05:24
Darren
05:24 PM
In the meantime do you have any idea how I can work around this? I could remove the field in my serializer before I commit in the cloud function, I guess ...
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
05:24 PM
Could you try adding grades explicitly to the schema as an array field?
05:25
Jason
05:25 PM
I suspect this might be an issue with just auto-schema detection
05:29
Darren
05:29 PM
Ok, I'll manually set the schema and post back if it makes a difference
05:38
Darren
05:38 PM
Jason So does seem its a bug. When turning off auto schema and setting type of _grades and _people to string[] that record still fails. But, when I add a different record FIRST that has data (see below) then after that I can indeed save the record that was failing

...here is that record that succeeds first time

>  parse {
>    _location: null,
>    description: null,
>    label: 'Grade Eight',
>    type: 'grade',
>    meta: { created: { at: null, by: null }, updated: { at: null, by: null } },
>    _grades: [],
>    _people: [
>      'tenants/one/people/eli-schreiber',
>      'tenants/one/people/miriam-schreiber',
>      'tenants/one/people/nathaniel-ewen',
>      'tenants/one/people/talia-ewen',
>      'tenants/one/people/seth-schwartzberg',
>      'tenants/one/people/maya-schwartzberg',
>      'tenants/one/people/michael-meyer',
>      'tenants/one/people/sammi-meyer',
>      'tenants/one/people/jeremiah-engel',
>      'tenants/one/people/ruth-engel',
>      'tenants/one/people/jeremy-maltzman',
>      'tenants/one/people/deborah-maltzman',
>      'tenants/one/people/matt-glaser',
>      'tenants/one/people/chava-glaser',
>      'tenants/one/people/sam-aaronson',
>      'tenants/one/people/chana-aaronson',
>      'tenants/one/people/isaac-feldman',
>      'tenants/one/people/sarah-feldman',
>      'tenants/one/people/josh-feinberg',
>      'tenants/one/people/eve-feinberg'
>    ],
>    tenant: 'one',
>    collection: 'groups',
>    id: 'grade-eight-all'
>  }

Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
05:41 PM
Oh boy! Worse than I thought... Could you add this to the github issue?
Darren
Photo of md5-fa83de4db3e1444aecaf7a587e35eb16
Darren
05:57 PM
Sure ...
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
07:11 PM
Will take a look and keep you posted
Dec 23, 2021 (25 months ago)
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
05:26 AM
Darren I'm able to reproduce the bug with the auto field. However, with an explicit string[] type definition, empty values are allowed as the very first document.

See here: https://gist.github.com/kishorenc/131476e9fb898381821f8aa026ec3fd0

(I tested this on Typesense 0.22.1)

If you can create a similar snippet for reproduction, it will help. Thanks.
Dec 26, 2021 (25 months ago)
Darren
Photo of md5-fa83de4db3e1444aecaf7a587e35eb16
Darren
11:03 PM
Sorry for slow reply, will take a look at this next week ... Kishore Nallan

Typesense

Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI

Indexed 3015 threads (79% resolved)

Join Our Community

Similar Threads

Cold Start Problem with Dynamic Collections

Adrian reported cold start issues with dynamic collections. Jason suggested using wildcard `*` for query_by parameters, upgrading to `0.25.0.rc34`, and clarified conventions. Adrian's issues were resolved but they reported a limitation that will potentially be addressed.

6

39
6mo

Issues with Schema Creation and Nested Fields

Sean encountered a problem with schema creation involving auto nested fields. Kishore Nallan suggested checking the API response for errors and adding problematic fields to the schema as optional. Sean confirmed the advice.

1

9
2mo

Troubleshooting 400 Error When Upgrading Typesense Firestore Extension

Orion experienced a `400` error after updating the Typesense Firestore extension, causing issues with cloud functions. They traced the issue back to a data type conflict in their Typesense collection schema after updating. With help from Jason and Kishore Nallan, they resolved the issue by recreating the collection.

5

96
14mo

Resolving Error with Incorrect Field Type during Schema Update

Samuel encountered an error updating the schema for an existing collection. Kishore Nallan suggested specifying a concrete type, and later found inconsistent data within the collection that caused the error.

7

43
9mo

Issue with Embedding Error in Version 0.25.0.rc63

Bill reported a bug in version 0.25.0.rc63 regarding a problem with updating or emplacing a document and receiving an embedding error. This was resolved in version 0.25.0.rc65, but further discussion ensued regarding the function of 'index' in the update feature.

5

63
4mo