#community-help

Troubleshooting Invalid Field Error in Firestore Document Indexing

TLDR Darren receives an error when indexing Firestore documents with empty array in "_grades" field. Jason suggests submitting a bug report and manually setting the schema. The user still experiences issues. Kishore Nallan reproduces the bug, but suggests a solution might exist with an explicit 'string[]' type definition. Further investigation is needed.

25mo

Join the chat

Dec 22, 2021 (25 months ago)

Photo of md5-fa83de4db3e1444aecaf7a587e35eb16

Darren

05:15 PM

Can anyone help with an error I don't understand:

The document (from Firestore) I'm trying to index looks like this:

{
>    _location: null,
>    description: null,
>    label: 'Kindergarten',
>    type: 'grade',
>    meta: { created: { at: null, by: null }, updated: { at: null, by: null } },
>    _grades: [],
>    _people: [],
>    tenant: 'one',
>    collection: 'groups',
>    id: 'grade-kindergarten-all'
}

The collection schema is:

{
    "created_at": 1640193019,
    "default_sorting_field": "",
    "fields": [
        {
            "facet": false,
            "index": true,
            "name": ".*",
            "optional": true,
            "type": "auto"
        },
        {
            "facet": false,
            "index": true,
            "name": "label",
            "optional": true,
            "type": "string"
        }
    ],
    "name": "groups",
    "num_documents": 0,
    "symbols_to_index": [],
    "token_separators": []

... and the error is:

>  #push RequestMalformed [Error]: Request failed with HTTP code 400 | Server said: Type of field `_grades` is invalid.

So seems like the _grades field (which has avalue of an empty array [] ) is causing a problem ... is this expected?

Photo of md5-8813087cccc512313602b6d9f9ece19f

Jason

05:16 PM

Could you share the exact error you see?

Darren

05:18 PM

Sorry, I posted early ... I was updating when you posted. Thanks Jason

Jason

05:20 PM

Hmm, I wonder if there are documents where grades is not an array, so docs with a different data type are being rejected. Could you do a GET /collections and share the schema output from there? That should show what grades's auto-detected type

Darren

05:21 PM

I already posted that

05:21

Darren

05:21 PM

See above.

"fields": [
        {
            "facet": false,
            "index": true,
            "name": ".*",
            "optional": true,
            "type": "auto"
        },
        {
            "facet": false,
            "index": true,
            "name": "label",
            "optional": true,
            "type": "string"
        }
    ],

05:22

Darren

05:22 PM

FYI the collection is currently empty

Jason

05:22 PM

Oh I see

05:23

Jason

05:23 PM

This sounds like a bug then... We should definitely allow empty arrays to be sent through. Could you open a Github issue describing this?

Darren

05:23 PM

GET /collections/groups

{
    "facet_counts": [],
    "found": 0,
    "hits": [],
    "out_of": 0,
    "page": 1,
    "request_params": {
        "collection_name": "groups",
        "per_page": 10,
        "q": "*"
    },
    "search_cutoff": false,
    "search_time_ms": 0
}

05:23

Darren

05:23 PM

Yeah

05:24

Darren

05:24 PM

In the meantime do you have any idea how I can work around this? I could remove the field in my serializer before I commit in the cloud function, I guess ...

Jason

05:24 PM

Could you try adding grades explicitly to the schema as an array field?

05:25

Jason

05:25 PM

I suspect this might be an issue with just auto-schema detection

Darren

05:28 PM

https://github.com/typesense/typesense-js/issues/93

05:29

Darren

05:29 PM

Ok, I'll manually set the schema and post back if it makes a difference

05:38

Darren

05:38 PM

Jason So does seem its a bug. When turning off auto schema and setting type of _grades and _people to string[] that record still fails. But, when I add a different record FIRST that has data (see below) then after that I can indeed save the record that was failing

...here is that record that succeeds first time

>  parse {
>    _location: null,
>    description: null,
>    label: 'Grade Eight',
>    type: 'grade',
>    meta: { created: { at: null, by: null }, updated: { at: null, by: null } },
>    _grades: [],
>    _people: [
>      'tenants/one/people/eli-schreiber',
>      'tenants/one/people/miriam-schreiber',
>      'tenants/one/people/nathaniel-ewen',
>      'tenants/one/people/talia-ewen',
>      'tenants/one/people/seth-schwartzberg',
>      'tenants/one/people/maya-schwartzberg',
>      'tenants/one/people/michael-meyer',
>      'tenants/one/people/sammi-meyer',
>      'tenants/one/people/jeremiah-engel',
>      'tenants/one/people/ruth-engel',
>      'tenants/one/people/jeremy-maltzman',
>      'tenants/one/people/deborah-maltzman',
>      'tenants/one/people/matt-glaser',
>      'tenants/one/people/chava-glaser',
>      'tenants/one/people/sam-aaronson',
>      'tenants/one/people/chana-aaronson',
>      'tenants/one/people/isaac-feldman',
>      'tenants/one/people/sarah-feldman',
>      'tenants/one/people/josh-feinberg',
>      'tenants/one/people/eve-feinberg'
>    ],
>    tenant: 'one',
>    collection: 'groups',
>    id: 'grade-eight-all'
>  }

Jason

05:41 PM

Oh boy! Worse than I thought... Could you add this to the github issue?

Darren

05:57 PM

Sure ...

Jason

07:11 PM

Will take a look and keep you posted

Dec 23, 2021 (25 months ago)

Photo of md5-4e872368b2b2668460205b409e95c2ea

Kishore Nallan

05:26 AM

Darren I'm able to reproduce the bug with the auto field. However, with an explicit string[] type definition, empty values are allowed as the very first document.

See here: https://gist.github.com/kishorenc/131476e9fb898381821f8aa026ec3fd0

(I tested this on Typesense 0.22.1)

If you can create a similar snippet for reproduction, it will help. Thanks.

Dec 26, 2021 (25 months ago)

Darren

11:03 PM

Sorry for slow reply, will take a look at this next week ... Kishore Nallan

Typesense

Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI

Indexed 3015 threads (79% resolved)

Join Our Community

Similar Threads

Cold Start Problem with Dynamic Collections

Adrian reported cold start issues with dynamic collections. Jason suggested using wildcard `*` for query_by parameters, upgrading to `0.25.0.rc34`, and clarified conventions. Adrian's issues were resolved but they reported a limitation that will potentially be addressed.

6mo

Issues with Schema Creation and Nested Fields

Sean encountered a problem with schema creation involving auto nested fields. Kishore Nallan suggested checking the API response for errors and adding problematic fields to the schema as optional. Sean confirmed the advice.

2mo

Troubleshooting 400 Error When Upgrading Typesense Firestore Extension

Orion experienced a `400` error after updating the Typesense Firestore extension, causing issues with cloud functions. They traced the issue back to a data type conflict in their Typesense collection schema after updating. With help from Jason and Kishore Nallan, they resolved the issue by recreating the collection.

14mo

Resolving Error with Incorrect Field Type during Schema Update

Samuel encountered an error updating the schema for an existing collection. Kishore Nallan suggested specifying a concrete type, and later found inconsistent data within the collection that caused the error.

9mo

Issue with Embedding Error in Version 0.25.0.rc63

Bill reported a bug in version 0.25.0.rc63 regarding a problem with updating or emplacing a document and receiving an embedding error. This was resolved in version 0.25.0.rc65, but further discussion ensued regarding the function of 'index' in the update feature.

4mo