#community-help

Issue with "Search Firestore with Typesense" Extension

TLDR Rajbek had an issue synchronizing data between Firestore and Typesense collection. Jason helped diagnose the issue and the problem was resolved by Rajbek increasing the memory allocation of the specific cloud function.

2mo

Join the chat

Sep 17, 2023 (2 months ago)

Photo of md5-fa2b564839c22b335c78e8bf9f3f7c67

Rajbek

03:19 AM

Hello everyone! I have an issue with "Search Firestore with Typesense" extension. I have succesfully installed and synchronized Typesense cloud cluster collection with my Firestore collection. However, the number of documents after backfill hits into 1000 ceil. Now after adding new docs into my Firestore collection the number stuck at 1016 docs. I see the logs of "ext-firestore-typesense-search-indexToTypesenseOnFirestoreWrite" cloud function after manually updating a random doc in my Firestore collection and it seems fine "Function execution took 412 ms, finished with status: 'ok'". But the updated document is still abscent in the Typesense cloud collection with the total ceil of 1016.

Photo of md5-8813087cccc512313602b6d9f9ece19f

Jason

03:23 AM

Could you post the complete logs from Firestore? Even if the function execution has completed, it might have logged errors before that?

Rajbek

03:25 AM

Thank you, Sir! That's the last logs

03:27

Rajbek

03:27 AM

Is csv ok? I can send the same info in JSON.

03:39

Rajbek

03:39 AM

Initially, I tested the extension only with one field. First, I didn't have such an issue with the number of docs in the Typsesense collection. After, I reinvented my Firestore collection scheme, deleted the extension in the Firebase and the collection in the Typesense cloud.

03:39

Rajbek

03:39 AM

Then, I installed the both agagin and now I have the issue.

03:41

Rajbek

03:41 AM

In the first try I had a field "id" in my Firestore collection, after I deceided to rename it into "registerID" to not mess with the typical "id" field of the Typsense.

Jason

03:45 AM

There should be logs for another backfill function, could you share those? Since the 1k doc issue is with the backfill right?

Rajbek

03:50 AM

Sure, one moment please🙏 But what about not upserting/synchronising doc after updating it in the firestore?

03:57

Rajbek

03:57 AM

ext-firestore-typesense-search-backfillToTypesenseFromFirestore last logs

10:29

Rajbek

10:29 AM

Jason Thank you. The issue is resolved. As you noticed the problem was indeed with the backfill. I increased the memory of the cloud function "ext-firestore-typesense-search-backfillToTypesenseFromFirestore" from 2Gb to 4Gb.🙂👍

Typesense

Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI

Indexed 3011 threads (79% resolved)

Join Our Community

Similar Threads

Discussing Firestore Extension and Collection Schema in Typesense

Loic asks about settings in Firestore Extension and experiences issues with the collection schema in Typesense. Discussion with Jason is ongoing.

3mo

Understanding Firestore Extension's Backfill Functionality

David asked about the backfill process of the firestore extension for different collections. Jason detailed the default backfill behavior and suggested running the backfill from a local computer for large collections to avoid function timeouts or memory issues.

1mo

Issue with Query Expectations on Typesense Search

Sean was having an issue with their search query on Typesense. Kishore Nallan suggested adjusting the 'drop_tokens_threshold' parameter. After making the adjustment, Sean found an issue with the order of the results, which was resolved by updating Typesense version.

2mo

Syncing records issue from BigQuery to Typesense using Airbyte

Jamshid had a problem syncing records from BigQuery to Typesense via Airbyte, with only a partial number of records syncing. Jason suggested checking AirByte logs for Typesense API responses. They found an issue with BigQuery's connector on Airbyte handling repeated fields and concluded to consider building their own sync script to mitigate the situation.

2mo

Issues with Semantic Search in OpenAI Embeddings

Semyon was having issues while creating a schema for semantic search with OpenAI embeddings. Jason gave multiple suggestions for troubleshooting but none of them worked. The error was narrowed down to possibly being related to Semyon's specific OpenAI account or the OpenAI API key. The thread ended with Jason suggesting Semyon to check billing and make a direct API call to OpenAI.

2mo