Cold Start Problem with Dynamic Collections
TLDR Adrian reported cold start issues with dynamic collections. Jason suggested using wildcard * for query_by parameters, upgrading to 0.25.0.rc34, and clarified conventions. Adrian's issues were resolved but they reported a limitation that will potentially be addressed.
6May 26, 2023 (4 months ago)
Adrian
09:46 PMSome context:
We have a variety of types of objects that we index in one collection. To keep things simple for now (and because we will have dynamic fields in the future), all non shared fields are dynamically defined in the schema. Ie
{
Name: ".*",
Type: "auto",
Optional: &_true,
},The problem:
At query time we are getting this error:
status: 404 response: {"message": "Could not find a field named $FIELD_NAME in the schema."} . This is happening when we have not indexed an object of the type that has FIELD_NAME yet in a given collection. This situation will happen often since we create new collections per customer tenant. Is there a way to have this FIELD_NAME ignored in the queryBy params if it does not exist, instead of throwing an error? Or is there a better approach here?
Jason
09:49 PMWhen a collection is searched without any documents, how do you determine the field names to use in query_by?
Adrian
09:50 PMJason
09:50 PM.* field?Adrian
09:52 PMJason
09:52 PMJason
09:53 PM{
Name: "known_field_1",
Type: "string",
Optional: &_true,
},
{
Name: "known_field_2",
Type: "string",
Optional: &_true,
},
{
Name: ".*",
Type: "auto",
Optional: &_true,
},Adrian
09:53 PMAdrian
09:54 PMJason
09:57 PMBut, I completely forgot that we actually added support for wild card field names in query by in recent builds of 0.25!
Adrian
09:58 PMJason
09:58 PM0.25.0.rc32 and setting query_by: .*1Jason
09:58 PMAdrian
09:58 PMAdrian
09:59 PMJason
10:01 PM1May 30, 2023 (4 months ago)
Adrian
03:08 PMstatus: 404 response: {"message": "No string or string array field found matching the pattern document_names.* in the schema."}Adrian
03:09 PMAdrian
03:10 PMJason
03:10 PMCould you adapt this set of curl commands to replicate this issue and post it as a comment in this issue.
1Adrian
03:25 PM0.25.0.rc32 , but I also don't see that image in dockerhubAdrian
03:25 PM0.25.0.rc30 ?Jason
03:26 PM1Jason
03:27 PM0.25.0.rc34?1Adrian
03:46 PMexport TYPESENSE_API_KEY=xyz
curl "" \
-X POST \
-H "Content-Type: application/json" \
-H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}" \
-d '{
"name": "companies",
"fields": [
{"name": "company_name", "type": "string" },
{"name": "num_employees", "type": "int32" },
{"name": "additional_data_1", "type": "string", "optional": true }
],
"default_sorting_field": "num_employees"
}'
curl "" \
-H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}" \
-H "Content-Type: text/plain" \
-X POST \
-d '{"id": "124","company_name": "Stark Industries","num_employees": 5215}
{"id": "125","company_name": "Acme Corp","num_employees": 2133}
{"id": "126","company_name": "Stark Industries","num_employees": 5215,"additional_data_1": "data"}'
curl "" \
-X POST \
-H "Content-Type: application/json" \
-H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}" \
-d '{
"searches": [
{
"collection": "companies",
"q": "stark",
"query_by": "company_name, additional_data_.*"
}
]
}' Adrian
03:46 PM{"results":[{"code":404,"error":"No string or string array field found matching the pattern additionaldata.* in the schema."}]}%Adrian
03:46 PM
Kishore Nallan
04:07 PMadditional_data_*Kishore Nallan
04:08 PM.* is to be used only for nested field, for e.g. person.name -- we had to come up with this convention to differentiate between the two cases.Adrian
05:15 PMJason
05:17 PM*, for performance reasonsJason
05:18 PMAdrian
06:51 PMAdrian
06:51 PMAdrian
06:53 PMAdrian
08:30 PMAdrian
08:31 PM*_additional_dataMay 31, 2023 (4 months ago)
Kishore Nallan
02:55 AM1
Typesense
Indexed 2764 threads (79% resolved)
Similar Threads
Phrase Search Relevancy and Weights Fix
Jan reported an issue with phrase search relevancy using Typesense Instantsearch Adapter. The problem occurred when searching phrases with double quotes. The team identified the issue to be related to weights and implemented a fix, improving the search results.
6Troubleshooting Issues with DocSearch Hits and Scraper Configuration
Rubai encountered issues with search result priorities and ellipsis. Jason helped debug the issue and suggested using different versions of typesense-docsearch.js, updating initialization parameters, and running the scraper on a Linux-based environment. The issues related to hits structure and scraper configuration were resolved.
7Firestore to Typesense Backfill Issue with Dynamic Paths
Greg experienced issues with Firestore to Typesense backfill not working, and Jason determined it might be related to dynamic paths in Firestore collections not being supported. An RC version of the extension with dynamic path support was considered but needs further review before being shared with Greg.
1Discussions on Typesense, Collections, and Dynamic Fields
Tugay shares plans to use Typesense for their SaaS platform and asks about collection sizes and sharding. Jason clarifies Typesense's capabilities and shares a beta feature. They discuss using unique collections per customer and new improvements. Kishore Nallan and Gabe comment on threading and data protection respectively.
3Handling Kinesis Stream Event Batching with Typesense
Dui had questions about how to handle Kinesis stream events with Typesense. Kishore Nallan suggested using upsert mode for creation/update and differentiating with logical deletion. After various discussions including identifying and resolving a bug, they finalized to introduce an `emplace` action in Typesense v0.23.
8