Issues with Cluster Upgrade and Embedding Field
TLDR Gustavo had issues upgrading their cluster and their embedding field wasn't being filled. Jason helped to solve the upgrade issue and advised re-indexing the documents to solve the embedding field issue. Both problems were successfully resolved.
Powered by Struct AI
7
1 72
1mo
Solved
Aug 02, 2023 (1 month ago)
Gustavo
Gustavo
07:14 PMI've initiated an upgrade in my cluster about 20 minutes ago. How much time does it use to take?
Jason
Jason
07:23 PMFor your dataset it usually takes about 30 minutes from what I remember…
Gustavo
Gustavo
07:23 PMI think something went wrong and it's rolling back
07:24
Gustavo
07:24 PMJason
Jason
07:24 PMYeah, it looks like the RC build you had chosen to upgrade to has a bug that made the process go into a crash-restart loop. We’re looking into it
07:24
Jason
07:24 PMThe rollback should be done in about 30 minutes
Gustavo
Gustavo
07:25 PMThe reason I tried to upgrade is because I noted the my embedding field wasn't being filled. No idea why.
Jason
Jason
07:26 PMFor new records you mean?
Gustavo
Gustavo
07:26 PMYes, but I don't now when it started to behave like that.
07:28
Gustavo
07:28 PMThe
embedding field is computed from the topics field which is an array of strings. After looking for a bug in my search functionality, I noted that a document in Typesense didn't have the embedding field when I retrived it with the SDK.07:29
Gustavo
07:29 PMI'm waiting for the cluster to be up so I can continue investigating.
07:33
Gustavo
07:33 PMIt finished to roll back, but it's unhealthy. Edit: it's healthy now. Edit: and unhealthy again. 😅
Jason
Jason
07:34 PMIt might be a bit flaky for a few minutes right after an upgrade completes, as DNS propagation happens
1Gustavo
Gustavo
07:48 PMAre you able to access my
posts-v1 collection? There's a document there with ID z3xMqOyt8d0xdTeEaPyb that contains the topics field, but not the embedding.07:48
Gustavo
07:48 PMCluster:
v601y2x3upjea4tipJason
Jason
08:01 PMNeed to try a few things which involve restarting your node. Is it ok if there’s sporadic 10 min downtimes?
Gustavo
Gustavo
08:02 PMYes
Jason
Jason
08:13 PMTried a few things and didn’t work unfortunately… Your cluster is back up, but need to look closer in a few hours
1Aug 03, 2023 (1 month ago)
Jason
Jason
04:10 AMIt looks like this might have been an issue from early RC builds where if an OpenAI API call fails, we still indexed the document without the embeddings. In builds after end of June, we added a validation to fail the document during import if the auto-embedding generation fails.
04:10
Jason
04:10 AMSo it’s likely that these documents that don’t have embeddings were created in earlier RC builds without this validation
04:11
Jason
04:11 AMI would recommend either deleting them and adding them back, or add a space at the end of each source field that’s used for embedding generation which will then cause the embedding generation to run
Gustavo
Gustavo
10:22 AMI’m on RC53. Upgrading to RC58 failed, but I see there’s the option for RC55 in the list. Is the problem fixed in it?
Jason
Jason
02:32 PMRC53 doesn’t have the missing validation issue.
However builds after that have a different bug that’s causing your particular dataset to not load in them. So I’d recommend not upgrading up from rc53 for now. We’re working on the fix.
However builds after that have a different bug that’s causing your particular dataset to not load in them. So I’d recommend not upgrading up from rc53 for now. We’re working on the fix.
Gustavo
Gustavo
04:20 PMI was having the issue on RC53 🤔
Jason
Jason
04:21 PMRight, but the issue started well before when you were running an older RC build. So those documents were probably just carried over with the missing embeddings from the older RC builds, even as you upgraded
04:21
Jason
04:21 PMEven once you upgrade to the upcoming RC build, the embeddings will still be missing, unless you do this: https://typesense-community.slack.com/archives/C01P749MET0/p1691035880028789?thread_ts=1691003672.362599&cid=C01P749MET0
04:241
Jason
04:24 PMJust published the new RC build. Scheduled the upgrade to your cluster
1Gustavo
Gustavo
04:26 PMI thought I updated a document using RC53 and the embedding was still missing, but it's probably because I didn't actually change the source field
Jason
Jason
04:31 PMYeah there needs to be some change to the field (eg: adding a space / period, etc)
Gustavo
Gustavo
05:01 PMIt finished to upgrade at 13:41 UTC-3, the dashboard consistently says it's healthy when I refresh and I can search from there, but my requests are still failing (20min later). Is it expected?
Jason
Jason
05:48 PMI do see searches coming into your cluster… Are you still seeing issues?
Gustavo
Gustavo
06:11 PMNo, it's working now
108:001
Gustavo
08:00 PMI've reindexed all docs and so far it seems the embedding field is working as expected. 👍
1Aug 14, 2023 (1 month ago)
Gustavo
Gustavo
10:16 PM Jason Caught more docs missing the embedding field since I sent the message above. The cluster is on RC59 all this time.
Jason
Jason
10:17 PMCould you give me the ID of one such document?
Gustavo
Gustavo
10:19 PMgZKGF5rItp8UWLdmKzIg: had the problem, but I deleted and recreated, so now it's OKAbHg5oQacJ1Ufw9JjI9v: still missing the embedding fieldJason
Jason
10:24 PMThe document was created on August 1 (based on the
See this: https://typesense-community.slack.com/archives/C01P749MET0/p1691079700751459?thread_ts=1691003672.362599&cid=C01P749MET0
createdAt timestamp), but the cluster was upgraded to rc59 only on Aug 3. So this is an old document that was created on a prior RC build which didn’t have the error handling for this particular case.See this: https://typesense-community.slack.com/archives/C01P749MET0/p1691079700751459?thread_ts=1691003672.362599&cid=C01P749MET0
10:26
Jason
10:26 PMUnrelated side note: rc66 is now out which I’d recommend upgrading to
Gustavo
Gustavo
10:28 PMStrange because the last thing I did that day was I deleted all docs and reindexed. 🤔
10:28
Gustavo
10:28 PMI'll update to RC66, then delete all docs and reindex. Let's see.
Jason
Jason
10:29 PMCould you make sure that the collection is fully empty before you re-index?
Best to just delete the whole collection and recreate it
Best to just delete the whole collection and recreate it
Gustavo
Gustavo
10:30 PMI think that's what I did, but I'll make sure this time.
11:01
Gustavo
11:01 PMUpgraded to RC66.
11:01
Gustavo
11:01 PMDeleted all docs.
11:01
Gustavo
11:01 PMReindexing them...
Jason
Jason
11:01 PMYou mean deleted the collection and created it right?
Gustavo
Gustavo
11:01 PMNo, just deleted the docs.
11:02
Gustavo
11:02 PMDo I have to delete the collection?
Jason
Jason
11:02 PMCould you delete the full collection and create it again, just to be sure (that it’s not an issue with older collections)?
Gustavo
Gustavo
11:02 PMYes
11:04
1
Gustavo
11:04 PMDeleted and recreated the collection.
1Aug 16, 2023 (1 month ago)
Gustavo
Gustavo
04:46 PM Jason I'm having another problem. It doesn't seem to be the same thing, but may be related. The embedding is generated from the
Weirdly, when I make a semantic search with "Auxílio maternidade", it doesn't appear.
topics array. I have a document with: id: 'WtSy5f9KgdvxGB0i6HGz',
topics: [
'Direito Trabalhista',
'Direito Previdenciário',
'Benefícios trabalhistas',
'Auxílio maternidade',
'Licença maternidade'
]Weirdly, when I make a semantic search with "Auxílio maternidade", it doesn't appear.
04:47
Gustavo
04:47 PMHow can it be that searching for an exact match in
topics doesn't include it? I wonder if there's something wrong with the embedding itself.04:48
Gustavo
04:48 PMHere's the definition of the embedding in the schema:
"embed": {
"from": [
"topics"
],
"model_config": {
"api_key": "...",
"model_name": "openai/text-embedding-ada-002"
}
},
"facet": false,
"index": true,
"infix": false,
"locale": "",
"name": "embedding",
"num_dim": 1536,
"optional": false,
"sort": false,
"type": "float[]"
},04:48
Gustavo
04:48 PMCould it be related to the
locale because the topics include diacritics?04:55
Gustavo
04:55 PMHmm, doesn't seem to be the case. I updated the document to
topics: ['Software development'] just to make it have a topic that's different from all other documents in the collection. I search for "Software development" and it still doesn't appear in the first 50 results. There's no other document with anything even close to that topic.04:56
Gustavo
04:56 PMIt's like this particular document it just broken.
Jason
Jason
04:57 PMI’ve seen this happen with different embedding models. Somehow verbatim searches are not considered “close enough” by the model… This is where hybrid search helps. Typesense will do a keyword search and that will put exact matches on top, followed by semantic matches
04:57
Jason
04:57 PMSo you can do something like
query_by: topics,embeddingGustavo
Gustavo
04:59 PMWait, I deleted the document, recreated and now it appears when I search.
05:00
Gustavo
05:00 PMBut it's not the same issue as before because before the embedding field was missing. This time, it was there.
05:01
Gustavo
05:01 PMThis is from my terminal, printed before I deleted I recreated the document, so you can see the embedding field was present.
Jason
Jason
05:02 PMHmmm
Gustavo
Gustavo
05:02 PMThis is the document now:
05:02
Gustavo
05:02 PMThe embedding is indeed not identical.
Jason
Jason
05:02 PMStrange! The embedding values are different
Gustavo
Gustavo
05:03 PMOh, nevermind, the reason the embedding is different is because the
topics field is different.105:05
Gustavo
05:05 PMBut you can see the
topics is the same as how it was before I manually changed it to test.05:05
05:06
Gustavo
05:06 PMSo here's what I'm sure about:
Same topics, same search.
Before: didn't appear in the results.
Now: appears normally.
Same topics, same search.
Before: didn't appear in the results.
Now: appears normally.
05:07
Gustavo
05:07 PMIf this problem happens again, do you want me to do something specific? Maybe send you the ID and leave it untouched so you can inspect it?
Jason
Jason
05:11 PMYeah, if you can send me the search query (curl request with all the search params, minus the API key) and the document ID that you expect to be returned but not returned, we can take a closer look
1
Typesense
Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI
Indexed 2764 threads (79% resolved)
Similar Threads
Utilizing Vector Search and Word Embeddings for Comprehensive Search in Typesense
Bill sought clarification on using vector search with multiple word embeddings in Typesense and using them instead of OpenAI's embedding. Kishore Nallan and Jason informed him that their development version 0.25 supports open source embedding models. They also resolved Bill's concerns regarding search performance, language support, and limitations in the search parameters.
11 225
2mo
Solved
Troubleshooting Typesense Node Embedding Field Error after Upgrade
Denny was having an error with the embedding field after upgrading a typesense node. Jason revealed that this was a bug in the v0.25.0 version. They suggested upgrading to 0.25.1.rc4 to fix the issue, which Denny did successfully.
1 12
1mo
Solved
Handling Kinesis Stream Event Batching with Typesense
Dui had questions about how to handle Kinesis stream events with Typesense. Kishore Nallan suggested using upsert mode for creation/update and differentiating with logical deletion. After various discussions including identifying and resolving a bug, they finalized to introduce an `emplace` action in Typesense v0.23.
8 91
20mo
Troubleshooting 400 Error When Upgrading Typesense Firestore Extension
Orion experienced a `400` error after updating the Typesense Firestore extension, causing issues with cloud functions. They traced the issue back to a data type conflict in their Typesense collection schema after updating. With help from Jason and Kishore Nallan, they resolved the issue by recreating the collection.
5 96
11mo
Solved
Discussing Indexing and Embedding Performance in Typesense
Dima had queries about indexing with embedding in Typesense. Kishore Nallan and Jason provided solutions, including reducing documents sent in an API call and running embeddings on a GPU. They facilitated Dima with the latest RC.
1 12
2mo
Solved