urgent issue, scaled our typsense collection and n...
# community-help
h
urgent issue, scaled our typsense collection and now one of our collections is unreachable, with an error about a missing synonym set and curation - i think scaling the size up might have deleted the curation and synonym sets and maybe even the connection?
k
Can you please share the cluster ID?
h
ended up creating a new collection
events_search_v8_curation is just gone (and the v8 synonym set)
k
Ok we will investigate what happened here.
I see that previously you did a change request on December 15th too. This problem did not occur then?
Do you know if this rotation was performed right after these curation and synonym sets were created or were these present for some time before the scaling operation?
h
problem did not occur on dec 15
curation and synonym sets have been around for weeks if not months
one thing we did do was change a collection alias in the middle of the scale-up
k
Ok thanks for the info. I will post an update based on what we find from our analysis.
events_search_v8_curation is just gone (and the v8 synonym set)
Need a clarification on this. When you mean by "gone", did you check via the synonym set / curation set listing API whether those sets were listed? Or, are you going by the error message that you saw when the collection was unreachable?
h
the collection was unavailable because it said it the synonym/curation set were not to be found
i kept seeing a 40x error on the cloud UI
i did not check via the api, but sdk calls to the collection were returning errors
k
Ok before rotation we took a snapshot of the data. Inside that I can see the curation and synonym sets being stored correctly. I suspect the error is because of alias. I'll check if we are not handling alias resolution for synonym set mapping correctly.
h
any luck on recovering the curation and synonym sets? i still am unable to access them
k
I found the issue and the sets has been restored. I will be fixing the root cause tomorrow. It happens during a clustered rotation. Will update when we have the new version that fixes this bug.
thankyou 1