Our production cluster decided to upgrade itself f...
# community-help
n
Our production cluster decided to upgrade itself from v30.1 to v30.2.RC7. This was not scheduled by one of us. Do any of you know, how could this be?
a
Hi @Nicholas Tjørnelund Since yesterday, two nodes of your cluster were crashing due to a unhigienized query. We have been working on a fix since then and just managed to release a build with the fix (v30.2.rc7). Since both versions are backwards compatible and your cluster just had all three nodes crashed, we went ahead with the upgrade to avoid extended downtime. We were planning to wait until it stabilized to reach out, but should have let you know beforehand, I apologize.
n
@Alan Martini > Since yesterday, two nodes of your cluster were crashing due to a unhigienized query 2 things. 1. Can you explain what was happening with our nodes? 2. During the upgrade we experienced down time. we have a proxy in front. we round robin directly to each node outside the load balance endpoint. If unhealthy node detected, our setup will just take it out of scheduling and call the others for a while. During the upgrade we experienced all nodes being unavailable/timeout, the code default to the load balance endpoint if a healthy node does not exist, and obviously that also returned unavailable. How can this be? in theory it should be possible to do a rolling upgrade?
a
1. There was a edge case related to the preset parameter. We are unsure if a created preset had "dirty" typing or the ones passed in the query had, but it was causing a crash due to failure in parsing. 2. To clarify: it was not the version upgrade that caused it. The cluster was in a flaky state and we were rushing to create a patch for it. We completed it and was going to communicate to find the best time to upgrade it when all three nodes crashed due to this error and then we immediatly triggered the upgrade, since it was down anyway.
🙌 1
n
@Henrik Drachmann
h
Hi Alan. We are writing a post-mortem for our customers on the outage. One of the actions going forward is to improve how we communicate issues when they occur. I understand that this was an unplanned fix, so it's difficult to inform anything up front. We would like a way to be informed when unplanned events occur. This would help us understand what actions to take (and what not to do) when we see our alerting system light up. Additionally, we are looking for is a way for you to communicate when updates are rolled out that should not impact operations: for example, planned service windows. What approach do you normally have ? Could we agree on email address that is always informed ?
@Alan Martini Do you have an idea to how to ensure communication for issues only related to our cluster?
a
Hi @Henrik Drachmann Aside the automatic alerts that are sent to the email defined in
Account -> Alerts email
, we usually only send emails when there is some action necessary. I can make sure to always warn you about some unforeseen situation, like this case.
h
that would be great. Thank you. Would it be possible to write our alerting email (alert@raptor.dk) because it sends to all relevant parties?
a
Yeah it will send to the email defined in your Alerts Email
h
perfect.