#community-help

Resolving Unhealthy Typesense Cluster and JSON Parsing Bug

TLDR Masahiro reported an unhealthy Typesense cluster. The cause was a parsing bug related to boolean values in JSON schemas. Jason resolved the issue by clearing node data and upgrading the server to v0.20, which resolved the issue and Masahiro's team decided to use Typesense.

Powered by Struct AI
+12
smile2
grin1
man-gesturing-ok1
raised_hands1
26
29mo
Solved
Join the chat
Apr 23, 2021 (29 months ago)
Masahiro
Photo of md5-366dff6b5f9b1a7d0f404fdc3261e573
Masahiro
01:33 AM
My cluster (gxu1iajypedl2z0tp-1.a1.typesense.net) turned into unhealthy.
Any ways to fix this issue? If there are documentations about this, it will be helpful 😄
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
01:33 AM
Looking
Masahiro
Photo of md5-366dff6b5f9b1a7d0f404fdc3261e573
Masahiro
01:34 AM
thanks!
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
01:39 AM
Could you share the exact collection schema you used?
Masahiro
Photo of md5-366dff6b5f9b1a7d0f404fdc3261e573
Masahiro
01:40 AM
{
        'name': 'users',
        'fields': [
          {'name': 'displayName', 'type': 'string' },
          {'name': 'userId', 'type': 'string' },
          {'name': 'favGameTitle1','optional':'true', 'type': 'string' },
      
          {'name': 'gender', 'optional':'true', 'type': 'int32' },

        ],
        'default_sorting_field': 'gender'
      }
01:41
Masahiro
01:41 AM
After adding optional parameter, the cluster might stop working.
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
01:42 AM
Ah, so the issue is that Typesense crashed because there's an uncaught exception in the json parsing around boolean values, in Typesense core.

We currently expect optional to have values of true or false. Looks like a string value of "true" for optional crashes the server 😞
01:44
Jason
01:44 AM
We'll fix the bug shortly, in the meantime could you use bool values instead of string values for true and false in the schema?
Masahiro
Photo of md5-366dff6b5f9b1a7d0f404fdc3261e573
Masahiro
01:44 AM
Of course, ok!
01:44
Masahiro
01:44 AM
Thanks 😸
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
01:45 AM
Now to recover the node, we'd unfortunately need to clear the bad data in the server... Would that be ok?
smile1
01:45
Jason
01:45 AM
(Separately we're also working on a way to skip over bad data in v0.20, so having to clear data when bad data makes it in, is only an issue with v0.19)
man-gesturing-ok1
01:48
Jason
01:48 AM
> Now to recover the node, we'd unfortunately need to clear the bad data in the server... Would that be ok?
Could you confirm that you're ok with this Masahiro
Masahiro
Photo of md5-366dff6b5f9b1a7d0f404fdc3261e573
Masahiro
01:50 AM
unfortunately no, still unhealthy..
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
01:50 AM
Sorry, I'd need explicit confirmation that you're ok with me clearing the data on the node and then restart the server
01:50
Jason
01:50 AM
Only then will it turn healthy
01:51
Jason
01:51 AM
Could you reply with an explicit OK to clear the data?
Masahiro
Photo of md5-366dff6b5f9b1a7d0f404fdc3261e573
Masahiro
01:51 AM
Oh sorry, ok!
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
01:52 AM
Ok, clearing data now
01:54
Jason
01:54 AM
Alright, node should be back up now
Masahiro
Photo of md5-366dff6b5f9b1a7d0f404fdc3261e573
Masahiro
01:57 AM
healthy now, thank you!
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
01:59 AM
Sorry about that! Bug fix for the original issue should be out in v0.20 soon
+11
raised_hands1
Apr 29, 2021 (29 months ago)
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
07:08 AM
Hi Masahiro I've upgraded this cluster to v0.20, now that it's out. It should be largely transparent to you!
grin1
Masahiro
Photo of md5-366dff6b5f9b1a7d0f404fdc3261e573
Masahiro
07:12 AM
Thank you so much😆😆🎉
07:14
Masahiro
07:14 AM
And finally our team has decided to use Typesense😊
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
07:59 AM
Yaaay! 🎉
+11
smile1