Hi everyone!
I'm running Typesense in a HA fashio...
# community-helpa
Alex Holt
09/04/2024, 6:48 PMHi everyone!
I'm running Typesense in a HA fashion and seeing one of my 3 nodes doesn't appear to be applying filters correctly. Any suggestions on how I can debug?
Alex Holt
09/04/2024, 6:48 PMThis is an example of the query I'm running and the inconsistent results:
Copy code
search_parameters = {
"q": "*",
"query_by": "name",
"infix": "off",
"offset": 0,
"limit": 31,
}
# 'found': 15, 'hits': [{'document': {'_schema': '0.0.1', 'created_at_unix_secs': 1725475012, 'id': 'lqPWmVfWmWKqVjNWKJhg', 'name': 'test3', 'user_id': 'wsU5zkU2R4SHGeiTvzO85wMACiq1', ....
search_parameters = {
"q": "*",
"query_by": "name",
"infix": "off",
"offset": 0,
"limit": 31,
"filter_by": "user_id:=wsU5zkU2R4SHGeiTvzO85wMACiq1",
}
# 'found': 0, 'hits': [], 'out_of': 15, 'page': 1, 'request_params': ......Alex Holt
09/04/2024, 6:49 PMWeirdly seeing other nodes are returning different results with the same query:
Copy code
When running against a good node
search_parameters = {
"q": "*",
"query_by": "name",
"infix": "off",
"offset": 0,
"limit": 31,
"filter_by": "user_id:=wsU5zkU2R4SHGeiTvzO85wMACiq1",
}
# 'found': 3, 'hits': [{'document': {'_schema': '0.0.1', 'created_at_unix_secs': 1725475012, 'id': 'lqPWmVfWmWKqVjNWKJhg', 'name': 'test3', 'user_id': 'wsU5zkU2R4SHGeiTvzO85wMACiq1'Alex Holt
09/04/2024, 6:50 PMHave tried restarting the bad node but sadly didn't help
Alex Holt
09/04/2024, 6:55 PMthe debug, metrics, stats endpoints don't seem to share anything interesting either so a bit at a loss as to what to look at next
j
Jason Bosco
09/05/2024, 2:50 AMCould you try replicating this in v27 of Typesense?
a
Alex Holt
09/05/2024, 9:33 AMIt's currently working as expected (same code) in our Dev cluster which is also running 25,2
Alex Holt
09/05/2024, 9:35 AMSo I presume it is specifically something about our prod cluster itself which is why I was trying to work out if there was any other way to debug/get more information an existing implementation
k
Kishore Nallan
09/05/2024, 11:37 AMSee if you there are any errors in the logs that might offer a hint for why it diverged. To recover this node, you can stop it, delete the data directory and then start it back so that it catches up.
a
Alex Holt
09/05/2024, 1:08 PMWould there be a way to do this for just a single collection?
k
Kishore Nallan
09/05/2024, 1:09 PMNo that's not possible because cluster state is global
a
Alex Holt
09/05/2024, 1:09 PMGotcha, thanks
Alex Holt
09/05/2024, 2:49 PMKilling the single node and wiping it's data dir seemed to do the trick, thanks so much!
👍 1