#community-help

Self-Hosted Cluster Down, Logs Reveal "No matching field found for token"

TLDR Imtiaz had a self-hosted server cluster of 3 instances crash. Kishore Nallan suggested it could be a bug or edge case, and recommended enabling query logging in Typesense for future issues.

Powered by Struct AI

1

Dec 13, 2022 (10 months ago)
Imtiaz
Photo of md5-5c86b23fcf7d52313d7a346d8cc3e86b
Imtiaz
02:15 PM
Hi,

I just had a self hosted cluster of 3 instances go down. It was preceded by a spam in the logs of "No matching field found for token: xxxxxx". The servers have been running stably for close to a year prior to this with the occasional restart of just the service for upgrades. Currently on 0.23.1.

Got the cluster back up by restarting the nodes. Any hint as to what I could look for as a root cause? (for context, the log entry was occurring around 32 times per second when the server went down).
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
02:17 PM
Probably a bug or an unaccounted edge case. Did the process crash or just hang?
Imtiaz
Photo of md5-5c86b23fcf7d52313d7a346d8cc3e86b
Imtiaz
02:17 PM
I could no longer call the /health endpoint on all 3 servers
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
02:17 PM
If it crashed there will be stack trace in the log or in system journal
02:18
Kishore Nallan
02:18 PM
Do you log your queries? Anyway to replay queries leading to this issue?
Imtiaz
Photo of md5-5c86b23fcf7d52313d7a346d8cc3e86b
Imtiaz
02:19 PM
No there isnt a log of the queries - we did do a sanity check to see if we are using a search query as a field name from the frontend - did not spot anything obvious but I have not fully ruled that out yet.
02:20
Imtiaz
02:20 PM
This was also accompanied by a spike in CPU on all 3 servers
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
02:21 PM
The No matching field found for token need not always been an error, but since it ran so many times here, it could be.
02:22
Kishore Nallan
02:22 PM
If this happens again, you can enable query logging in Typesense to identify the offending query. Just set a low value like 1 for the slow request logging: https://typesense.org/docs/0.23.1/api/cluster-operations.html#toggle-slow-request-log
Imtiaz
Photo of md5-5c86b23fcf7d52313d7a346d8cc3e86b
Imtiaz
02:25 PM
Thank you - will monitor and if we learn anything or get more info I will share it here.

1