Self-Hosted Cluster Down, Logs Reveal "No matching field found for token"
TLDR Imtiaz had a self-hosted server cluster of 3 instances crash. Kishore Nallan suggested it could be a bug or edge case, and recommended enabling query logging in Typesense for future issues.
Dec 13, 2022 (10 months ago)
I just had a self hosted cluster of 3 instances go down. It was preceded by a spam in the logs of "No matching field found for token: xxxxxx". The servers have been running stably for close to a year prior to this with the occasional restart of just the service for upgrades. Currently on
Got the cluster back up by restarting the nodes. Any hint as to what I could look for as a root cause? (for context, the log entry was occurring around 32 times per second when the server went down).
Kishore Nallan02:17 PM
/healthendpoint on all 3 servers
Kishore Nallan02:17 PM
Kishore Nallan02:18 PM
Kishore Nallan02:21 PM
No matching field found for tokenneed not always been an error, but since it ran so many times here, it could be.
Kishore Nallan02:22 PM
1for the slow request logging: https://typesense.org/docs/0.23.1/api/cluster-operations.html#toggle-slow-request-log
Indexed 2779 threads (79% resolved)
Troubleshooting Typesense Cluster Mode Error Logs
gaurav was having troubles with Typesense error logs in cluster mode. Kishore Nallan explained that it can occur due to master's location troubles and node restarts or state resets, especially in a 3-node setup.
Problem with Loading Records on Typesense Server
Rafael had an issue with Typesense server loop during startup after loading 8M of 15M records. Kishore Nallan identified the issue as memory shortage and advised Rafael to allocate more server memory.
Stability Issues with Self-Hosted Typesense Server on AWS ECS
Ryan reports stability issues with a self-hosted typesense server, whilst Jason suggests various potential factors to consider, including configuration, CPU allocation, modifying the healthcheck, and testing on EC2 instances. However, no definitive solution is reached.
Addressing Typesense Server Issues and Optimization Needs
Robert had an issue with a 'stuck' typesense server. Jason and Kishore Nallan gave advice on handling writes, configuration for high search volumes, and running multiple typesense instances. They also recommended monitoring CPU usage and updating the server version for bug fixes.
Discussion About Typesense Nodes Not Synchronizing Correctly
Erick experienced an issue where documents weren't updated properly in a Typesense instance running on 3 nodes. Upon requesting debug logs and configs, Jason identified that these nodes weren't part of the same cluster. They couldn't resolve the nodes' failure to connect issue and recommended a fresh installation.