#community-help

Self-Hosted Cluster Down, Logs Reveal "No matching field found for token"

TLDR Imtiaz had a self-hosted server cluster of 3 instances crash. Kishore Nallan suggested it could be a bug or edge case, and recommended enabling query logging in Typesense for future issues.

10mo

Join the chat

Dec 13, 2022 (10 months ago)

Photo of md5-5c86b23fcf7d52313d7a346d8cc3e86b

Imtiaz

02:15 PM

Hi,

I just had a self hosted cluster of 3 instances go down. It was preceded by a spam in the logs of "No matching field found for token: xxxxxx". The servers have been running stably for close to a year prior to this with the occasional restart of just the service for upgrades. Currently on 0.23.1.

Got the cluster back up by restarting the nodes. Any hint as to what I could look for as a root cause? (for context, the log entry was occurring around 32 times per second when the server went down).

Photo of md5-4e872368b2b2668460205b409e95c2ea

Kishore Nallan

02:17 PM

Probably a bug or an unaccounted edge case. Did the process crash or just hang?

Imtiaz

02:17 PM

I could no longer call the /health endpoint on all 3 servers

Kishore Nallan

02:17 PM

If it crashed there will be stack trace in the log or in system journal

02:18

Kishore Nallan

02:18 PM

Do you log your queries? Anyway to replay queries leading to this issue?

Imtiaz

02:19 PM

No there isnt a log of the queries - we did do a sanity check to see if we are using a search query as a field name from the frontend - did not spot anything obvious but I have not fully ruled that out yet.

02:20

Imtiaz

02:20 PM

This was also accompanied by a spike in CPU on all 3 servers

Kishore Nallan

02:21 PM

The No matching field found for token need not always been an error, but since it ran so many times here, it could be.

02:22

Kishore Nallan

02:22 PM

If this happens again, you can enable query logging in Typesense to identify the offending query. Just set a low value like 1 for the slow request logging: https://typesense.org/docs/0.23.1/api/cluster-operations.html#toggle-slow-request-log

Imtiaz

02:25 PM

Thank you - will monitor and if we learn anything or get more info I will share it here.

Typesense

Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI

Indexed 2779 threads (79% resolved)

Join Our Community Manage subscription

Similar Threads

Troubleshooting Typesense Cluster Mode Error Logs

gaurav was having troubles with Typesense error logs in cluster mode. Kishore Nallan explained that it can occur due to master's location troubles and node restarts or state resets, especially in a 3-node setup.

10mo

Problem with Loading Records on Typesense Server

Rafael had an issue with Typesense server loop during startup after loading 8M of 15M records. Kishore Nallan identified the issue as memory shortage and advised Rafael to allocate more server memory.

15mo

Stability Issues with Self-Hosted Typesense Server on AWS ECS

Ryan reports stability issues with a self-hosted typesense server, whilst Jason suggests various potential factors to consider, including configuration, CPU allocation, modifying the healthcheck, and testing on EC2 instances. However, no definitive solution is reached.

1mo

Addressing Typesense Server Issues and Optimization Needs

Robert had an issue with a 'stuck' typesense server. Jason and Kishore Nallan gave advice on handling writes, configuration for high search volumes, and running multiple typesense instances. They also recommended monitoring CPU usage and updating the server version for bug fixes.

12mo

Discussion About Typesense Nodes Not Synchronizing Correctly

Erick experienced an issue where documents weren't updated properly in a Typesense instance running on 3 nodes. Upon requesting debug logs and configs, Jason identified that these nodes weren't part of the same cluster. They couldn't resolve the nodes' failure to connect issue and recommended a fresh installation.

21mo