Troubleshooting Typesense Error on Mac Mini
TLDR Darren reported errors while testing Typesense locally on a Mac mini. Kishore Nallan suggested the issue might stem from not having enough open files and instructed to increase it.
Aug 24, 2022 (16 months ago)
Darren
02:44 PM$ curl
{"ok":false}
$ curl -H "X-TYPESENSE-API-KEY: xyz" ""
{ "message": "Not Ready or Lagging"}
I restarted the service and followed the logs, which turned up 600k+ occurrences of this line:
E20220824 10:35:34.819658 418111488 collection.cpp:344] Write to disk failed. Will restore old document
Both disk and memory appear to be available on the mini. Any ideas how to recover the server? Log attached (with the spammy error lines removed for brevity).
Kishore Nallan
03:17 PMKishore Nallan
03:18 PMDarren
03:46 PMDarren
04:31 PMI20220824 10:34:36.802707 434208768 collection_manager.cpp:240] Initializing batched indexer from snapshot state...
I20220824 10:34:36.802758 434208768 batched_indexer.cpp:408] Restored 0 in-flight requests from snapshot.
I20220824 10:34:36.802770 434208768 raft_server.cpp:486] Finished loading collections from disk.
I20220824 10:34:36.802786 434208768 raft_server.h:299] Configuration of this group is 192.168.0.16:8107:8108
I20220824 10:34:36.804388 434208768 snapshot_executor.cpp:264] node default_group:192.168.0.16:8107:8108 snapshot_load_done, last_included_index: 17093 last_included_term: 2 peers: "192.168.0.16:8107:8108"
I20220824 10:34:36.804724 393965568 raft_meta.cpp:521] Loaded single stable meta, path /opt/homebrew/var/lib/typesense/state/meta term 5 votedfor 192.168.0.16:8107:8108 time: 190
I20220824 10:34:36.804749 393965568 node.cpp:576] node default_group:192.168.0.16:8107:8108 init, term: 5 last_log_id: (index=25168,term=5) conf: 192.168.0.16:8107:8108 old_conf:
I20220824 10:34:36.804800 393965568 node.cpp:1549] node default_group:192.168.0.16:8107:8108 term 5 start vote and grant vote self
I20220824 10:34:36.805931 393965568 raft_meta.cpp:546] Saved single stable meta, path /opt/homebrew/var/lib/typesense/state/meta term 6 votedfor 192.168.0.16:8107:8108 time: 610
I20220824 10:34:36.805953 393965568 node.cpp:1783] node default_group:192.168.0.16:8107:8108 term 6 become leader of group 192.168.0.16:8107:8108
I20220824 10:34:36.806133 393965568 raft_server.cpp:129] Node last_index: 25168
I20220824 10:34:36.806149 393965568 typesense_server_utils.cpp:301] Typesense peering service is running on 192.168.0.16:8107
I20220824 10:34:36.806159 393965568 typesense_server_utils.cpp:302] Snapshot interval configured as: 3600s
I20220824 10:34:36.806171 393965568 typesense_server_utils.cpp:303] Snapshot max byte count configured as: 4194304
W20220824 10:34:36.806387 393965568 controller.cpp:1454] SIGINT was installed with 1
I20220824 10:34:36.806587 393965568 raft_server.cpp:534] Term: 6, last_index index: 25169, committed_index: 0, known_applied_index: 17093, applying_index: 0, queued_writes: 0, pending_queue_size: 1, local_sequence: 50416002
W20220824 10:34:36.806711 393965568 node.cpp:811] [default_group:192.168.0.16:8107:8108 ] Refusing concurrent configuration changing
E20220824 10:34:36.806732 393965568 raft_server.cpp:608] 8076 lagging entries > healthy read lag of 1000
E20220824 10:34:36.806871 393965568 raft_server.cpp:620] 8076 lagging entries > healthy write lag of 500
>>>>> E20220824 10:34:36.807055 429916160 raft_server.h:62] Peer refresh failed, error: Doing another configuration change
E20220824 10:34:37.779659 418111488 collection.cpp:344] Write to disk failed. Will restore old document
E20220824 10:34:37.780156 418111488 collection.cpp:344] Write to disk failed. Will restore old document
E20220824 10:34:37.780210 418111488 collection.cpp:344] Write to disk failed. Will restore old document
E20220824 10:34:37.780268 418111488 collection.cpp:344] Write to disk failed. Will restore old document
E20220824 10:34:37.780308 418111488 collection.cpp:344] Write to disk failed. Will restore old document
Darren
04:35 PMRefusing concurrent configuration changing
? I'm not sure what that means.Aug 25, 2022 (16 months ago)
Kishore Nallan
04:01 AMDoing another configuration change
need not always be an error. When the DB is loading from disk, like in this instance, this message can be ignored.I think the issue here is likely not having enough open files. Can you try increasing it? https://apple.stackexchange.com/a/73688
Typesense
Indexed 3015 threads (79% resolved)
Similar Threads
Issue Resolution and Upgrade Problems in Typesense Version 0.26rc16
Ankit reported an issue with Typesense, which was addressed by Kishore Nallan and Jason. However, Ankit experienced difficulties while trying to upgrade, with the server status showing as "Not ready or lagging" 503. The resolution remains incomplete.
Discussion About Typesense Nodes Not Synchronizing Correctly
Erick experienced an issue where documents weren't updated properly in a Typesense instance running on 3 nodes. Upon requesting debug logs and configs, Jason identified that these nodes weren't part of the same cluster. They couldn't resolve the nodes' failure to connect issue and recommended a fresh installation.
Troubleshooting 400 Error When Upgrading Typesense Firestore Extension
Orion experienced a `400` error after updating the Typesense Firestore extension, causing issues with cloud functions. They traced the issue back to a data type conflict in their Typesense collection schema after updating. With help from Jason and Kishore Nallan, they resolved the issue by recreating the collection.