#community-help

Troubleshooting Typesense Error on Mac Mini

TLDR Darren reported errors while testing Typesense locally on a Mac mini. Kishore Nallan suggested the issue might stem from not having enough open files and instructed to increase it.

Powered by Struct AI
Aug 24, 2022 (16 months ago)
Darren
Photo of md5-0ed5c38d6c4a8f3bfa3e783c727a7651
Darren
02:44 PM
Hi everyone, I am testing Typesense locally on a Mac mini using the the homebrew installation. After creating a collection and loading several million documents (id and short string field), the HTTP service is returning errors:
$ curl 
{"ok":false}

$ curl -H "X-TYPESENSE-API-KEY: xyz" ""
{ "message": "Not Ready or Lagging"}

I restarted the service and followed the logs, which turned up 600k+ occurrences of this line:
E20220824 10:35:34.819658 418111488 collection.cpp:344] Write to disk failed. Will restore old document

Both disk and memory appear to be available on the mini. Any ideas how to recover the server? Log attached (with the spammy error lines removed for brevity).
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
03:17 PM
Definitely some disk related error.
03:18
Kishore Nallan
03:18 PM
Do you see any other logs before that?
Darren
Photo of md5-0ed5c38d6c4a8f3bfa3e783c727a7651
Darren
03:46 PM
from what I can tell disk has 60gb+ free
04:31
Darren
04:31 PM
Everything seems healthy until the line prefixed with >>>>> :
I20220824 10:34:36.802707 434208768 collection_manager.cpp:240] Initializing batched indexer from snapshot state...
I20220824 10:34:36.802758 434208768 batched_indexer.cpp:408] Restored 0 in-flight requests from snapshot.
I20220824 10:34:36.802770 434208768 raft_server.cpp:486] Finished loading collections from disk.
I20220824 10:34:36.802786 434208768 raft_server.h:299] Configuration of this group is 192.168.0.16:8107:8108
I20220824 10:34:36.804388 434208768 snapshot_executor.cpp:264] node default_group:192.168.0.16:8107:8108 snapshot_load_done, last_included_index: 17093 last_included_term: 2 peers: "192.168.0.16:8107:8108"
I20220824 10:34:36.804724 393965568 raft_meta.cpp:521] Loaded single stable meta, path /opt/homebrew/var/lib/typesense/state/meta term 5 votedfor 192.168.0.16:8107:8108 time: 190
I20220824 10:34:36.804749 393965568 node.cpp:576] node default_group:192.168.0.16:8107:8108 init, term: 5 last_log_id: (index=25168,term=5) conf: 192.168.0.16:8107:8108 old_conf:
I20220824 10:34:36.804800 393965568 node.cpp:1549] node default_group:192.168.0.16:8107:8108 term 5 start vote and grant vote self
I20220824 10:34:36.805931 393965568 raft_meta.cpp:546] Saved single stable meta, path /opt/homebrew/var/lib/typesense/state/meta term 6 votedfor 192.168.0.16:8107:8108 time: 610
I20220824 10:34:36.805953 393965568 node.cpp:1783] node default_group:192.168.0.16:8107:8108 term 6 become leader of group 192.168.0.16:8107:8108
I20220824 10:34:36.806133 393965568 raft_server.cpp:129] Node last_index: 25168
I20220824 10:34:36.806149 393965568 typesense_server_utils.cpp:301] Typesense peering service is running on 192.168.0.16:8107
I20220824 10:34:36.806159 393965568 typesense_server_utils.cpp:302] Snapshot interval configured as: 3600s
I20220824 10:34:36.806171 393965568 typesense_server_utils.cpp:303] Snapshot max byte count configured as: 4194304
W20220824 10:34:36.806387 393965568 controller.cpp:1454] SIGINT was installed with 1
I20220824 10:34:36.806587 393965568 raft_server.cpp:534] Term: 6, last_index index: 25169, committed_index: 0, known_applied_index: 17093, applying_index: 0, queued_writes: 0, pending_queue_size: 1, local_sequence: 50416002
W20220824 10:34:36.806711 393965568 node.cpp:811] [default_group:192.168.0.16:8107:8108 ] Refusing concurrent configuration changing
E20220824 10:34:36.806732 393965568 raft_server.cpp:608] 8076 lagging entries > healthy read lag of 1000
E20220824 10:34:36.806871 393965568 raft_server.cpp:620] 8076 lagging entries > healthy write lag of 500
>>>>> E20220824 10:34:36.807055 429916160 raft_server.h:62] Peer refresh failed, error: Doing another configuration change
E20220824 10:34:37.779659 418111488 collection.cpp:344] Write to disk failed. Will restore old document
E20220824 10:34:37.780156 418111488 collection.cpp:344] Write to disk failed. Will restore old document
E20220824 10:34:37.780210 418111488 collection.cpp:344] Write to disk failed. Will restore old document
E20220824 10:34:37.780268 418111488 collection.cpp:344] Write to disk failed. Will restore old document
E20220824 10:34:37.780308 418111488 collection.cpp:344] Write to disk failed. Will restore old document

04:35
Darren
04:35 PM
Actually maybe earlier at Refusing concurrent configuration changing? I'm not sure what that means.
Aug 25, 2022 (16 months ago)
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
04:01 AM
Doing another configuration change need not always be an error. When the DB is loading from disk, like in this instance, this message can be ignored.

I think the issue here is likely not having enough open files. Can you try increasing it? https://apple.stackexchange.com/a/73688