#community-help

Troubleshooting Typesense Error on Mac Mini

TLDR Darren reported errors while testing Typesense locally on a Mac mini. Kishore Nallan suggested the issue might stem from not having enough open files and instructed to increase it.

16mo

Join the chat

Aug 24, 2022 (16 months ago)

Photo of md5-0ed5c38d6c4a8f3bfa3e783c727a7651

Darren

02:44 PM

Hi everyone, I am testing Typesense locally on a Mac mini using the the homebrew installation. After creating a collection and loading several million documents (id and short string field), the HTTP service is returning errors:

$ curl 
{"ok":false}

$ curl -H "X-TYPESENSE-API-KEY: xyz" ""
{ "message": "Not Ready or Lagging"}

I restarted the service and followed the logs, which turned up 600k+ occurrences of this line:

E20220824 10:35:34.819658 418111488 collection.cpp:344] Write to disk failed. Will restore old document

Both disk and memory appear to be available on the mini. Any ideas how to recover the server? Log attached (with the spammy error lines removed for brevity).

Photo of md5-4e872368b2b2668460205b409e95c2ea

Kishore Nallan

03:17 PM

Definitely some disk related error.

03:18

Kishore Nallan

03:18 PM

Do you see any other logs before that?

Darren

03:46 PM

from what I can tell disk has 60gb+ free

04:31

Darren

04:31 PM

Everything seems healthy until the line prefixed with >>>>> :

I20220824 10:34:36.802707 434208768 collection_manager.cpp:240] Initializing batched indexer from snapshot state...
I20220824 10:34:36.802758 434208768 batched_indexer.cpp:408] Restored 0 in-flight requests from snapshot.
I20220824 10:34:36.802770 434208768 raft_server.cpp:486] Finished loading collections from disk.
I20220824 10:34:36.802786 434208768 raft_server.h:299] Configuration of this group is 192.168.0.16:8107:8108
I20220824 10:34:36.804388 434208768 snapshot_executor.cpp:264] node default_group:192.168.0.16:8107:8108 snapshot_load_done, last_included_index: 17093 last_included_term: 2 peers: "192.168.0.16:8107:8108"
I20220824 10:34:36.804724 393965568 raft_meta.cpp:521] Loaded single stable meta, path /opt/homebrew/var/lib/typesense/state/meta term 5 votedfor 192.168.0.16:8107:8108 time: 190
I20220824 10:34:36.804749 393965568 node.cpp:576] node default_group:192.168.0.16:8107:8108 init, term: 5 last_log_id: (index=25168,term=5) conf: 192.168.0.16:8107:8108 old_conf:
I20220824 10:34:36.804800 393965568 node.cpp:1549] node default_group:192.168.0.16:8107:8108 term 5 start vote and grant vote self
I20220824 10:34:36.805931 393965568 raft_meta.cpp:546] Saved single stable meta, path /opt/homebrew/var/lib/typesense/state/meta term 6 votedfor 192.168.0.16:8107:8108 time: 610
I20220824 10:34:36.805953 393965568 node.cpp:1783] node default_group:192.168.0.16:8107:8108 term 6 become leader of group 192.168.0.16:8107:8108
I20220824 10:34:36.806133 393965568 raft_server.cpp:129] Node last_index: 25168
I20220824 10:34:36.806149 393965568 typesense_server_utils.cpp:301] Typesense peering service is running on 192.168.0.16:8107
I20220824 10:34:36.806159 393965568 typesense_server_utils.cpp:302] Snapshot interval configured as: 3600s
I20220824 10:34:36.806171 393965568 typesense_server_utils.cpp:303] Snapshot max byte count configured as: 4194304
W20220824 10:34:36.806387 393965568 controller.cpp:1454] SIGINT was installed with 1
I20220824 10:34:36.806587 393965568 raft_server.cpp:534] Term: 6, last_index index: 25169, committed_index: 0, known_applied_index: 17093, applying_index: 0, queued_writes: 0, pending_queue_size: 1, local_sequence: 50416002
W20220824 10:34:36.806711 393965568 node.cpp:811] [default_group:192.168.0.16:8107:8108 ] Refusing concurrent configuration changing
E20220824 10:34:36.806732 393965568 raft_server.cpp:608] 8076 lagging entries > healthy read lag of 1000
E20220824 10:34:36.806871 393965568 raft_server.cpp:620] 8076 lagging entries > healthy write lag of 500
>>>>> E20220824 10:34:36.807055 429916160 raft_server.h:62] Peer refresh failed, error: Doing another configuration change
E20220824 10:34:37.779659 418111488 collection.cpp:344] Write to disk failed. Will restore old document
E20220824 10:34:37.780156 418111488 collection.cpp:344] Write to disk failed. Will restore old document
E20220824 10:34:37.780210 418111488 collection.cpp:344] Write to disk failed. Will restore old document
E20220824 10:34:37.780268 418111488 collection.cpp:344] Write to disk failed. Will restore old document
E20220824 10:34:37.780308 418111488 collection.cpp:344] Write to disk failed. Will restore old document

04:35

Darren

04:35 PM

Actually maybe earlier at Refusing concurrent configuration changing? I'm not sure what that means.

Aug 25, 2022 (16 months ago)

Kishore Nallan

04:01 AM

Doing another configuration change need not always be an error. When the DB is loading from disk, like in this instance, this message can be ignored.

I think the issue here is likely not having enough open files. Can you try increasing it? https://apple.stackexchange.com/a/73688

Typesense

Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI

Indexed 3015 threads (79% resolved)

Join Our Community

Similar Threads

Issue Resolution and Upgrade Problems in Typesense Version 0.26rc16

Ankit reported an issue with Typesense, which was addressed by Kishore Nallan and Jason. However, Ankit experienced difficulties while trying to upgrade, with the server status showing as "Not ready or lagging" 503. The resolution remains incomplete.

2mo

Discussion About Typesense Nodes Not Synchronizing Correctly

Erick experienced an issue where documents weren't updated properly in a Typesense instance running on 3 nodes. Upon requesting debug logs and configs, Jason identified that these nodes weren't part of the same cluster. They couldn't resolve the nodes' failure to connect issue and recommended a fresh installation.

23mo

Troubleshooting 400 Error When Upgrading Typesense Firestore Extension

Orion experienced a `400` error after updating the Typesense Firestore extension, causing issues with cloud functions. They traced the issue back to a data type conflict in their Typesense collection schema after updating. With help from Jason and Kishore Nallan, they resolved the issue by recreating the collection.

14mo