also get this in ealier

Stack trace

point seem cant get the store

store

DEFAULT 2025-07-17T05:49:43.080379Z E20250717 05:49:43.079523 14 backward.hpp:4200] Stack trace (most recent call last):
DEFAULT 2025-07-17T05:49:43.080395Z E20250717 05:49:43.079578 14 backward.hpp:4200] #7 Object "/usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2", at 0xffffffffffffffff, in
DEFAULT 2025-07-17T05:49:43.080399Z E20250717 05:49:43.079586 14 backward.hpp:4200] #6 Object "/opt/typesense-server", at 0x55941e7d002d, in _start
DEFAULT 2025-07-17T05:49:43.080403Z E20250717 05:49:43.079591 14 backward.hpp:4200] #5 Object "/usr/lib/x86_64-linux-gnu/libc.so.6", at 0x7f1edd100e3f, in __libc_start_main
DEFAULT 2025-07-17T05:49:43.080407Z E20250717 05:49:43.079596 14 backward.hpp:4200] #4 Object "/usr/lib/x86_64-linux-gnu/libc.so.6", at 0x7f1edd100d8f, in
DEFAULT 2025-07-17T05:49:43.080411Z E20250717 05:49:43.079602 14 backward.hpp:4200] #3 Source "src/main/typesense_server.cpp", line 224, in main [0x55941e7db1bd]
DEFAULT 2025-07-17T05:49:43.080415Z E20250717 05:49:43.079608 14 backward.hpp:4200] #2 Source "src/typesense_server_utils.cpp", line 506, in run_server [0x55941ec943cf]
DEFAULT 2025-07-17T05:49:43.080419Z E20250717 05:49:43.079613 14 backward.hpp:4200] #1 Source "src/ratelimit_manager.cpp", line 498, in init [0x55941ec0a447]
DEFAULT 2025-07-17T05:49:43.080422Z E20250717 05:49:43.079618 14 backward.hpp:4200] #0 Source "src/store.cpp", line 101, in get [0x55941ec3302f]
DEFAULT 2025-07-17T05:49:43.080426Z Segmentation fault (Address not mapped to object [(nil)])

Hey sorry @Fanis Tharropoulos , do you have any idea on this one 🙌

I constantly get this one

Write to disk failed. Will restore old document

each reboot

Try dropping the index and re-indexing

Which Typesense version are you using?

im using v29

I also encounter some error cant init store, but it's not happen this time when write to disk failed https://typesense-community.slack.com/archives/C01P749MET0/p1752726993065609?thread_ts=1752658980.051089&cid=C01P749MET0

I can try as non-root user

Error while initializing store: IO error: Stale file handle

Error while initializing store: IO error: No such file or directory: While opening a file for sequentially reading: /root/lib/typesense/data/db/MANIFEST-000035: No such file or directory

Does it cause an issue? when a MANIFEST file dont exist? then leak to issue in stack trace?

But in this case why it didnt re-init a fresh db?

image.png

Ok from log I guess, there was 2 instances, start up nearly same time and mount to single GCS one of them is starting up and the rest one will show

It seems like the data directory /root/lib/typesense/data/meta is already being used by another Typesense server.

But i didnt clear why it's ccausing a stack trace error and container was force exit because of this

DEFAULT 2025-07-17T08:45:56.169490Z E20250717 08:45:56.168789 13 typesense_server.cpp:159] Typesense 29.0 is terminating abruptly.
DEFAULT 2025-07-17T08:45:56.178313Z Segmentation fault
WARNING 2025-07-17T08:45:56.190706Z Container called exit(139).

Just a question: I didn’t manually delete anything. Was it intentionally deleted by process from TS

] Error while initializing store: IO error: No such file or directory: While opening a file for sequentially reading: /root/lib/typesense/data/db/MANIFEST-000020: No such file or directory

I wrote a small tests, which perm 100 request API call into 3 instances and yeah obviously the data is not sync between them, even all of them are mounting into only one GCS bucket

image.png

Hey @Fanis Tharropoulos thanks for all your help, Just wanted to give back to the community in case someone else wants to try this approach like I did. In Cloud Run, that's just how Google Cloud Run works. • we can only expose a single port. So if we expose

8108

for the API address, we can't also expose

8107

for peering • I*nstance-to-instance communication* isn’t possible in Cloud Run natively My workaround was to deploy two separate Cloud Run services: • One Cloud Run service acts as the leader, handling write requests. It's configured to run as a single instance. • Another Cloud Run service handles read-only requests, using the

--skip-write

argument. This one can scale out to multiple instances.

Hey @Fanis Tharropoulos Do we have any way to talk to Raft or DB to do this index whenever DB data-dir change?

raft_server.cpp:692] Term: 3, pending_queue: 0, last_index: 243, committed: 243, known_applied: 243, applying: 0, pending_writes: 0, queued_writes: 0, local_sequence: 15364

Since I start up a read-only server. and have external mount - that shared w other TS server (read-write)