https://typesense.org logo
Join Slack
Powered by
hey guys, has anyone managed to get typesense (hig...
# community-help
k
hey guys, has anyone managed to get typesense (high availability) working with ecs on fargate? if so, help would be greatly appreciated!
r
I would wonder the usecase of running typesense on a serverless environment.
I’ve done it on EC2 and left it running. Works fine on a 2GB ram with my workflow. T4G
k
Yea our initial setup was in EC2 as well but the final decision from upper management was to go with Fargate as there's less configuration and maintenance in the future
@Rohit Gupta Hi mate, for EC2, when defining the nodes.txt file - how did you get the private IP of all the other nodes? I'm able to get the current node IP but not the others my aws_instance resource in terraform looks like this atm
Copy code
user_data = <<-EOF
    # download & unarchive typesense
    curl -O <https://dl.typesense.org/releases/${var.typesense_version}/typesense-server-${var.typesense_version}-linux-arm64.tar.gz>
    tar -xzf typesense-server-${var.typesense_version}-linux-arm64.tar.gz -C /opt/typesense

    # remove archive
    rm typesense-server-${var.typesense_version}-linux-arm64.tar.gz

    # get instance private IP
    EC2_INSTANCE_IP=$(curl -s <http://instance-data/latest/meta-data/local-ipv4>)

    # create typesense folders
    mkdir -p /opt/typesense/data
    mkdir -p /opt/typesense/log

    # create the nodes file
    echo "$EC2_INSTANCE_IP:${var.peering_port}:${var.app_port}" > /opt/typesense/nodes

    # create typesense server config file
    echo "[server]
    api-key = ""
    data-dir = /opt/typesense/data
    log-dir = /opt/typesense/log
    enable-cors = true
    api-port = ${var.app_port}
    peering-port = ${var.peering_port}
    peering-address = $EC2_INSTANCE_IP
    nodes = /opt/typesense/nodes" > /opt/typesense/typesense.ini

    # create typesense service
    echo "[Unit]
    Description=Typesense service
    After=network.target
    [Service]
    Type=simple
    Restart=always
    RestartSec=5
    User=ec2-user
    ExecStart=/opt/typesense/typesense-server --config=/opt/typesense/typesense.ini
    [Install]
    WantedBy=default.target" > /etc/systemd/system/typesense.service

    # start typesense service
    systemctl start typesense

    # enable typesense daemon
    systemctl enable typesense
  EOF
r
I haven’t worked on terraform yet. but the core should be to keep a load balancer outside, the request should only be passed through load balancer
k
were you able to find a way to keep the nodes in sync with each other? i.e with autoscaling and spinning up and tearing down instances (updating the nodes.txt file for each other node)
r
y
Going to reply here since it’s the best/only thread I found about HA. Coming back to running on ECS Fargate with HA: I see that TypeSense doesn’t like it when the number of ECS tasks is scaled up without specifying other master 
--nodes
for HA (screenshot). I was hoping it would be that easy. So my Q: do nodes need to know about each other if the clients are only ever reading from typesense? Does typesense have a read-only mode, of sorts? Such that it wouldn’t care about filesystem locks since no writes are happening.
The concrete use case is: Fargate ECS with Elastic File System network mount for underlying storage. EFS would be shared between all ECS nodes, and there can be as many nodes as needed to support search / read load.
I’m guessing the problem with what I’m saying is that each time a node is added, all other nodes need to know. I was hoping this wasn’t the case. Perhaps I’m putting on my developer hat at the wrong time, and treating typesense too much as a stateless app. Guess I need to put my infra hat back on, and look at it more as a database.
j
do nodes need to know about each other if the clients are only ever reading from typesense?
Yes, for data to be accepted into a node it has to be part of a stable cluster, and having a stable cluster means that all nodes files are identical across all the nodes.
There is no read-only node in Typesense
Perhaps I’m putting on my developer hat at the wrong time, and treating typesense too much as a stateless app. Guess I need to put my infra hat back on, and look at it more as a database.
Haha! That's a nice way to put it 🙂 The statefulness of a DB, more specifically a distributed system, definitely has more implications infra-wise than a stateless web app
y
Thanks for confirming!
👍 1
Do I understand it correctly that TS nodes can share the same underlying filesystem? From my testing I’m seeing that they can’t. I have 3 EC2 servers sharing the same 
/data
folder mounted from one AWS EFS mount. In 
/etc/typesense/typesense-server.ini
I have
Copy code
[server]

api-address = 0.0.0.0
api-port = 8108
data-dir = /data
api-key = <snip>
log-dir = /var/log/typesense
log-slow-requests-time-ms = 2000
nodes = /etc/typesense/nodes
peering-address = 10.133.34.184 # diff IP for other 2 servers
peering-port = 8107
/etc/typesense/nodes
Copy code
10.133.33.89:8107:8108,10.133.32.43:8107:8108,10.133.34.184:8107:8108
Then I run
Copy code
systemctl restart typesense-server.service && journalctl -f -u typesense-server.service
on all nodes at the same time. The first node that starts up first locks up the filesystem and other nodes report this error:
Copy code
May 22 19:18:19 ip-10-133-33-89 systemd[1]: Started typesense-server.service - Typesense Server.
May 22 19:18:19 ip-10-133-33-89 typesense-server[35141]: Log directory is configured as: /var/log/typesense
May 22 19:18:19 ip-10-133-33-89 typesense-server[35141]: E20250522 19:18:19.734864 35141 store.cpp:54] Error while initializing store: IO error: While lock file: /data/db/LOCK: Resource temporarily unavailable
May 22 19:18:19 ip-10-133-33-89 typesense-server[35141]: E20250522 19:18:19.734895 35141 store.cpp:56] It seems like the data directory /data/db is already being used by another Typesense server.
May 22 19:18:19 ip-10-133-33-89 typesense-server[35141]: E20250522 19:18:19.734901 35141 store.cpp:58] If you are SURE that this is not the case, delete the LOCK file in the data db directory and try again.
May 22 19:18:19 ip-10-133-33-89 typesense-server[35141]: E20250522 19:18:19.781569 35141 store.cpp:54] Error while initializing store: IO error: While lock file: /data/meta/LOCK: Resource temporarily unavailable
May 22 19:18:19 ip-10-133-33-89 typesense-server[35141]: E20250522 19:18:19.781616 35141 store.cpp:56] It seems like the data directory /data/meta is already being used by another Typesense server.
May 22 19:18:19 ip-10-133-33-89 typesense-server[35141]: E20250522 19:18:19.781621 35141 store.cpp:58] If you are SURE that this is not the case, delete the LOCK file in the data db directory and try again.
May 22 19:18:19 ip-10-133-33-89 typesense-server[35141]: E20250522 19:18:19.987128 35141 backward.hpp:4200] Stack trace (most recent call last):
May 22 19:18:19 ip-10-133-33-89 typesense-server[35141]: E20250522 19:18:19.987164 35141 backward.hpp:4200] #7    Object "", at 0xffffffffffffffff, in
May 22 19:18:19 ip-10-133-33-89 typesense-server[35141]: E20250522 19:18:19.987169 35141 backward.hpp:4200] #6    Object "/usr/bin/typesense-server", at 0x55d04973c02d, in _start
May 22 19:18:19 ip-10-133-33-89 typesense-server[35141]: E20250522 19:18:19.987174 35141 backward.hpp:4200] #5    Object "/usr/lib64/libc.so.6", at 0x7f8cc923ff8f, in __libc_start_main@@GLIBC_2.34
May 22 19:18:19 ip-10-133-33-89 typesense-server[35141]: E20250522 19:18:19.987177 35141 backward.hpp:4200] #4    Object "/usr/lib64/libc.so.6", at 0x7f8cc923fedf, in __libc_start_call_main
May 22 19:18:19 ip-10-133-33-89 typesense-server[35141]: E20250522 19:18:19.987185 35141 backward.hpp:4200] #3    Source "src/main/typesense_server.cpp", line 214, in main [0x55d04974684d]
May 22 19:18:19 ip-10-133-33-89 typesense-server[35141]: E20250522 19:18:19.987187 35141 backward.hpp:4200] #2    Source "src/typesense_server_utils.cpp", line 484, in run_server [0x55d049b70e27]
May 22 19:18:19 ip-10-133-33-89 typesense-server[35141]: E20250522 19:18:19.987191 35141 backward.hpp:4200] #1    Source "src/ratelimit_manager.cpp", line 498, in init [0x55d049af0e37]
May 22 19:18:19 ip-10-133-33-89 typesense-server[35141]: E20250522 19:18:19.987195 35141 backward.hpp:4200] #0    Source "src/store.cpp", line 102, in get [0x55d049b19b5f]
May 22 19:18:19 ip-10-133-33-89 typesense-server[35141]: Segmentation fault (Address not mapped to object [(nil)])
May 22 19:18:20 ip-10-133-33-89 typesense-server[35141]: E20250522 19:18:20.189659 35141 typesense_server.cpp:149] Typesense 28.0 is terminating abruptly.
j
> Do I understand it correctly that TS nodes can share the same underlying filesystem? Nope, this is not the case. Each Typesense node in a cluster needs to have a unique data dir location and cannot share the same directory
For fault-tolerance, you need to have completely independent nodes that just sync data with each other via Raft, instead of sharing things at the file system level
y
Ok gotchu thanks. I mistakenly thought that there is one writes and readers need to have access to the writer’s filesystem. Sounds like all nodes are writers and readers, and sync data via Raft
j
Sounds like all nodes are writers and readers, and sync data via Raft
that's correct
🙌 1
3 Views
Open in Slack
PreviousNext
Powered by