Setting Up HA Cluster on Fly with Typesense
TLDR Jordan needed help setting up an HA cluster on Fly for Typesense. Jason assisted in troubleshooting the configuration and 6tunnel usage. Eventually, a solution was found in the Fly community forum.
1
May 19, 2023 (7 months ago)
Jordan
08:59 PM6tunnel
since Fly.io doesnt natively support IPv4 but I have a bit of confusion for what I need to be using as the IPs for the nodes file and also the peering addresses (since they cant be IPv4…right?)I posted a question on the Fly.io forum but I think it’s a bit too typesense specific so was wondering if we could work through it here? Please let me know what else I can provide
Current raft error I see is:
typesense-server | W0519 20:38:53.494565 652 external/com_github_brpc_braft/src/braft/node.cpp:1559] node default_group:127.0.0.1:8107:8063 request PreVote from 127.0.0.1:8107:8062 error: [E2][172.19.128.34:8107][E2]peer_id not exist
Most likely has to do with my nodes file using localhost since I thought that’s how I could use 6tunnel to communicate from port to other node :thinking_face:
https://community.fly.io/t/how-to-run-a-typesense-ha-cluster-on-fly/12994
Jason
10:31 PMJordan
10:37 PMlocalhost:8107:8062,localhost:8107:8063
Also for context for multiple nodes, the below is hardcoded for now but thought I could do something like the code below for each region I deploy in:
So in this case Port 8062 is for LAX region and 8063 is ported to SJC region.
if typesenseVars.Region == "lax" {
exec.Command("6tunnel", "-6", "-l", "::", "8108", "localhost", "8062").Run()
exec.Command("6tunnel", "-6", "-l", "lax.foundry-typesense-cluster.internal", "8062", "localhost", "8062").Run()
exec.Command("6tunnel", "-6", "-l", "lax.foundry-typesense-cluster.internal", "8107", "localhost", "8107").Run()
exec.Command("6tunnel", "-4", "-l", "localhost", "8063", "sjc.foundry-typesense-cluster.internal").Run()
exec.Command("6tunnel", "-4", "-l", "localhost", "8107", "sjc.foundry-typesense-cluster.internal").Run()
svisor.AddProcess(
"typesense-server",
"doppler run -- typesense-server --data-dir=/data --api-key=xxx --api-port=8062 --peering-address 127.0.0.1 --peering-port 8107 --nodes=/etc/typesense-nodes --reset-peers-on-error",
supervisor.WithRestart(0, 1*time.Second),
)
}
if typesenseVars.Region == "sjc" {
exec.Command("6tunnel", "-6", "-l", "::", "8108", "localhost", "8063").Run()
exec.Command("6tunnel", "-6", "-l", "sjc.foundry-typesense-cluster.internal", "8063", "localhost", "8063").Run()
exec.Command("6tunnel", "-6", "-l", "sjc.foundry-typesense-cluster.internal", "8107", "localhost", "8107").Run()
exec.Command("6tunnel", "-4", "-l", "localhost", "8062", "lax.foundry-typesense-cluster.internal").Run()
exec.Command("6tunnel", "-4", "-l", "localhost", "8107", "lax.foundry-typesense-cluster.internal").Run()
svisor.AddProcess(
"typesense-server",
"doppler run -- typesense-server --data-dir=/data --api-key=xxx --api-port=8063 --peering-address 127.0.0.1 --peering-port 8107 --nodes=/etc/typesense-nodes --reset-peers-on-error",
supervisor.WithRestart(0, 1*time.Second),
)
}
Also more than happy to write some documentation for how to currently handle HA on Fly as I’m sure it’ll be fairly common!
May 20, 2023 (6 months ago)
Jason
12:32 AMAre you able to setup different local ip addresses on each node via 6tunnel?
For eg on node 1, setup
127.0.0.1
as the IP, on node 2, setup 127.0.0.2
, and node 3 setup 127.0.0.3
Jordan
12:47 AMlocalhost
’s in the 6tunnel
to 127.0.0.1 for LAX and 127.0.0.2 for SJCif typesenseVars.Region == "lax" {
exec.Command("6tunnel", "-6", "-l", "::", "8108", "127.0.0.1", "8062").Run()
exec.Command("6tunnel", "-6", "-l", "lax.foundry-typesense-cluster.internal", "8062", "127.0.0.1", "8062").Run()
exec.Command("6tunnel", "-6", "-l", "lax.foundry-typesense-cluster.internal", "8107", "127.0.0.1", "8107").Run()
exec.Command("6tunnel", "-4", "-l", "127.0.0.1", "8063", "sjc.foundry-typesense-cluster.internal").Run()
exec.Command("6tunnel", "-4", "-l", "127.0.0.1", "8107", "sjc.foundry-typesense-cluster.internal").Run()
svisor.AddProcess(
"typesense-server",
"doppler run -- typesense-server --data-dir=/data --api-key=xxx --api-port=8062 --peering-address 127.0.0.1 --peering-port 8107 --nodes=/etc/typesense-nodes --reset-peers-on-error",
supervisor.WithRestart(0, 1*time.Second),
)
}
if typesenseVars.Region == "sjc" {
exec.Command("6tunnel", "-6", "-l", "::", "8108", "127.0.0.2", "8063").Run()
exec.Command("6tunnel", "-6", "-l", "sjc.foundry-typesense-cluster.internal", "8063", "127.0.0.2", "8063").Run()
exec.Command("6tunnel", "-6", "-l", "sjc.foundry-typesense-cluster.internal", "8107", "127.0.0.2", "8107").Run()
exec.Command("6tunnel", "-4", "-l", "127.0.0.2", "8062", "lax.foundry-typesense-cluster.internal").Run()
exec.Command("6tunnel", "-4", "-l", "127.0.0.2", "8107", "lax.foundry-typesense-cluster.internal").Run()
svisor.AddProcess(
"typesense-server",
"doppler run -- typesense-server --data-dir=/data --api-key=xxx --api-port=8063 --peering-address 127.0.0.2 --peering-port 8107 --nodes=/etc/typesense-nodes --reset-peers-on-error",
supervisor.WithRestart(0, 1*time.Second),
)
}
And then I updated my nodes file like this for LAX and SJC respectively:
127.0.0.1:8107:8062,127.0.0.2:8107:8063
As a result, I get an error from LAX stating that it cannot
listen to 127.0.0.1:8107
which means that I haven’t yet setup 6tunnel
correctly to listen on that addressJason
12:51 AMJordan
12:51 AMJason
12:51 AMJason
12:52 AMJason
12:52 AMJason
12:52 AMip:api_port:peering_port
for each nodeJason
12:54 AMlocalhost:8107:8062,localhost:8107:8063
Could you try using doing something like this instead:
127.0.0.1:8107:8062,127.0.0.1:8107:8063
Jason
12:54 AMJordan
12:55 AMJason
12:56 AMJordan
12:57 AMJason
12:59 AMJordan
01:01 AMJason
01:02 AMlocalhost
in the 6tunnel config, and only use 127.0.0.1
in the Typesense nodes file?Jordan
01:04 AMJordan
01:05 AMJordan
01:07 AMJordan
01:07 AMJason
01:07 AMJason
01:08 AM--api-address 127.0.0.1
as an additional command line arg to typesense-server
?Jordan
01:10 AMJason
01:11 AMexec.Command("6tunnel", "-6", "-l", "sjc.foundry-typesense-cluster.internal", "8063", "127.0.0.2", "8063").Run()
exec.Command("6tunnel", "-6", "-l", "sjc.foundry-typesense-cluster.internal", "8107", "127.0.0.2", "8107").Run()
Jason
01:11 AMJordan
01:12 AMlocalhost
based from your message here https://typesense-community.slack.com/archives/C01P749MET0/p1684544521145089?thread_ts=1684529975.170459&cid=C01P749MET0Jason
01:12 AMJordan
01:12 AMJordan
01:13 AMJason
01:15 AMJason
01:15 AMJordan
01:18 AMalright lets see how i can change up those ports 😄 EDIT: of which i already do for the api ports, just need to do it for the peering
Jordan
01:32 AMJason
02:43 AMJason
02:45 AMJason
02:45 AMJordan
02:46 AMJordan
02:47 AMJason
02:48 AMJordan
02:51 AMexec.Command("6tunnel", "-6", "-l", "::", "8107", "localhost", "8107").Run()
since that’s the port I’m using in my
typesense-server
command and it’ll throw a conflict so I need to add an intermediary port:exec.Command("6tunnel", "-6", "-l", "::", "8117", "localhost", "8107").Run()
^ like that and then from my other nodes send to 8117
..i think
sighs
Jason
02:52 AMJason
02:52 AMJordan
02:53 AM1
Jordan
07:10 PMnodes
file. Let me explain…1. I’ve added a new IPv6 listener that routes requests from Port
8117
to 8107
(my peering port defined as flag)2. I route localhost
8119
requests from LAX to SJC (in the same vain where 8119
is the intermediary port that points to SJC’s 8109
peering port).Here’s the LAX block, added
new!
two these new updatesif typesenseVars.Region == "lax" {
exec.Command("6tunnel", "-6", "-l", "::", "8108", "localhost", "8062").Run()
exec.Command("6tunnel", "-6", "-l", "::", "8117", "localhost", "8107").Run() // new!
exec.Command("6tunnel", "-6", "-l", "lax.foundry-typesense-cluster.internal", "8062", "localhost", "8062").Run()
exec.Command("6tunnel", "-6", "-l", "lax.foundry-typesense-cluster.internal", "8107", "localhost", "8107").Run()
exec.Command("6tunnel", "-4", "-l", "localhost", "8063", "sjc.foundry-typesense-cluster.internal").Run()
exec.Command("6tunnel", "-4", "-l", "localhost", "8119", "sjc.foundry-typesense-cluster.internal").Run() // new!
svisor.AddProcess(
"typesense-server",
"doppler run -- typesense-server --data-dir=/data --api-key=xxx --api-port=8062 --peering-address=127.0.0.1 --peering-port 8107 --nodes=/etc/typesense-nodes --reset-peers-on-error",
supervisor.WithRestart(0, 1*time.Second),
)
}
For typesense to actually choose who votes, we need to update the
nodes
file too, and since we need the requests to be sent to the intermediary ports we need to update our file like so:localhost:8117:8062,localhost:8119:8063 (8117 instead of 8107 etc)
Which now means Typesense throws an error:
typesense-server | W0520 19:07:48.512210 652 external/com_github_brpc_braft/src/braft/node.cpp:1589] node default_group:127.0.0.1:8107:8062 can't do pre_vote as it is not in 127.0.0.1:8117:8062,127.0.0.1:8119:8063
Since the IP/port (127.0.0.1:8107) defined in the server flag is not in the nodes file…I have been stumped again.
Jordan
10:04 PMMay 21, 2023 (6 months ago)
Jason
06:02 PMTypesense
Indexed 3015 threads (79% resolved)
Similar Threads
Troubleshooting Typesense Cluster Multi-node Leadership Error
Bill experienced a problem with a new typesense cluster, receiving an error about no leader and health status issues. Jason and Kishore Nallan provided troubleshooting steps and determined it was likely due to a communication issue between nodes. Kishore Nallan identified a potential solution involving resetting the data directory. Following this, Bill reported the error resolved.
Troubleshooting Multi-Node Setup in Docker
Harpreet experienced issues running a multi-node setup on Docker and received troubleshooting advice and alternative solution from Kishore Nallan.
Typesense Node Stuck in Segfault Loop After Stress Test
Adrian encountered a segfault loop when stress testing a Typesense cluster. Kishore Nallan recommended trying a newer RC build and suggested potential issues with hostname resolution.
Deploying to Fly.io via Dockerfile
andrew asks how to launch to Fly.io with Docker. Jason guides them on the environment setup, specifically mentioning persistent VMs, and approves andrew's Dockerfile setup.
Troubleshooting IP Update on Kubernetes Typesense
Alessandro and Damien are having issues with old IP addresses in a Kubernetes Typesense cluster not being updated. Kishore Nallan provides possible troubleshooting solutions, and mentioned the need for a fix for DNS retries. A suggested update strategy was shared by Aljosa.