we used nearest_node before but was explained before that having that, it will always use that node, what we want is to distribute all incoming queries equally to all nodes

like if node 1 used now, next query should use node 2 even if node 1 is not yet loaded

@Fanis Tharropoulos how about my question regardung write process, does it always use the leader node? which is always node 1?

Yes, the leader will take the writes and distribute them, you can reelect a leader though, it won't always be node 1

If you want to find the leader, you can call out to the

/debug

route. If the

state

key is 1, then it's the leader

The other nodes will have a state of 4, being follower nodes

@Fanis Tharropoulos is there cases where the leader node changes? or it is fixed to a certain node?

@Fanis Tharropoulos because of that we create our own round robin, we have 3 clients, where for each the order of the nodes are different

typesense_client_b_config = {
        'nodes': [
            {'host': f'{host_unique_key}-1.{host_domain}', 'port': '443', 'protocol': 'https'},
            {'host': f'{host_unique_key}-2.{host_domain}', 'port': '443', 'protocol': 'https'},
            {'host': f'{host_unique_key}-3.{host_domain}', 'port': '443', 'protocol': 'https'},
        ]


typesense_client_b_config = {
        'nodes': [
            {'host': f'{host_unique_key}-2.{host_domain}', 'port': '443', 'protocol': 'https'},
            {'host': f'{host_unique_key}-3.{host_domain}', 'port': '443', 'protocol': 'https'},
            {'host': f'{host_unique_key}-1.{host_domain}', 'port': '443', 'protocol': 'https'},
        ],
        'api_key': Bootstrapper.config.typesense_api_key,
        'connection_timeout_seconds': typesense_connection_timeout
    }
    typesense_client_b = typesense.Client(typesense_client_b_config)


 typesense_client_c_config = {
        'nodes': [
            {'host': f'{host_unique_key}-3.{host_domain}', 'port': '443', 'protocol': 'https'},
            {'host': f'{host_unique_key}-1.{host_domain}', 'port': '443', 'protocol': 'https'},
            {'host': f'{host_unique_key}-2.{host_domain}', 'port': '443', 'protocol': 'https'},
        ],
        'api_key': Bootstrapper.config.typesense_api_key,
        'connection_timeout_seconds': typesense_connection_timeout
    }
typesense_client_c = typesense.Client(typesense_client_c_config)

with this, we use them linearly for each query like round robin, if first query use client_a, then next query will use client_b, to evenly distribute tasks

@Fanis Tharropoulos but somehow, as the tasks are running, there will be times where 1 node becomes 100 CPU usage causing like 48 seconds query

any idea on why is that? maybe on your side?

@Fanis Tharropoulos would you be able to check on your side if there's anything you may find weird or any idea what might be causing the issue? Case: • we have a worker with 4 concurrencies that process a schedule task typesense search 4 in parallel, at first we ran it at the Demo Typesense (not multi node) and didn't encounter any performance issues, then we deployed it to our prod typesense(multi node), somehow, in prod, one node is getting overloaded, causing query time to reached as high as 48seconds, this is the first time we encounter this, since before we only run the scheduled task linearly (so 1 at a time) • difference between demo and prod is that, in prod, we have our actual users who access our website that performs other typesense queries/write, demo is more of a testing app • also this time, the scheduled task is running typesense query 4 in parallel, which before, only 1 • 1 node is reaching as high as 100 CPU