Hi team, We're running a self-hosted Typesense cl...
# community-help
a
Hi team, We're running a self-hosted Typesense cluster (v30.1, 5-node Raft, 4 CPU / 16 GiB per node) and experiencing intermittent 503s under our current write load. We operate a precious metals marketplace where product prices are adjusted in real time based on spot price changes. We use the bulk import endpoint to push ~1,700 document updates per minute, and we need those updates to be queryable within seconds. 1. 503 errors during write bursts, despite what we believe should be sufficient resources. 2. Write-to-read latency — updates take several minutes before they're visible to clients, whereas we need near real-time propagation. - Is 1,700 updates/min via bulk import within expected throughput for this hardware profile, or do we need to scale differently? - Are there recommended tuning parameters (batch size, Raft settings, resource allocation) to reduce write-to-read latency for this kind of real-time pricing use case? - Any general guidance on achieving low-latency writes with Typesense at this volume? Happy to share config details, logs, or metrics if helpful. Thanks in advance.
a
Hi Armando, Are you in cloud? Could you dm me your cluster id?
a
Hi Alan were not in cloud were self hosting.
a
@Armando Villanueva, sorry for the wait. Usually self hosted clusters are hardly to deeply investigate, since we don't have access to many tools we do in cloud. But there is some general guidelines on handling big write patterns: The hardware is in principle adequate for ~28 writes/sec, but this workload is bottlenecked on the single-threaded Raft leader. The 503s and latency almost always come from (a) too many concurrent small import calls, (b) too-small batches, (c) all traffic hitting one node, (d) a 5-node cluster (vs 3) doubling the per-write replication cost, and/or (e) schema work per doc (embeddings, many indexed fields). Fix by batching larger, capping concurrency at N-2 workers, configuring all node hostnames client-side, raising client timeout, and verifying schema cost before scaling hardware. You'll also want to check this: https://typesense.org/docs/guide/syncing-data-into-typesense.html#high-volume-writes