Hi Context: We are running a six node typesense cluster in our production environment with each node n2d-highcpu-48 (48-cores 48gb-memory) hosted in GCP for our search engine. This is the search request we are performing on typesnse DB.

curl --location '<http://localhost:8108/multi_search?q=hey>' \
--header 'Content-Type: application/json' \
--header 'X-TYPESENSE-API-KEY: xyz\
--data '{
    "searches": [
       {
                "collection": "contents",
                "drop_tokens_threshold": 2,
                "filter_by": "visible: true && contentType: [Movies] && vendor: ![vkjdsda] && ((playbackType: playback && provider: [netflix,disney,hotstar]) || (provider:[primeVideo,aha,sony,colors,TataNeu,netflix,disney,hotstar]) || (playbackType: deeplink && provider: [disney]))",
                "max_candidates": 3,
                "min_len_1typo": 2,
                "num_typos": "5,0,0,2",
                "per_page": 10,
                "query_by": "title, language, genres, starcast",
                "query_by_weights": "5, 3, 2, 2",
                "sort_by": "_text_match(buckets: 10):desc, score1(missing_values: last):desc, score2(missing_values: last):desc",
                "typo_tokens_threshold": 20,
                "use_cache": true,
                "page_no": 1
        }
    ]
}'

Collection Schema: There are 3 Million records in the collection.

{
  "name": "contents",
  "fields": [
    {
      "name": "starcast",
      "type": "string[]",
      "facet": false,
      "optional": false,
      "index": true,
      "sort": false,
      "infix": false,
      "locale": "",
      "stem": false
    },
    {
      "name": "genres",
      "type": "string[]",
      "facet": false,
      "optional": false,
      "index": true,
      "sort": false,
      "infix": false,
      "locale": "",
      "stem": false
    },
    {
      "name": "title",
      "type": "string",
      "facet": false,
      "optional": false,
      "index": true,
      "sort": false,
      "infix": false,
      "locale": "",
      "stem": false
    },
    {
      "name": "editorial_index",
      "type": "float",
      "facet": false,
      "optional": true,
      "index": true,
      "sort": true,
      "infix": false,
      "locale": "",
      "stem": false
    },
    {
      "name": "data_engg_index",
      "type": "float",
      "facet": false,
      "optional": true,
      "index": true,
      "sort": true,
      "infix": false,
      "locale": "",
      "stem": false
    },
    {
      "name": "reco_index",
      "type": "float",
      "facet": false,
      "optional": true,
      "index": true,
      "sort": true,
      "infix": false,
      "locale": "",
      "stem": false
    },
    {
      "name": "telecast_date_ts",
      "type": "int64",
      "facet": false,
      "optional": false,
      "index": true,
      "sort": true,
      "infix": false,
      "locale": "",
      "stem": false
    },
    {
      "name": "contentType",
      "type": "string",
      "facet": true,
      "optional": false,
      "index": true,
      "sort": true,
      "infix": false,
      "locale": "",
      "stem": false
    },
    {
      "name": "language",
      "type": "string",
      "facet": false,
      "optional": false,
      "index": true,
      "sort": true,
      "infix": false,
      "locale": "",
      "stem": false
    },
    {
      "name": "maturity_rating",
      "type": "string",
      "facet": false,
      "optional": false,
      "index": true,
      "sort": false,
      "infix": false,
      "locale": "",
      "stem": false
    },
    {
      "name": "directors",
      "type": "string[]",
      "facet": false,
      "optional": false,
      "index": true,
      "sort": false,
      "infix": false,
      "locale": "",
      "stem": false
    },
    {
      "name": "searchTags",
      "type": "string[]",
      "facet": false,
      "optional": false,
      "index": true,
      "sort": false,
      "infix": false,
      "locale": "",
      "stem": false
    },
    {
      "name": "visible",
      "type": "bool",
      "facet": false,
      "optional": false,
      "index": true,
      "sort": false,
      "infix": false,
      "locale": "",
      "stem": false
    },
    {
      "name": "vendor",
      "type": "string",
      "facet": false,
      "optional": true,
      "index": true,
      "sort": false,
      "infix": false,
      "locale": "",
      "stem": false
    },
    {
      "name": "masterId",
      "type": "string",
      "facet": false,
      "optional": true,
      "index": true,
      "sort": false,
      "infix": false,
      "locale": "",
      "stem": false
    },
    {
      "name": "provider",
      "type": "string",
      "facet": true,
      "optional": true,
      "index": true,
      "sort": false,
      "infix": false,
      "locale": "",
      "stem": false
    },
    {
      "name": "playbackType",
      "type": "string",
      "facet": false,
      "optional": true,
      "index": true,
      "sort": false,
      "infix": false,
      "locale": "",
      "stem": false
    },
    {
      "name": "shareWithPartner",
      "type": "bool",
      "facet": false,
      "optional": true,
      "index": true,
      "sort": false,
      "infix": false,
      "locale": "",
      "stem": false
    }
  ],
  "default_sorting_field": "telecast_date_ts",
  "enable_nested_fields": true,
  "symbols_to_index": [],
  "token_separators": []
}

Despite utilizing a high-performance CPU with 48 cores, our production environment is currently able to handle only 300 requests per second (RPS), averaging approximately 60 RPS per node. This is notably low, considering we route all search requests exclusively to the follower nodes (5 nodes in total). Simultaneously, we are also limiting 100–150 write operations per second directly to leader Node. Could you suggest ways to optimize this performance? Please suggest improvements in query parameters or in schema Data size: 3+ millions Most of the time when incident occurs, its due to CPU utilization goes above 100% And if we get more search requests in the peak time, Nodes are going down and Getting the below log messages in typesense logs and typesense DB is shutting down automatically. Errors found • W20250119 135803.441246 440569 node.cpp:1559] node default_group10.130.133.798107:8108 request PreVote from 10.130.133.10681078108 error: [E2][10.130.133.106:8107][E2]peer_id not exist • W20250119 135803.629504 440408 raft_server.cpp:721] Multi-node with no leader: refusing to reset peers. • W20250119 135303.197127 440408 controller.cpp:1550] SIGINT was installed with 1 • E20250119 135234.970660 472737 raft_server.cpp:780] 6195 lagging entries > healthy write lag of 3000 • E20250119 135234.970606 472737 raft_server.cpp:768] 6195 lagging entries > healthy read lag of 5000

Responded here: https://typesense-community.slack.com/archives/C01P749MET0/p1737408239846909?thread_ts=1737222488.154589&cid=C01P749MET0 Let's continue the conversation in that thread, so it's all in once place.