Connection Refused Issue with Typesense and Nginx Reverse Proxy

TLDR Karthik experiences a connection refused problem with Typesense and Nginx reverse proxy. Jason helps troubleshoot, but the issue remains unresolved.

Photo of Karthik
Karthik
Fri, 17 Mar 2023 23:07:00 UTC

we are running the following on ubuntu on vultr : nginx reverse proxy, nextjs apps on node on localhost:3000; typesense on localhost:8108; Here is my current nginx config; location / { #try_files $uri $uri/ =404; proxy_pass ; proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection 'upgrade'; proxy_set_header Host $host; proxy_cache_bypass $http_upgrade; } I an getting a connection refused with typsense; xhr.js?4381:210 POST net::ERR_CONNECTION_REFUSED I am assuming I should also add typesense to proxy_pass; is that correct? if so how do i add?Any help will be appreciated to resolve this issue

Photo of Jason
Jason
Fri, 17 Mar 2023 23:18:14 UTC

If you’re trying to connect to Typesense running on 8108, you don’t need to run it through nginx

Photo of Jason
Jason
Fri, 17 Mar 2023 23:18:41 UTC

It sounds like typesense server is not running based on the connection refused error

Photo of Jason
Jason
Fri, 17 Mar 2023 23:19:07 UTC

Does `px aux | grep typesense-server` give you any results?

Photo of Srini
Srini
Sat, 18 Mar 2023 00:22:58 UTC

Jason replying as Karthik left home; typesense is running. We are trying to configure the nextjs app on nodejs proxied by nginx and the app has a typesnse searchbar

Photo of Srini
Srini
Sat, 18 Mar 2023 00:27:03 UTC

Jason ● typesense-server.service - Typesense Server Loaded: loaded (/etc/systemd/system/typesense-server.service; enabled; preset: enabled) Active: active (running) since Sat 2023-03-18 00:20:35 UTC; 5min ago Docs: Main PID: 100800 (typesense-serve) Tasks: 107 (limit: 2225) Memory: 23.7M CPU: 350ms CGroup: /system.slice/typesense-server.service └─100800 /usr/bin/typesense-server --config=/etc/typesense/typesense-server.ini Mar 18 00:20:35 acureach systemd[1]: Started Typesense Server. Mar 18 00:20:35 acureach typesense-server[100800]: Log directory is configured as: /var/log/typesense Mar 18 00:20:35 acureach typesense-server[100800]: E20230318 00:20:35.443630 100938 raft_server.h:62] Peer refresh failed, error: Doing another configuration change

Photo of Jason
Jason
Sat, 18 Mar 2023 00:48:33 UTC

Could you run and share the output of this: ```sudo netstat -nlp | grep :8108```

Photo of Jason
Jason
Sat, 18 Mar 2023 00:49:48 UTC

Could you also share the output of the following: ```date ; tail -100 /var/log/typesense/typesense.log```

Photo of Jason
Jason
Sat, 18 Mar 2023 00:50:48 UTC

And also this: ```curl ```

Photo of Srini
Srini
Sat, 18 Mar 2023 00:59:38 UTC

Jason lsof -i :8108

Photo of Srini
Srini
Sat, 18 Mar 2023 00:59:39 UTC

COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME typesense 158688 root 23u IPv4 789450 0t0 TCP *:8108 (LISTEN)

Photo of Srini
Srini
Sat, 18 Mar 2023 01:00:31 UTC

{"ok":true}

Photo of Jason
Jason
Sat, 18 Mar 2023 01:01:13 UTC

Ok so Typesense is running and is healthy

Photo of Jason
Jason
Sat, 18 Mar 2023 01:01:47 UTC

From the very first message: > net::ERR_CONNECTION_REFUSED That will only show up if somehow port 8108 is not accessible by whatever app is throwing that error

Photo of Jason
Jason
Sat, 18 Mar 2023 01:02:02 UTC

Are you running on docker?

Photo of Srini
Srini
Sat, 18 Mar 2023 01:02:36 UTC

no we are notu using docker

Photo of Srini
Srini
Sat, 18 Mar 2023 01:02:39 UTC

sudo netstat -nlp | grep :8108 tcp 0 0 0.0.0.0:8108 0.0.0.0:* LISTEN 158688/typesense-se

Photo of Srini
Srini
Sat, 18 Mar 2023 01:03:12 UTC

you can access to find what happening with chrome debugger

Photo of Jason
Jason
Sat, 18 Mar 2023 01:04:45 UTC

It looks like the Typesense hostname is just configured as , but then that’s where you primary website is also running

Photo of Jason
Jason
Sat, 18 Mar 2023 01:05:06 UTC

I see what you’re trying to do now

Photo of Jason
Jason
Sat, 18 Mar 2023 01:05:30 UTC

You probably want to have Typesense behind say `` and have nginx reverse proxy just that path to Typesense

Photo of Jason
Jason
Sat, 18 Mar 2023 01:06:14 UTC

You want something like this in your nginx config:

Photo of Jason
Jason
Sat, 18 Mar 2023 01:06:39 UTC

```location /search { rewrite /search/(.*) /$1 break; proxy_pass ; proxy_redirect off; proxy_set_header Host $host; }```

Photo of Srini
Srini
Sat, 18 Mar 2023 01:08:28 UTC

I already did this

Photo of Srini
Srini
Sat, 18 Mar 2023 01:08:43 UTC

location / { #try_files $uri $uri/ =404; proxy_pass ; proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection 'upgrade'; proxy_set_header Host $host; proxy_cache_bypass $http_upgrade; } location /multi_search { proxy_pass ; proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection 'upgrade'; proxy_set_header Host $host; proxy_cache_bypass $http_upgrade; }

Photo of Jason
Jason
Sat, 18 Mar 2023 01:11:17 UTC

Could you post the last few logs from the nginx error log file?

Photo of Srini
Srini
Sat, 18 Mar 2023 01:11:18 UTC

just a note this works perfectly fine in local laptop it only happens in cloud wuth nginx reverse proxy

Photo of Jason
Jason
Sat, 18 Mar 2023 01:11:53 UTC

I see a HTTP 500 thrown for that path, so should show some error in the nginx log file

Photo of Srini
Srini
Sat, 18 Mar 2023 01:13:16 UTC

24.44.64.232 - - [18/Mar/2023:01:12:32 +0000] "GET /_next/static/webpack/webpack.7a7305f32c49b41c.hot-update.js HTTP/1.1" 200 858 "" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 168.149.146.41 - - [18/Mar/2023:01:12:32 +0000] "GET /_next/static/chunks/pages/_error.js?ts=1679101951333 HTTP/1.1" 200 1445 "" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 168.149.146.41 - - [18/Mar/2023:01:12:33 +0000] "GET /_next/static/development/_buildManifest.js?ts=1679101951333 HTTP/1.1" 200 364 "" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 168.149.146.41 - - [18/Mar/2023:01:12:33 +0000] "GET /_next/static/chunks/main.js?ts=1679101951333 HTTP/1.1" "" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 168.149.146.41 - - [18/Mar/2023:01:12:33 +0000] "GET /_next/static/development/_ssgManifest.js?ts=1679101951333 HTTP/1.1" 200 76 "" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 168.149.146.41 - - [18/Mar/2023:01:12:33 +0000] "GET /_next/static/chunks/pages/_app.js?ts=1679101951333 HTTP/1.1" 200 888937 "" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 24.44.64.232 - - [18/Mar/2023:01:12:35 +0000] "GET /_next/webpack-hmr HTTP/1.1" 101 158 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 168.149.146.41 - - [18/Mar/2023:01:12:36 +0000] "GET /favicon.ico HTTP/1.1" 304 0 "" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 168.149.146.41 - - [18/Mar/2023:01:12:36 +0000] "GET /_next/static/development/_devMiddlewareManifest.json HTTP/1.1" 200 12 "" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 24.44.64.232 - - [18/Mar/2023:01:12:37 +0000] "GET /_next/webpack-hmr HTTP/1.1" 101 156 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 24.44.64.232 - - [18/Mar/2023:01:12:40 +0000] "GET /_next/webpack-hmr HTTP/1.1" 101 158 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 24.44.64.232 - - [18/Mar/2023:01:12:46 +0000] "GET /_next/webpack-hmr HTTP/1.1" 101 158 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 24.44.64.232 - - [18/Mar/2023:01:12:46 +0000] "GET /_next/webpack-hmr HTTP/1.1" 101 314 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 24.44.64.232 - - [18/Mar/2023:01:12:46 +0000] "GET /_next/webpack-hmr HTTP/1.1" 101 470 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 24.44.64.232 - - [18/Mar/2023:01:12:46 +0000] "GET /_next/webpack-hmr HTTP/1.1" 101 659 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36"

Photo of Srini
Srini
Sat, 18 Mar 2023 01:13:27 UTC

that is from access.log

Photo of Jason
Jason
Sat, 18 Mar 2023 01:13:57 UTC

You want to look at error.log

Photo of Srini
Srini
Sat, 18 Mar 2023 01:14:09 UTC

this is from error.log

Photo of Srini
Srini
Sat, 18 Mar 2023 01:14:10 UTC

2023/03/18 01:12:11 [error] 8671#8671: *26307 connect() failed (111: Connection refused) while connecting to upstream, client: 24.44.64.232, server: , request: "GET /_next/webpack-hmr HTTP/1.1", upstream: "http://[::1]:3000/_next/webpack-hmr", host: "" 2023/03/18 01:12:11 [error] 8671#8671: *26307 connect() failed (111: Connection refused) while connecting to upstream, client: 24.44.64.232, server: , request: "GET /_next/webpack-hmr HTTP/1.1", upstream: "", host: "" 2023/03/18 01:12:12 [error] 8671#8671: *26310 no live upstreams while connecting to upstream, client: 24.44.64.232, server: , request: "GET /_next/webpack-hmr HTTP/1.1", upstream: "", host: "" 2023/03/18 01:12:12 [error] 8672#8672: *26311 connect() failed (111: Connection refused) while connecting to upstream, client: 24.44.64.232, server: , request: "GET /_next/webpack-hmr HTTP/1.1", upstream: "http://[::1]:3000/_next/webpack-hmr", host: "" 2023/03/18 01:12:12 [error] 8672#8672: *26311 connect() failed (111: Connection refused) while connecting to upstream, client: 24.44.64.232, server: , request: "GET /_next/webpack-hmr HTTP/1.1", upstream: "", host: "" 2023/03/18 01:12:13 [error] 8671#8671: *26314 no live upstreams while connecting to upstream, client: 24.44.64.232, server: , request: "GET /_next/webpack-hmr HTTP/1.1", upstream: "", host: "" 2023/03/18 01:12:14 [error] 8671#8671: *26315 no live upstreams while connecting to upstream, client: 24.44.64.232, server: , request: "GET /_next/webpack-hmr HTTP/1.1", upstream: "", host: "" 2023/03/18 01:12:17 [error] 8671#8671: *26316 no live upstreams while connecting to upstream, client: 24.44.64.232, server: , request: "GET /_next/webpack-hmr HTTP/1.1", upstream: "", host: "" 2023/03/18 01:12:17 [error] 8672#8672: *26317 no live upstreams while connecting to upstream, client: 24.44.64.232, server: , request: "GET /_next/webpack-hmr HTTP/1.1", upstream: "", host: "" 2023/03/18 01:12:18 [error] 8672#8672: *26319 no live upstreams while connecting to upstream, client: 168.149.146.41, server: , request: "GET /_next/webpack-hmr HTTP/1.1", upstream: "", host: "" 2023/03/18 01:12:19 [error] 8672#8672: *26320 no live upstreams while connecting to upstream, client: 24.44.64.232, server: , request: "GET /_next/webpack-hmr HTTP/1.1", upstream: "", host: "" 2023/03/18 01:12:20 [error] 8672#8672: *26321 no live upstreams while connecting to upstream, client: 24.44.64.232, server: , request: "GET /_next/webpack-hmr HTTP/1.1", upstream: "", host: "" 2023/03/18 01:12:20 [error] 8672#8672: *26322 no live upstreams while connecting to upstream, client: 24.44.64.232, server: , request: "GET /_next/webpack-hmr HTTP/1.1", upstream: "", host: "" 2023/03/18 01:12:22 [error] 8672#8672: *26323 no live upstreams while connecting to upstream, client: 52.129.0.102, server: , request: "GET /_next/webpack-hmr HTTP/1.1", upstream: "", host: "" 2023/03/18 01:12:27 [error] 8672#8672: *26328 no live upstreams while connecting to upstream, client: 24.44.64.232, server: , request: "GET / HTTP/1.1", upstream: "", host: "", referrer: "" 2023/03/18 01:12:30 [error] 8671#8671: *26353 no live upstreams while connecting to upstream, client: 168.149.146.41, server: , request: "GET /_next/static/webpack/d31f7b7411543aac.webpack.hot-update.json HTTP/1.1", upstream: "", host: "", referrer: ""

Photo of Jason
Jason
Sat, 18 Mar 2023 01:15:25 UTC

I don’t see any requests to multi_search here… Could you first visit the site, generate a multi_search request, and then post the lines from the logs?

Photo of Srini
Srini
Sat, 18 Mar 2023 01:15:25 UTC

do need to add upstream tag in nginx config?

Photo of Jason
Jason
Sat, 18 Mar 2023 01:16:01 UTC

My nginx knowledge might be outdated, but afaik, that shouldn’t be needed

Photo of Srini
Srini
Sat, 18 Mar 2023 01:16:45 UTC

btw Thank you so much for helping us on a Friday evening

Photo of Srini
Srini
Sat, 18 Mar 2023 01:17:00 UTC

I really appreciate yoru help

Photo of Srini
Srini
Sat, 18 Mar 2023 01:17:19 UTC

nothing is coming in error.log when id do the search

Photo of Jason
Jason
Sat, 18 Mar 2023 01:18:46 UTC

Is something else in front of nginx that could be catching that request and throwing a 500?

Photo of Jason
Jason
Sat, 18 Mar 2023 01:19:21 UTC

Because the multi_search request I see generated by the frontend has to show up either in access log or error log

Photo of Jason
Jason
Sat, 18 Mar 2023 01:19:29 UTC

if it doesn’t show up in either places, it’s not reaching this nginx server

Photo of Srini
Srini
Sat, 18 Mar 2023 01:20:10 UTC

nothing is infront of nginx : the config is like this :--> nginx - nodejs nginx - typenses-adapter

Photo of Jason
Jason
Sat, 18 Mar 2023 01:20:32 UTC

Could you try debug just this:

Photo of Jason
Jason
Sat, 18 Mar 2023 01:20:51 UTC

search the logs for a POST request to multi_search

Photo of Srini
Srini
Sat, 18 Mar 2023 01:24:05 UTC

I don;t think it is a nginx problem, it is somewhere in config of Nextjs and typsense adapter

Photo of Srini
Srini
Sat, 18 Mar 2023 01:32:15 UTC

Jason have a good night.