#community-help

Connection Refused Issue with Typesense and Nginx Reverse Proxy

TLDR Karthik experiences a connection refused problem with Typesense and Nginx reverse proxy. Jason helps troubleshoot, but the issue remains unresolved.

Powered by Struct AI
+13
smile1
Mar 17, 2023 (6 months ago)
Karthik
Photo of md5-b7a258bf917f26602c9d5efc88bfc6f5
Karthik
11:07 PM
we are running the following on ubuntu on vultr : nginx reverse proxy, nextjs apps on node on localhost:3000; typesense on localhost:8108; Here is my current nginx config; location / {
#try_files $uri $uri/ =404;
proxy_pass http://localhost:3000;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
} I an getting a connection refused with typsense; xhr.js?4381:210 POST http://localhost:8108/multi_search?x-typesense-api-key=xyz net::ERR_CONNECTION_REFUSED I am assuming I should also add typesense to proxy_pass; is that correct? if so how do i add?Any help will be appreciated to resolve this issue
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
11:18 PM
If you’re trying to connect to Typesense running on 8108, you don’t need to run it through nginx
11:18
Jason
11:18 PM
It sounds like typesense server is not running based on the connection refused error
11:19
Jason
11:19 PM
Does px aux | grep typesense-server give you any results?
Mar 18, 2023 (6 months ago)
Srini
Photo of md5-9cd23c7211ae4451cdff9bd79e03bbbc
Srini
12:22 AM
Jason replying as Karthik left home; typesense is running. We are trying to configure the nextjs app on nodejs proxied by nginx and the app has a typesnse searchbar
12:27
Srini
12:27 AM
Jason ● typesense-server.service - Typesense Server
Loaded: loaded (/etc/systemd/system/typesense-server.service; enabled; preset: enabled)
Active: active (running) since Sat 2023-03-18 00:20:35 UTC; 5min ago
Docs: https://typesense.org
Main PID: 100800 (typesense-serve)
Tasks: 107 (limit: 2225)
Memory: 23.7M
CPU: 350ms
CGroup: /system.slice/typesense-server.service
└─100800 /usr/bin/typesense-server --config=/etc/typesense/typesense-server.ini

Mar 18 00:20:35 acureach systemd[1]: Started Typesense Server.
Mar 18 00:20:35 acureach typesense-server[100800]: Log directory is configured as: /var/log/typesense
Mar 18 00:20:35 acureach typesense-server[100800]: E20230318 00:20:35.443630 100938 raft_server.h:62] Peer refresh failed, error: Doing another configuration change
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
12:48 AM
Could you run and share the output of this:

sudo netstat -nlp | grep :8108
12:49
Jason
12:49 AM
Could you also share the output of the following:

date ; tail -100 /var/log/typesense/typesense.log
12:50
Jason
12:50 AM
And also this:

curl 
Srini
Photo of md5-9cd23c7211ae4451cdff9bd79e03bbbc
Srini
12:59 AM
Jason lsof -i :8108
12:59
Srini
12:59 AM
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
typesense 158688 root 23u IPv4 789450 0t0 TCP *:8108 (LISTEN)
01:00
Srini
01:00 AM
{"ok":true}
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
01:01 AM
Ok so Typesense is running and is healthy
01:01
Jason
01:01 AM
From the very first message:

> http://localhost:8108/multi_search?x-typesense-api-key=xyz net::ERR_CONNECTION_REFUSED
That will only show up if somehow port 8108 is not accessible by whatever app is throwing that error
01:02
Jason
01:02 AM
Are you running on docker?
Srini
Photo of md5-9cd23c7211ae4451cdff9bd79e03bbbc
Srini
01:02 AM
no we are notu using docker
01:02
Srini
01:02 AM
sudo netstat -nlp | grep :8108
tcp 0 0 0.0.0.0:8108 0.0.0.0:* LISTEN 158688/typesense-se
01:03
Srini
01:03 AM
you can access acureach.in to find what happening with chrome debugger
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
01:04 AM
It looks like the Typesense hostname is just configured as https://acureach.in, but then that’s where you primary website is also running
01:05
Jason
01:05 AM
I see what you’re trying to do now
01:05
Jason
01:05 AM
You probably want to have Typesense behind say <https://acureach.in/search> and have nginx reverse proxy just that path to Typesense
01:06
Jason
01:06 AM
You want something like this in your nginx config: https://serverfault.com/questions/379675/nginx-reverse-proxy-url-rewrite
01:06
Jason
01:06 AM
location  /search {
  rewrite /search/(.*) /$1  break;
  proxy_pass         ;
  proxy_redirect     off;
  proxy_set_header   Host $host;
}
Srini
Photo of md5-9cd23c7211ae4451cdff9bd79e03bbbc
Srini
01:08 AM
I already did this
01:08
Srini
01:08 AM
location / {
#try_files $uri $uri/ =404;
proxy_pass http://localhost:3000;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
}

location /multi_search {
proxy_pass http://localhost:8108;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
}
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
01:11 AM
Could you post the last few logs from the nginx error log file?
Srini
Photo of md5-9cd23c7211ae4451cdff9bd79e03bbbc
Srini
01:11 AM
just a note this works perfectly fine in local laptop it only happens in cloud wuth nginx reverse proxy
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
01:11 AM
I see a HTTP 500 thrown for that path, so should show some error in the nginx log file
Srini
Photo of md5-9cd23c7211ae4451cdff9bd79e03bbbc
Srini
01:13 AM
24.44.64.232 - - [18/Mar/2023:01:12:32 +0000] "GET /_next/static/webpack/webpack.7a7305f32c49b41c.hot-update.js HTTP/1.1" 200 858 "https://acureach.in/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36"
168.149.146.41 - - [18/Mar/2023:01:12:32 +0000] "GET /_next/static/chunks/pages/_error.js?ts=1679101951333 HTTP/1.1" 200 1445 "https://acureach.in/multi_search?x-typesense-api-key=xyx" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36"
168.149.146.41 - - [18/Mar/2023:01:12:33 +0000] "GET /_next/static/development/_buildManifest.js?ts=1679101951333 HTTP/1.1" 200 364 "https://acureach.in/multi_search?x-typesense-api-key=xyx" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36"
168.149.146.41 - - [18/Mar/2023:01:12:33 +0000] "GET /_next/static/chunks/main.js?ts=1679101951333 HTTP/1.1" 200 1114746 "https://acureach.in/multi_search?x-typesense-api-key=xyx" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36"
168.149.146.41 - - [18/Mar/2023:01:12:33 +0000] "GET /_next/static/development/_ssgManifest.js?ts=1679101951333 HTTP/1.1" 200 76 "https://acureach.in/multi_search?x-typesense-api-key=xyx" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36"
168.149.146.41 - - [18/Mar/2023:01:12:33 +0000] "GET /_next/static/chunks/pages/_app.js?ts=1679101951333 HTTP/1.1" 200 888937 "https://acureach.in/multi_search?x-typesense-api-key=xyx" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36"
24.44.64.232 - - [18/Mar/2023:01:12:35 +0000] "GET /_next/webpack-hmr HTTP/1.1" 101 158 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36"
168.149.146.41 - - [18/Mar/2023:01:12:36 +0000] "GET /favicon.ico HTTP/1.1" 304 0 "https://acureach.in/multi_search?x-typesense-api-key=xyx" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36"
168.149.146.41 - - [18/Mar/2023:01:12:36 +0000] "GET /_next/static/development/_devMiddlewareManifest.json HTTP/1.1" 200 12 "https://acureach.in/multi_search?x-typesense-api-key=xyx" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36"
24.44.64.232 - - [18/Mar/2023:01:12:37 +0000] "GET /_next/webpack-hmr HTTP/1.1" 101 156 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36"
24.44.64.232 - - [18/Mar/2023:01:12:40 +0000] "GET /_next/webpack-hmr HTTP/1.1" 101 158 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36"
24.44.64.232 - - [18/Mar/2023:01:12:46 +0000] "GET /_next/webpack-hmr HTTP/1.1" 101 158 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36"
24.44.64.232 - - [18/Mar/2023:01:12:46 +0000] "GET /_next/webpack-hmr HTTP/1.1" 101 314 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36"
24.44.64.232 - - [18/Mar/2023:01:12:46 +0000] "GET /_next/webpack-hmr HTTP/1.1" 101 470 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36"
24.44.64.232 - - [18/Mar/2023:01:12:46 +0000] "GET /_next/webpack-hmr HTTP/1.1" 101 659 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36"
01:13
Srini
01:13 AM
that is from access.log
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
01:13 AM
You want to look at error.log
Srini
Photo of md5-9cd23c7211ae4451cdff9bd79e03bbbc
Srini
01:14 AM
this is from error.log
01:14
Srini
01:14 AM
2023/03/18 01:12:11 [error] 8671#8671: *26307 connect() failed (111: Connection refused) while connecting to upstream, client: 24.44.64.232, server: acureach.in, request: "GET /_next/webpack-hmr HTTP/1.1", upstream: "http://[::1]:3000/_next/webpack-hmr", host: "acureach.in"
2023/03/18 01:12:11 [error] 8671#8671: *26307 connect() failed (111: Connection refused) while connecting to upstream, client: 24.44.64.232, server: acureach.in, request: "GET /_next/webpack-hmr HTTP/1.1", upstream: "http://127.0.0.1:3000/_next/webpack-hmr", host: "acureach.in"
2023/03/18 01:12:12 [error] 8671#8671: *26310 no live upstreams while connecting to upstream, client: 24.44.64.232, server: acureach.in, request: "GET /_next/webpack-hmr HTTP/1.1", upstream: "http://localhost/_next/webpack-hmr", host: "acureach.in"
2023/03/18 01:12:12 [error] 8672#8672: *26311 connect() failed (111: Connection refused) while connecting to upstream, client: 24.44.64.232, server: acureach.in, request: "GET /_next/webpack-hmr HTTP/1.1", upstream: "http://[::1]:3000/_next/webpack-hmr", host: "acureach.in"
2023/03/18 01:12:12 [error] 8672#8672: *26311 connect() failed (111: Connection refused) while connecting to upstream, client: 24.44.64.232, server: acureach.in, request: "GET /_next/webpack-hmr HTTP/1.1", upstream: "http://127.0.0.1:3000/_next/webpack-hmr", host: "acureach.in"
2023/03/18 01:12:13 [error] 8671#8671: *26314 no live upstreams while connecting to upstream, client: 24.44.64.232, server: acureach.in, request: "GET /_next/webpack-hmr HTTP/1.1", upstream: "http://localhost/_next/webpack-hmr", host: "acureach.in"
2023/03/18 01:12:14 [error] 8671#8671: *26315 no live upstreams while connecting to upstream, client: 24.44.64.232, server: acureach.in, request: "GET /_next/webpack-hmr HTTP/1.1", upstream: "http://localhost/_next/webpack-hmr", host: "acureach.in"
2023/03/18 01:12:17 [error] 8671#8671: *26316 no live upstreams while connecting to upstream, client: 24.44.64.232, server: acureach.in, request: "GET /_next/webpack-hmr HTTP/1.1", upstream: "http://localhost/_next/webpack-hmr", host: "acureach.in"
2023/03/18 01:12:17 [error] 8672#8672: *26317 no live upstreams while connecting to upstream, client: 24.44.64.232, server: acureach.in, request: "GET /_next/webpack-hmr HTTP/1.1", upstream: "http://localhost/_next/webpack-hmr", host: "acureach.in"
2023/03/18 01:12:18 [error] 8672#8672: *26319 no live upstreams while connecting to upstream, client: 168.149.146.41, server: acureach.in, request: "GET /_next/webpack-hmr HTTP/1.1", upstream: "http://localhost/_next/webpack-hmr", host: "acureach.in"
2023/03/18 01:12:19 [error] 8672#8672: *26320 no live upstreams while connecting to upstream, client: 24.44.64.232, server: acureach.in, request: "GET /_next/webpack-hmr HTTP/1.1", upstream: "http://localhost/_next/webpack-hmr", host: "acureach.in"
2023/03/18 01:12:20 [error] 8672#8672: *26321 no live upstreams while connecting to upstream, client: 24.44.64.232, server: acureach.in, request: "GET /_next/webpack-hmr HTTP/1.1", upstream: "http://localhost/_next/webpack-hmr", host: "acureach.in"
2023/03/18 01:12:20 [error] 8672#8672: *26322 no live upstreams while connecting to upstream, client: 24.44.64.232, server: acureach.in, request: "GET /_next/webpack-hmr HTTP/1.1", upstream: "http://localhost/_next/webpack-hmr", host: "acureach.in"
2023/03/18 01:12:22 [error] 8672#8672: *26323 no live upstreams while connecting to upstream, client: 52.129.0.102, server: acureach.in, request: "GET /_next/webpack-hmr HTTP/1.1", upstream: "http://localhost/_next/webpack-hmr", host: "acureach.in"
2023/03/18 01:12:27 [error] 8672#8672: *26328 no live upstreams while connecting to upstream, client: 24.44.64.232, server: acureach.in, request: "GET / HTTP/1.1", upstream: "http://localhost/", host: "acureach.in", referrer: "https://acureach.in/"
2023/03/18 01:12:30 [error] 8671#8671: *26353 no live upstreams while connecting to upstream, client: 168.149.146.41, server: acureach.in, request: "GET /_next/static/webpack/d31f7b7411543aac.webpack.hot-update.json HTTP/1.1", upstream: "http://localhost/_next/static/webpack/d31f7b7411543aac.webpack.hot-update.json", host: "acureach.in", referrer: "https://acureach.in/multi_search?x-typesense-api-key=xyx"
01:15
Srini
01:15 AM
do need to add upstream tag in nginx config?
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
01:15 AM
I don’t see any requests to multi_search here…

Could you first visit the site, generate a multi_search request, and then post the lines from the logs?
01:16
Jason
01:16 AM
My nginx knowledge might be outdated, but afaik, that shouldn’t be needed
Srini
Photo of md5-9cd23c7211ae4451cdff9bd79e03bbbc
Srini
01:16 AM
btw Thank you so much for helping us on a Friday evening
+11
smile1
01:17
Srini
01:17 AM
I really appreciate yoru help
+11
01:17
Srini
01:17 AM
nothing is coming in error.log when id do the search
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
01:18 AM
Is something else in front of nginx that could be catching that request and throwing a 500?
01:19
Jason
01:19 AM
Because the multi_search request I see generated by the frontend has to show up either in access log or error log
01:19
Jason
01:19 AM
if it doesn’t show up in either places, it’s not reaching this nginx server
Srini
Photo of md5-9cd23c7211ae4451cdff9bd79e03bbbc
Srini
01:20 AM
nothing is infront of nginx : the config is like this :-->

nginx - nodejs
nginx - typenses-adapter
01:20
Jason
01:20 AM
search the logs for a POST request to multi_search
Srini
Photo of md5-9cd23c7211ae4451cdff9bd79e03bbbc
Srini
01:24 AM
I don;t think it is a nginx problem, it is somewhere in config of Nextjs and typsense adapter
01:32
Srini
01:32 AM
Jason have a good night.
+11