Server IP Address and Port Issues with Typesense

TLDR Alex encountered problems with Typesense server configuration when setting IP and port using --api-address and --api-port, facing additional problems with CORS and version inequities. Kishore Nallan addressed each point, ultimately finding a bug with the command line parser affecting subsequent commands. The advised workaround was to use `--enable-cors=true`.

Photo of Alex
Alex
Fri, 08 Apr 2022 09:39:05 UTC

IP weirdness with TS server: when I start typesense specifying an IP / PORT like so: --api-address 159.203.31.163 \ --api-port=80 but then I get this: typesense TCP 10.20.0.5:8107 (LISTEN) typesense TCP 159.203.31.163:80 (LISTEN) and in the TS log I can see the 10.x IP being used, while there is nothing listening on that IP on port 80: CURL failed. URL: , Code: 7, strerror: Couldn't connect to server

Photo of Kishore Nallan
Kishore Nallan
Fri, 08 Apr 2022 09:41:23 UTC

There are two address configs. One for api http service which is configured via --api-address and the other is for the raft port that listens on 8107 -- this raft listener uses an internal interface only by default.

Photo of Alex
Alex
Fri, 08 Apr 2022 09:42:05 UTC

ok, but why does TS try to connect to port 80 for a health check on that internal IP?

Photo of Kishore Nallan
Kishore Nallan
Fri, 08 Apr 2022 09:42:31 UTC

It shouldn't.

Photo of Kishore Nallan
Kishore Nallan
Fri, 08 Apr 2022 09:42:56 UTC

Check the logs to see if there are any errors with api port binding.

Photo of Kishore Nallan
Kishore Nallan
Fri, 08 Apr 2022 09:43:34 UTC

If you have SSL configured you need to use https while curling.

Photo of Alex
Alex
Fri, 08 Apr 2022 09:44:35 UTC

This is not something I'm doing, I'm assuming TS does some selfcheck on the wrong IP/PORT: CURL failed. URL: , Code: 7, strerror: Couldn't connect to server

Photo of Kishore Nallan
Kishore Nallan
Fri, 08 Apr 2022 09:45:08 UTC

Oh you see that in the logs?

Photo of Alex
Alex
Fri, 08 Apr 2022 09:45:56 UTC

yup it's in the nohup error log

Photo of Kishore Nallan
Kishore Nallan
Fri, 08 Apr 2022 09:46:22 UTC

Are you using clustering?

Photo of Alex
Alex
Fri, 08 Apr 2022 09:46:30 UTC

not in this case no.

Photo of Alex
Alex
Fri, 08 Apr 2022 09:47:00 UTC

nohup ./typesense-server --data-dir=/home/alex/tsdata --api-key=$TYPESENSE_API_KEY --enable-cors \ --api-address 159.203.31.163 \ --api-port=80 \ --log-dir=/home/alex/tsdata/log

Photo of Kishore Nallan
Kishore Nallan
Fri, 08 Apr 2022 09:47:23 UTC

Can you add a equals after --api-address flag?

Photo of Alex
Alex
Fri, 08 Apr 2022 09:47:32 UTC

sure

Photo of Alex
Alex
Fri, 08 Apr 2022 11:29:39 UTC

So when I started version 23.rc47 it seemed to ignore the api-address line altogether and bind to * . I didn't see any of the healthcheck errors in the log, but also binding to * might explain that.

Photo of Kishore Nallan
Kishore Nallan
Fri, 08 Apr 2022 11:35:41 UTC

Oh okay, we did make one recent change related to this part. I will take a look.

Photo of Alex
Alex
Fri, 08 Apr 2022 11:37:06 UTC

my JS client had a issues connecting to rc47 too

Photo of Kishore Nallan
Kishore Nallan
Fri, 08 Apr 2022 11:38:01 UTC

We've other customers using rc47 though. Maybe they don't use custom API address.

Photo of Kishore Nallan
Kishore Nallan
Fri, 08 Apr 2022 11:38:50 UTC

Do you have multiple public IPs? Otherwise binding to * is generally fine since you do intend to expose it to internet.

Photo of Alex
Alex
Fri, 08 Apr 2022 11:58:26 UTC

So when I remove the --api-address line then 23.RC47 also ignores --api-port=80 and binds to: *:8108

Photo of Alex
Alex
Fri, 08 Apr 2022 12:07:55 UTC

ApiCall.ts:203 Request #1649419591544: Request to Node 0 failed due to "undefined Network Error" performRequest @ ApiCall.ts:203 ApiCall.ts:205 Request #1649419591544: Sleeping for 0.1s and then retrying request... performRequest @ ApiCall.ts:205 localhost/multi_search?x-typesense-api-key=xyz:1 Failed to load resource: net::ERR_CONNECTION_REFUSED ApiCall.ts:203 Request #1649419591544: Request to Node 0 failed due to "undefined Network Error" performRequest @ ApiCall.ts:203 ApiCall.ts:205 Request #1649419591544: Sleeping for 0.1s and then retrying request... performRequest @ ApiCall.ts:205 localhost/multi_search?x-typesense-api-key=xyz:1 Failed to load resource: net::ERR_CONNECTION_REFUSED

Photo of Alex
Alex
Fri, 08 Apr 2022 12:08:31 UTC

but curl connects just fine

Photo of Kishore Nallan
Kishore Nallan
Fri, 08 Apr 2022 12:25:56 UTC

Ok will look at this and get back to you.

Photo of Kishore Nallan
Kishore Nallan
Sat, 09 Apr 2022 01:29:40 UTC

We're indeed using the internal IP for an internal health call which happens after a snapshot. But for single node setup, this is not critical and should not affect the server ops. If you just remove the api-address config, Typesense will listen on all interfaces or can also explicitly use: ```--api-address=0.0.0.0```

Photo of Kishore Nallan
Kishore Nallan
Sat, 09 Apr 2022 02:08:59 UTC

This is how Typesense has always been, so maybe you've never set this config directly before so didn't notice this problem? Despite that error, single node operations won't be anwyay affected.

Photo of Alex
Alex
Sat, 09 Apr 2022 05:42:41 UTC

Well they are affected: My JS code which comes from your Ecommerce example is no longer able to connect to the single 23.RC47 node. And If I do remove the --api-address line from startup, --api-port=80 is ignored. so 23.RC47 is unable to have a port set unless an --api-address line is in the config

Photo of Kishore Nallan
Kishore Nallan
Sat, 09 Apr 2022 05:46:33 UTC

I'm just trying this locally and removing api address has no impact on port. So very strange... Let me try again on a Linux box. Was trying on local mac.

Photo of Alex
Alex
Sat, 09 Apr 2022 06:52:20 UTC

I'm thinking it also might be ignoring the --enable-cors switch as the error I get with RC47 is: Access to XMLHttpRequest at '' from origin '' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource.

Photo of Alex
Alex
Sat, 09 Apr 2022 06:53:02 UTC

Same code works fine with 22.2

Photo of Alex
Alex
Sat, 09 Apr 2022 06:53:43 UTC

nohup ./typesense-server --data-dir=/home/alex/tsdata --api-key=$TYPESENSE_API_KEY --enable-cors \ --api-address=0.0.0.0 \ --api-port=80 \ --log-dir=/home/alex/tsdata/log

Photo of Kishore Nallan
Kishore Nallan
Sat, 09 Apr 2022 07:16:04 UTC

Oh right it might be a cors issue.

Photo of Alex
Alex
Sat, 09 Apr 2022 07:18:42 UTC

I could try with a different 23.RC maybe?

Photo of Alex
Alex
Sat, 09 Apr 2022 11:04:06 UTC

Alternatively is there a way to check if cors is enabled? like some sort of display current config command that can be issued to TS?

Photo of Kishore Nallan
Kishore Nallan
Sat, 09 Apr 2022 12:10:59 UTC

The response header should have access-control-allow-origin as * I've been AFK so haven't been able to investigate yet, sorry. Will keep you posted.

Photo of Alex
Alex
Sat, 09 Apr 2022 14:46:52 UTC

no rush :wink:

Photo of Kishore Nallan
Kishore Nallan
Sun, 10 Apr 2022 03:08:49 UTC

Turned out to be an annoying bug with the command line parser. People were trying to enable cors by send a true value, like `--enable-cors=true` which the argument parsing library that Typesense uses does not support. In trying to fix that, ended up introducing a bug where any flag sent after the enable-cors option becomes interpreted as a value of the enable-cors flag, so when your command is: `--enable-cors --api-address=0.0.0.0` it set the enable-cors value as "`--api-address=0.0.0.0`" :face_palm: Until I fix this parsing issue in a future build, please use `--enable-cors=true` so it's explicit.