#community-help

Typesense Certificate Error in Docker Compose

TLDR Suraj encountered an issue with typesense not finding SSL certificates in docker compose, even though certs were mounted successfully. After several suggestions by Jason, the issue was resolved by Suraj by mounting the correct folder in the docker compose file.

Powered by Struct AI
+11
pray1
smile1
55
13mo
Solved
Join the chat
Aug 22, 2022 (13 months ago)
Suraj
Photo of md5-a7605b093448e06efb0a1c150153cad6
Suraj
03:34 PM
Hey guys, I'm using typesense in docker compose with cerbot for ssl, the certs get mounted in a shared volume successfully but typesense throws an "error while trying to load server certificate file: path/full chain.pem" and failed to listen on 0.0.0.0:443 - No such file or directory. Any guidance?
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
05:09 PM
Could you share the full docker command you’re using?
Suraj
Photo of md5-a7605b093448e06efb0a1c150153cad6
Suraj
05:12 PM
Thanks for the quick response!

version: "3.9"
services:
  typesense:
    image: typesense/typesense:0.23.1
    environment:
      - API_KEY=${API_KEY}
    entrypoint: sh -c "/opt/typesense-server --data-dir=/data --api-key=${API_KEY} --enable-cors --api-port=443 --ssl-certificate=/etc/letsencrypt/live/test.domain.io/fullchain.pem --ssl-certificate-key=/etc/letsencrypt/live/test.domain.io/privkey.pem"
    ports:
      - "443:443"
      - "8108:8108"
    restart: unless-stopped
    volumes:
      - typesense-data:/data
      - certbot-etc:/etc/letsencrypt/live
    depends_on:
      - certbot
  certbot:
    image: certbot/certbot
    container_name: certbot
    ports:
      - "80:80"
    volumes:
      - certbot-etc:/etc/letsencrypt/live
    command: certonly --standalone --agree-tos --redirect --email   -d 

volumes:
  certbot-etc:
  typesense-data:
    driver: local
    driver_opts:
      type: none
      device: ./typesense-data/
      o: bind
05:14
Suraj
05:14 PM
That's my docker-compose file with placeholder domain and email, when testing with real information I have the certs mounted in the correct directory but typesense server is unable to find them even though they are in the folder
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
05:14 PM
I wonder if there’s a delay between when the certbot container starts and the actual SSL cert is available in /etc/letsencrypt/live for then Typesense to pickup when starting?
05:15
Jason
05:15 PM
As soon as the Typesense container is up, could you exec bash into it and check if the certs are there?
Suraj
Photo of md5-a7605b093448e06efb0a1c150153cad6
Suraj
05:15 PM
I tried stopping the container and restarting it to see if it finds it but no luck
05:15
Suraj
05:15 PM
They are there when using docker exec ls
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
05:16 PM
It’s interesting that the error says “error while trying to load server certificate file: path/full chain.pem
05:16
Jason
05:16 PM
Wonder where it’s getting “path/full chain.pem” from
Suraj
Photo of md5-a7605b093448e06efb0a1c150153cad6
Suraj
05:16 PM
I shortened it there as an example it actually says the path specified in the run command "/etc/letsencrypt/live/test.domain.io/fullchain.pem"
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
05:16 PM
Ah ok
05:18
Jason
05:18 PM
Suraj
Photo of md5-a7605b093448e06efb0a1c150153cad6
Suraj
05:19 PM
Yup
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
05:19 PM
Could you exec bash into the Typesense container, and then run the entrypoint typesense-server command from that prompt and see if it works?
Suraj
Photo of md5-a7605b093448e06efb0a1c150153cad6
Suraj
05:21 PM
The server is constantly resetting so can't really bash into it, could only use ls during start of container as there are multiple attempts made to start the server
05:23
Suraj
05:23 PM
and I am also able to docker cp the certs out to make sure they are there
06:08
Suraj
06:08 PM
still getting this after stopping container, overriding the entrypoint to /bin/bash and checking if the certs are there and manually starting the server but still get the same error
E20220822 18:05:29.061584    24 http_server.cpp:1057] An error occurred while trying to load server certificate file: /etc/letsencrypt/live/tests.surajpatel.org/fullchain.pem
E20220822 18:05:29.061868    24 http_server.cpp:174] Failed to listen on 0.0.0.0:443 - No such file or directory
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
06:09 PM
Could you try starting a Typesense server using one of the native binaries without Docker, with those certs?
Suraj
Photo of md5-a7605b093448e06efb0a1c150153cad6
Suraj
06:10 PM
Sure will let you know how it goes
06:41
Suraj
06:41 PM
works if I do it without docker
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
06:43 PM
Hmmm, I wonder if it is some permissions issue then may be?
Suraj
Photo of md5-a7605b093448e06efb0a1c150153cad6
Suraj
06:49 PM
Any idea what I could try?
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
06:50 PM
Could you try cating the contents of the SSL cert / key files from inside the Typesense container?
Suraj
Photo of md5-a7605b093448e06efb0a1c150153cad6
Suraj
06:51 PM
The SSL cert and Key are there, I can copy them out of the container too
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
06:52 PM
Let’s try this:

How about changing entry point to something like this:

entrypoint: sh -c "cat /etc/letsencrypt/live/test.domain.io/fullchain.pem && /opt/typesense-server --data-dir=/data --api-key=${API_KEY} --enable-cors --api-port=443 --ssl-certificate=/etc/letsencrypt/live/test.domain.io/fullchain.pem --ssl-certificate-key=/etc/letsencrypt/live/test.domain.io/privkey.pem"
06:53
Jason
06:53 PM
And then run docker logs to see if you see the output of the cat command
Suraj
Photo of md5-a7605b093448e06efb0a1c150153cad6
Suraj
06:53 PM
Thanks trying it now
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
06:54 PM
I’m essentially wondering if the permissions on those files somehow prevent them from being read by the Typesense process
Suraj
Photo of md5-a7605b093448e06efb0a1c150153cad6
Suraj
06:55 PM
Yeah I was about to send you them using should work
docker-compose exec typesense ls -la /etc/letsencrypt/live
07:16
Suraj
07:16 PM
returns
root@test-typesense:~# docker-compose exec typesense ls -la /etc/letsencrypt/live
total 16
drwxr-xr-x 3 root root 4096 Aug 22 19:15 .
drwxr-xr-x 3 root root 4096 Aug 22 19:16 ..
-rw-r--r-- 1 root root  740 Aug 22 19:15 README
drwxr-xr-x 2 root root 4096 Aug 22 19:15 
07:22
Suraj
07:22 PM
however with your entrypoint I get
Attaching to root_typesense_1
typesense_1  | cat: /etc/letsencrypt/live/testing.surajpatel.org/fullchain.pem: No such file or directory
07:22
Suraj
07:22 PM
running docker-compose logs typesense
07:26
Suraj
07:26 PM
but if i /bin/bash in the files are there
07:28
Suraj
07:28 PM
also tried manually starting it from inside the container and pointing to the certs but getting the same error
07:29
Suraj
07:29 PM
E20220822 19:28:24.086433    28 http_server.cpp:1057] An error occurred while trying to load server certificate file: /etc/letsencrypt/live/testing.surajpatel.org/fullchain.pem
E20220822 19:28:24.086486    28 http_server.cpp:174] Failed to listen on 0.0.0.0:443 - No such file or directory

Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
08:10 PM
> https://typesense-community.slack.com/archives/C01P749MET0/p1661196130203849?thread_ts=1661182497.082429&cid=C01P749MET0
Ok, then the issue is definitely timing related.

I think by the time you run docker-compose exec typesense ls -la the SSL certs have already been generated.
08:10
Jason
08:10 PM
To quickly test this theory out, could you try changing the entrypoint to:

entrypoint: sh -c "sleep 120 && cat /etc/letsencrypt/live/test.domain.io/fullchain.pem && /opt/typesense-server --data-dir=/data --api-key=${API_KEY} --enable-cors --api-port=443 --ssl-certificate=/etc/letsencrypt/live/test.domain.io/fullchain.pem --ssl-certificate-key=/etc/letsencrypt/live/test.domain.io/privkey.pem"
08:11
Jason
08:11 PM
We’re essentially trying to see if running cat a little while later returns the actual cert
Suraj
Photo of md5-a7605b093448e06efb0a1c150153cad6
Suraj
08:21 PM
Makes sense, but it didn't work haha
08:22
Suraj
08:22 PM
still getting
typesense_1  | cat: /etc/letsencrypt/live/testing.surajpatel.org/fullchain.pem: No such file or directory
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
08:24 PM
Ok, looks like depends_on has been deprecated by docker compose.

Could you try the health-check approach mentioned here: https://stackoverflow.com/a/41854997/123545

In your case the health check in the certbot container would be the presence of the SSL cert and key file.
Suraj
Photo of md5-a7605b093448e06efb0a1c150153cad6
Suraj
08:32 PM
Tried this also tried it with docker compose 2+ with
    depends_on:
      certbot:
        condition: service_completed_successfully

08:32
Suraj
08:32 PM
same in both cases, cerbot runs fine and gets the cert/key but typesense server can't find them
08:32
Suraj
08:32 PM
I'm pretty new to docker so maybe i'm just an idiot lol
08:33
Suraj
08:33 PM
Thank you for your help so far!
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
09:14 PM
Did you try with the health-checks? I think that’s the one that waits for those files to be written
Suraj
Photo of md5-a7605b093448e06efb0a1c150153cad6
Suraj
09:45 PM
I added the following to certbot in docker-compose.yaml
09:45
Suraj
09:45 PM
    healthcheck:
      test: ["CMD-SHELL", "test -f /etc/letsencrypt/live/testing.surajpatel.org/fullchain.pem"]
      interval: 1m30s
      timeout: 1m
      retries: 5
      start_period: 30s
09:45
Suraj
09:45 PM
but I can't seem to get the test to return as healthy although the certbot setup suceeds
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
10:08 PM
Hmmm, I’m afraid I’m at the end of my Docker knowledge here…
10:08
Jason
10:08 PM
Since the cat command fails when starting up the typesense container, this doesn’t look like a Typesense specific issue. So I’d recommend posting on stackoverflow to see if a Docker expert can help
Suraj
Photo of md5-a7605b093448e06efb0a1c150153cad6
Suraj
10:10 PM
No worries dude, thank you for you time and help! If I figure it out i'll add to the docs and if not I'll go without docker haha
+11
pray1
smile1
Aug 25, 2022 (13 months ago)
Suraj
Photo of md5-a7605b093448e06efb0a1c150153cad6
Suraj
07:37 AM
Figured it out and running in production! Should mount letsencrypt folder not the domain folder within, can post on guides if you think it will useful
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
02:42 PM
Awesome! Yeah if you’re able to put together something like this, but for Docker Compose, that would be great: https://typesense.org/docs/guide/docker-swarm-high-availability.html