Hey everyone, I am trying to create a dockerfile for typesense with GPU support, so far I got this:

FROM nvidia/cuda:11.8.0-cudnn8-devel-ubuntu22.04

RUN apt update
RUN apt install -y curl

RUN curl -O <https://dl.typesense.org/releases/26.0/typesense-server-26.0-linux-amd64.tar.gz>
RUN tar -xzf typesense-server-26.0-linux-amd64.tar.gz

RUN curl -O <https://dl.typesense.org/releases/26.0/typesense-gpu-deps-26.0-amd64.deb>
RUN apt install ./typesense-gpu-deps-26.0-amd64.deb

ENTRYPOINT ["./typesense-server"]

but when I run

nvidia-smi

no processes are shown, and the GPU usage is not changed... I do get a message:

2024-04-22 17:38:37.365215677 [W:onnxruntime:, <http://session_state.cc:1136|session_state.cc:1136> VerifyEachNodeIsAssignedToAnEp] Some nodes were not assigned to the preferred execution providers which may or may not have an negative impact on performance. e.g. ORT explicitly assigns shape related ops to CPU to improve perf.
2024-04-22 17:38:37.365235639 [W:onnxruntime:, <http://session_state.cc:1138|session_state.cc:1138> VerifyEachNodeIsAssignedToAnEp] Rerunning with verbose output on a non-minimal build will show node assignments.

Could you try creating a collection with an auto-embedding field with a built-in model, and then index one document in the collection? And then check nvidia-smi as you're doing this?

Yeah, I actually tried indexing with e5-small-v2 but no processes showed up in nvidia-smi, I think I have access to GPU since nvidia-smi inside the pod outputs the driver and the hardware

Interesting, good to know