https://typesense.org logo
Join Slack
Powered by
Is there a way to check if a GPU on a Typesense se...
# community-help
j
Is there a way to check if a GPU on a Typesense server is installed correctly based on steps here (https://typesense.org/docs/guide/install-typesense.html#using-a-gpu-optional), and if the GPU is currently being used?
e
If you have direct control of the terminal on the server, 
nvidia-smi
might tell you something interesting. namely the amount of vram being used and which process is using
➕ 1
j
I do have access to nvidia-smi - during real time embedding, no processes are listed here, so I want to see on the Typesense side if the GPU is being used
e
that's what I mean. If nothing is consuming VRAM, then Typesense isn't using it
j
Gotcha. How can I troubleshoot this to determine why Typesense isn’t using the GPU?
e
Does the Typesense Dashboard tell you anything?
j
I am self-hosting so no Typesense Dashboard, just the server Typesense is hosted on
e
https://bfritscher.github.io/typesense-dashboard You can save it offline and connect to your local instance. Super handy tool
🙌 1
I don't have a GPU instance with Typesense at the moment to help look, but it might reveal something to you
f
Worst case scenario something like 
btop
can show real time GPU usage. On my laptop for example, with an RTX ADA 2000, normal use typically has no usage for the dedicated GPU, so a bump in usage would indicate that it's being used
o
@John Sokol could you check your logs while loading the model to memory. If your server do not use the GPU, you will see this log: > ONNX shared libs: off Before: > Loading model from disk: PATH If you can't see that log about the shared libs, that means Typesense were able to load model to your GPU successfully.
j
@Ozan Armağan How do I access the logs to check this?
o
@John Sokol You mentioned that you are self-hosting, are there any chance to connect and inspect logs on the terminal that runs Typesense on the server you are running?
j
Gotcha, I accessed the logs from the directory specified in
Copy code
/etc/typesense/typesense-server.ini
I found ONNX shared libs: off in the log file
Copy code
I20250313 02:12:26.131770  4040 text_embedder.cpp:20] ONNX shared libs: off
I20250313 02:12:26.134429  4040 text_embedder.cpp:33] Loading model from disk: olddrive/var/lib/typesense/models/ts_nomic-embed-text-v1.5/model.on>
I20250313 02:12:27.609331  4040 embedder_manager.cpp:20] Finished initializing local model: ts/nomic-embed-text-v1.5
I20250313 02:12:27.609414  4040 collection_manager.cpp:135] Model init done.
I’ll restart the Typesense service to see if it uses the GPU
o
Before restarting, can we make sure of these points?: • You have CUDA 11.8 on the server • Downloaded our GPU dependency packages • Dynamic libraries inside the package are reachable with 
LD_LIBRARY_PATH
j
Sure thing:
Copy code
1. root@admin:/olddrive/var/log/typesense# ls /usr/local/cuda-11.8/bin/nvcc
/usr/local/cuda-11.8/bin/nvcc
Copy code
2. root@admin:/olddrive/var/log/typesense# dpkg -l | grep -E 'typesense-gpu-deps'
ii  typesense-gpu-deps                   28.0                                    amd64        Typesense GPU Dependencies
Copy code
3. root@admin:/olddrive/var/log/typesense# echo $LD_LIBRARY_PATH
/usr/local/cuda-11.8/lib64:
o
Ok this looks good to go, could you try again now?
j
Sure, do I need to add the environment variables PATH, LD_LIBRARY_PATH and CUDA_HOME to /etc/profile.d/cuda-path.sh as specified in the docs? Currently this .sh file is empty, and:
Copy code
1. echo $PATH
/usr/local/cuda-11.8/bin:
Copy code
2. echo $LD_LIBRARY_PATH
/usr/local/cuda-11.8/lib64:
Copy code
3. echo $CUDA_HOME
/usr/local/cuda-11.8
I declared these env variables in ~/.bashrc
o
Bashrc also should be fine since it is executed for every terminal
👍 1
j
Cool, will restart
It’s been 3 hours since restart - all collections except for the one with the vectorizer have loaded. Going to restart again
It appears all the collections except for the one with the Nomic embeddings model are loading. I just see “Running GC for aborted requests” in the Typesense log when Typesense should be loading that collection with the embeddings
Found the error: Failed to load library libonnxruntime_providers_cuda.so with error: libcublasLt.so.11: cannot open shared object file: No such file or directory. FYI, this error didn’t come up in the current Typesense instance logs upon reloading, I had to redirect the log and database libraries in the init file, and attempt to create a new collection for this error to appear.
Then I ran /usr/local/cuda/lib64 to your system’s linker config:
Copy code
echo "/usr/local/cuda/lib64" | sudo tee /etc/ld.so.conf.d/cuda.conf
sudo ldconfig
This solved the issue
Open in Slack
PreviousNext
Powered by