Hi! I am on Typesense Cloud v30.rca34 I was hoping...
# community-help
g
Hi! I am on Typesense Cloud v30.rca34 I was hoping it would allow me to define a Natural Language Search model that is a local llama.cpp -openai format I am able to define a conversational model (RAG) that way using the
openai_url
property; it doesn't seem to be available for NLS? I was hoping to use something like this:
Copy code
{
  "id": "NL-llama.cpp-ministral-3-14B",
  "model_name": "openai/Ministral-3-14B-Instruct-2512-Q4_K_M.gguf",
  "api_key": "NOT_NEEDED",
  "openai_url": "http://xxxxxxxx:5000",
  "max_bytes": 16000,
  "temperature": 0,
  "system_prompt": "Be precise and accurate in parsing queries"
}
Any other options? Thank you!
a
Hey Gauthier, Could you try some things? 1. Use api_url instead of openai_url 2. Use vllm namespace instead of openai ("model_name": "<namespace>/Ministral-3-14B-Instruct-2512-Q4_K_M.gguf") You'll want to try 1. with both versions of 2. (openai and vllm).
g
Thank you. Results: • using
api_url
and no other changes:
Request failed with HTTP code 400 | Server said: OpenAI API error: File Not Found
• using
api_url
and `vllm namspace`:
Request failed with HTTP code 400 | Server said: vLLM API error: File Not Found
Interestingly, if I don't include
api_url
or
openai_url
then the error is
Incorrect API key provided
so it seems that including a URL does "something"
I think I've figured it out:
Copy code
{
  "id": "NL-llama.cpp-ministral-3-14B",
  "model_name": "openai/Ministral-3-14B-Instruct-2512-Q4_K_M.gguf",
  "api_key": "NOT_NEEDED",
  "api_url": "http://xxxxx:5000/v1/chat/completions",
  "max_bytes": 16000,
  "temperature": 0,
  "system_prompt": "Be precise and accurate in parsing queries"
}
/v1/chat/completions
was missing. Now the model is created successfully. I'll check if it works
111 Views