has anyone build tracing on top of typenses which ...
# community-help
p
has anyone build tracing on top of typenses which uses openai - We are seeing a huge burn on our tokens and trying to understand where the tokens are getting burnt.
f
Hey Praneeth, What ML features are you using Typesense with?
p
Hi Fanis.. we are started with a simple NL Search. - We are using the instant js search, right now we are burning too many tokens and decided last night to totally remove the natural langugage feature and rely only on filters. We are hosting our data in supbase and have a function to sync data as upsert (to handle duplicates). We only new this when we are looking at the openai bill as don't have a way to set tracing. All our rest of app have otel or langfuse embeded to trace anywhere LLM tokens are used.
Also realized there is no way to change the base TS prompt as it added the whole collection to the chatcompletion prompt. we assumed that will be embedding but seems like we either did it wrong or have the wrong understanding of TS
f
Hey Praneeth, Natural Language Search relies on chat completions, as there aren't any embeddings involved in the search. The prompt itself is cached in memory for 24h, though it is being sent on every single search request
p
thanks Fanis. Is there a way to setup tracing for this? We currently using Langfuse and Azure Insights(OTel)
f
I don't have any experience there myself, but prompt caching can be set up on the LLM side as well, as some newer GPT models support it. Some documentation: https://platform.openai.com/docs/guides/prompt-caching