Using Typesense for In-house Log Search
TLDR Gokay asked if using Typesense for a log search project is feasible. Jason replied it may become expensive due to being in-memory and missing some features but it's possible in some contexts.
Oct 14, 2022 (14 months ago)
I have a question for you. I'm trying to setup a log search for my side project. ELK stack is too heavy and overkill for my project and pain in the ass to setup. I'm considering an inhouse solution and I've found typesense. Do you think it'll work for my case? I'll store access logs.
Also, Typesense was designed more searching through structured data for sites / app search, so you might find some aggregation features that you’d typically find in ES missing in Typesense.
That said, I do know a few users who use Typesense for log search - but in a specialized context. Their users generate logs in their application, and they allow a portion of those logs (recent ones) to be searched for / filtered by their end users. So it’s still site/app search, but the underlying data their searching is user-generated logs (think audit events, app events, etc)
Indexed 3015 threads (79% resolved)
Discussing Typesense Use for Project
Fulvio suggested starting a Kibana-like project with Typesense. Kishore Nallan explained that Typesense isn't usually used for logs due to the large volume.
Timeseries Data and Typesense for Large Datasets
Josh asks if Typesense is suitable for their use case with timeseries data, high qps, and 40TB+ datasets. Jason suggests considering Elasticsearch for such scale.
Managing Memory and Storage in Typesense
Joe inquired about storing recent data logs in RAM and the rest in storage. Jason confirmed it's possible but would need to be manually managed.