Using Typesense for In-house Log Search
TLDR Gokay asked if using Typesense for a log search project is feasible. Jason replied it may become expensive due to being in-memory and missing some features but it's possible in some contexts.
Powered by Struct AI
2
14mo
Oct 14, 2022 (14 months ago)
Gokay
Gokay
06:13 PMGreat product!
I have a question for you. I'm trying to setup a log search for my side project. ELK stack is too heavy and overkill for my project and pain in the ass to setup. I'm considering an inhouse solution and I've found typesense. Do you think it'll work for my case? I'll store access logs.
I have a question for you. I'm trying to setup a log search for my side project. ELK stack is too heavy and overkill for my project and pain in the ass to setup. I'm considering an inhouse solution and I've found typesense. Do you think it'll work for my case? I'll store access logs.
Jason
Jason
06:15 PMTypesense is an in-memory search engine, so depending on the volume of your logs, it might become cost prohibitive to store all logs in RAM, since RAM is still more expensive than disk.
Also, Typesense was designed more searching through structured data for sites / app search, so you might find some aggregation features that you’d typically find in ES missing in Typesense.
That said, I do know a few users who use Typesense for log search - but in a specialized context. Their users generate logs in their application, and they allow a portion of those logs (recent ones) to be searched for / filtered by their end users. So it’s still site/app search, but the underlying data their searching is user-generated logs (think audit events, app events, etc)
Also, Typesense was designed more searching through structured data for sites / app search, so you might find some aggregation features that you’d typically find in ES missing in Typesense.
That said, I do know a few users who use Typesense for log search - but in a specialized context. Their users generate logs in their application, and they allow a portion of those logs (recent ones) to be searched for / filtered by their end users. So it’s still site/app search, but the underlying data their searching is user-generated logs (think audit events, app events, etc)
Typesense
Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI
Indexed 3015 threads (79% resolved)
Similar Threads
Discussing Typesense Use for Project
Fulvio suggested starting a Kibana-like project with Typesense. Kishore Nallan explained that Typesense isn't usually used for logs due to the large volume.
3
27mo
Timeseries Data and Typesense for Large Datasets
Josh asks if Typesense is suitable for their use case with timeseries data, high qps, and 40TB+ datasets. Jason suggests considering Elasticsearch for such scale.
27
10mo
Managing Memory and Storage in Typesense
Joe inquired about storing recent data logs in RAM and the rest in storage. Jason confirmed it's possible but would need to be manually managed.
2
34mo