Hey folks, I've been doing a deep dive on our AWS ...
# community-help
j
Hey folks, I've been doing a deep dive on our AWS bill and I'm finding fairly high data transfer cost for copying data over to the Typesense API which I believe could likely be almost completely eliminated if we had a proper private link endpoint. Is this something that's on the Typesense cloud roadmap at all? Does anyone have any sneaky tricks for working around this problem outside of self hosting?
➕ 1
j
We briefly explored private link / vpc endpoints, but it required extensive changes to our architecture so we decided to not pursue it.
✅ 1
For context, may I know how much data you're currently transferring into Typesense per month?
j
We basically have crons that refresh everything periodically at the moment. Some of these can be pretty easily optimized but our largest collection now has a ranking system that needs to periodically decay it's relevance score based on various criteria so needs to be regularly refreshed from our Clickhouse data.
I can't get an accurate representation of the total quantity of data for a month since our VPC flow log has only been switched on ~1 day but since this data all needs to flow out our NAT gateway we also get extra cost there
j
Ah ok. One thing I'd recommend testing out is to enable gzip compression on both the request and responses when making the
import
API calls to Typesense
This is disabled by default, but you can enable it using the standard http headers for content encoding and accept encoding
✅ 1
j
Thanks @Jason Bosco - i'll have a play with that to see how much a difference it makes!
j
Key thing to keep an eye out for is CPU usage on both the Typesense cluster side and also on your own app servers
Since gzipping on the fly is going to take additional CPU cycles
✅ 1