#community-help

Addressing Cluster Issue due to Excessive Data

TLDR Andrew had trouble with cluster operations due to excessive data and collections. Jason advised flushing the data and stated that the upcoming update will remedy such issues. Both agreed to stick to v0.19 and not to fill the cluster excessively.

Powered by Struct AI

2

2

1

40
33mo
Solved
Join the chat
Mar 11, 2021 (33 months ago)
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
06:42 PM
I was just looking at the cluster issue you emailed about

1

Andrew
Photo of md5-08f6fb4c00b4a074647988ce90a07f5c
Andrew
06:43 PM
Writes are working, reads are not ๐Ÿ˜ฑ
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
06:43 PM
So strange
Andrew
Photo of md5-08f6fb4c00b4a074647988ce90a07f5c
Andrew
06:44 PM
Feeling like something is up, because at the beginning, reads and writes were not working, but writes are returning successfully.

Seems I canโ€™t create/destroy indexes.

The worst part is I canโ€™t log in to the cloud portal, because Iโ€™m locked out of my github account from missing 2FA tokens ๐Ÿคฏ (thatโ€™ll get fixed in a couple days haha)
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
06:47 PM
Found the issue, it looks like the OS ended up killing the Typesense process due to resource constraints, and after several successive restarts and crashes it stopped trying to restart the process back up
06:48
Jason
06:48 PM
I'm guessing you might have been able to write in the brief moments when the process had started back up, before it crashed again due to resource constraints
06:48
Jason
06:48 PM
May I know what size of a dataset and number of collections you were trying to index?
Andrew
Photo of md5-08f6fb4c00b4a074647988ce90a07f5c
Andrew
06:48 PM
Any ETA on how long it will take to resolve? Did I just put too much data in this small cluster?
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
06:49 PM
> Did I just put too much data in this small cluster?
Yeah, that's likely the issue
Andrew
Photo of md5-08f6fb4c00b4a074647988ce90a07f5c
Andrew
06:49 PM
I created a lot of collections, each with not very much data
06:49
Andrew
06:49 PM
I was just using this to test the new serivce I was spinning up (constantly destroying/recreating data, but just in postgres, not cleaning up the cluster)
06:49
Andrew
06:49 PM
You can flush all the data in the cluster
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
06:50 PM
That's definitely the issue. Till v0.19 we create 4 threads per collection, so with sufficient collections, it will max out the OS threads limits on the cluster you have
06:51
Jason
06:51 PM
In v0.20 we're switching to a thread pool model, which will allow for more efficient (re)use of threads even on small clusters
06:51
Jason
06:51 PM
> You can flush all the data in the cluster
Ok cool, this will resolve the issue right away
Andrew
Photo of md5-08f6fb4c00b4a074647988ce90a07f5c
Andrew
06:51 PM
:thumbsup: , are you willing to flush all of the data out of the cluster? I shouldnโ€™t be creating a ton of collections in the future
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
06:52 PM
Yup, I can flush the data

1

Andrew
Photo of md5-08f6fb4c00b4a074647988ce90a07f5c
Andrew
06:52 PM
(Iโ€™m actually shifting focus to ecommerce, so really a lot of documents in one products collection
06:52
Andrew
06:52 PM
Thanks dude!

1

Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
06:52 PM
Let me know if you'd also like me to upgrade your cluster to v0.20. So if you do need to create a ton of collections, the upper limit of the number of collections you can create will at least be increased on the same cluster size, because of the shared thread pool
06:53
Andrew
06:53 PM
V0.20 is out?
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
06:53 PM
No, I have a nightly RC build
06:54
Jason
06:54 PM
We're still working on a few features before we can release it publicly
Andrew
Photo of md5-08f6fb4c00b4a074647988ce90a07f5c
Andrew
06:54 PM
yeah letโ€™s stick with 0.19, def was just wiping my db, and creating a new collection in typesense every 20 minutes or so over the past couple of days while building, that wonโ€™t happen again
06:54
Andrew
06:54 PM
Nightly builds scare me
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
06:55 PM
Ha ha, ok!
06:55
Jason
06:55 PM
Alright, your cluster should be back up
Andrew
Photo of md5-08f6fb4c00b4a074647988ce90a07f5c
Andrew
06:56 PM
Typesense::Error::RequestUnauthorized (Forbidden - a valid x-typesense-api-key header must be sent.)
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
06:57 PM
You'd need to regenerate a new API key
Andrew
Photo of md5-08f6fb4c00b4a074647988ce90a07f5c
Andrew
06:57 PM
hmm
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
06:57 PM
Old API keys would have also been wiped out with the data wipe, sorry if that wasn't clear
06:57
Jason
06:57 PM
Oh shoot, you said you don't have access to the cloud console because of Github login ๐Ÿ˜ž
Andrew
Photo of md5-08f6fb4c00b4a074647988ce90a07f5c
Andrew
06:58 PM
Would you be willing to shoot me a regular one and a search one in a private message ๐Ÿ˜ฌ ?
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
07:00 PM
Alright, only because you asked! Could you send me your GPG public key?

For the record, (for others who might be reading this in the future), our standard policy is to not access customer clusters and data on Typesense Cloud clusters, unless specifically asked by customers like above.
07:01
Jason
07:01 PM
If you're on a mac, I'd recommend this GUI-based GPG tool: https://gpgtools.org/
Andrew
Photo of md5-08f6fb4c00b4a074647988ce90a07f5c
Andrew
07:06 PM
-----BEGIN PGP PUBLIC KEY BLOCK-----

mQGNBGBKacQBDADSGCqUaKPL4IKtCzV/cUN+DL7cT1z8DszIYYAQsproKzBgqJnH
c/6MOqrqBg2u+fXNaJ9YKSNyGZb5NPpZuMv3vRkkjyjdZHmNcF67HeKBlXgvFmzN
6uIU0U64fQOlE7Qf1K0s9CPY2o3vPoDZ+u2YfaYor0Wd4dgYNBrK5+xnXB93g2NI
S9wyqPCXGEmOg8YAXkHdBF1lLvGBAv9PHeQ6a+BGy7NBqAyUcOvu+SMAuccHRExn
ty4losnpO0cDMfIG9WtLcfrbtds0h4QPrPeZvV0rC/Nx3mPDHwpxRGJ7Q//EU4hm
msIKZt3FjlRXhBSurSJO7uGeLp/kQI7fM4KxzROHPnRxhhYGo9KXimnN/1HC+kvM
4wW6KEHnpNZVCZSdexafXqx7jfwgOVdtKkvmiOkRBjySQcn5UmUrUOcmhKfehGrE
kvQMYeP9r+K6zG9sB7XNaxMskzol5iL/JKmrawUuGvCvG9MvTlnc6BNBSy13vL11
C1lQw2Bi70mqW0sAEQEAAbQfYW5kcmV3IDxhbmRyZXcuZGVudGFAZ21haWwuY29t
PokB1AQTAQgAPhYhBCpg+e8BNVYAf16MUJOUukY9oGenBQJgSmnEAhsDBQkAA/SA
BQsJCAcCBhUKCQgLAgQWAgMBAh4BAheAAAoJEJOUukY9oGen5EAMAK6b075z5jkW
YaLW+50N7n/3yDRHktrF+cz2S0mT2CnSK4fCiYIMaoQ0eCHE7ARyYRnt+0BK1Tp8
4ReWPcj2e4VOWzzWSF7k4VDs5PqGLZ149A3+Ia9ZYzc4sN1gsEShKc1FBUZOcGEQ
wl3W/g2ChaqubEGkD5AlyRuW3em1ZL6wEIKnFk82vVE0XWACQtYkMlh0JEBQnd62
9K25nBTv8EIPOAy1g1FTa1PSS3TXlHaWdSUHKGy984zdK2YEB1MHqRiqi6oh5pO2
Kd+8Yn12+v7wk2//j1LDZUVBwwAvkm1rIQtZpojOc+Jo8VLhoFwQ0lIG8Wv6RBWp
G4tQxelb0/dWFcxL/7JIYtSmXYuKHAF2wKHA4iOkUbx5ARE3hDiZ8sqSghB6QwPR
djCYW+Ay36rOwhaMnoz/9kUG6mdSRLyhLKLh302R/o2sngrxL6Akwhh/frG9XqUq
Tb6W4kSz5ZxJArlkdhd58RqZ6HoZhkl1crN98UySkjTwcvKEaK/kXrkBjQRgSmnE
AQwAwUyd+9k5hoeVtIdASHIxkAS3yJjIm/RaP4Vkqbh6X2YgVeRKIDnyOB9uVk7Q
WvtLVsYopEBg35R/uzW/nwk/LYMob0BZDyIx3xEqXke83eY3p0qQrouDiSO7Hjsb
9oqzxsEP5CBg2nwdVXMfY3LhtHu9N7MpCij4sGcjYeCrxhB/LtbCy1FGC502u8C7
qhU42mCThmJq+m7zVMAvVOyq+FvuHHLTisFWmhXxDpTFol0qCvI4QJSYExkEApVM
CTdWi/J+rW1O7xhrXuC3Q/Mh9z6Ee4s2hPKMdqJP3BFp+JeVcwQ1c7YqUbPmfhaC
wV1c33qdMwhI273vfYKoxwvDjQcTpuqcRBHkcnuN0b8OfsH8P3KgrQVBH92NxMMv
DMY2riB6Qjuj4cKSrWrQLx6mC3f1siQSQOdJHku5rQ9jXGjygXoAKj0Cpz19XG2B
O6k+1hlq3Jr8eti+qW1y7c6g6hL+35tvv15RdOvItow9M3kmxhtfeLK/qYAbfx7O
dbBxABEBAAGJAbwEGAEIACYWIQQqYPnvATVWAH9ejFCTlLpGPaBnpwUCYEppxAIb
DAUJAAP0gAAKCRCTlLpGPaBnp/bMDAC5lK1eeualAhLmxOI+8zD7nfnEDO0ys+om
QcrpOrAHPqvRxScQk1rRU3Gnzx8kn14nV/5tOFfw169EQ6/aE02jWbTLJFKk/s2l
83eOl9FGRvFPpXGPME66qsyNaNMFGBzYiFHr8F859qBd+slwA8rELlTCSScK2fYY
9TcNYssjnLMeDxNhhyzZk6HpTHFXeEJZiOkMccn0hcHbA7CSkvzw3dqq88xTHCRq
UDUGx7ZcZgSzWymeByVJ0OeF071/TWb8735jirEncDu5p6YUHwJDxcB5LpNtVka0
YjdL/Eo1TlETI4f+r069mGH4EX2YTMYhV8dbtTY68RFXxwdrMkvAjAWIm4Fcx6/o
YS1KOfKT4SYlD18/6PIPsLXJdrAa2T7KuAtre7gWAUwrwzhoTPUR0XZmIiFLpEzJ
4Uq40tINX0vnNszJcQQAkdjGwebWEdznasSWsqUCLxtnqFr1dEszEl/NYf6ywqwA
/vmrlvkBeRkxk78bUnaA4tveLI3+xHU=
=YAj+
-----END PGP PUBLIC KEY BLOCK-----

1

Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
07:07 PM
Cool, I'll reply to your email with the encrypted keys

1

Andrew
Photo of md5-08f6fb4c00b4a074647988ce90a07f5c
Andrew
07:17 PM
thanks dude!
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
07:20 PM
Happy to help! Emailed you