Resolving Typesense Cloud Cluster Issue with Cron Job
TLDR Issei reported a problem with an unhealthy Typesense Cloud cluster. With the particular help of Jason and Kishore Nallan, they discovered that a problematic cron job was responsible. A solution, using a different endpoint for data export, was agreed on and implemented.
5
Apr 17, 2021 (32 months ago)
Issei
03:38 AMIssei
03:39 AMJason
03:39 AMIssei
03:39 AMbw8oxjzkvrs3l562p
Jason
03:40 AMJason
03:49 AMIssei
03:51 AMclips
Jason
04:06 AMCluster should be back up in about 10 mins
Jason
04:11 AMJason
04:11 AMIssei
04:12 AM1
Jason
04:15 AMIssei
04:16 AMIssei
04:18 AMJason
04:18 AMJason
04:28 AMIssei
04:28 AMJason
04:30 AMIssei
04:30 AM/documents/search
Issei
04:33 AMJason
04:34 AMIssei
04:41 AM while (true) {
$page++;
$params = [
'q' => '*',
'filter_by' => 'vc:>100',
'include_fields' => 'id,b.l,at',
'page' => $page,
'per_page' => 250,
];
...
}
Issei
04:44 AM 'filter_by' => 'vc:>1000',
Jason
04:44 AMJason
04:45 AM'filter_by' => 'vc:>1000'
, before the changeJason
04:45 AMJason
04:46 AMIssei
04:47 AMIssei
04:49 AM'filter_by' => 'vc:>1000'
{
"facet_counts": [],
"found": 207397,
'filter_by' => 'vc:>100'
{
"facet_counts": [],
"found": 927449,
Jason
04:53 AMvc:>100
? I'd like to see if we can replicate the issue.Issei
04:56 AM1
Jason
04:56 AMIssei
04:57 AM1
Jason
05:09 AMJason
05:13 AMIssei
05:15 AMIssei
05:18 AMJason
05:19 AMIssei
05:19 AM/documents/search?q=hello&query_by=t&per_page=10
Jason
05:19 AMJason
05:25 AMIssei
05:30 AMJason
05:33 AMIssei
05:34 AMJason
05:35 AMJason
05:35 AMIssei
05:36 AMIssei
05:39 AM> Does the order of records returned matter for your use case?
I donโt care in this case.
Jason
05:42 AMfilter_by
param to the documents/export
endpoint, that might be better suited for your use caseJason
05:44 AMJason
05:44 AMIssei
05:45 AMJason
05:48 AMinclude_fields
param in your search query. Do you use or see a need for any other search param to be supported in the export endpoint?Issei
05:54 AMfilter_by
and include_fields
are enough for me!Jason
05:55 AMIssei
06:03 AM1
Jun 28, 2021 (29 months ago)
Kishore Nallan
03:47 AMIf you can give it a spin locally, and want to upgrade, I can upgrade your cloud cluster.
Jul 05, 2021 (29 months ago)
Issei
03:43 AMIssei
03:45 AMKishore Nallan
03:48 AMKishore Nallan
03:49 AMKishore Nallan
03:51 AMJason
04:44 PMIssei Could you update your integration to use the documents/export endpoint as described here: https://github.com/typesense/typesense/issues/283#issuecomment-869189177
Jul 06, 2021 (29 months ago)
Issei
01:13 AMIssei
02:19 AM1
Typesense
Indexed 2779 threads (79% resolved)
Similar Threads
Optimizing Typesense Implementation for Large Collections
Oskar faced performance issues with his document collection in Typesense due to filter additions. Jason suggested trying a newer Typesense build and potentially partitioning the data into country-wise collections. They also discussed reducing network latency with CDN solutions.
Large JSONL Documents Import Issue & Resolution
Suraj was having trouble loading large JSONL documents into Typesense server. After several discussions and attempts, it was discovered that the issue was due to data quality. Once the team extracted the data again, the upload process worked smoothly.
Troubleshooting Indexing Duration in Typesense Import
Alan asked about lengthy indexing times for importing documents to Typesense. Jason suggested various potential causes, including network connectivity and system resources. They later identified the problem to be an error in Alan's code.
Slow, High CPU Write Operations After Collection Drop in Typesense
Himank discussed an issue in Typesense where deleting and recreating a collection led to slow write operations and high CPU usage. Kishore Nallan suggested using an alias to avoid this issue. Numerous tests and debugging was conducted as pboros contributed with local testing. Kishore Nallan aimed to start implementing a range delete and full db compaction after deletion to potentially solve the issue.
Resolving Typesense v0.22.0 Import Issues
Anton encountered issues importing documents in batches using Typesense v0.22.0. Kishore Nallan suggested using atomic import and proposed a debug build. After multiple trials, they were able to reproduce and fix the issue. Anton confirmed the solution was working.