Issues with Aggregation and Analytic Rules
TLDR Pawel experienced issues with aggregation in a complex data collection. Kishore Nallan identified it as a bug and provided an updated software build to solve the problem.
Powered by Struct AI
3
36
1mo
Solved
Aug 23, 2023 (1 month ago)
Pawel
Pawel
11:29 AMHi!
I have an issue with aggregation (analytic rules)
I would like to have separate aggregation for each base collection.
I created aggregation collections as mentioned in the documentation
I added rules for each collection:
After this, every single time, only the last added rule is working (firs in the rules)
I tested it on two, three and 20 collections, In this case only magento_2_default_reviews_meta_aggregation is collection queries
Also In server logs I see only this one:
I have an issue with aggregation (analytic rules)
I would like to have separate aggregation for each base collection.
I created aggregation collections as mentioned in the documentation
'name' => $name,
'fields' => [
['name' => 'q', 'type' => 'string'],
['name' => 'count', 'type' => 'int32'],
],I added rules for each collection:
{"rules":[{"name":"magento_2_default_reviews_meta_aggregation_rule","params":{"destination":{"collection":"magento_2_default_reviews_meta_aggregation"},"limit":1000,"source":{"collections":["magento_2_default_reviews_meta_1692788921"]}},"type":"popular_queries"},{"name":"magento_2_czech_market_reviews_aggregation_rule","params":{"destination":{"collection":"magento_2_czech_market_reviews_aggregation"},"limit":1000,"source":{"collections":["magento_2_czech_market_reviews_1692788919"]}},"type":"popular_queries"},{"name":"magento_2_default_urls_aggregation_rule","params":{"destination":{"collection":"magento_2_default_urls_aggregation"},"limit":1000,"source":{"collections":["magento_2_default_urls_1692788923"]}},"type":"popular_queries"},{"name":"magento_2_czech_market_reviews_meta_aggregation_rule","params":{"destination":{"collection":"magento_2_czech_market_reviews_meta_aggregation"},"limit":1000,"source":{"collections":["magento_2_czech_market_reviews_meta_1692788922"]}},"type":"popular_queries"},{"name":"magento_2_default_reviews_aggregation_rule","params":{"destination":{"collection":"magento_2_default_reviews_aggregation"},"limit":1000,"source":{"collections":["magento_2_default_reviews_1692788917"]}},"type":"popular_queries"},{"name":"magento_2_czech_market_eav_attributes_aggregation_rule","params":{"destination":{"collection":"magento_2_czech_market_eav_attributes_aggregation"},"limit":1000,"source":{"collections":["magento_2_czech_market_eav_attributes_1692788915"]}},"type":"popular_queries"},{"name":"magento_2_default_categories_aggregation_rule","params":{"destination":{"collection":"magento_2_default_categories_aggregation"},"limit":1000,"source":{"collections":["magento_2_default_categories_1692788517"]}},"type":"popular_queries"},{"name":"magento_2_default_products_children_aggregation_rule","params":{"destination":{"collection":"magento_2_default_products_children_aggregation"},"limit":1000,"source":{"collections":["magento_2_default_products_children_1692788636"]}},"type":"popular_queries"},{"name":"magento_2_czech_market_products_aggregation_rule","params":{"destination":{"collection":"magento_2_czech_market_products_aggregation"},"limit":1000,"source":{"collections":["magento_2_czech_market_products_1692788586"]}},"type":"popular_queries"},{"name":"magento_2_default_products_aggregation_rule","params":{"destination":{"collection":"magento_2_default_products_aggregation"},"limit":1000,"source":{"collections":["magento_2_default_products_1692788523"]}},"type":"popular_queries"},{"name":"magento_2_czech_market_urls_aggregation_rule","params":{"destination":{"collection":"magento_2_czech_market_urls_aggregation"},"limit":1000,"source":{"collections":["magento_2_czech_market_urls_1692788954"]}},"type":"popular_queries"},{"name":"magento_2_default_cms_pages_aggregation_rule","params":{"destination":{"collection":"magento_2_default_cms_pages_aggregation"},"limit":1000,"source":{"collections":["magento_2_default_cms_pages_1692788906"]}},"type":"popular_queries"},{"name":"magento_2_czech_market_products_children_aggregation_rule","params":{"destination":{"collection":"magento_2_czech_market_products_children_aggregation"},"limit":1000,"source":{"collections":["magento_2_czech_market_products_children_1692788809"]}},"type":"popular_queries"},{"name":"magento_2_default_cms_blocks_aggregation_rule","params":{"destination":{"collection":"magento_2_default_cms_blocks_aggregation"},"limit":1000,"source":{"collections":["magento_2_default_cms_blocks_1692788902"]}},"type":"popular_queries"},{"name":"magento_2_czech_market_categories_aggregation_rule","params":{"destination":{"collection":"magento_2_czech_market_categories_aggregation"},"limit":1000,"source":{"collections":["magento_2_czech_market_categories_1692788521"]}},"type":"popular_queries"},{"name":"magento_2_czech_market_cms_blocks_aggregation_rule","params":{"destination":{"collection":"magento_2_czech_market_cms_blocks_aggregation"},"limit":1000,"source":{"collections":["magento_2_czech_market_cms_blocks_1692788904"]}},"type":"popular_queries"},{"name":"magento_2_czech_market_cms_pages_aggregation_rule","params":{"destination":{"collection":"magento_2_czech_market_cms_pages_aggregation"},"limit":1000,"source":{"collections":["magento_2_czech_market_cms_pages_1692788908"]}},"type":"popular_queries"},{"name":"magento_2_default_gdpr_cookies_aggregation_rule","params":{"destination":{"collection":"magento_2_default_gdpr_cookies_aggregation"},"limit":1000,"source":{"collections":["magento_2_default_gdpr_cookies_1692788909"]}},"type":"popular_queries"},{"name":"magento_2_czech_market_gdpr_cookies_aggregation_rule","params":{"destination":{"collection":"magento_2_czech_market_gdpr_cookies_aggregation"},"limit":1000,"source":{"collections":["magento_2_czech_market_gdpr_cookies_1692788911"]}},"type":"popular_queries"},{"name":"magento_2_default_eav_attributes_aggregation_rule","params":{"destination":{"collection":"magento_2_default_eav_attributes_aggregation"},"limit":1000,"source":{"collections":["magento_2_default_eav_attributes_1692788912"]}},"type":"popular_queries"}]}After this, every single time, only the last added rule is working (firs in the rules)
I tested it on two, three and 20 collections, In this case only magento_2_default_reviews_meta_aggregation is collection queries
Also In server logs I see only this one:
I20230823 11:24:09.380872 1402 analytics_manager.cpp:297] Query aggregation for collection: magento_2_default_reviews_meta_aggregation
I20230823 11:24:09.588793 1402 analytics_manager.cpp:311] Top K aggregation for collection: magento_2_default_reviews_meta_aggregationKishore Nallan
Kishore Nallan
11:31 AM👋 Let me take a look and get back to you.
1
Pawel
Pawel
11:37 AMPlease note, that when I checked this with super simple collections from example on typesense site - it was working
Kishore Nallan
Kishore Nallan
11:38 AMWhen you create just 2 or 3 simple collections did it work?
Pawel
Pawel
11:38 AMon two collection it was working
11:39
Pawel
11:39 AMbut when I created mine 2 collections much more complicated, the behavior is the same as I described
Kishore Nallan
Kishore Nallan
11:39 AMActually I think this might be an issue that I fixed a few days ago. Hold on, I will get you another build you can try on.
11:40
Kishore Nallan
11:40 AMtypesense/typesense:0.25.1.rc2Pawel
Pawel
11:41 AMIs it available on deb packages?
Kishore Nallan
Kishore Nallan
11:42 AMYes, just change the version numbers on the URL to
0.25.1.rc2Pawel
Pawel
11:42 AMGive me a sec
11:43
Pawel
11:43 AMlike this?
11:43
Pawel
11:43 AMtypesense-server-0.25.1.rc2-amd64.deb
11:44
Pawel
11:44 AMIf yes, I see 404
Kishore Nallan
Kishore Nallan
11:46 AM11:46
Kishore Nallan
11:46 AM(updated arm -> amd)
Pawel
Pawel
11:50 AM404 Sorry - we could not find this page. Ironic isn't it?12:07
Pawel
12:07 PMOh, I see it now - give me a few minutes, I'm checking this
Kishore Nallan
Kishore Nallan
12:08 PMMaybe the cdn cache took some time :thinking_face:
Pawel
Pawel
12:13 PMUnfortunately it is not working
12:14
Pawel
12:14 PMonly last rule (first in the query /analytics/rules) is adding stats to aggregation collection
12:15
Pawel
12:15 PM Kishore Nallan
12:17
Pawel
12:17 PMI can give you direct access to api if needed
Kishore Nallan
Kishore Nallan
12:17 PMI would need the data directory itself
12:17
Kishore Nallan
12:17 PMThat's the only way to preserve the state of the bug entirely.
Pawel
Pawel
12:18 PMhow can I do that?
12:18
Pawel
12:18 PMshould I gzip and send it to you ?
Kishore Nallan
Kishore Nallan
12:18 PMIf you can reproduce on throw away or sample data, that would be best. Just stop typesense server and zip the data directory and share.
Pawel
Pawel
12:18 PMok, give me a moment
1
12:20
Pawel
12:20 PMI shared the archive on private channell
1
Kishore Nallan
Kishore Nallan
12:26 PMSo the way to test this will be to try sending queries to collections other than the rule that targets
magento_2_default_reviews_meta_aggregation ?Pawel
Pawel
12:27 PMyes
Kishore Nallan
Kishore Nallan
01:41 PMFound the issue, will share a build with the fix shortly!
02:16
Kishore Nallan
02:16 PMFixed in
0.25.1.rc3. Please try it out and confirm. Thank you.Aug 24, 2023 (1 month ago)
Pawel
Pawel
06:14 AMwow, you are the best!
It is working now as expected.
Thank you Kishore Nallan so much!
It is working now as expected.
Thank you Kishore Nallan so much!
Kishore Nallan
Kishore Nallan
06:23 AMGreat, you're welcome!
Typesense
Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI
Indexed 2779 threads (79% resolved)
Similar Threads
Resolving Typesense Analytics "top_queries" Collection Issue
Zain faced issues with typesense analytics and an empty "top_queries" collection. Jason and Kishore Nallan assisted in troubleshooting, identified a bug, and suggested a fix in `0.25.0.rc66`. The fix worked for Zain.
29
1mo
Solved
Troubleshooting Typesense API Analytics Query Suggestions
Md was confused about implementing Typesense's Analytics Query Suggestions and experienced issues with collections returning no hits. Assistance from Kishore Nallan eventually led to the identification that analytics had to be enabled. They also discussed tracking duplicate and empty queries, resulting in Md creating a Github issue.
27
1mo
Solved
Typesense Server Crash and Counting Issue
Dan reported a server crash and counting issues with Typesense server version 0.26.0.rc1. Jason suggested an upgrade to `0.26.0.rc13` while Kishore Nallan committed to investigate the issue based on a sample dataset from Dan.
11
1mo
Troubleshooting 400 Error When Upgrading Typesense Firestore Extension
Orion experienced a `400` error after updating the Typesense Firestore extension, causing issues with cloud functions. They traced the issue back to a data type conflict in their Typesense collection schema after updating. With help from Jason and Kishore Nallan, they resolved the issue by recreating the collection.
96
12mo
Solved
Troubleshooting Typesense Connection Issue and Data Retrieval
Felix encountered issues connecting to a server-created typesense and retrieving data collections. Kishore Nallan and Jason provided guidance, highlighting errors in the original code and suggesting changes to the URL and curl command.
63
22mo
Solved