#community-help

Issues with Aggregation and Analytic Rules

TLDR Pawel experienced issues with aggregation in a complex data collection. Kishore Nallan identified it as a bug and provided an updated software build to solve the problem.

Powered by Struct AI

3

36
1mo
Solved
Join the chat
Aug 23, 2023 (1 month ago)
Pawel
Photo of md5-4b83cee8145bcbcaa53578a2c37da6d6
Pawel
11:29 AM
Hi!
I have an issue with aggregation (analytic rules)
I would like to have separate aggregation for each base collection.

I created aggregation collections as mentioned in the documentation
'name' => $name,
            'fields' => [
                ['name' => 'q', 'type' => 'string'],
                ['name' => 'count', 'type' => 'int32'],
            ],

I added rules for each collection:
{"rules":[{"name":"magento_2_default_reviews_meta_aggregation_rule","params":{"destination":{"collection":"magento_2_default_reviews_meta_aggregation"},"limit":1000,"source":{"collections":["magento_2_default_reviews_meta_1692788921"]}},"type":"popular_queries"},{"name":"magento_2_czech_market_reviews_aggregation_rule","params":{"destination":{"collection":"magento_2_czech_market_reviews_aggregation"},"limit":1000,"source":{"collections":["magento_2_czech_market_reviews_1692788919"]}},"type":"popular_queries"},{"name":"magento_2_default_urls_aggregation_rule","params":{"destination":{"collection":"magento_2_default_urls_aggregation"},"limit":1000,"source":{"collections":["magento_2_default_urls_1692788923"]}},"type":"popular_queries"},{"name":"magento_2_czech_market_reviews_meta_aggregation_rule","params":{"destination":{"collection":"magento_2_czech_market_reviews_meta_aggregation"},"limit":1000,"source":{"collections":["magento_2_czech_market_reviews_meta_1692788922"]}},"type":"popular_queries"},{"name":"magento_2_default_reviews_aggregation_rule","params":{"destination":{"collection":"magento_2_default_reviews_aggregation"},"limit":1000,"source":{"collections":["magento_2_default_reviews_1692788917"]}},"type":"popular_queries"},{"name":"magento_2_czech_market_eav_attributes_aggregation_rule","params":{"destination":{"collection":"magento_2_czech_market_eav_attributes_aggregation"},"limit":1000,"source":{"collections":["magento_2_czech_market_eav_attributes_1692788915"]}},"type":"popular_queries"},{"name":"magento_2_default_categories_aggregation_rule","params":{"destination":{"collection":"magento_2_default_categories_aggregation"},"limit":1000,"source":{"collections":["magento_2_default_categories_1692788517"]}},"type":"popular_queries"},{"name":"magento_2_default_products_children_aggregation_rule","params":{"destination":{"collection":"magento_2_default_products_children_aggregation"},"limit":1000,"source":{"collections":["magento_2_default_products_children_1692788636"]}},"type":"popular_queries"},{"name":"magento_2_czech_market_products_aggregation_rule","params":{"destination":{"collection":"magento_2_czech_market_products_aggregation"},"limit":1000,"source":{"collections":["magento_2_czech_market_products_1692788586"]}},"type":"popular_queries"},{"name":"magento_2_default_products_aggregation_rule","params":{"destination":{"collection":"magento_2_default_products_aggregation"},"limit":1000,"source":{"collections":["magento_2_default_products_1692788523"]}},"type":"popular_queries"},{"name":"magento_2_czech_market_urls_aggregation_rule","params":{"destination":{"collection":"magento_2_czech_market_urls_aggregation"},"limit":1000,"source":{"collections":["magento_2_czech_market_urls_1692788954"]}},"type":"popular_queries"},{"name":"magento_2_default_cms_pages_aggregation_rule","params":{"destination":{"collection":"magento_2_default_cms_pages_aggregation"},"limit":1000,"source":{"collections":["magento_2_default_cms_pages_1692788906"]}},"type":"popular_queries"},{"name":"magento_2_czech_market_products_children_aggregation_rule","params":{"destination":{"collection":"magento_2_czech_market_products_children_aggregation"},"limit":1000,"source":{"collections":["magento_2_czech_market_products_children_1692788809"]}},"type":"popular_queries"},{"name":"magento_2_default_cms_blocks_aggregation_rule","params":{"destination":{"collection":"magento_2_default_cms_blocks_aggregation"},"limit":1000,"source":{"collections":["magento_2_default_cms_blocks_1692788902"]}},"type":"popular_queries"},{"name":"magento_2_czech_market_categories_aggregation_rule","params":{"destination":{"collection":"magento_2_czech_market_categories_aggregation"},"limit":1000,"source":{"collections":["magento_2_czech_market_categories_1692788521"]}},"type":"popular_queries"},{"name":"magento_2_czech_market_cms_blocks_aggregation_rule","params":{"destination":{"collection":"magento_2_czech_market_cms_blocks_aggregation"},"limit":1000,"source":{"collections":["magento_2_czech_market_cms_blocks_1692788904"]}},"type":"popular_queries"},{"name":"magento_2_czech_market_cms_pages_aggregation_rule","params":{"destination":{"collection":"magento_2_czech_market_cms_pages_aggregation"},"limit":1000,"source":{"collections":["magento_2_czech_market_cms_pages_1692788908"]}},"type":"popular_queries"},{"name":"magento_2_default_gdpr_cookies_aggregation_rule","params":{"destination":{"collection":"magento_2_default_gdpr_cookies_aggregation"},"limit":1000,"source":{"collections":["magento_2_default_gdpr_cookies_1692788909"]}},"type":"popular_queries"},{"name":"magento_2_czech_market_gdpr_cookies_aggregation_rule","params":{"destination":{"collection":"magento_2_czech_market_gdpr_cookies_aggregation"},"limit":1000,"source":{"collections":["magento_2_czech_market_gdpr_cookies_1692788911"]}},"type":"popular_queries"},{"name":"magento_2_default_eav_attributes_aggregation_rule","params":{"destination":{"collection":"magento_2_default_eav_attributes_aggregation"},"limit":1000,"source":{"collections":["magento_2_default_eav_attributes_1692788912"]}},"type":"popular_queries"}]}

After this, every single time, only the last added rule is working (firs in the rules)
I tested it on two, three and 20 collections, In this case only magento_2_default_reviews_meta_aggregation is collection queries

Also In server logs I see only this one:
I20230823 11:24:09.380872  1402 analytics_manager.cpp:297] Query aggregation for collection: magento_2_default_reviews_meta_aggregation
I20230823 11:24:09.588793  1402 analytics_manager.cpp:311] Top K aggregation for collection: magento_2_default_reviews_meta_aggregation
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
11:31 AM
👋 Let me take a look and get back to you.

1

Pawel
Photo of md5-4b83cee8145bcbcaa53578a2c37da6d6
Pawel
11:37 AM
Please note, that when I checked this with super simple collections from example on typesense site - it was working
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
11:38 AM
When you create just 2 or 3 simple collections did it work?
Pawel
Photo of md5-4b83cee8145bcbcaa53578a2c37da6d6
Pawel
11:38 AM
on two collection it was working
11:39
Pawel
11:39 AM
but when I created mine 2 collections much more complicated, the behavior is the same as I described
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
11:39 AM
Actually I think this might be an issue that I fixed a few days ago. Hold on, I will get you another build you can try on.
11:40
Kishore Nallan
11:40 AM
typesense/typesense:0.25.1.rc2
Pawel
Photo of md5-4b83cee8145bcbcaa53578a2c37da6d6
Pawel
11:41 AM
Is it available on deb packages?
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
11:42 AM
Yes, just change the version numbers on the URL to 0.25.1.rc2
Pawel
Photo of md5-4b83cee8145bcbcaa53578a2c37da6d6
Pawel
11:42 AM
Give me a sec
11:43
Pawel
11:43 AM
like this?
11:43
Pawel
11:43 AM
typesense-server-0.25.1.rc2-amd64.deb
11:44
Pawel
11:44 AM
If yes, I see 404
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
11:46 AM
11:46
Kishore Nallan
11:46 AM
(updated arm -> amd)
Pawel
Photo of md5-4b83cee8145bcbcaa53578a2c37da6d6
Pawel
11:50 AM
404 Sorry - we could not find this page. Ironic isn't it?

12:07
Pawel
12:07 PM
Oh, I see it now - give me a few minutes, I'm checking this
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
12:08 PM
Maybe the cdn cache took some time :thinking_face:
Pawel
Photo of md5-4b83cee8145bcbcaa53578a2c37da6d6
Pawel
12:13 PM
Unfortunately it is not working
12:14
Pawel
12:14 PM
only last rule (first in the query /analytics/rules) is adding stats to aggregation collection
12:15
Pawel
12:15 PM
Kishore Nallan
12:17
Pawel
12:17 PM
I can give you direct access to api if needed
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
12:17 PM
I would need the data directory itself
12:17
Kishore Nallan
12:17 PM
That's the only way to preserve the state of the bug entirely.
Pawel
Photo of md5-4b83cee8145bcbcaa53578a2c37da6d6
Pawel
12:18 PM
how can I do that?
12:18
Pawel
12:18 PM
should I gzip and send it to you ?
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
12:18 PM
If you can reproduce on throw away or sample data, that would be best. Just stop typesense server and zip the data directory and share.
Pawel
Photo of md5-4b83cee8145bcbcaa53578a2c37da6d6
Pawel
12:18 PM
ok, give me a moment

1

12:20
Pawel
12:20 PM
I shared the archive on private channell

1

Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
12:26 PM
So the way to test this will be to try sending queries to collections other than the rule that targets magento_2_default_reviews_meta_aggregation ?
Pawel
Photo of md5-4b83cee8145bcbcaa53578a2c37da6d6
Pawel
12:27 PM
yes
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
01:41 PM
Found the issue, will share a build with the fix shortly!
02:16
Kishore Nallan
02:16 PM
Fixed in 0.25.1.rc3. Please try it out and confirm. Thank you.
Aug 24, 2023 (1 month ago)
Pawel
Photo of md5-4b83cee8145bcbcaa53578a2c37da6d6
Pawel
06:14 AM
wow, you are the best!
It is working now as expected.
Thank you Kishore Nallan so much!
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
06:23 AM
Great, you're welcome!