This message was deleted.
# community-help
s
This message was deleted.
k
@Harpreet Sangar Can you please check what could be happening here?
πŸ‘€ 1
πŸ™ 1
h
@Dhia Can you check if the response to
price: [61..1000000]
filter takes the same time? The second value should be the largest value of your dataset.
d
sure
Copy code
filter_by	hits	timing	facet(price) total_values
price:>61	7331753	~4200ms	18787
price:[61..499900000]	7331992	~4500ms	18788
price:[61..4999000]	7331969	~4500ms	18763
price:[61..499900]	7331104	~4500ms	18105
price:[61..49990]	7296188	~3600ms	9792
price:[61..4999]	6809221	~850ms	3436
i also run facet on price (in seperate query to not impact timing) to check how many values in each range
"facet_by":"price"
and fetched
results[0].facet_counts[0].stats.total_values
h
@Kishore Nallan With ~70M values, I think we're spending too much time copying values.
It's actually 7M right?
d
yes 7M
h
I'll try to replicate the issue and let you know.
d
it doesn't seem to be linear for example in test 5
price:[61..49990]
compared to test 4
price:[61..499900]
β€’ we have hits: 7.2M vs 7.3M 0.5% less β€’ we have timing: 3.6s vs 4.5s 20% less in test 6
price:[61..4999]
westill have 6.8M hit and timing did drop by 80% it seems like some kind of cliff, it could be cache spilling/trashing that create the cliff or another compeonet that i didn't account for
side note the dataset is 15M lines
k
The main problem with range query is that we have an index that maps individual price values to one or more document IDs. Much of the time is spent in aggregating these small lists into a giant list:
Copy code
61: [100]
62: [99, 1000, 2034]
63: [38483]
So the actual range matters because of the distribution of IDs within those individual numbers in the range.
d
ok, than why it change completly depending on the direction of filter > or <?
Copy code
filter_by	hits	timing	facet(price) total_values
price:<499900000	14844977	~800ms	18826
price:<4999000	14844951	~800ms	18801
price:<499900	14844080	~800ms	18143
price:<49990	14809157	~900ms	9829
Price:<4999	14322170	~900ms	3473
k
That's the mystery to figure out. Iteration order should not matter.
This is why we introduced the
range_index
option. It will consolidate those IDs and the queries will be very fast (memory cpu trade-off)
d
and it's set on the field
k
Got it.
d
i ll try to see if i could reproduce with more minimal configuration (like simpler schema)
h
The schema shouldn't matter in this scenario since we're dealing with single field.
@Dhia Is it possible for you to share your dataset? I would just need the price field. You can use the export api to only include the price field.
d
yes doeing this (also running the test locally)
βœ… 1
i m able to reproduce with only prices and id in docuemnt. i m generating a snapshot and i ll send it
βœ… 1
at 1M doc i do not see much diff(but response is quick may be it's below noise) . at 5M the diff is clear (<61 220ms >61 960ms)
here the export with ~8M entries . one note we have some skew in dataset and some prices are more represented than others (0 for example). i don't know if it has uan impact
and here the collection conf
Copy code
"created_at": 1748862323,
  "default_sorting_field": "",
  "enable_nested_fields": true,
  "fields": [
    {
      "facet": true,
      "index": true,
      "infix": false,
      "locale": "",
      "name": "price",
      "optional": true,
      "range_index": true,
      "sort": true,
      "stem": false,
      "stem_dictionary": "",
      "store": true,
      "type": "int64"
    }
  ],
  "name": "dhia_test_price_2",
  "num_documents": 8016392,
  "symbols_to_index": [],
  "token_separators": []
}
βœ… 1
playing with the query a bit it seems i can preproduce in both ways (with
:>
or
:<
, it'a a matter of what data is fetched. this trace perf seems interesting to me 82% of time spent in
id_list_t::uncompress(std::vector<unsigned int, std::allocator<unsigned int> >&)
and it causes a lot of page faults
Copy code
-   98,75%     0,00%  typesense-serve  libc.so.6          [.] 0x0000727d1ad5dee8                                                                                                                                                                                               β—†
     0x727d1ad5dee8
   - std::__future_base::_State_baseV2::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*)
      - 86,39% std::_Function_handler<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> (), std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, std::_
         - 86,37% post_multi_search(std::shared_ptr<http_req> const&, std::shared_ptr<http_res>const&)
            - 86,36% CollectionManager::do_search(std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::chβ–’
               - 86,35% Collection::search[abi:cxx11](collection_search_args_t&) const
                  - 86,33% Index::run_search(search_args*)
                     - 86,33% Index::search(std::vector<query_tokens_t, std::allocator<query_tokens_t> >&, std::vector<search_field_t, std::allocator<search_field_t> > const&, text_match_type_t, std::unique_ptr<filter_node_t, std::default_delete<filter_node_t> >&, std::vβ–’
                        - 86,28% filter_result_iterator_t::filter_result_iterator_t(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, Index const*, filter_node_t const*, bool const&, unsigned long const&, unsigned long, unsigned longβ–’
                             filter_result_iterator_t::init(bool const&, bool const&)
                           - NumericTrie::search_greater_than(long const&, bool const&, unsigned int*&, unsigned int&)
                              - 85,80% NumericTrie::Node::search_greater_than(long const&, char const&, unsigned int*&, unsigned int&)
                                 - 82,09% id_list_t::uncompress(std::vector<unsigned int, std::allocator<unsigned int> >&)
                                    - 32,26% 0x727d1ae64be7
                                       - 31,21% asm_exc_page_fault
                                          - 27,58% exc_page_fault
i guess you are able to see that in your testing also. please ping me if you need me to dig further
h
Regarding the
>61
and
<61
case, these queries are matching 512 nodes and 35 nodes respectively. It explains the difference in response time.
id_list_t::uncompress
is called on each node.
d
disclaimer: I m not expert in C++ i m surprised that reserving 512 time is taking few seconds , i expect it to be in the single digit ms per call only exception i see: β€’ we are reserving few MB each time (and may be over provisionning or using a lot of data) β€’ we are miss reserving the size and the vector get extended frequently, but i guess we should see that in perf trace Am i missing something?
k
Will need to sync up with Harpreet, but the 512 nodes each have a document Id list associated with them, which must be compressed and aggregated. Still doesn't sound something that will explain the delta. Will get back to you once we have a clear picture.
βœ… 2
I've published
29.0.rc28
that contains a fix for this issue.
d
hello, Thanks. i ll be testing today in large cluster (in test env seems much better indeed)
πŸ‘ 1
it's much better now. that's really cool thank you very much i have a multisearch query with 5 search with facting that was taking 24s and now it runs in 7s (still high from what i target but that's another story)
πŸ™ŒπŸΌ 1
k
Enabling approximate faceting will also help.
πŸ‘ 1