hey, I'm looking for a pointer to the best directi...
# community-help
j
hey, I'm looking for a pointer to the best direction. My application was a mongodb/elasticsearch backed trip search. My goal is to use Typesense to replace all the searching in one place. I created a little Example of my data:
Copy code
{
  "title": "test",
  "dates": [
    {
      "from_date": "2024-08-26",
      "till_date": "2024-08-30",
      "available": true,
      "preview": false,
      "mp": 2
    },
    {
      "from_date": "2024-09-26",
      "till_date": "2024-09-30",
      "available": true,
      "preview": false,
      "mp": 2
    },
    {
      "from_date": "2024-10-26",
      "till_date": "2024-10-30",
      "available": true,
      "preview": false,
      "mp": 2
    },
    {
      "from_date": "2024-11-02",
      "till_date": "2024-11-07",
      "available": false,
      "preview": true,
      "mp": 2
    }
  ]
}
My goal is to find all the documents matching a query like "from_date and till_date in Oktober 2024" (or any other range). Or combinations like in october an preview == true. What I understood from the documentation and git issues: There is currently no good solution to do this without flattening the data. Flatten the Array and include multiple entries is no problem and works great (I think). But: I lose a lot of possiblities with the result set. Because every Trip is in the result 1-20 times. so all counts are off. is there a good way to "unflatten" the result back in typesense? I would think this is a common problem, but did not find a good solution. Thank you!
f
Typesense provides sane defaults out of the box, so migrating won't prove much of a hassle. You can fine-tune your configuration and get up to speed quickly. I would suggest you take a look at the documentation guide to get a general overview of how Typesense behaves. Afterwards, the Collections documentation should be your first stฮฟp in your Typesense journey. I'll provide some more detail now that you've posted a sample of your data to provide some more insight
j
Thank you for your quick first answer. I gone through the documentation yesterday but was stuck at this point. The search with flattened data is running. The main problem is, that flattening introduces a postprocess step, which takes many of typesense counts ad absurdum. Just to clearify the basic setup works, it's a detail problem on one aspect I dont understand enough ๐Ÿ™‚
f
Typesense supports nested object data structures . From the docs:
To simplify traversing the data in the results, you might want to send both the flattened and unflattened version of the nested fields into Typesense, and only set the flattened keys as indexed in the collection's schema and use them for search/filtering/faceting. At display time when parsing the results, you can then use the nested version.
You could then take a look at the Curation feature, with which you can set organic filtering by query (for example, if a query contains the sentence
available
inside, filter by only available events etc. There's a more visual way of handling curation over at Typesense Cloud (the hosted SaaS product of Typesense), which has a Dashboard that includes many more features other than curation
j
I'm adding the date fields to my "main object" right now. The problem is not, to get a untainted/unflattend object back. It's more to get only 1 instance of that trip. Kind of a .reduce() on the result-set.
most of the interesting data is in the main object and dates are only a part of the filter-matrix.
On the nested structures: If i understood the documentation correct, my search would not be possible in that structure:
from_date:> '2024-10-01 as timestamp && from_date:< '2024-10-31 as stimestamp' && preview == true
๐Ÿ‘ 1
f
To get a behavior similar to
reduce()
you could use some grouping parameters https://typesense.org/docs/26.0/api/search.html#grouping-parameters if you plan to aggregate results
If i understood the documentation correct, my search would not be possible in that structure:
from_date:> '2024-10-01 as timestamp && from_date:< '2024-10-31 as stimestamp' && preview == true
You can also use ranges for numeric filtering:
Numeric Filtering:
Filter documents with numeric values between a min and max value, using the range operator
[min..max]
or using simple comparison operators
>
,
>=
<
,
<=
,
=
.
You can enable
"range_index": true
on the numerical field schema for fast range queries (will incur additional memory usage for the index though).
Examples:
-
num_employees:[10..100]
-
num_employees:<40
-
num_employees:[10..100,40]
(Filter docs where value is between 10 to 100 or exactly 40).
j
sorry, for the confusion. i used the range option, but wanted to illustrate the needed filter above, by having a "argument" more
f
Should work either way!
๐Ÿ‘ 1
You can filter while the user types by calling a function that formats their input to a unix timestamp, either by regex, or by using a library like
date-fns
Mind you, I would avoid doing this on blur, as it can be computationally intensive and lead to lag, but it is possible
j
{"q":"Zucker","query_by":"*","group_by":"nr","group_limit":1,"per_page":30,"filter_by":"fd:[1725148800..1727740800]"}
is my last attempt on using the grouping feature. the problem here is that the result is now grouped into 1 element arrays like this
result["grouped_hits"][0]["hits"]
i would need to
result['grouped_hits'].map{ |gh| gh['hits'].first }
but with this i lost all of your sorting and counting options, correct?
Mind you, I would avoid doing this on blur, as it can be computationally intensive and lead to lag, but it is possible
what do you mean with "doing it on blur"?
f
Have it activate when the element loses focus
j
okok, my head was not prepared for frontend topics ๐Ÿ™‚
f
There's an example implementation that incorporates grouping (for JOINs, but it still applies) with query parameters . The results are passed down as normal and still preserve the grouping
j
Hm, maybe I'm missing the point in this example. Is flattening the best approach, or should a use a different tactic? I want to use the full power of typesense.
f
Flattening would be advisable for searching purposes
j
This use-case for availability based filtering has come up in the past. Here's prior discussion on this topic for reference: https://threads.typesense.org/2K1c77