Docker Upgrade and Indexing Data Issues for Travel App

Is there any way to upgrade to a rc from docker and keeping the indexing data?

Yes simply point the new Docker image to the same data directory and Typesense will just start and index the on disk back again and be ready. There will be a small downtime during this restart process.

thanks!

Do you need a RPM or DEB?

DEB

Okay, here's the latest RC DEB: https://dl.typesense.org/releases/0.21.0.rc14/typesense-server-0.21.0.rc14-amd64.deb

thanks!

Welcome, let me know how it works.

how should we install it and keep the index?

1. Take a back up of the data dir just in case.
2. Stop service.
3. Install deb with:

apt -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" -y install new-typsense.deb

thanks!

Are you querying across multiple collections using multi_search?

Ioan-Andrei can answer this

we don't, we query each collection and merge the results

When you merge the results, how do you sort them? Based on the text_match score value?

yes

The issue here is that the query paris is not an exact match with paris, france -- Typesense does not rank strings that are shorter ahead of strings that are longer. i.e. we only look at the number of tokens matched, whether there are typos and the number of fields matching in a record against the query.

Exact matching requires an exact match of the token, i.e. paris query will match a field with string Paris .

would multi search be a better option?

No, multi search will just return independent per-collection results -- it just parallelizes the query.

just a heads up, the name of the destination is just "Paris"

I see. In that case, can you give me a sample data set where this problem can be illustrated?

giving you a exact response json is ok?

Yes, exact JSON response of a search against a single collection is fine.

`

[
    {
        "document": {
            "attraction_name": "Mosquee de Paris",
            "coordinates": "nan",
            "country_id": "82",
            "country_name": "France",
            "destination_name": "Paris",
            "id": "880",
            "parent_destination_name": "Lhasa"
        },
        "highlights": [
            {
                "field": "destination_name",
                "matched_tokens": [
                    "Paris"
                ],
                "snippet": "<mark>Paris</mark>"
            },
            {
                "field": "attraction_name",
                "matched_tokens": [
                    "Paris"
                ],
                "snippet": "Mosquee de <mark>Paris</mark>"
            }
        ],
        "text_match": 2203368317191,
        "type": "attraction"
    },
    {
        "document": {
            "attraction_name": "The Paris Catacombs",
            "coordinates": "nan",
            "country_id": "82",
            "country_name": "France",
            "destination_name": "Paris",
            "id": "906",
            "parent_destination_name": "Lhasa"
        },
        "highlights": [
            {
                "field": "destination_name",
                "matched_tokens": [
                    "Paris"
                ],
                "snippet": "<mark>Paris</mark>"
            },
            {
                "field": "attraction_name",
                "matched_tokens": [
                    "Paris"
                ],
                "snippet": "The <mark>Paris</mark> Catacombs"
            }
        ],
        "text_match": 2203368317191,
        "type": "attraction"
    },
    {
        "document": {
            "attraction_name": "Eglise Saint-Etienne-du-Mont de Paris",
            "coordinates": "nan",
            "country_id": "82",
            "country_name": "France",
            "destination_name": "Paris",
            "id": "831",
            "parent_destination_name": "Lhasa"
        },
        "highlights": [
            {
                "field": "destination_name",
                "matched_tokens": [
                    "Paris"
                ],
                "snippet": "<mark>Paris</mark>"
            },
            {
                "field": "attraction_name",
                "matched_tokens": [
                    "Paris"
                ],
                "snippet": "Eglise Saint-Etienne-du-Mont de <mark>Paris</mark>"
            }
        ],
        "text_match": 2203368317191,
        "type": "attraction"
    },
    {
        "document": {
            "coordinates": "2.3522,48.8566",
            "country_id": "82",
            "country_name": "France",
            "destination_name": "Paris",
            "id": "42",
            "parent_destination_name": "Lhasa"
        },
        "highlights": [
            {
                "field": "destination_name",
                "matched_tokens": [
                    "Paris"
                ],
                "snippet": "<mark>Paris</mark>"
            }
        ],
        "text_match": 1103840043779,
        "type": "destination"
    },
    {
        "document": {
            "coordinates": "25.160855,37.080582",
            "country_id": "92",
            "country_name": "Greece",
            "destination_name": "Paros",
            "id": "986"
        },
        "highlights": [
            {
                "field": "destination_name",
                "matched_tokens": [
                    "Paros"
                ],
                "snippet": "<mark>Paros</mark>"
            }
        ],
        "text_match": 4328350465,
        "type": "destination"
    },
    {
        "document": {
            "coordinates": "10.3280833,44.8013678",
            "country_id": "114",
            "country_name": "Italy",
            "destination_name": "Parma",
            "id": "676"
        },
        "highlights": [
            {
                "field": "destination_name",
                "matched_tokens": [
                    "Parma"
                ],
                "snippet": "<mark>Parma</mark>"
            }
        ],
        "text_match": 4328284929,
        "type": "destination"
    },
    {
        "document": {
            "id": "211",
            "name": "Sri Lanka"
        },
        "highlights": [
            {
                "field": "name",
                "matched_tokens": [
                    "Sri"
                ],
                "snippet": "<mark>Sri</mark> Lanka"
            }
        ],
        "text_match": 33317888,
        "type": "country"
    },
    {
        "document": {
            "id": "196",
            "name": "San Marino"
        },
        "highlights": [
            {
                "field": "name",
                "matched_tokens": [
                    "Marino"
                ],
                "snippet": "San <mark>Marino</mark>"
            }
        ],
        "text_match": 33317888,
        "type": "country"
    },
    {
        "document": {
            "full_name": "LarisaNegreanu",
            "id": "98",
            "username": "larisa.negreanu"
        },
        "highlights": [
            {
                "field": "username",
                "matched_tokens": [
                    "larisa.negreanu"
                ],
                "snippet": "<mark>larisa.negreanu</mark>"
            },
            {
                "field": "full_name",
                "matched_tokens": [
                    "LarisaNegreanu"
                ],
                "snippet": "<mark>LarisaNegreanu</mark>"
            }
        ],
        "text_match": 4344668419,
        "type": "user"
    },
    {
        "document": {
            "full_name": "LarisaNegreanu",
            "id": "110",
            "username": "larisanegreanu"
        },
        "highlights": [
            {
                "field": "username",
                "matched_tokens": [
                    "larisanegreanu"
                ],
                "snippet": "<mark>larisanegreanu</mark>"
            },
            {
                "field": "full_name",
                "matched_tokens": [
                    "LarisaNegreanu"
                ],
                "snippet": "<mark>LarisaNegreanu</mark>"
            }
        ],
        "text_match": 4344668419,
        "type": "user"
    },
    {
        "document": {
            "full_name": "MariusIonescu",
            "id": "285",
            "username": "marius.ionescu"
        },
        "highlights": [
            {
                "field": "username",
                "matched_tokens": [
                    "marius.ionescu"
                ],
                "snippet": "<mark>marius.ionescu</mark>"
            },
            {
                "field": "full_name",
                "matched_tokens": [
                    "MariusIonescu"
                ],
                "snippet": "<mark>MariusIonescu</mark>"
            }
        ],
        "text_match": 4344471811,
        "type": "user"
    }
]

If I can't find anything obvious, I might still need a representative dataset that reproduces the issue so I can debug further locally.

here is our response after querring multiple collections, getting the top responses, then merging the results and sorting by text_match

Can you also please tell me your exact query?

one sec

Which record in that JSON you would like to appear first?

`

const searchParameters = {
    q: key,
    query_by: 'attraction_name, destination_name, parent_destination_name',
  }

I think the issue is with multi-field matching.

for countries is just
`

const searchParameters = {
    q: key,
    query_by: 'name',
  } 

Yes, the issue is because in countries collection you can have a match with only 1 single field, but when you query the other collection there can be many fields than can match.

this would be just query_by_weights: 1, in this case where we are querying for only one field

It would be query_by_weights: 10 when querying a single field but something like query_by_weights: 4,3,2 when querying multiple fields. The values will depend on your exact use case. The basic gist is using weights to control relative popularity.

we modified and got a better result, now we have users apearing higher as a score because they are indexed bu full name and username and they get 2 matches probably

A single field match will be treated as a lower score than two field.

We modified our queries to go after single field in the fact that we search throw all collections, after that sorting using the text match. Results are somewhat better but some results we don't understand
searching for romania we get
`

{
        "document": {
            "coordinates": "12.4964,41.9028",
            "country_id": "114",
            "country_name": "Italy",
            "destination_name": "Rome",
            "id": "44"
        },
        "highlights": [
            {
                "field": "destination_name",
                "matched_tokens": [
                    "Rome"
                ],
                "snippet": "<mark>Rome</mark>"
            }
        ],
        "text_match": 4328219393,
        "type": "destination"
    },
    {
        "document": {
            "id": "187",
            "name": "Romania"
        },
        "highlights": [
            {
                "field": "name",
                "matched_tokens": [
                    "Romania"
                ],
                "snippet": "<mark>Romania</mark>"
            }
        ],
        "text_match": 33514498,
        "type": "country"
    },
    {
        "document": {
            "attraction_name": "Museum of the National Bank of Romania",
            "coordinates": "nan",
            "country_id": "187",
            "country_name": "Romania",
            "destination_name": "Bucharest",
            "id": "4690"
        },
        "highlights": [
            {
                "field": "attraction_name",
                "matched_tokens": [
                    "Romania"
                ],
                "snippet": "Museum of the National Bank of <mark>Romania</mark>"
            }
        ],
        "text_match": 33514496,
        "type": "attraction"
    },

What's the weight you are using for the field destination_name ?

in the destinations collection, there is no weight on it. Only weight we have is on the country collection with a weight of 10

Can you please paste your updated query parameters again for all the 3 collections?

after conidering what you said this morning, i just changed the destination, instead of query for deatination_name and parent_destination_name, just one field

You mean combine the multiple fields to a single field now?

yeah, i modified the search in each collection to only use one field

Can you place the updated query params for each collection here? I can then see if I can explain the oddities.

ok

Got it. And the Romania example above is with a single search param correct? It's late here so I might get back to you tomorrow to resume this convo.

Ioan-Andrei Can you post the JSON response value of the first result record when you query for romania on just the destination collection against destination_name field?

We found out why the text match was hifh. It is because we were dearching for 2 fields. Query by one field returned normal

Cool 👍