This message was deleted.
# community-help
s
This message was deleted.
s
Posted here per @Kishore Nallan request. Applied this to
food
collection
Copy code
{
  "fields": [
    {
      "name": "foodBrandId",
      "drop": true
    },
    {
      "name": "foodBrandId",
      "type": "string",
      "optional": true,
      "facet": false,
      "sort": false,
      "reference": "foodBrands.id"
    }
  ]
}
Which it said was saved successfully, but I don't think the web UI knows how to show it yet, as that field disappears from the list of the schema when this change was applied (still seems to be found on the actual data). Trying to make subsequent changes shows: "Error: e: Request failed with HTTP code 400 | Server said: Field
foodBrandId
is not part of collection schema." so seems like all it ended up doing was dropping the field (I just added it back to schema). Furthermore in the sole example given, can we do it this direction for the reference instead to accomplish the same thing: Schema of
Products
collection:
Copy code
{
  "name": "Products",
  "fields": [
    {
      "name": "product_id",
      "type": "string",
      "reference": "CustomerProductPrices.product_id"
    },
    {
      "name": "product_name",
      "type": "string"
    },
    {
      "name": "product_description",
      "type": "string"
    }
  ]
}
CustomerProductPrices collectionโ€™s schema would store the reference to the product and its associated price:
Copy code
{
  "name": "CustomerProductPrices",
  "fields": [
    {
      "name": "customer_id",
      "type": "string"
    },
    {
      "name": "product_id",
      "type": "string"
    },
    {
      "name": "product_price",
      "type": "float"
    }
  ]
}
Or that wont work? As when I tried to add the reference in my case about, I get this when trying to query: "RequestMalformed: Request failed with HTTP code 400 | Server said: Failed to apply reference filter on
foodBrands
collection: Could not find any field in
foodBrands
referencing the collection
food
." However, I can't add the reference in the
foodBrands
needs to be foods, as many foods could ref/share the same brand, so it needs to be
food.foodBrandId
pointing to
foodBrands.id
[12:26 PM] My query as follows:
"filter_by": "$foodBrands(id:=foodBrandId)"
k
@Harpreet Sangar cc
๐Ÿ‘ 1
h
We don't support updating the fields with
reference
yet so you'll have to re-index the data. Regarding the
Products
collection having the reference field, it won't work since we would be searching for products. If I understood your problem correctly, you have
food
and
foodBrands
collections and you would want to search on
food
collection and filter by specific
foodBrands
?
s
What you mean, re-index? As in delete all the records in the TS collection, then reload them ?
h
Yeah
s
Ok understood, will that be changed later on, so we can update with a
reference
property and not have to delete everything ?
h
Yeah, I think this would be nice feature to support.
โœ… 1
s
Copy code
If I understood your problem correctly, you have food and foodBrands collections and you would want to search on food collection and filter by specific foodBrands?
Yes we have both collections. Both do get searched and currently
food
can filter by
foodBrandId
already. We want to add the reference to
foodBrands
so that when a food is returned (in any way) we can also grab the
foodBrands.name
for example from the other collection (currently food just has the foreign key).
h
Give me some time to think about this.
s
ok got it
Any ideas ?
h
Hey @Shaun Byrnes we have a few ideas but working out what would be the most efficient way to implement this. I'll keep you posted.
s
ok cool
๐Ÿ‘๐Ÿผ 1
Please come back to me when you know next
any progress @Harpreet Sangar
h
Hey Shaun, I'm currently working on adding support for sorting by a reference field. I'll implement your feature request next.
s
ah ok makes sense - the sort by ref field is also useful I think
Ok, I will await that being done before further testing
๐Ÿ‘ 1
h
We want to add the reference to
foodBrands
so that when a food is returned (in any way) we can also grab the
foodBrands.name
for example from the other collection (currently food just has the foreign key).
This should work by adding
include_fields: $foodBrands(name)
when searching on
food
collection. https://github.com/typesense/typesense/pull/1156
k
Let me build that change.
s
sweet ๐Ÿ˜„
k
I've published
0.26.0.rc14
where you can try this out.
s
ah ok sweet
any docs on way to use it? I dont see it as an option to upgrade to either
k
Haven't exposed it yet. We are a bit careful about exposing builds at the start of the cycle. Can you please tell me the cluster ID? I will upgrade it.
s
ok np
sh376gjtdxq8b9mop-1.a1.typesense.net [https: 443]
And any docs on how to use new features ?
k
This contains only the addition that Harpreet has mentioned above.
I'm running the upgrade now
s
thanks
but his messages on how to use are very limited
not sure I follow 100%
If you or he can elaborate would be good thanks
h
With the latest build you should be able to perform the following operation:
Copy code
{
  "name": "foods",
  "fields": [
    {
      "name": "name",
      "type": "string"
    },
    {
      "name": "food_brand_id",
      "type": "string",
      "reference": "foodBrands.id"
    },
    {
      "name": "calories",
      "type": "int32"
    }
  ]
}
Copy code
{
  "name": "foodBrands",
  "fields": [
    {
      "name": "brand_name",
      "type": "string"
    },
    {
      "name": "brand_category",
      "type": "string"
    }
  ]
}
Multi-search Query:
Copy code
{
  "collection": "foods",
  "query_by": "name",
  "q": "apple",
  "include_fields": "name,calories,$foodBrands(brandName)"
}
s
ok thanks will check it
tried to run this on the
food
collection:
Copy code
{
  "fields": [
    {
      "name": "foodBrandId",
      "drop": true
    },
    {
      "facet": false,
      "index": true,
      "infix": false,
      "locale": "",
      "name": "foodBrandId",
      "optional": true,
      "sort": false,
      "type": "string",
      "reference": "foodBrands.id"
    }
  ]
}
Got this returned:
Error: e: Request failed with HTTP code 400 | Server said: There are duplicate field names in the schema.
If I am not mistaken, the schema update I tried to run should be correct to be able to test, per instructions above? @Harpreet Sangar
But could also be TS cloud doesn't support this properly yet? As after I ran that, I now see: Error: 404 - Could not find a field named
foodBrandId
in the schema. When trying to even run the search
If I drop
foodBrandId
from searching field, results are returned, I think it's because the
drop
operation worked, but adding the field back with
reference
failed.
Despite the fact the
foodBrandId
still exists ...
What should I do next ?
k
I'm not sure we support schema updates for reference fields yet. I will recommend trying on a fresh set of collections created with the reference upfront.
โœ… 1
s
ah ok, so we cant use exisiting dataset to try test it on ?
k
Not yet. We have to wait for the join related API to stabilize a bit before we can implement alter support. For now you have to create collections fresh, reindex data and then can use join.
s
ok makes sense
will create a new collection to test against
and provide feedback
๐Ÿ‘ 1
I have tried creating 2 new test collections which have the reference in them as suggested, then run this:
Copy code
let searchParameters = {
    "q": "",
    "query_by": "name",
    "filter_by": "$brands1(uid:=brandId)"
  };
  
  let results = await client.collections("food1").documents().search(searchParameters);
I get RequestMalformed: Request failed with HTTP code 400 | Server said: Failed to apply reference filter on
brands1
collection: Could not find any field in
brands1
referencing the collection
food1
.
They are in my TS dev cluster:
sh376gjtdxq8b9mop-1.a1.typesense.net [https: 443]
food1 and brands1
Can you please check it @Kishore Nallan @Harpreet Sangar
k
We've a fix for this. Will share an updated build by tomorrow.
Can we upgrade your dev cluster to the latest build?
s
yes sure thing @Kishore Nallan thanks
has any other changes been made to it other than this fix I should be aware of? And does typesense cloud support references yet ?
to make changes to collections or still need to create the new ones from scratch for now
k
I'm running an upgrade now to
0.26.0.rc17
. It should complete shortly (you'll get an email). 1. Typesense Cloud UI still does not support references. 2. Schema alter still does not support adding references. We will get to both of those as soon as we iron out these issues we are finding. Sorry and thank you for the support and patience!
s
yeah i got the email just now
1 & 2 - understood
Yeah, makes total sense - get the lib working correctly first before update the web UI/UX
Still showing this RequestMalformed: Request failed with HTTP code 400 | Server said: Could not find a reference to
food1
in
brands1
collection.
is there anything else I have to do?
k
Did you create your collections fresh?
s
not since the upgrade was performed - i just tried to re-run the test query I had setup from before
k
Can you try dropping existing collections and try creating from scratch?
s
can do yea
Just re-created and added some test records and same error when I re-run
do you want to check the collections/data out and see if there is anything I have done wrong ?
brands1
and
food1
k
Ok let me look and get back to you.
s
cheers
h
Hey Shaun could you share your collection schema and query that you are trying?
s
Sure
brands1-schema-06-09-2023--15-25-57.json
These are the 2 test collections
Copy code
let searchParameters = {
    "q": "",
    "query_by": "name",
    "filter_by": "$brands1(uid:=brandId)"
  };
  
  let results = await client.collections("food1").documents().search(searchParameters);

  console.dir(results, { depth: null });
h
Thanks I'll try to reproduce the issue. AFK right now.
s
ok
thanks
h
Isn't
brandId
already present in
food1
collection? you can do
Copy code
let searchParameters = {
    "q": "",
    "query_by": "name",
    "filter_by": "brandId:=foo"
  };
If you want to include
brands1.name
, you can do
Copy code
let searchParameters = {
    "q": "",
    "query_by": "name",
    "filter_by": "brandId:=foo",
    "include_fields": "$brands1(name)"
  };
Just realized there's a bug in
include_fields
for this scenario, you'll have to send
*, $brands1(name)
in the meantime.
s
Got it
Thanks for the clarification, when you think that bug getting fixed @Harpreet Sangar?
Also, when I try as suggested:
Copy code
let searchParameters = {
    "q": "",
    "query_by": "name",
    "include_fields": "*, $brands1(name)"
  };
  
  let results = await client.collections("food1").documents().search(searchParameters);
the issue seems to be the
food1.name
is overwritten by
brands1.name
๐Ÿ˜•
Copy code
{
  facet_counts: [],
  found: 2,
  hits: [
    {
      document: {
        brandId: '1',
        brandId_sequence_id: 0,
        name: 'Brand one',
        uid: '2'
      },
      highlight: {},
      highlights: [],
      text_match: 100,
      text_match_info: {
        best_field_score: '0',
        best_field_weight: 12,
        fields_matched: 4,
        score: '100',
        tokens_matched: 0
      }
    },
    {
      document: {
        brandId: '1',
        brandId_sequence_id: 0,
        name: 'Brand one',
        uid: '1'
      },
      highlight: {},
      highlights: [],
      text_match: 100,
      text_match_info: {
        best_field_score: '0',
        best_field_weight: 12,
        fields_matched: 4,
        score: '100',
        tokens_matched: 0
      }
    }
  ],
  out_of: 2,
  page: 1,
  request_params: { collection_name: 'food1', per_page: 10, q: '' },
  search_cutoff: false,
  search_time_ms: 23
}
k
Sorry about this. Now that we have the exact schema and query, we will spend more time addressing this issue and testing it further before getting back to you again with a new build. We will get back to you mid next week.
โœ… 1
s
Notice the
name: 'Brand one'
coming back for each is correct, but the
food1.name
values are not returned
THanks @Kishore Nallan
My thoughts on this is maybe if the additional data is included via
$collectionName(prop1, prop2)
we should have it come back as
collectionName.propN
in the document obj? This way it prevents same name properties from being clobbered
Like if we do
"include_fields": "*, $brands1(name, uid)"
both the name and uid field in food1 are overwritten with the brands1 name+uid values
k
Yeah this is something to think about. We might need an equivalent of SQL's "as" keyword.
โœ… 1
s
yeah exactly
you could do something like
"include_fields": "* as f, $brands1(name, uid) as b"
maybe
๐Ÿ‘ 1
especially seeing if we have multiple collections included in a query, the likelyhood of common field names like name,id etc clashing is high:
"include_fields": "* as food, $foodBrands(name, uid) as brands, $userFoods(count) as userFoods"
how are we going with this @Kishore Nallan @Harpreet Sangar
k
Hi Shaun, we're still working on this.
๐Ÿ‘€ 1
s
ok thanks for the update
k
We've published a new build
0.26.0.rc18
that will address the issues you faced. Can I upgrade the
sh376gjtdxq8b9mop
cluster?
s
yes please
let me know if there is anything else in the new release we should be aware of or need to do different with the query code
@Kishore Nallan
h
Hey Shaun, we've added the ability to provide an alias for the reference include fields. So this would work now:
"include_fields": "$foodBrands(name, uid) as brands, $userFoods(count) as userFoods"
Also, you'll be able to join on
foodBrands
collection when searching on
foods
โœ… 1
k
sh376gjtdxq8b9mop
has now been upgraded.
s
yep saw the email thanks
Will give the query a try that @Harpreet Sangar suggested and let you know
๐Ÿ‘ 1
๐Ÿ‘๐Ÿผ 1
Hey guys, sorry for the delay on testing this further and providing feedback. We have now done some more testing and have the following:
Copy code
let searchParameters = {
    "q": "",
    "query_by": "name",
    "include_fields": "$brands1(name) as brands, $userFood1(count, userId) as userFood"
  };
  
  let results = await client.collections("food1").documents().search(searchParameters);
Copy code
{
  facet_counts: [],
  found: 2,
  hits: [
    {
      document: {
        brandId: '1',
        brandId_sequence_id: 0,
        'brands.name': 'Brand one',
        id: '1',
        name: 'Food 2',
        uid: '2'
      },
      highlight: {},
      highlights: [],
      text_match: 100,
      text_match_info: {
        best_field_score: '0',
        best_field_weight: 12,
        fields_matched: 4,
        score: '100',
        tokens_matched: 0
      }
    },
    {
      document: {
        brandId: '1',
        brandId_sequence_id: 0,
        'brands.name': 'Brand one',
        id: '0',
        name: 'Food 1',
        uid: '1'
      },
      highlight: {},
      highlights: [],
      text_match: 100,
      text_match_info: {
        best_field_score: '0',
        best_field_weight: 12,
        fields_matched: 4,
        score: '100',
        tokens_matched: 0
      }
    }
  ],
  out_of: 2,
  page: 1,
  request_params: { collection_name: 'food1', per_page: 10, q: '' },
  search_cutoff: false,
  search_time_ms: 0
}
Brands is working well and results are returned fine as you can see above, but for some reason there is no
userFood
data being returned - doesnt error out either ๐Ÿ˜•
@Kishore Nallan @Harpreet Sangar can you take a look and see if i am missing something? I have created a schema for the
userFood1
collection and loaded some dummy docs into it with a reference to
food1.uid
h
There should've been one more
*_sequence_id
field like
brandId_sequence_id
in the response. My guess is for some reason the reference to
food1
is not created.
Can you share the schemas and a few documents so I can check where the issue is?
s
sure thing
Those are the docs in it
h
If I remember correctly, this collection is supposed to store the user's favorite foods, right?
s
store stats about food usage per user
so right now i just made a test collection as a simulation
which has a
count
field with some int data in it
so show there was some "amount" per userId
h
Okay. Since the
food1
collection doesn't have the
reference
field to
userFood1
, you'll have to specify a "join" in the
filter_by
like:
Copy code
"filter_by": "$userFood1(userId:foo)",
to get the data of a particular user or
Copy code
"filter_by": "$userFood1(id:*)",
to get all the related docs.
Also, you should upgrade to
v0.26.0.rc25
. I'm not sure if
v0.26.0.rc11
supports the
id: *
syntax.
s
ok ill do the upgrade now
I have a reference field in the userFood1 collection tho
isnt that is what needed as I am using the
$collectionName()
syntax which is where the reference has to be ?
I cant select rc25 either - newest is rc24
which is what I am going to
โœ… 1
So should I remove the
Copy code
"reference": "food1.uid",
from userFoods1
and put that in the
food1
collection instead ?
h
A single user can be related to multiple foods here, right?
s
basically a food (such a apple) could have multiple users "use" it, which would increase the count
userA <> apple (2) userB <> apple (100)
so we have users, foods, userFoods
userFoods would be a join table in a SQL DB
with foreign keys userId and foodId
So could do users.id users.name userFoods.count (userFoods.userId = users.id & userFoods.foodId = food.id) food.id food.name
I updated to rc24 and then tried:
"include_fields": "$brands1(name) as brands, $userFood1(foodId:*) as userFood"
Doesnt seem to do anything
h
Understood. That change was supposed to go in
filter_by
. Try this out:
Copy code
let searchParameters = {
    "q": "",
    "query_by": "name",
    "filter_by": "$userFood1(id:*)",
    "include_fields": "$brands1(name) as brands, $userFood1(count, userId) as userFood"
  };
  
  let results = await client.collections("food1").documents().search(searchParameters);
k
Let me enable rc26 which is the latest so we have the latest reference point
s
ok
thanks
k
Done, it's actually
0.26.0.rc25
s
yep
upgrading it now
done
Same issue when I run:
"include_fields": "$brands1(name) as brands, $userFood1(foodId:*) as userFood"
Dont get any diff results returned
h
That change was supposed to go in
filter_by
. Try this out:
Copy code
let searchParameters = {
"q": "",
"query_by": "name",
"filter_by": "$userFood1(id:*)",
"include_fields": "$brands1(name) as brands, $userFood1(count, userId) as userFood"
};
  
let results = await client.collections("food1").documents().search(se
s
Sorry, my bad
will adjust and try again
Copy code
{
  facet_counts: [],
  found: 2,
  hits: [
    {
      document: {
        brandId: '1',
        brandId_sequence_id: 0,
        'brands.name': 'Brand one',
        id: '1',
        name: 'Food 2',
        uid: '2',
        'userFood.count': [ 10, 1 ],
        'userFood.userId': [ 's', 'b' ]
      },
      highlight: {},
      highlights: [],
      text_match: 100,
      text_match_info: {
        best_field_score: '0',
        best_field_weight: 12,
        fields_matched: 4,
        score: '100',
        tokens_matched: 0
      }
    },
    {
      document: {
        brandId: '1',
        brandId_sequence_id: 0,
        'brands.name': 'Brand one',
        id: '0',
        name: 'Food 1',
        uid: '1',
        'userFood.count': 5,
        'userFood.userId': 's'
      },
      highlight: {},
      highlights: [],
      text_match: 100,
      text_match_info: {
        best_field_score: '0',
        best_field_weight: 12,
        fields_matched: 4,
        score: '100',
        tokens_matched: 0
      }
    }
  ],
  out_of: 2,
  page: 1,
  request_params: { collection_name: 'food1', per_page: 10, q: '' },
  search_cutoff: false,
  search_time_ms: 0
}
now we getting somewhere!
๐Ÿ™Œ๐Ÿผ 1
Copy code
let searchParameters = {
    "q": "",
    "query_by": "name",
    "filter_by": "$userFood1(id:*)",
    "include_fields": "$brands1(name) as brands, $userFood1(count, userId) as userFood"
  };
  
  let results = await client.collections("food1").documents().search(searchParameters);
But what is this for:
"$userFood1(id:*)",
i dont get what we are supposed to put in there ? id:* ? how is this syntax supposed to work
h
Since the
food1
collection doesn't have the
reference
field to
userFood1
, we have to specify a "join" in the
filter_by
like:
Copy code
"filter_by": "$userFood1(userId:foo)",
to get the data of a particular user or
Copy code
"filter_by": "$userFood1(id:*)",
to get all the related docs.
s
so id:* will give those arrays
where it does 1 <> many
but if i change to userId: abc
h
Or any other filter query like
Copy code
"filter_by": "$userFood1(count:>100)",
s
then it would only return matches for that user?
โœ… 1
got it
so can use any supported filters in that query inside there
h
Yeah
s
what if i remove the reference in the userFood collection to food
and put it in food collection
do we still need that ?
does thtat mean we can leave references out of the collection config and just join anything on the fly ?
as previously a reference field in the collection schema was requirement
h
what if i remove the reference in the userFood collection to food
and put it in food collection
Then the "join" won't be necessary in the
filter_by
but I think in your scenario it wouldn't make sense to do that.
s
Oh and why's that? Currently I have this in the
userFood1
collection:
Copy code
{
      "facet": false,
      "index": true,
      "infix": false,
      "locale": "",
      "name": "foodId",
      "optional": false,
      "reference": "food1.uid",
      "sort": false,
      "type": "string"
    },
Do i even need the reference: food1.uid ?
h
Yeah, it's required. Check the response of this request
Copy code
let searchParameters = {
    "q": "",
    "query_by": "name",
    "filter_by": "$userFood1(id:*)",
    "include_fields": "$brands1(name) as brands, $userFood1(*) as userFood"
  };
  
  let results = await client.collections("food1").documents().search(searchParameters);
You'll see
userFood.foodId_sequence_id
field that's used internally to link the referenced docs.
s
mmm
just has:
Copy code
document: {
        brandId: '1',
        brandId_sequence_id: 0,
        'brands.name': 'Brand one',
        id: '0',
        name: 'Food 1',
        uid: '1',
        'userFood.count': 5,
        'userFood.userId': 's'
      },
I see the
brandId_sequence_id
but no
foodId_sequence_id
h
I think you did not update the query. I see that field is present in your screenshot you shared earlier.
s
I just dropped the foodId field in the userFood collection which had the reference and re-added
without reference
and get the same results with the filter_by you suggested
h
Because
foodId_sequence_id
field is still present in the
userFood
docs that have already been created. Try indexing new docs in
userFood
, they won't be returned in the response..
s
how do i force a re-index on the userFood ?
h
At the moment you cannot. You'll have to re-create
userFood
s
ok got it
I get
ObjectNotFound: Request failed with HTTP code 404 | Server said: Could not find a field named
userFood.count
in the schema for sorting.
When trying to add: "sort_by": "userFood.count:desc"
h
The syntax is
$userFood(count:desc)
s
gotcha
thanks
๐Ÿ™Œ๐Ÿผ 1
obv cos this is still new in testing there is no docs
so alot is trying to guess the syntax
h
k
As a general guideline, whenever you want to reference anything on a reference collection, you have to use the
$
syntax. Like
$ReferenceCollection(field_name)
.
s
yeah
I now get
RequestMalformed: Request failed with HTTP code 400 | Server said: Multiple references found to sort by on
userFood1.count
.
With the sort, when there is an array of results
I have:
Copy code
"q": "",
    "query_by": "name",
    "filter_by": "$userFood1(id:*)",
    "include_fields": "$brands1(name) as brands, $userFood1(count, userId) as userFood",
    "sort_by": "$userFood1(count:desc)"
Bascially trying to sort by all userFood counts highest to lowest
regardless of user
(show most popular first)
if I drop the sort its ok, and simply returns like:
Copy code
{
      document: {
        brandId: '2',
        brandId_sequence_id: 1,
        'brands.name': 'Brand two',
        id: '2',
        name: 'Food 3',
        uid: '3',
        'userFood.count': 11,
        'userFood.userId': 'b'
      },
      highlight: {},
      highlights: [],
      text_match: 100,
      text_match_info: {
        best_field_score: '0',
        best_field_weight: 12,
        fields_matched: 4,
        score: '100',
        tokens_matched: 0
      }
    },
    {
      document: {
        brandId: '1',
        brandId_sequence_id: 0,
        'brands.name': 'Brand one',
        id: '1',
        name: 'Food 2',
        uid: '2',
        'userFood.count': [ 5, 3 ],
        'userFood.userId': [ 's', 'b' ]
      },
      highlight: {},
      highlights: [],
      text_match: 100,
      text_match_info: {
        best_field_score: '0',
        best_field_weight: 12,
        fields_matched: 4,
        score: '100',
        tokens_matched: 0
      }
    },
    {
      document: {
        brandId: '1',
        brandId_sequence_id: 0,
        'brands.name': 'Brand one',
        id: '0',
        name: 'Food 1',
        uid: '1',
        'userFood.count': [ 10, 0 ],
        'userFood.userId': [ 's', 'b' ]
      },
      highlight: {},
      highlights: [],
      text_match: 100,
      text_match_info: {
        best_field_score: '0',
        best_field_weight: 12,
        fields_matched: 4,
        score: '100',
        tokens_matched: 0
      }
    }
but in that way I dont get the results ordered correctly
h
This is because we can't determine the sort score of the document when there are multiple values present in it. For example,
'userFood.count': [ 10, 0 ],
and
'userFood.count': [ 5, 3 ],
Bascially trying to sort by all userFood counts highest to lowest
You can try
Copy code
"q": "*",
"filter_by": "$food1(id:*)",
"include_fields": "$brands1(name) as brands, $food1(brandId, id, name, uid) as foods",
"sort_by": "count:desc"
Copy code
let results = await client.collections("userFood1").documents().search(searchParameters);
s
yeah but that doesnt return brand names:
Copy code
{
  facet_counts: [],
  found: 5,
  hits: [
    {
      document: {
        count: 11,
        'food.brandId': '2',
        'food.id': '2',
        'food.name': 'Food 3',
        'food.uid': '3',
        foodId: '3',
        foodId_sequence_id: 2,
        id: '4',
        userId: 'b'
      },
      highlight: {},
      highlights: []
    },
    {
      document: {
        count: 10,
        'food.brandId': '1',
        'food.id': '0',
        'food.name': 'Food 1',
        'food.uid': '1',
        foodId: '1',
        foodId_sequence_id: 0,
        id: '0',
        userId: 's'
      },
      highlight: {},
      highlights: []
    },
    {
      document: {
        count: 5,
        'food.brandId': '1',
        'food.id': '1',
        'food.name': 'Food 2',
        'food.uid': '2',
        foodId: '2',
        foodId_sequence_id: 1,
        id: '1',
        userId: 's'
      },
      highlight: {},
      highlights: []
    },
    {
      document: {
        count: 3,
        'food.brandId': '1',
        'food.id': '1',
        'food.name': 'Food 2',
        'food.uid': '2',
        foodId: '2',
        foodId_sequence_id: 1,
        id: '3',
        userId: 'b'
      },
      highlight: {},
      highlights: []
    },
    {
      document: {
        count: 0,
        'food.brandId': '1',
        'food.id': '0',
        'food.name': 'Food 1',
        'food.uid': '1',
        foodId: '1',
        foodId_sequence_id: 0,
        id: '2',
        userId: 'b'
      },
      highlight: {},
      highlights: []
    }
  ],
  out_of: 5,
  page: 1,
  request_params: { collection_name: 'userFood1', per_page: 10, q: '*' },
  search_cutoff: false,
  search_time_ms: 0
}
h
I'll test this locally in some time.
s
ok
thanks
Also, in this example here:
Copy code
let searchParameters = {
    "q": "",
    "query_by": "name",
    "filter_by": "$userFood1(userId:s)",
    "include_fields": "$brands1(name) as brands, $userFood1(count, userId) as userFood",
    "sort_by": "$userFood1(count:desc)"
  };
  
  let results = await client.collections("food1").documents().search(searchParameters);
HOw to chnage "query_by": "name" to query other collection fields?
Such as to query
$brands1(name)
(fails)
"query_by": "name, brands.name", ObjectNotFound: Request failed with HTTP code 404 | Server said: Could not find a field named
brands.name
in the schema.
"query_by": "name, $brands1(name)", Error: timeout of 3000ms exceeded at createError (/Users/shaunbyrnes/Software/shredded-platform/shredded-firebase/utils/temp/node_modules/axios/lib/core/createError.js1615) at RedirectableRequest.handleRequestTimeout (/Users/shaunbyrnes/Software/shredded-platform/shredded-firebase/utils/temp/node_modules/axios/lib/adapters/http.js36916)
h
That is not supported at the moment. You can try
filter_by: $brands(name: foo)
s
gotcha np
ok ill try that
pretty sure you dont have any other functions supported currently like
min() max() count() sum() avg()
etc right ?
that we can use in query syntax
h
Not yet, no.
s
ok np
would be a great feature in future, once the joins is working
allow us to do some pretty cool stuff
๐Ÿ™‚
h
Yeah. Next in line is
facet_by
reference field and then other features.
s
oh ok got it - how will the facet_by reference is supposed to work ?
Also, I have the following to the
food
collection schema:
Copy code
{
  "facet": false,
  "index": true,
  "infix": false,
  "locale": "",
  "name": "portions",
  "optional": true,
  "sort": false,
  "type": "object[]"
}
which produces this type of output when returning results
Copy code
{
  facet_counts: [],
  found: 2,
  hits: [
    {
      document: {
        brandId: '1',
        brandId_sequence_id: 0,
        'brands.name': 'Brand one',
        id: '0',
        name: 'Food 1',
        portions: [ { portionId: 'a' } ],
        uid: '1',
        'userFood.count': 10,
        'userFood.userId': 's'
      },
      highlight: {},
      highlights: [],
      text_match: 100,
      text_match_info: {
        best_field_score: '0',
        best_field_weight: 12,
        fields_matched: 4,
        score: '100',
        tokens_matched: 0
      }
    },
    {
      document: {
        brandId: '1',
        brandId_sequence_id: 0,
        'brands.name': 'Brand one',
        id: '1',
        name: 'Food 2',
        portions: [ { portionId: 'c' }, { portionId: 'b' } ],
        uid: '2',
        'userFood.count': 5,
        'userFood.userId': 's'
      },
      highlight: {},
      highlights: [],
      text_match: 100,
      text_match_info: {
        best_field_score: '0',
        best_field_weight: 12,
        fields_matched: 4,
        score: '100',
        tokens_matched: 0
      }
    }
  ],
  out_of: 3,
  page: 1,
  request_params: { collection_name: 'food1', per_page: 10, q: '' },
  search_cutoff: false,
  search_time_ms: 0
}
Is it currently possible to have
portions[].portionId
match the
portions
collection via reference field ?
h
how will the facet_by reference is supposed to work ?
For example, faceting on brand name when searching on foods.
Is it currently possible to have
portions[].portionId
match the
portions
collection via reference field ?
You'll have to include this field in the schema, like:
Copy code
"name": "portions.portionId", "type": "string[]", "reference": "foo.bar"
Is there a reason for having an array of objects though?
โœ… 1
s
Ah nice on the facet by - would be good
Well each one of those portionId's represents the possible portion sizes for a food, we have the actual portion data in another collection, which has the portion name for example
So we would want to match
food.portions.portionId = portions.id
so we can get the
portions.name
h
Wouldn't having a
string[]
field of just portion ids work?
s
what you mean sorry?
Source data look like:
Copy code
"portionSize": {
    "displayFormat": 0,
    "portionsPerItem": null,
    "selectedPortionIndex": 1,
    "unit": "g",
    "value": 1
  },
  "portions": [
    {
      "amount": 100,
      "measureUnitId": "REzO2pZ0K1J6MFrCnWAi",
      "multiple": 1,
      "portionSizeId": "GEMDwd7SBYFZ6T9Ui0s2",
      "type": 0
    },
    {
      "amount": 604,
      "measureUnitId": "GmwdaGtivG85KDyjohtj",
      "multiple": 6.04,
      "portionSizeId": "9UvEjhgOXKTITezpsPGu",
      "type": 0
    }
  ],
h
Oh, I thought there was only one field
portionId
in each object. So I suggested this
Copy code
portionIds: [ 'c', 'b' ],
s
yes, but we can use array of objects as our current structure correct ?
Seems to work when i deleted temp collection of food1 and remade it with reference for portions.portionId as string[]
Copy code
{
  facet_counts: [],
  found: 2,
  hits: [
    {
      document: {
        brandId: '1',
        brandId_sequence_id: 0,
        'brands.name': 'Brand one',
        id: '0',
        name: 'Food 1',
        'portionSizes.name': [ '1 cup', '1 glass' ],
        'portions.portionId': [ 'a', 'b' ],
        'portions.portionId_sequence_id': [ 0, 1 ],
        uid: '1',
        'userFood.count': 10,
        'userFood.userId': 's'
      },
      highlight: {},
      highlights: [],
      text_match: 100,
      text_match_info: {
        best_field_score: '0',
        best_field_weight: 12,
        fields_matched: 4,
        score: '100',
        tokens_matched: 0
      }
    },
    {
      document: {
        brandId: '1',
        brandId_sequence_id: 0,
        'brands.name': 'Brand one',
        id: '1',
        name: 'Food 2',
        'portionSizes.name': [ '1 glass', '100ml' ],
        'portions.portionId': [ 'b', 'c' ],
        'portions.portionId_sequence_id': [ 1, 2 ],
        uid: '1',
        'userFood.count': 5,
        'userFood.userId': 's'
      },
      highlight: {},
      highlights: [],
      text_match: 100,
      text_match_info: {
        best_field_score: '0',
        best_field_weight: 12,
        fields_matched: 4,
        score: '100',
        tokens_matched: 0
      }
    }
  ],
  out_of: 3,
  page: 1,
  request_params: { collection_name: 'food1', per_page: 10, q: '' },
  search_cutoff: false,
  search_time_ms: 0
}
Based on this query:
Copy code
let searchParameters = {
    "q": "",
    "query_by": "name",
    "filter_by": "$userFood1(userId:s)",
    "include_fields": "$brands1(name) as brands, $userFood1(count, userId) as userFood, $portionSizes1(name) as portionSizes",
    "sort_by": "$userFood1(count:desc)"
  };
Except it wants this data format:
Copy code
[
  {
    "uid": "String Value",
    "name": "String Value",
    "brandId": "String Value",
    "portions.portionId": [
      "String Value 1 ",
      "String Value 2"
    ]
  },
  {
    "uid": "String Value",
    "name": "String Value",
    "brandId": "String Value",
    "portions.portionId": [
      "String Value 1 ",
      "String Value 2"
    ]
  }
]
Which is just an array of strings, not an array of objects
Can we have it as:
Copy code
[
  {
    "uid": "String Value",
    "name": "String Value",
    "brandId": "String Value",
    "portions": [
      { "portionId": "String Value 1 ", "other": 1},
      { "portionId": "String Value 2", "other": 2}
    ]
  },
  {
    "uid": "String Value",
    "name": "String Value",
    "brandId": "String Value",
    "portions": [
      { "portionId": "String Value 1 ", "other": 1},
      { "portionId": "String Value 2", "other": 2}
    ]
  }
]
As this is what we need
h
Yeah, we can definitely return an array of reference objects in the response but we'll need a flag in the
include_by
syntax to specify this.
s
ok so how should i structure the schema when creating it and how should I make query look ?
I followed what you suggested and this is current schema:
Copy code
{
  "created_at": 1698141520,
  "default_sorting_field": "",
  "enable_nested_fields": true,
  "fields": [
    {
      "facet": false,
      "index": true,
      "infix": false,
      "locale": "",
      "name": "uid",
      "optional": false,
      "sort": false,
      "type": "string"
    },
    {
      "facet": false,
      "index": true,
      "infix": false,
      "locale": "",
      "name": "name",
      "optional": false,
      "sort": false,
      "type": "string"
    },
    {
      "facet": false,
      "index": true,
      "infix": false,
      "locale": "",
      "name": "brandId",
      "optional": false,
      "reference": "brands1.uid",
      "sort": false,
      "type": "string"
    },
    {
      "facet": false,
      "index": true,
      "infix": false,
      "locale": "",
      "name": "portions.portionId",
      "optional": false,
      "reference": "portionSizes1.uid",
      "sort": false,
      "type": "string[]"
    }
  ],
  "name": "food1",
  "num_documents": 3,
  "symbols_to_index": [],
  "token_separators": []
}
This is current query:
Copy code
let searchParameters = {
    "q": "",
    "query_by": "name",
    "filter_by": "$userFood1(userId:s)",
    "include_fields": "$brands1(name) as brands, $userFood1(count, userId) as userFood, $portionSizes1(name) as portionSizes",
    "sort_by": "$userFood1(count:desc)"
  };
  
  let results = await client.collections("food1").documents().search(searchParameters);
h
There should be no change required in the schema. Let me think about what can be done in the
include_fields
syntax.
s
Got it. But currently the portions.portionId is just an array of strings not an array of objects with string field: portionId in each object ๐Ÿค”
h
An object[] field is indexed as
type[]
internally with the name
object_name.field_name
.
s
So have I got it correct? Why are they then being saved as an array of strings not array of objects? Just seeing what I need to change for our source data which is like below, to work and be able to translate the portionSizeIds by reference: Source data look like:
Copy code
"portionSize": {
    "displayFormat": 0,
    "portionsPerItem": null,
    "selectedPortionIndex": 1,
    "unit": "g",
    "value": 1
  },
  "portions": [
    {
      "amount": 100,
      "measureUnitId": "REzO2pZ0K1J6MFrCnWAi",
      "multiple": 1,
      "portionSizeId": "GEMDwd7SBYFZ6T9Ui0s2",
      "type": 0
    },
    {
      "amount": 604,
      "measureUnitId": "GmwdaGtivG85KDyjohtj",
      "multiple": 6.04,
      "portionSizeId": "9UvEjhgOXKTITezpsPGu",
      "type": 0
    }
  ],
k
Each field inside a nested object or nested array of objects are flattened and stored as primitive field types.
s
Got it
But will our requirement work ?
k
Sorry I haven't followed the full convo above, can you summarise what you are trying to achieve? One gotcha / limitation with the current filtering is that there is no way to filter within a given array of objects because filtering is applied globally across records. See this: https://github.com/typesense/typesense/issues/828
s
We arenโ€™t talking about filtering at all
If you read Above there is the example of our source data
And itโ€™s regarding having an array of objects where we want to use the reference field
To match the foreign key IDs in each obj
With the other collection which needs the join
Itโ€™s explained above if you read the last messages
k
Got it, looks we have to account for nesting while formatting the response. We will look into it.
s
Ok let me know
h
@Shaun Byrnes can you provide an example with curl commands so I can better understand the situation?
s
i provided the exact scenario just above ?
Did you read the messages I sent?
@Harpreet Sangar @Kishore Nallan I have re-summarised the question/issue again below. As I said in the other messages above, I have the following schema, based on Harpreet suggestion:
Copy code
...
,
    {
      "facet": false,
      "index": true,
      "infix": false,
      "locale": "",
      "name": "portions.portionId",
      "optional": false,
      "reference": "portionSizes1.uid",
      "sort": false,
      "type": "string[]"
    }
  ],
  "name": "food1",
...
which produces this in TS:
Copy code
[
  {
    "uid": "String Value",
    "name": "String Value",
    "brandId": "String Value",
    "portions.portionId": [
      "String Value 1 ",
      "String Value 2"
    ]
  },
...
]
With the above, I am able to reference the portion size names based on
"reference": "portionSizes1.uid",
as shown here in my results:
Copy code
document: {
        brandId: '1',
        brandId_sequence_id: 0,
        'brands.name': 'Brand one',
        id: '1',
        name: 'Food 2',
        'portionSizes.name': [ '1 glass', '100ml' ],
        'portions.portionId': [ 'b', 'c' ],
        'portions.portionId_sequence_id': [ 1, 2 ],
        uid: '1',
        'userFood.count': 5,
        'userFood.userId': 's'
      },
However, with that schema, TS is altering my source format of an array of objects to:
Copy code
"portions.portionId": [
      "portionSizeId1",
      "portionSizeId2"
    ]
Whereas my source data looks like this:
Copy code
"portions": [
    {
      "amount": 100,
      "measureUnitId": "REzO2pZ0K1J6MFrCnWAi",
      "multiple": 1,
      "portionSizeId": "GEMDwd7SBYFZ6T9Ui0s2",
      "type": 0
    },
    {
      "amount": 604,
      "measureUnitId": "GmwdaGtivG85KDyjohtj",
      "multiple": 6.04,
      "portionSizeId": "9UvEjhgOXKTITezpsPGu",
      "type": 0
    }
  ],
As you can see our data being saved to TS from the source DB is an array of objects not an array of strings. We need to maintain the other values in each "portion" object under
portions[]
with the intent being to create reference to
portionSizeId
in each object, as that is matched to our TS
portions
collection, so we can obtain the name for each portion size. As you can see in the results returned we are able to get the name matched (per Harpreet suggested query), but the schema is incorrect. Wondering how to keep the array of objects with each
portionSizeId
referenced and matched to the
portions
collection? The query we are using is as follows:
Copy code
let searchParameters = {
    "q": "",
    "query_by": "name",
    "filter_by": "$userFood1(userId:s)",
    "include_fields": "$brands1(name) as brands, $userFood1(count, userId) as userFood, $portionSizes1(name) as portionSizes",
    "sort_by": "$userFood1(count:desc)"
  };
  
  let results = await client.collections("food1").documents().search(searchParameters);
So not sure if this is normal behaviour etc, but looking at the docs on https://typesense.org/docs/0.25.1/api/collections.html#notes-on-indexing-common-types-of-data isn't super clear how exactly to structure the schema for type
object
or
object[]
We actually originally has
object[]
for the
portions.portionId
thinking it should be that way given an array of objects which each contain a
portionSizeId
but Harpreet said to write it as above. Hopefully this is all clear now what we are asking?
h
What I'm unable to understand is that you've declared
portions.portionId
in your
food1
schema but your source data does not contain
portionId
field in it.
Copy code
"portions": [
    {
      "amount": 100,
      "measureUnitId": "REzO2pZ0K1J6MFrCnWAi",
      "multiple": 1,
      "portionSizeId": "GEMDwd7SBYFZ6T9Ui0s2",
      "type": 0
    },
    {
      "amount": 604,
      "measureUnitId": "GmwdaGtivG85KDyjohtj",
      "multiple": 6.04,
      "portionSizeId": "9UvEjhgOXKTITezpsPGu",
      "type": 0
    }
  ],
Even then somehow the docs end up with these values
Copy code
"portions.portionId": [
      "portionSizeId1",
      "portionSizeId2"
    ]
I'm trying these commands to index the data
Copy code
curl "http://localhost:8108/collections" \
       -X POST \
       -H "Content-Type: application/json" \
       -H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}" \
       -d '{
  "name": "food1",
  "fields": [
    {
      "name": "portions.portionId",
      "reference": "portionSizes1.uid",
      "type": "string[]"
    }
  ],
  "enable_nested_fields": true
}'

curl "http://localhost:8108/collections/food1/documents/import?action=create" \
        -H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}" \
        -H "Content-Type: text/plain" \
        -X POST \
        -d '{"portions": [{"amount": 100,"measureUnitId": "REzO2pZ0K1J6MFrCnWAi","multiple": 1,"portionSizeId": "GEMDwd7SBYFZ6T9Ui0s2","type": 0},{"amount": 604,"measureUnitId": "GmwdaGtivG85KDyjohtj","multiple": 6.04,"portionSizeId": "9UvEjhgOXKTITezpsPGu","type": 0}]}'
As expected, I get:
Copy code
"error":"Missing the required reference field `portions.portionId` in the document."
s
Gotcha
So what I wrote up to summarise made sense this time at least?
As we are getting somewhat of a reference name from the portions collection, returned in the results, funnily enough... As you can see, we got the portion names referenced through in the
'portionSizes.name': [ '1 glass', '100ml' ],
array... Why would that have happened then? Furthermore, because the TS cloud currently doesnt support changes with the reference fields to my knowledge, I expect to fix the
portions.portionId
>
portions.portionSizeId
change, we would need to delete the test
food1
collection and re-create with correct field name, correct?
k
Shuan, just to be sure we are all on the same page, would it be possible for you to provide a sample dataset, the collection schema and the query? This way we can exactly reproduce the setup locally.
As for your question, yes we don't support any schema changes for reference yey
โœ… 1
s
I have just above
Its a long message re-iterating the entire question/problem, with the query and schema
๐Ÿ‘ 1
Did you read my summary above ?
@Kishore Nallan
k
Yes, I will work with Harpreet to create a local example with that schema and sample doc and query. Will get back to you.
s
You can also check my cluster if you want to look at it: โ€ข
sh376gjtdxq8b9mop-1.a1.typesense.net [https: 443]
Its a dev environment and per your instructions, the collections for testing these new features have been created specially - so as to not mess with the dev data, while the joins functionality is in alpha
all of them are suffix with a 1 on the end such as
food1 brands1 userFood1 portionSizes1
k
Ok
s
Because we cant edit the schemas on this references stuff, I keep deleting these test collections and re-creating them, then reloading them with a few dummy records to simulate the basics of what we need
once we know what can or cant be done and what the roadmap is (also pending each time we have encountered bugs whcih you guys patched), we can then look at rebuilding the real dev env data and re-seeding it from source DB
h
@Shaun Byrnes Is this the response that you're expecting?
Copy code
{
  "brandId": "1",
  "brandId_sequence_id": 0,
  "brands.name": "Brand one",
  "id": "1",
  "name": "Food 2",
  "portionSizes": [
    {
      "name": "1 glass",
      "uid": "b"
    },
    {
      "name": "100ml",
      "uid": "c"
    }
  ],
  "portions.portionId_sequence_id": [
    1,
    2
  ],
  "uid": "2",
  "userFood.count": 5,
  "userFood.userId": "s"
}
s
correct!
h
Understood! I'll get back with a possible solution.
s
kk
thanks
@Kishore Nallan @Harpreet Sangar hows it going on this?
h
This will land in the next RC build https://github.com/typesense/typesense/pull/1341#issue-1964860534 Here's the query that would produce the desired result:
Copy code
curl 'http://localhost:8108/multi_search' -X POST -H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}" -d '
  {"searches":[{
    "collection": "food1",
    "q": "",
    "query_by": "name",
    "filter_by": "$userFood1(userId:s)",
    "include_fields": "$brands1(name :merge) as brands, $userFood1(count, userId :merge) as userFood, $portionSizes1(*:nest) as portionSizes",
    "sort_by": "$userFood1(count:desc)"
}]}' | jq
s
oh wow, yep
quote a lot of new syntax we can now use!
Can you explain a bit more what these do: โ€ข :merge โ€ข *:nest
h
I've added a summary in the PR. I'll add detailed examples in the Joins document.
s
Ok cool. You can link me to the detailed examples? When you think release is happening ?
k
I've just published
0.26.0.rc26
which contains these changes.
s
Ah ok cool. I can update to it myself?
k
Yes, it's available on the dashboard
s
Ok will do. Is there any docs written up yet on how to use new features?
k
We will update the docs by Monday
โœ… 1
s
Ok cool
Thatโ€™s all the docs ?
k
Yes
s
ok cool will take a look
Ive upgraded the cluster, tried re-creating the colleciton:
Copy code
{
  "created_at": 1698659635,
  "default_sorting_field": "",
  "enable_nested_fields": true,
  "fields": [
    {
      "facet": false,
      "index": true,
      "infix": false,
      "locale": "",
      "name": "uid",
      "optional": false,
      "sort": false,
      "type": "string"
    },
    {
      "facet": false,
      "index": true,
      "infix": false,
      "locale": "",
      "name": "name",
      "optional": false,
      "sort": false,
      "type": "string"
    },
    {
      "facet": false,
      "index": true,
      "infix": false,
      "locale": "",
      "name": "brandId",
      "optional": false,
      "reference": "brands1.uid",
      "sort": false,
      "type": "string"
    },
    {
      "facet": false,
      "index": true,
      "infix": false,
      "locale": "",
      "name": "portions.portionSizeId",
      "optional": false,
      "reference": "portionSizes1.uid",
      "sort": false,
      "type": "string[]"
    }
  ],
  "name": "food1",
  "num_documents": 0,
  "symbols_to_index": [],
  "token_separators": []
}
Then try and add this:
Copy code
[
  {
    "uid": "1",
    "name": "Food 1",
    "brandId": "1",
    "portions": [
      {
        "portionSizeId": "a",
        "amount": 10
      },
            {
        "portionSizeId": "b",
        "amount": 590
      }
    ]
  }
]
And get this error:
Copy code
There were some errors importing your documents. 

[
  {
    "code": 400,
    "document": "{\"uid\":\"1\",\"name\":\"Food 1\",\"brandId\":\"1\",\"portions\":[{\"portionSizeId\":\"1\",\"amount\":10},{\"portionSizeId\":\"2\",\"amount\":590}]}",
    "error": "Missing the required reference field `portions.portionSizeId` in the document.",
    "success": false
  }
]
I understand the way the reference field was setup, its expecting an string array of portionSizeId's, but as described above we need to be able to create an array of object's then reference a field in each object - what is the correct way the structure schema for this?
@Kishore Nallan
h
You'll have to extract the portion data out of the food schema and create a
foodPortionPrice
collection that has reference to both
food
and
portion
. That collection would contain the
price
of a certain food's portion.
Copy code
food		foodPortionPrice		  portion
uid		    food_uid_reference        uid
		    portion_uid_reference
		    price
following will get you the portions and their prices for the foods that match the query.
Copy code
collection: food
q: *
filter_by: $foodPortionPrice(*)
include_fields: $foodPortionPrice(price), $portion(*)
s
Im confused? Since last we spoke it was almost there and working with the schema we have and you replied with https://typesense-community.slack.com/archives/C01P749MET0/p1698396768980619?thread_ts=1691473538.701649&cid=C01P749MET0
We were almost there with the previous state we had
Mapping array of objects isnt that obscure surely?
h
In the schema you provided at the time, there was no concept of price per food portion.
Or I'm understanding this incorrectly?
Copy code
{
  "portionSizeId": "a",
  "amount": 10
}
s
its not a price btw
im just simulating some other value
I showed the schema we needed which had more than just the portionSizeId - hence an object used for storing values - otherwise obviously we would just use an array of strings or ints
I shared it several times, but again, see below:
Copy code
"portionSize": {
    "displayFormat": 0,
    "portionsPerItem": null,
    "selectedPortionIndex": 1,
    "unit": "g",
    "value": 1
  },
  "portions": [
    {
      "amount": 100,
      "measureUnitId": "REzO2pZ0K1J6MFrCnWAi",
      "multiple": 1,
      "portionSizeId": "GEMDwd7SBYFZ6T9Ui0s2",
      "type": 0
    },
    {
      "amount": 152,
      "measureUnitId": "GmwdaGtivG85KDyjohtj",
      "multiple": 1.52,
      "portionSizeId": "9UvEjhgOXKTITezpsPGu",
      "type": 0
    }
  ],
For each
portionSizeId
we would wanna match the reference field to the
portionSizes
collection and merge in the portion name etc
We have many very similar situations, such as:
Copy code
"nutrients": [
    {
      "amount": 6.69,
      "nutrientId": "ejXlabfZ7hM3N9oHF0KJ"
    },
    {
      "amount": 1.25,
      "nutrientId": "qlRUwsJ69Oyyy0DWfS07"
    },
    {
      "amount": 42.8,
      "nutrientId": "7eCtoVIpp0bVDtpi0aSG"
    },
...
]
The above is also in the same food document and we would match each
nutrientId
to get its name from the
nutrients
collection, merged in, as they are not just
string[]
h
I'm pretty sure you'll have to breakup this data and use a junction collection but let me discuss if there could be another simpler solution.
s
got it
we have alot of data which is arrays of objects like this
and having to break that up will be very tedious - currently we have the Typesense Firestore extensions to auto write docs over from Firebase to Typesense, but this would necessitate a much more convoluted amount of collections etc than just being able to map IDs inside an object of arrays
Something like this i envision:
Copy code
{
  "name": "portions.portionSizeId",
  "reference": "portionSizes1.uid",
  "type": "object[]"
}
๐Ÿ‘€ 1
or another way could be:
Copy code
{
  "name": "portions[].portionSizeId",
  "reference": "portionSizes1.uid",
  "type": "string[]"
}
but I dont think the name syntax is designed to support arrays in there, only object notation
k
I agree that creating intermediate collections is not convenient. Let's get back to you on this.
s
Yeah ok cool
how we going with that @Kishore Nallan
k
We've arrived at an approach. Implementing it now. Will update when we have a build to test.
s
ok sounds good thanks
hey @Kishore Nallan hows this progressing, any news?
k
Still actively working on it. I don't have an ETA to share. Will update this thread when we have a build ready.
s
ok sounds good man
thanks
hey guys hows this progressing @Kishore Nallan
k
Hi Shaun, we just finished the implementation yesterday. Will share a build and docs in a few hours.
Sorry about the delay. You can try this build:
0.26.0.rc34
We have updated instruction for using objects as references here: https://gist.github.com/kishorenc/037314619c3104970ae703b25aff2f51#reference-field-inside-an-object
s
Ah ok good to hear. Updating now and will check it out
This is working much better now:
Copy code
let searchParameters = {
    "q": "",
    "query_by": "name",
    "filter_by": "$userFood1(userId:s)",
    "include_fields": "$brands1(name) as brands, $portionSizes1(name) as portionSizes, $userFood1(count, userId) as userFood",
    "sort_by": "$userFood1(count:desc)"
  };
  
  let results = await client.collections("food1").documents().search(searchParameters);
Produces this result:
Copy code
{
  facet_counts: [],
  found: 2,
  hits: [
    {
      document: {
        brandId: '1',
        brands: { name: 'Brand one' },
        id: '0',
        name: 'Food 1',
        portions: [
          {
            amount: 10,
            portionSizeId: 'a',
            portionSizes: { name: '1 cup' }
          },
          {
            amount: 590.508,
            portionSizeId: 'b',
            portionSizes: { name: '1 glass' }
          }
        ],
        uid: '1',
        userFood: { count: 20, userId: 's' }
      },
      highlight: {},
      highlights: [],
      text_match: 100,
      text_match_info: {
        best_field_score: '0',
        best_field_weight: 12,
        fields_matched: 4,
        score: '100',
        tokens_matched: 0
      }
    },
    {
      document: {
        brandId: '2',
        brands: { name: 'Brand two' },
        id: '2',
        name: 'Food 2',
        portions: [
          {
            amount: 1,
            portionSizeId: 'c',
            portionSizes: { name: '100ml' }
          },
          {
            amount: 5,
            portionSizeId: 'a',
            portionSizes: { name: '1 cup' }
          },
          {
            amount: 3,
            portionSizeId: 'b',
            portionSizes: { name: '1 glass' }
          }
        ],
        uid: '2',
        userFood: { count: 4, userId: 's' }
      },
      highlight: {},
      highlights: [],
      text_match: 100,
      text_match_info: {
        best_field_score: '0',
        best_field_weight: 12,
        fields_matched: 4,
        score: '100',
        tokens_matched: 0
      }
    }
  ],
  out_of: 2,
  page: 1,
  request_params: { collection_name: 'food1', per_page: 10, q: '' },
  search_cutoff: false,
  search_time_ms: 0
}
Pretty cool!
Need to do some more tests for feedback
k
Good to hear!
s
yep
So with the following schema:
Copy code
{
  "created_at": 1700091091,
  "default_sorting_field": "count",
  "enable_nested_fields": true,
  "fields": [
    {
      "facet": false,
      "index": true,
      "infix": false,
      "locale": "",
      "name": "userId",
      "optional": false,
      "sort": false,
      "type": "string"
    },
    {
      "facet": false,
      "index": true,
      "infix": false,
      "locale": "",
      "name": "count",
      "optional": false,
      "sort": true,
      "type": "int32"
    },
    {
      "facet": false,
      "index": true,
      "infix": false,
      "locale": "",
      "name": "foodId",
      "optional": false,
      "reference": "food1.uid",
      "sort": false,
      "type": "string"
    },
    {
      "facet": false,
      "index": true,
      "infix": false,
      "locale": "",
      "name": "dateLastUsed",
      "optional": false,
      "sort": true,
      "type": "int64"
    },
    {
      "facet": false,
      "index": true,
      "infix": false,
      "locale": "",
      "name": "addedToMeals",
      "optional": false,
      "sort": false,
      "type": "string[]"
    }
  ],
  "name": "userFood1",
  "num_documents": 4
}
Is there a way to do like
SUM(count) GROUP BY foodId
? In that way we group all the foods which have been added and sum up the added count of each food, which would allow to see most added food's irrespective of user
k
Have you tried doing
group_by=foodId
along with
facet_by=count
? There will be a stats field returned as part of the facet response, which will have a sum.
s
As soon as I add
"group_by": "$userFood1(foodId)",
It causes an infinite loop:
Copy code
Request #1700104343962: Request to Node 0 failed due to "ECONNABORTED timeout of 3000ms exceeded"
Request #1700104343962: Sleeping for 0.1s and then retrying request...
Request #1700104343962: Request to Node 0 failed due to "ECONNABORTED timeout of 3000ms exceeded"
Request #1700104343962: Sleeping for 0.1s and then retrying request...
/Users/shaunbyrnes/Software/shredded-platform/shredded-firebase/utils/temp/node_modules/axios/lib/core/createError.js:16
  var error = new Error(message);
              ^

Error: timeout of 3000ms exceeded
    at createError (/Users/shaunbyrnes/Software/shredded-platform/shredded-firebase/utils/temp/node_modules/axios/lib/core/createError.js:16:15)
    at RedirectableRequest.handleRequestTimeout (/Users/shaunbyrnes/Software/shredded-platform/shredded-firebase/utils/temp/node_modules/axios/lib/adapters/http.js:369:16)
    at RedirectableRequest.emit (node:events:513:28)
    at Timeout.<anonymous> (/Users/shaunbyrnes/Software/shredded-platform/shredded-firebase/utils/temp/node_modules/follow-redirects/index.js:169:12)
    at listOnTimeout (node:internal/timers:564:17)
    at process.processTimers (node:internal/timers:507:7) {
  config: {
    transitional: {
      silentJSONParsing: true,
      forcedJSONParsing: true,
      clarifyTimeoutError: false
    },
    adapter: [Function: httpAdapter],
    transformRequest: [ [Function: transformRequest] ],
    transformResponse: [ [Function (anonymous)] ],
    timeout: 3000,
    xsrfCookieName: 'XSRF-TOKEN',
    xsrfHeaderName: 'X-XSRF-TOKEN',
    maxContentLength: Infinity,
    maxBodyLength: Infinity,
k
My bad, we haven't added support for group by, facet by and query by yet. Still working through adding support for all the sub features.
s
gotcha
k
Will keep you posted. Alter and alias as well. Now the foundational stuff are in place for joins, these should be straight forward to add.
s
yeah makes sense
definately starting to come together
k
Really appreciate all the feedback and help!
s
Yep youโ€™re welcome !
@Kishore Nallan question, now we have these joins, how would you propose best way to combine 2 similar, but seperate collections into one single order result. Screen looks as attached, we have collections for
food
(docs in orange) and
meals
(docs in purple). Documents in both can optionally have a
userId
value, which would associate them with an end user who created them. When we want to present a list of aggregated results that match the user such as
userId=2
from both collections, which would be best approach? At first glance federated search seems useful, but as its simple 2 seperate searches, with their own pagination, its not really useful for this. We need the results grouped into a single result set, so they can be sorted accordingly and paginated correctly. As it's simply a single list with "scroll down to load more" used to paginate.
Thus, could we use some sort of join logic to search both collections and join them on userId maybe, despite having different schemas?
k
This will be a merge operation and we've a feature request open for it: https://github.com/typesense/typesense/issues/381 Join cannot be used for this because joining requires the collections to have the same record entities, but with just varying fields.
s
Ahh ok - so the federated search will take place then aggregate results server side, thus allowing for pagination and querying of combined collections being searched ?
k
Correct. We haven't yet implemented this though. We can tackle this only after we wrap up joins.
s
Yeah makes sense. I expect the joins feature will still work in this - as it would be in each query used in federated search - then their results will just be amalgamated into one result set?
k
Yes correct
s
Yea would be great. Is that next in the runway after the joins is completed?
k
Yes
โœ… 1
s
Ok got it. Well once joins is stable and you have an early working version of this, let me know, so I can test it as that is the final piece of the puzzle we need to complete for our search
๐Ÿ‘ 1
hey @Kishore Nallan hows this one progressing?
k
Still working on adding those faceting features for joins. Given the holidays, I expect this to take a few more weeks. We need to incorporate join related semantics into all existing features like faceting, grouping etc.
s
ah ok got it
makes sense
hey @Kishore Nallan just checking in how the sub features are going? would be great to see us move towards https://github.com/typesense/typesense/issues/381 so that we can overhaul the current implementation we have with the new release, especially as there is another section of the applications that also needs to be refactored with Typesense and want to apply the first section first, get it working with the joins, then mirror this to other location with similar UX
k
We are not going to be able to get to that aggregate feature until at least 2 months. There is still a lot of work left with joins that we are uncovering. Merge also has only some overlap with joins so there's still unknowns to figure out. So we can't commit to a timeline yet. For now you have do the aggregation client side.
s
ok gotcha, thanks for the transparency - makes sense. Yes I agree the joins are quite similar in ways to the aggregation - it is after all combining multiple collections together into a result set, so could potentially integrate into that new feature set in some way given the similarity.
Is there any extra testing on my end can do to help with the stuff you are hitting into?
k
What's left is now integrating things like facet_by, group_by etc. into join by allowing them to play nice with joined collections. I will post an update here once we have a few of those knocked out. Appreciate all the help and feedback from testing.
s
No problem. Sounds good - I can rerun those example collection queries and try some of those new features to test if needed
๐Ÿ‘ 1
Happy new year guys ๐Ÿฅณ hows this progressing?
k
Happy New Year! We will be wrapping up the join related work in a week or two and then start focusing on the merge feature.
โœ… 1
s
awesome
good to hear
h
Bascially trying to sort by all userFood counts highest to lowest
@Shaun Byrnes In the latest RC build you can try this query now and it will return the brand names as well.
Copy code
curl 'http://localhost:8108/multi_search' -X POST -H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}" -d '
  {"searches":[{
    "collection": "userFood1",
    "q": "*",
    "filter_by": "$food1(id:*)",
    "include_fields": "$brands1(*), $food1(*)",
    "sort_by": "count:desc"
}]}' | jq
s
ah ok nice
hey guys how is the progress of this for stable release coming ?
@Kishore Nallan you see the last message by chance ?
k
Hey Shaun, sorry I missed responding earlier. We are still fixing edge cases and bugs people have been reporting so we are continuing to wrap up on the 0.26 release. I will update this thread when we pick up the merge feature. It's definitely the next big item on the road map but we have to stabilize and release 0.26 before we can fully focus on that.
s
No problem, thanks for the update and totally get it. Will part of 0.26 stable release allow TS cloud to add reference fields without having to rebuild schema or is this for down the track?
k
We have not looked into adding reference fields via alter schema yet.
s
gotcha. no problem
@Kishore Nallan with searching, is there a way to limit the max amount of records returned (diff to
per_page
/
limit
alias). As in we still want pagination - such as 10 per page. But max results of 50. Then we can do max of 5 pages - sorted as we want, but limit the amount of results returned from server ?
k
Yes, check
limit_hits
Maximum number of hits that can be fetched from the collection. Eg:
200
page * per_page
should be less than this number for the search request to return results.
Default: no limit
You'd typically want to generate a scoped API key with this parameter embedded and use that API key to perform the search, so it's automatically applied and can't be changed at search time.
s
gotcha
but if its not really a big deal security wise, you could just do this on the normal API key too ?
k
Yes if the goal is not about preventing scraping then it doesn't matter
s
yeah just to have pagination
but limit to a certain amount of records max - for UX perspective
let me try add
When I try:
Copy code
"limit_hits": 1,
    "page": 2,
    "per_page": 1
I get:
Copy code
ObjectUnprocessable: Request failed with HTTP code 422 | Server said: Only upto 1 hits can be fetched. Ensure that `page` and `per_page` parameters are within this range.
(there is 2 results possible to return from query,
page: 1
is happy, but changing to
page: 2
throws that error - correct me if im wrong, but it should go to the result on the second page - if working as suggested ?
k
By setting limit hits to 1 you are saying that you can't paginate past 1 record which is why the error is thrown..
s
gotcha
sorry haha
๐Ÿ™‚ 1
Copy code
"limit_hits": 10,
    "page": 2,
    "per_page": 1
This is what I need to simulate with 2 records
going to 2nd page
works fine - thanks
Separate random question - related to extensions in Firebase - the new version 1.3 changed how to do backfil with "upsert" but the docs doesnt seem to have an exact write up of the change from like this, to new way - unless I missed ?
j
All backfills use upsert now, thereโ€™s no additional configuration changes needed
s
what you mean sorry ?
do we still use that collection/document structure as pictured?
Whats the upsert change
j
Yup, that's backfill trigger collection structure is still the same
s
ahh gotcha - yep just adding the updates to be done in that fashion, cool
@Kishore Nallan looping back to the
limit_hits
and your suggestion for scoped API key. Quite a few questions or thoughts below as to how this could apply in our case ๐Ÿ˜„ Initially I wasn't going to bother on adding scoped key's as a further complexity if we didn't really need it (such as the limit fields is just for UX), but we also have some user related information - joined in to other collections - which gives the data for recently added/usage counts etc per user. Currently we don't have any custom backend reliance for Typesense, as the full search API key (and marketplace extensions) have been sufficient to use on the client side till now as all TS collections were full public. However, from what I understand from the docs, if you have reliance on scoped keys for some kind of security purpose, you should not use the default full search API key on client anywhere - as this would simply bypass the point of scoping keys. Correct? In which case, this key would be relocated to the backend to generate scoped keys from it correct? Given that 90% of all the collections are still completely public, I presume the scoped key would have the array of all collections listed except the one(s) which have
userId
etc. In this case it would mean that all the other collections which have no user related content and should be completely public would function as normal with scoped key returned, with the exception of those which have the reliance of the userId ? My concern was will those public collections get messed up with something like this returned for each user:
Copy code
return client.keys().generateScopedSearchKey(searchOnlyAPIKey, {
    "filter_by": `userId:=${userId}`, 
    // 1 month from now
    "expires_at": Utils.getUnixTimestamp() + 2592000
  });
Given `"filter_by":
userId:=${userId}
,` the filter_by would bake in the userId for all requests. Or will collections that don't have a userId in schema be unaffected? I presume they will try and query by that if included and either throw an error or return no results? How will this work with queries that join collections (which is what we are doing). Consider the following example on client side at present:
Copy code
const searchParameters = {
  "q": "",
  "query_by": "name",
  "filter_by": "$userFood(userId:Mb2L5bAc1AbaZ6xPEkIjl60UDbr2)",
  "include_fields": "$foodBrands(name) as foodBrand, $portionSizes(name) as portionSize, $userFood(count, dateLastUsed, addedToMeals)",
  "sort_by": "$userFood(count:desc)",
  "limit_hits": 50
};
  
const results = await client.collections("food").documents().search(searchParameters);
In the query above, only the
userFood
collection has a
userId
field - as it contains info about publicly accessible foods related to each user's use. Currently the userId is passed on client side - so technically anyone could just replace with another user's ID - to get their count for a food (which doesn't really matter, but I am thinking ahead for scenarios where this could have actual private data). If we were to go the way of scoped key, I assume we would set this `"filter_by":
$userFood(userId:${userId})
,` on the backend? I.e. Using the same syntax for reference collections, but just interpolating the user's ID server side, and will work ok? In that case I expected maybe we need to send back 2 keys - one with full public collections listed in it (excluding any which have userId) and every user can fetch that scoped from backend for public access (to replace the current full search API key). Followed by another key they can use for user scoped collections - where their userId is included in the filter_by ? Additionally, for
expires_at
it simply states in docs to pass a Unix timestamp - from what I could deduce, it should be in seconds - or it doesn't matter if its down to nanoseconds? Can we simply pass
0
for no expiry? Or must it be some time in the future always?
k
Use scoped api keys if your collection stores data from multiple users and you want to ensure that the api key in the front end has an embedded user filter such that one user can't use their api key to look up data of another. The filter clause embedded in the scoped API key is added as an
&&
condition to other filter clauses you add in the query.
Additionally, for
expires_at
it simply states in docs to pass a Unix timestamp - from what I could deduce, it should be in seconds - or it doesn't matter if its down to nanoseconds? Can we simply pass
0
for no expiry? Or must it be some time in the future always?
All the keys already have a default expires_at set far into the future, so you don't need to specify anything if you don't want your keys to expire.
s
Got it on the expires. As for the other questions, so the filter_by passed in the scoped key will work the same as if it was
&& condition
on client? I.e. we can use the same syntax for reference collections like
$collectionName(conditions)
in the scoped key ?
Like this: `"filter_by":
$userFood(userId:${userId})
,` to ensure that the referenced
userFood
collection matches only on userId set on backend to generate scoped key
k
Yes, correct
s
Ok cool makes sense
And what about for the way to do the scoped - given that 90% of the collections should still be full public - would we just send a scoped search key which instead of
*
explicitly lists all the full public collections - then use that in place on client side for all the full public access. Then send another scoped key with the userId baked in for the specific collections which need that ?
k
For fully public collections you can just use normal search api key, no scoping is even needed.
Scoping needed for queries targeting a single collection with a user_id filter or when joining with a collection where the user id filter is applied.
s
yeah got it - but what I mean, is now if we pass the default search key - which has access to all collections - this will circumvent the userId scoped key - as the client app would have the full search key and could just use that key to get all from a collection which we want scoped
So what I expect, is we need to create a new scoped key based off the full search one currently in use, which has only collections which should be full public (exclude those which have userId). Then replace the current full search key on client app with this one.
That will cover all the full public collections, but will break the ones which have userId. Then for those, client app fetches another scoped key - which has the collections explicit which need to enforce logged in user's ID. Swap out the current full search key on client for this one which is scoped per user. I expect this is the way to do it?
k
Yes correct. You need to do that.
Swap the key and delete the old key.
s
yeah ok
thought so
otherwise the old key could just allow access to bypass the scoped one anyhow
When creating keys, API docs state we can use regex for the collections.
Copy code
client.keys().create({
  "description": "Key for searching org collections.",
  "actions": [
    "documents:search"
  ],
  "collections": [
    "org_.*"
  ]
})
Are we free to use any regex? Like can we do:
Copy code
"collections": [
    "^(?!userFood).*$"
  ]
Ie any collection not starting with
userFood
?
Reason I am checking as otherwise we need to rename all the collections we want to allow on the public search key to prefix with something like
public_
except there seems to be no rename function for existing collections in TS Cloud, meaning we would have to delete each collection and re-create it with the new name like
public_collectionName
? Furthermore, each env has marketplace extensions setup for each collection, each of which would need to be re-configured with the new TS collection name, followed by backfilling all the docs, am I correct on this?
k
You are free to use a full regexp.
โœ… 1
We don't have a rename collection option but you can use an alias and make api keys use alias collection names.
s
good to know on both - can the regex refer to aliases as in if collection name is
food
aliased to
pub_food
can we do
Copy code
"collections": [
    "pub_.*"
  ]
k
When you query with an alias, the alias name must be present inside the api key to allow it access.
s
what do you mean sorry ?
can you give example ?
And what about my question regarding the regex for collections when creating api keys - is this supported to refer to aliases too ?
k
When you make a search query, you pass a collection name in the URL. This name can either be an alias or a real collection name. The authentication logic checks this name (either real name or alias) and validates it against the allowed values (either regexp or just strings) in the key.
To give a concrete example, if you have an alias mapping
foo
to real collection
bar
and then you make a query with
/collections/foo
-> the API key associated with the request MUST have
foo
in allowed collection name. (it could be a regexp like
f.*
also)
s
gotcha
so
f.*
matching that alias would be good - provided that was added to the collections[] of the key used for the query
k
Yes
s
ok got it
will check it out
When trying to backfill collection, I get a failure for a string field which is
null
Collection is setup with this for the field:
Copy code
{
      "facet": false,
      "index": true,
      "infix": false,
      "locale": "",
      "name": "foodBrandId",
      "optional": true,
      "reference": "foodBrands.uid",
      "sort": false,
      "type": "string"
    },
My understanding that if
optional: true
then
null
should be OK? doesn't need to be saved as
""
But I get this in logs:
Copy code
error: 'Field `foodBrandId` must have `string` value.',
      success: false
Can you confirm?
k
Hmm yes, optional field can be null.
Can you try upgrading to the latest 0.26 RC build?
s
Will do
I thoguht I was on .26 but maybe not latest
Will check when back at my computer
๐Ÿ‘ 1
do i need to re-create the collection
or should jsut work to back fill once upgraded
now my cluster looks like this (which I never saw before) and I cant access any collections to test @Kishore Nallan
k
If you have just rotated, the DNS resolution will be stale for a 1 min or so.
s
gotcha
so thats just normal
I have updated to
Typesense v0.26.0.rc59
still gives same error
Copy code
Import error ImportError: 3 documents imported successfully, 2 documents failed during import. Use `error.importResults` from the raised exception to get a detailed error reason for each document.
    at ImportError.TypesenseError [as constructor] (/workspace/node_modules/typesense/lib/Typesense/Errors/TypesenseError.js:23:28)
    at new ImportError (/workspace/node_modules/typesense/lib/Typesense/Errors/ImportError.js:25:28)
    at Documents.<anonymous> (/workspace/node_modules/typesense/lib/Typesense/Documents.js:160:39)
    at step (/workspace/node_modules/typesense/lib/Typesense/Documents.js:48:23)
    at Object.next (/workspace/node_modules/typesense/lib/Typesense/Documents.js:29:53)
    at fulfilled (/workspace/node_modules/typesense/lib/Typesense/Documents.js:20:58)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
  importResults: [
    { success: true },
    { success: true },
    {
      code: 400,
      document: '{"foodBrandId":null,"notes":null,"preparationTime":null,"verified":false,"userId":"kPKKgXOyiGXnv21I5DL4cHNXiFu2","cookingTime":null,"uid":"M8KWqkMWPttyxb9yN5Wv","baseUid":"M8KWqkMWPttyxb9yN5Wv","createdDate":1708911965,"deleted":true,"directions":[],"parts":[],"name":"Meal for good","foodCategories":[],"dietaryTypes":null,"portionSize":{"unit":"g","selectedPortionIndex":0,"displayFormat":0,"value":1,"portionsPerItem":null},"itemWeight":null,"portions":[{"amount":null,"multiple":1,"type":0,"measureUnitId":"REzO2pZ0K1J6MFrCnWAi","portionSizeId":"qO6fez6gHhO8SQ8riQ89"}],"id":"M8KWqkMWPttyxb9yN5Wv"}',
      error: 'Field `foodBrandId` must have `string` value.',
      success: false
    },
    { success: true },
    {
      code: 400,
      document: '{"foodBrandId":null,"verified":false,"preparationTime":null,"userId":"tS1RgdIiRYY5VFgWL5P1I7EI9UN2","cookingTime":null,"uid":"qP9oFHBbgBNPAEFnvw8d","directions":[],"name":"meal1","foodCategories":[],"dietaryTypes":null,"portionSize":{"unit":"g","selectedPortionIndex":0,"displayFormat":0,"value":1,"portionsPerItem":null},"itemWeight":null,"portions":[{"amount":null,"multiple":1,"type":0,"measureUnitId":"GmwdaGtivG85KDyjohtj","portionSizeId":"9L8ZLOBNTw4jUUxOhmQY"}],"parts":[{"uid":"R2EswNcBNwRtuVcQleuH","name":"Main","items":[{"uid":"HSOmGn34fwr0viVYlaPj","type":0},{"uid":"N9oRfnPxD1ECLuE4f7Xa","type":0}]}],"createdDate":1708336556,"deleted":false,"id":"qP9oFHBbgBNPAEFnvw8d"}',
      error: 'Field `foodBrandId` must have `string` value.',
      success: false
    }
  ]
}
k
Can you please check if this also happens on a new collection?
s
i deleted that collection and remade it
this happened when i tried to back fill it
(the newly created coll)
k
Ok we are able to reproduce this. Will have a fix shortly. thanks for reporting this.
s
no prob
cheers
lmk when i can apply update to cluster
k
Sure, will post an update here, the cycle will take a few hours.
โค๏ธ 1
s
any luck ?
k
We published the build but noticed an unrelated faceting related error that we have to still check on what's happening.
s
ah ok got it
m
Maybe same error @Kishore Nallan, but I have an optional float field in typesense, and when i try to use the /_documents/import?action=create_ endpoint I get an error "some_field must be a float." Some of my records have a null value for this field.
k
@Shaun Byrnes please use 0.26.0.rc62
@Malik please create a GitHub issues with a reproducible example we can try.
s
ok i will update cluster @Kishore Nallan and let you know if its working now
๐Ÿ‘ 1
I added "pub" prefix and similar issue: Getting something like this now: Request failed with status code: 400.Server said: Could not include the reference document of
pub_portionSizes
collection. Expected
portionSize
to be an array. Try renaming the alias.
Any ideas?
h
Can you share the request that you're making? Also, your schema if you've changed it. Most likely this has something to do with your alias name clashing with a field name.
s
Aliases
h
By alias name I meant what you're sending in the
include_fields
like
as foo
.
s
yeah im just showing whiuch ones are setup in TS cloud
Copy code
let searchParametersAll = {
    "q": "",
    "query_by": "name",
    "filter_by": "deleted:false",
    "include_fields": "$foodBrands(name) as foodBrand, $portionSizes(name) as portionSize"
  };
  
  let results = await client.collections("food").documents().search(searchParametersAll);
Copy code
RequestMalformed: Request failed with HTTP code 400 | Server said: Could not include the reference document of `portionSizes` collection. Expected `portionSize` to be an array. Try renaming the alias.
    at RequestMalformed.TypesenseError [as constructor] (/Users/shaunbyrnes/Software/shredded-platform/shredded-firebase/utils/temp/node_modules/typesense/lib/Typesense/Errors/TypesenseError.js:23:28)
    at new RequestMalformed (/Users/shaunbyrnes/Software/shredded-platform/shredded-firebase/utils/temp/node_modules/typesense/lib/Typesense/Errors/RequestMalformed.js:25:42)
    at ApiCall.customErrorForResponse (/Users/shaunbyrnes/Software/shredded-platform/shredded-firebase/utils/temp/node_modules/typesense/lib/Typesense/ApiCall.js:332:21)
    at /Users/shaunbyrnes/Software/shredded-platform/shredded-firebase/utils/temp/node_modules/typesense/lib/Typesense/ApiCall.js:199:98
    at step (/Users/shaunbyrnes/Software/shredded-platform/shredded-firebase/utils/temp/node_modules/typesense/lib/Typesense/ApiCall.js:33:23)
    at Object.next (/Users/shaunbyrnes/Software/shredded-platform/shredded-firebase/utils/temp/node_modules/typesense/lib/Typesense/ApiCall.js:14:53)
    at step (/Users/shaunbyrnes/Software/shredded-platform/shredded-firebase/utils/temp/node_modules/typesense/lib/Typesense/ApiCall.js:18:139)
    at Object.next (/Users/shaunbyrnes/Software/shredded-platform/shredded-firebase/utils/temp/node_modules/typesense/lib/Typesense/ApiCall.js:14:53)
    at fulfilled (/Users/shaunbyrnes/Software/shredded-platform/shredded-firebase/utils/temp/node_modules/typesense/lib/Typesense/ApiCall.js:5:58)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
  httpStatus: 400
}

Node.js v18.12.1
This only just started happening with upgrade to
Typesense v0.26.0.rc62
from
rc58
no other changes were made to collection schema, aliases, query etc
as soon as I remove the alias ie
$portionSizes(name) as portionSize
>
$portionSizes(name)
works again ๐Ÿ˜•
This also works
$pub_portionSizes(name)
But this does not
$pub_portionSizes(name) as portionSize
k
Yes, there was a bug with aliases we had to fix. Maybe that caused a regression. Earlier, aliases did not work with joins
s
Seems bugged
yeah
exactly, because previous before the upgrade to rc62 was happy with that alias
and its weird as the other join alias doesnt throw err
Ie this
$foodBrands(name) as foodBrand
is still working fine
h
Can you try changing
$pub_portionSizes(name) as portionSize
to
$pub_portionSizes(name) as p
or even
$portionSizes(name) as p
. Trying to determine if the alias name is actually causing the issue here.
s
ok will doi
Ok so both
$pub_portionSizes(name) as p
and
$portionSizes(name) as p
work
h
As I suspected. The
foods
document must have a field named
portionSize
and it must be referencing multiple portion sizes. Renaming the alias in
include_fields
should fix this. I'm surprized this was working in previous RC build.
s
gotcha
makes sense
yea, weird it worked before, as I think you are rigth
Correct:
Copy code
portionSize
{ "displayFormat": 0, "portionsPerItem": 25.4, "selectedPortionIndex": 0, "unit": "g", "value": 1 }
is example from a food doc
ok so its just a naming clash with the alias picked
h
Yeah
s
cool thanks, not sure why it worked before, but will update
maybe it was merging the results
from the alias'd field into the existing or something
h
Oh right. There was a change in the default method for including the referenced documents. We chose to have
nest
as the default. Earlier it was
merge
. So I think
$portionSizes(name :merge) as portionSize
will be a better fix for you.
s
as yes I see
no I think the change to nest as default is better
ie:
Copy code
"include_fields": "$pub_foodBrands(name, deleted, verified, userId) as foodBrand, $pub_portionSizes(name, verified, deleted, userId) as portionSize"
gives:
Copy code
portions: [
          {
            amount: 100,
            measureUnitId: 'zDyQbolmfqHvru7I8YHs',
            multiple: 1,
            portionSize: { deleted: false, name: '100ml', verified: true },
            portionSizeId: 'y7Y7sWBoCUJeVMj95RcC',
            type: 0
          },
          {
            amount: 450,
            measureUnitId: 'zDyQbolmfqHvru7I8YHs',
            multiple: 4.5,
            portionSize: { deleted: false, name: '15.2oz', verified: true },
            portionSizeId: 'pOVPgpOOtYclVT0u2Ocf',
            type: 0
          }
        ]
which is cleaner imo
h
Yeah
s
works nicely
ok i will begin to test other thing about backfill with nulls and let you know @Kishore Nallan if all good now
๐Ÿ‘ 1
there is still some import failures (these might be totally correct and just need fixing in source data), but the logs explorer just chops off the array of success and there seems to be no way to get the full results so you can see the errors to find out why they failed ...
k
It only shows a snippet. You can iterate on the results array and print the ones with success false.
s
how though?
in the logs explorer?
k
In your code, you can get the response text, split it on new line and go through every line.
s
this is for backfill
k
Firebase function?
s
so i didnt code it - just changed flag on backfill collection
in Firebase
k
Ah ok. Then that limits the options. You have to manually try importing from locally to catch the error.
Or if there is a way to run backfill on a smaller subset of data
s
what you mean
well i might not find the error then
i think best to update with an errors[]
since the google logger is garbage
obviously it should let you ignore truncation or some sort of download which shows all content
but doesnt seem to do so - even the download just has it as text and truncated lol
as i wont be able to find out what records have issues otherwise
k
Let me find out if there is a workaround here
s
ok cool
j
I would recommend taking the code here: https://github.com/typesense/firestore-typesense-search/blob/master/functions/src/backfill.js And running it from your local machine using an admin Firebase script That will be the fastest way.
In the meantime, we can add a flag to log the errors in a separate line, so it doesnโ€™t get truncated, sometime in the next 24hrs
s
Yeah Iโ€™ll just wait for that update is fine
is that flag ready ?
j
Not yet, got pulled into other stuff on Fri. Will get to this on Monday.
โค๏ธ 1
Here's a pre-release version with this change. You want to use the installation link in the release notes specifically: https://github.com/typesense/firestore-typesense-search/releases/tag/v1.4.1-rc.0
s
Thanks
Is that for the extensions ?
Also, when using the scoped key for TS request (swapped out the default public search key ) we get this Request failed with status code: 401.Server said: Forbidden - a valid
x-typesense-api-key
header must be sent.
Any ideas ?
j
Is that for the extensions ?
Correct
โœ… 1
Once you install that version, you should see just the import error messages logged in a more readable format in the Google Cloud logs explorer UI
Btw, could you post each question / topic in a separate thread in the main channel in #community-help. Slack takes forever to load a long thread for me
โœ… 1
Also, when using the scoped key for TS request (swapped out the default public search key ) we get this
Request failed with status code: 401.Server said: Forbidden - a valid
x-typesense-api-key
header must be sent.
You want to check if the parent API key has sufficient permissions to access the collection
s
yea the majority of this whole thread is on joins and getting that new feature functioning - i had just dropped in a few short random topics in there. will post other q's if they are unrelated in diff thread
The sscoped keys are part of the testing for the joins
THe parent key should be fine:
Copy code
Description	Key Prefix	Actions	Collections
Search only key for all public collections	zHAV	[ "documents:search" ]	[ "pub_.+" ]
aliases
I have double checked the parent key being used matches:
Copy code
const publicSearchAPIKey = typesensePublicSearchAPIKey.value(); // zHAV...
result.public = client.keys().generateScopedSearchKey(publicSearchAPIKey, paramaters);
k
I wonder if there is a gotcha with scoped API keys and joins. Can you please post the full search params you are using and the collection name?
s
ok will do
once my android dev working on this part is online i will get it from him and post here
๐Ÿ‘ 1
Copy code
//config
String publicKey = "aEdySm************************"
Configuration configuration = new Configuration(nodes, Duration.ofSeconds(2), publicKey);
client = new Client(configuration);
Copy code
//search query - search portionSize with given uid
SearchParameters searchParameters = new SearchParameters()
		.q("fH6H6i8g8Dplf4kPUf3b")
		.queryBy("uid")
		.filterBy("deleted:=false")
		.sortBy("name: asc")
		.page(1)
		.perPage(20);
return client.collections("portionSizes").documents().search(searchParameters);
Copy code
//error code:
org.typesense.api.exceptions.RequestUnauthorized:
Request failed with status code: 401.Server said: Forbidden - a valid `x-typesense-api-key` header must be sent.
Copy code
//note
this same request works with default API key โ€œNtWM***โ€
This one doesnโ€™t even have a join in it and still fails (although the portionSizes collection can be referenced when need be per schema )
k
Hmm, i would then need the actual key used and the collection name. Can you please DM me? I'll have to check what's happening. Perhaps the key is not created correctly.
โค๏ธ 1
s
And this one which have join (giving same error)
Copy code
//config
String publicKey = "aEdySm************************"
Configuration configuration = new Configuration(nodes, Duration.ofSeconds(2), publicKey);
client = new Client(configuration);
Copy code
//search query with join
SearchParameters searchParameters = new SearchParameters()
	.q("")
	.queryBy("name")
	.filterBy("deleted:=false")
	.includeFields("$pub_foodBrands(deleted, name, uid, verified, userId) as foodBrand, $pub_portionSizes(uid, name, verified, deleted, measureUnitId, userId) as portionSizes")
	.page(1)
	.perPage(20);
return client.collections("food").documents().search(searchParameters);
Copy code
//error code:
org.typesense.api.exceptions.RequestUnauthorized:
Request failed with status code: 401.Server said: Forbidden - a valid `x-typesense-api-key` header must be sent.
Copy code
//note
this same request works with default API key โ€œNtWM***โ€
Any idea whats happened with rc66 guys? @Kishore Nallan I used to have this (as above):
Copy code
"include_fields": "$pub_foodBrands(name, deleted, verified, userId) as foodBrand, $pub_portionSizes(name, verified, deleted, userId) as portionSize"
which gave this output (which was perfect):
Copy code
...
portions: [
          {
            amount: 100,
            measureUnitId: 'zDyQbolmfqHvru7I8YHs',
            multiple: 1,
            portionSize: { deleted: false, name: '100ml', verified: true },
            portionSizeId: 'y7Y7sWBoCUJeVMj95RcC',
            type: 0
          },
          {
            amount: 450,
            measureUnitId: 'zDyQbolmfqHvru7I8YHs',
            multiple: 4.5,
            portionSize: { deleted: false, name: '15.2oz', verified: true },
            portionSizeId: 'pOVPgpOOtYclVT0u2Ocf',
            type: 0
          }
        ]
...
now I get error as nesting doesnt seem to work anymore
Because the alias conflicts with another existing field, but when nesting was working (ie :nest) then it was fine. If I change the alias to say
portionSizess
to avoid conflict I now get this:
Copy code
portionSizess: [
          { deleted: false, name: '1 cup, pieces', verified: true },
          { deleted: false, name: '100g', verified: true },
          { deleted: false, name: '1 slice, NFS', verified: true },
          {
            deleted: false,
            name: '1 Guideline amount on regular sandwich',
            verified: true
          },
          {
            deleted: false,
            name: '1 Guideline amount on large sandwich',
            verified: true
          },
          {
            deleted: false,
            name: '1 Quantity not specified',
            verified: true
          }
        ],
        portions: [
          {
            amount: 100,
            measureUnitId: 'REzO2pZ0K1J6MFrCnWAi',
            multiple: 1,
            portionSizeId: 'GEMDwd7SBYFZ6T9Ui0s2',
            type: 0
          },
          {
            amount: 56,
            measureUnitId: 'GmwdaGtivG85KDyjohtj',
            multiple: 0.56,
            portionSizeId: 'Z14eBJrHfEDxqwUMfUhX',
            type: 0
          },
          {
            amount: 28,
            measureUnitId: 'GmwdaGtivG85KDyjohtj',
            multiple: 0.28,
            portionSizeId: 'HCU3MoLa3YFsCZsos9MD',
            type: 0
          },
          {
            amount: 84,
            measureUnitId: 'GmwdaGtivG85KDyjohtj',
            multiple: 0.84,
            portionSizeId: 'eoE3YE153eqJKmiSRSvL',
            type: 0
          },
          {
            amount: 140,
            measureUnitId: 'GmwdaGtivG85KDyjohtj',
            multiple: 1.4,
            portionSizeId: '68tKa9ViNktfxpKL4fh0',
            type: 0
          },
          {
            amount: 56,
            measureUnitId: 'GmwdaGtivG85KDyjohtj',
            multiple: 0.56,
            portionSizeId: 'mTrUr1NUxN8rfo3iTiOG',
            type: 0
          }
        ],
They are now 2 separate arrays (instead of as before)...
which not only broke things on client for us, but is much less useful that how it was just previously - can you shed any light? :merge still can be used in the join collection as before, but the :nest as default was much better for the structure being mapped
did you see this issue @Kishore Nallan
k
Yes, will investigate and get back to you.
What version were you using before?
s
64
If you see the old output with :nest was perfect
k
@Shaun Byrnes Are you sure you were on rc64 before? I don't see any relevant changes since then. If it's broken it would not have worked on rc64 as well.
s
I just did the upgrade from rc64 to 66
๐Ÿ‘ 1
But I mean maybe it was 62 or something - either way , the outputs have changed from the one I showed you (which was perfect) to now which isnโ€™t idealโ€ฆ
h
@Shaun Byrnes Could you share the full query that is producing this result?
s
Which result? I can no longer produce the correct output when I re-run the query I showed you
k
The full query, with the other fields apart from the
include_fields
that you have pasted above.
s
Gotcha
k
If you can share the full query it will be great, so we can directly run and check and can verify the fix later as well.
s
Yeah Iโ€™ll share it when back at my computer
Is the idea itโ€™s supposed to still nest the joined fields by default?
k
Yes, nest is default.
s
Copy code
let searchParametersAll = {
    "q": "",
    "query_by": "name",
    "filter_by": "deleted:false",
    "include_fields": "$foodBrands(name, deleted, verified, userId) as foodBrand, $portionSizes(name, verified, deleted, userId) as portionSizess"
  };
  
  let results = await client.collections("food").documents().search(searchParametersAll);
so instead of creating an object nested for each item in
food.portions[]
array called
portionSizess
we get an array of objects
food.portionSizes[]
๐Ÿ‘€ 1
h
Let me know what the response is with this:
Copy code
"include_fields": "$foodBrands(name, deleted, verified, userId) as foodBrand, $portionSizes(name, verified, deleted, userId : merge)"
s
ok
I get an error
if i add an alias to portionSizes such as p
I get
Copy code
'p.deleted': [ false, false, false, false, false, false ],
        'p.name': [
          '1 cup, pieces',
          '100g',
          '1 slice, NFS',
          '1 Guideline amount on regular sandwich',
          '1 Guideline amount on large sandwich',
          '1 Quantity not specified'
        ],
        'p.verified': [ true, true, true, true, true, true ],
which is basically what used to happen (when
:merge
was default behaviour)
But the
:nest
behavior was better as it would put an object inside the joined/referenced array
like
portions[].referenceCollectionName{}
with the fields for that item at each index
h
I'm having difficulty reproducing this error. I'm trying with this schema
Copy code
{
  "name": "Foods",
  "fields": [
    {
      "name": "name",
      "type": "string"
    },
    {
      "name": "portions",
      "type": "object[]"
    },
    {
      "name": "portions.portion_id",
      "type": "string[]",
      "reference": "Portions.portion_id",
      "optional": true
    }
  ],
  "enable_nested_fields": true
}
{
  "name": "Portions",
  "fields": [
    {
      "name": "portion_id",
      "type": "string"
    },
    {
      "name": "quantity",
      "type": "int32"
    },
    {
      "name": "unit",
      "type": "string"
    }
  ]
}
Indexed these documents:
Copy code
{
  "portion_id": "portion_a",
  "quantity": 500,
  "unit": "g"
}
{
  "portion_id": "portion_b",
  "quantity": 1,
  "unit": "lt"
}
{
  "portion_id": "portion_c",
  "quantity": 500,
  "unit": "ml"
}
and this food:
Copy code
{
  "name": "Milk",
  "portions": [
    {
      "portion_id": "portion_b",
      "count": 3
    },
    {
      "count": 3
    },
    {
      "portion_id": "portion_c",
      "count": 1
    }
  ]
}
For
Copy code
{"collection", "Foods"},
{"q", "*"},
{"include_fields", "$Portions(*:merge)"}
I receive
Copy code
{
  "name": "Milk",
  "portions": [
    {
      "count": 3,
      "id": "1",
      "portion_id": "portion_b",
      "quantity": 1,
      "unit": "lt"
    },
    {
      "count": 3
    },
    {
      "count": 1,
      "id": "2",
      "portion_id": "portion_c",
      "quantity": 500,
      "unit": "ml"
    }
  ]
}
with
:merge
.
Could you also share your Foods collection schema?
s
sure
Copy code
{
  "created_at": 1709820691,
  "default_sorting_field": "",
  "enable_nested_fields": true,
  "fields": [
    {
      "facet": false,
      "index": true,
      "infix": false,
      "locale": "",
      "name": "foodCategories",
      "optional": true,
      "sort": false,
      "stem": false,
      "type": "string[]"
    },
    {
      "facet": false,
      "index": true,
      "infix": false,
      "locale": "",
      "name": "name",
      "optional": false,
      "sort": true,
      "stem": false,
      "type": "string"
    },
    {
      "facet": false,
      "index": true,
      "infix": false,
      "locale": "",
      "name": "deleted",
      "optional": false,
      "sort": false,
      "stem": false,
      "type": "bool"
    },
    {
      "facet": false,
      "index": true,
      "infix": false,
      "locale": "",
      "name": "verified",
      "optional": false,
      "sort": false,
      "stem": false,
      "type": "bool"
    },
    {
      "facet": false,
      "index": true,
      "infix": false,
      "locale": "",
      "name": "water",
      "optional": false,
      "sort": false,
      "stem": false,
      "type": "bool"
    },
    {
      "facet": false,
      "index": true,
      "infix": false,
      "locale": "",
      "name": "supplement",
      "optional": false,
      "sort": false,
      "stem": false,
      "type": "bool"
    },
    {
      "facet": false,
      "index": true,
      "infix": false,
      "locale": "",
      "name": "fdcId",
      "optional": true,
      "sort": false,
      "stem": false,
      "type": "int32"
    },
    {
      "facet": false,
      "index": true,
      "infix": false,
      "locale": "",
      "name": "userId",
      "optional": true,
      "sort": false,
      "stem": false,
      "type": "string"
    },
    {
      "facet": false,
      "index": true,
      "infix": false,
      "locale": "",
      "name": "uid",
      "optional": false,
      "sort": false,
      "stem": false,
      "type": "string"
    },
    {
      "facet": false,
      "index": true,
      "infix": false,
      "locale": "",
      "name": "foodBrandId",
      "optional": true,
      "reference": "foodBrands.uid",
      "sort": false,
      "stem": false,
      "type": "string"
    },
    {
      "facet": false,
      "index": true,
      "infix": false,
      "locale": "",
      "name": "portions",
      "optional": false,
      "sort": false,
      "stem": false,
      "type": "object[]"
    },
    {
      "facet": false,
      "index": true,
      "infix": false,
      "locale": "",
      "name": "portions.portionSizeId",
      "optional": false,
      "reference": "portionSizes.uid",
      "sort": false,
      "stem": false,
      "type": "string[]"
    },
    {
      "facet": false,
      "index": true,
      "infix": false,
      "locale": "",
      "name": "portions.amount",
      "optional": true,
      "sort": false,
      "stem": false,
      "type": "float[]"
    },
    {
      "facet": false,
      "index": true,
      "infix": false,
      "locale": "",
      "name": "portions.multiple",
      "optional": false,
      "sort": false,
      "stem": false,
      "type": "float[]"
    },
    {
      "facet": false,
      "index": true,
      "infix": false,
      "locale": "",
      "name": "portions.type",
      "optional": false,
      "sort": false,
      "stem": false,
      "type": "int32[]"
    },
    {
      "facet": false,
      "index": true,
      "infix": false,
      "locale": "",
      "name": "portions.measureUnitId",
      "optional": true,
      "sort": false,
      "stem": false,
      "type": "string[]"
    }
  ],
  "name": "food",
  "num_documents": 20175,
  "symbols_to_index": [],
  "token_separators": []
}
k
If we are not able to reproduce the issue would you be fine in taking a copy of your data for debugging on our end? Maybe we can do staging data if same problem occurs there as well.
We will delete the data once we are done testing.
s
you can just login to our dev cluster and test with the data in there
๐Ÿ‘ 1
k
We've identified the issue and have a fix for it as well. Will share the build later today.
s
Got it. Thanks for the update !
k
Please try upgrading your dev cluster to
0.26.0.rc67
and check.
โœ… 1
s
i dont see 67 in the list of upgrades
k
Sorry forgot to update that, you should see it now.
s
Ok Iโ€™ll check
yep working perfectly now
for
:nest
and
:merge
thanks
๐Ÿ™Œ 1
hey guys
i suspect there might be another bug with scoped keys + join + alias
we have a scoped key created, based off parent key:
Copy code
ID โ†“	Description	Key Prefix	Actions	Collections	Expires At
8	Search only key for all user collections	u9RB	[ "documents:search" ]	[ "user_.+" ]	01/01/4021, 10:59:59
scoped key in use as follows:
QWJLMFVkVFlNeUx3dDh4aVBwbDJGbHlGdEFObEFNTmQ4bjYrL1dCNzRHRT11OVJCeyJleHBpcmVzX2F0IjoxNzE0MDMwMDI0LCJmaWx0ZXJfYnkiOiIkdXNlcl91c2VyRm9vZCh1c2VySWQ6PXRTMVJnZElpUllZNVZGZ1dMNVAxSTdFSTlVTjIpIiwibGltaXRfaGl0cyI6NTB9
Decodes as:
AbK0UdTYMyLwt8xiPpl2FlyFtANlAMNd8n6+/WB74GE=u9RB{"expires_at":1714030024,"filter_by":"$user_userFood(userId:=tS1RgdIiRYY5VFgWL5P1I7EI9UN2)","limit_hits":50}
I can see parent key is a match
But the filter_by clause doesnt work
We get this error thrown
Copy code
org.typesense.api.exceptions.RequestUnauthorized:             Request failed with status code: 401.Server said: Forbidden - a valid `x-typesense-api-key` header must be sent.
As far as I can see, it should work fine no?
The filter by has the collection to reference, properly aliased
and if we revert to simply using a parent API key which can search and add
&& $user_userFood(userId:=tS1RgdIiRYY5VFgWL5P1I7EI9UN2)
to the filter by clause sent locally from client, get results back - so I dont see its a malformed query either
THoughts @Kishore Nallan
k
Which cluster is this happening?
s
Dev one
Same one you did the testing on from before
k
Ok so you are saying that the same filter_by clause works when you make the same request with parent search API key right?
s
Yeah
But when itโ€™s inside the scoped key
We get that error
k
Can you verify that the parent API key indeed starts with
u9RB
s
So Iโ€™m not sure if we are doing something dumb or
Yeah I wrote that above
k
Ok got it, let me look and get back to you.
s
Thanks
You can test on the dev cluster
๐Ÿ‘ 1
any luck >?
k
Sorry was swamped yesterday. Looking at it now. Can you please post the full search query?
Or even the collection name being searched upon is fine. I see that the filter_by references
$user_userFood
--> what is name of the primary collection queried?
s
np, thanks for getting to it now
primary collection being queried is
pub_food
(thats the alias, real collection name is
food
)
but i know when using the scoped key with regex match, we need to use the aliases for all references or wont work
k
The interesting thing to me is that the regular search key works. Hold on let me try to reproduce.
s
Ok
k
Able to reproduce. Will debug what's happening.
s
Cool thanks
k
But even when i use the parent search api key, I see the same auth error, though.
So it's not that parent search api key works. The issue is that the searh api key has permission for
user_.+
but here we are querying against
pub_food
s
Right
So we need to update the regex to (pub|user)_.+
k
Yes that will work.
s
And should likely work
๐Ÿ‘ 1
As would need access to either aliased collections
Just hard as the error thrown is very limited on why - for security I guess
k
Yes we can't reveal the reason, which is annoying but can be used to leak vulnerability, if any
s
yah agreed
unless we can do a debug mode or something
that can provide more details in trusted env mayve
maybe
I get the same error for both parent api key and scoped api key, and that's the correct behavior given permission set on the key.
s
yeah
So when setting the regex, for collections allowed, do we need to have every one listed that may be referenced
including the actual collections queried
k
Right now we only auth against primary collection.
s
such as where "user" collections should have access on that key, but we also wanna include "public" collections too
ahh ok got it - so its the primary searched which needs to be in there?
k
Because when the primary collection references a secondary collection, we assume that the auth flows.
s
as in this case we search pub_food
k
Yes
s
but the reference collections is user_userFood
k
If you plan to query either and reference either then both are needed. Otherwise only primary collection needs to be given permission in API key.
s
so if its just the primary collection, wouldnt that be a hole?
as I could thus use the parent key with
[ "pub_.+" ]
as you said the auth will flow through
and the collection queried is
pub_food
only in the filter by (as sent inside scoped key), do we have
$user_userFood(userId:=x)
k
Since the collection schema needs to add a reference field to join on child collections, auth flow is fine because that's an explicit decision.
s
yeah but then a key with collection access to
pub_food
would allow ?
consider the following parent keys
k
Let's say there are 2 collections X and Y where X references Y. There is an API key A that allows access to X only. Now when you query X and then add Y in filter_by as reference join, it will work. But you can't use A to query Y directly.
s
yeah so for the food collection schema, which references userFood
access is granted with parent key aliased as
pub_food
references the userFood, which is supposed to be locked down with parent key only allowing aliased collections
user_
otherwise why would we even need a parent key with collections:
[ "user_.+" ]
as we never query
user_userFood
k
Are you saying you want be able to define a key which allows only access to
user_.+
and then be able to execute this query?
s
no i dont mean that
if i understand now correctly, I think the parent key would just have collections
[ "pub_.+" ]
thereby stopping any user from querying collections aliased with
user_
etc.
we then create scoped key off that with
$user_userFood(userId:=x)
included
we then query
pub_food
(which is allowed based on parent)
and the reference to
user_userFood
inside the filterby will work, because its already explicitly defined in food schema
k
Yes, correct.
s
but if someone tries to user that scoped key (or parent key for that matter), to query
userFood
/
user_userFood
alias directly, it wont work
given it only supports collections prefixed with
pub_
k
Correct
s
yeah gotcha
so we need to update the parent key used to generate the scoped it seems like
this key:
Copy code
8	Search only key for all user collections	u9RB	[ "documents:search" ]	[ "user_.+" ]
can still remain, but not in use
it would only be useful if we wanted to query one of those collections direct
k
Yes, create new search api key and discard the old one. use the new one to created new scoped key
No harm in keeping it.
s
yeah
well currently we have:
Copy code
paramaters = { ...defaultParamaters, ...{ filter_by: `$user_userFood(userId:=${userId})`, limit_hits: 50 } };
    result.userFood = client.keys().generateScopedSearchKey(userSearchAPIKey, paramaters);
which we need to change
userSearchAPIKey
>
publicSearchAPIKey
i think
k
Correct
s
then the scoped keys will all be made from the parent key with
pub_
allowed for collections, thus, searching
pub_food
or
pub_meals
will succeed and then the referenced collections should be all good as they were defined
in theory then, we can refer to the referenced collections simply
userFood
and not via alias of
user_userFood
, i would think that should work
๐Ÿ‘ 1
like is there any good reason to use the aliases for reference collections
i guess it keeps it consistent
it obv doesnt harm to do so
k
Gives you flexibility in future to create a new collection with new schema and swap to that without changing app code.
s
how so?
i dont follow
Additionally, another question on the joins output, we have collection schema with reference field
types
of
string[]
because the field is array of string ids. When joining with
"include_fields": "$pub_exerciseTypes(name, deleted :nest) as exerciseTypes"
we get like this:
Copy code
exerciseTypes: [
          { deleted: false, name: 'Crossfit' },
          { deleted: false, name: 'Recreational' },
          { deleted: false, name: 'Casual' },
          { deleted: false, name: 'Cardio' }
        ],
        types: [
          'n1NByFcILax1z1qyZ8H3',
          'WMCsYQMLLj2FvjW8do6k',
          '06svI9qtNw4oIBwGXsGv',
          '1LO22SsEC0UEvDV2Daif'
        ],
Which is correct as we had 4 ids, each one gets an object which matched and the respective data based on match. However, is there any way to present/map the results more like this instead:
Copy code
exerciseTypes: [ // or could be 'types' too, doesn't matter
          { uid: 'n1NByFcILax1z1qyZ8H3', deleted: false, name: 'Crossfit' },
          { uid: 'WMCsYQMLLj2FvjW8do6k', deleted: false, name: 'Recreational' },
          { uid: '06svI9qtNw4oIBwGXsGv', deleted: false, name: 'Casual' },
          { uid: '1LO22SsEC0UEvDV2Daif', deleted: false, name: 'Cardio' }
        ],
Tried changing schema for
types
to
object[]
to get it more like the above, but of course it doesn't work as
types
is not an array of objects, just an array of strings. As you know, something like the above/desired is possible with type
object[]
and
:merge
assuming the referenced field started out in an array of objects. But we have no need here to over complicate an array of simple data type (string/int) into an array of objects like
types: [{ uid: 'a' }, { uid: 'b' }]
as this is overkill and also interferes with filtering on ID in types field anyhow. I guess the idea is rather than 2 arrays returned - source which is array of simple values like int/string and then another array of objects, containing each field, in same position, merging in the original array of simple values into the result would present the data much more cohesively. Thoughts?
k
The
types
field refers to the
id
field of exericeTypes collection?
s
ya does
k
Then you can include the
id
as well in
include_fields
?
s
what do you mean sorry
k
Like this:
Copy code
"include_fields": "$pub_exerciseTypes(id, name, deleted :nest)
s
no no
the exercise collection has a property
types
like
Copy code
types: [
          'n1NByFcILax1z1qyZ8H3',
          'WMCsYQMLLj2FvjW8do6k',
          '06svI9qtNw4oIBwGXsGv',
          '1LO22SsEC0UEvDV2Daif'
        ],
per doc
thats it
we are using join to grab the matching data from another collection
exerciseTypes
for each ID in types[]
hence in results we end up with
types[]
and
exerciseTypes[]
as shown in output above
k
So you want to map this array with the corresponding array of objects that you get from join
s
yeah
ideally
as you can see above in real output vs next message where i put a proposed type output
just throwing the situation out for discussion
as an array of simple ids which need to join is common enough, but then we currently get back the joined data in a new array
and so results are a little disjointed (even though I know each index is matching from A <> B)
k
This will be a new feature and that will be orthogonal to join because one could easily make the case for doing this post processing even within a collection's fields.
s
yeah exactly
i mean its easy enough to map/merge them post
im just throwing it out there
like we could do on server side maybe
:merge=key
k
But for this particular case, you can store
n1NByFcILax1z1qyZ8H3
as part of the
{ deleted: false, name: 'Crossfit' }
object?
s
and that would apply merge, giving each simple value the property name of
key
in merged array
i dont think you follow
{ deleted: false, name: 'Crossfit' }
this is the results back from query after join
source data is like:
Copy code
{
  "allowAutoIncrement": false,
  "allowSets": false,
  "allowWarmupSets": false,
  "amrapVisibility": 2,
  "bodyParts": [
    "kWEtuAPIgWoqWbf3jauS",
    "OXCLMw5spVo9Xnacjhzr",
    "PFWvobeFyYHBDHiiklgJ",
    "m1EWX9asdXtGDPJB9iWN",
    "Uqz1pdhGaJXtabsBlaU0"
  ],
  "deleted": false,
  "displayWeightCalculator": false,
  "distance": {
    "precision": 0
  },
  "distanceVisibility": 0,
  "equipment": [],
  "height": {
    "precision": null
  },
  "heightVisibility": 2,
  "isBodyweight": true,
  "isLap": false,
  "lapsVisibility": 1,
  "name": "Swimming",
  "predefined": true,
  "repsVisibility": 2,
  "setting": 2,
  "tempoVisibility": 0,
  "trackrpeVisibility": false,
  "types": [
    "n1NByFcILax1z1qyZ8H3",
    "WMCsYQMLLj2FvjW8do6k",
    "8TGOKPmPlyGsMMmFs8KD",
    "1LO22SsEC0UEvDV2Daif"
  ],
  "uid": "b5kF7VC2pTsVAISDtPmx",
  "verified": true,
  "weightVisibility": 2
}
k
Maybe create a github issue with 2 simple collections and schema? Happy to consider as a feature request.
We can also use the issue for tracking.
s
query is like:
Copy code
"include_fields": "$pub_bodyParts(name, deleted :nest) as bodyPartsData, $pub_exerciseTypes(name, deleted :nest) as exerciseTypes"
schema have like:
Copy code
{
      "facet": false,
      "index": true,
      "infix": false,
      "locale": "",
      "name": "bodyParts",
      "optional": false,
      "reference": "bodyParts.uid",
      "sort": false,
      "stem": false,
      "type": "string[]"
    },
so the join works on each string id in bodyParts[] or types[]
but we then have the original array of strings
with another array of values which are added from the join
i was just thinking of a way to merge the 2
k
Why wouldn't this work?
Copy code
"include_fields": "$pub_bodyParts(name, deleted :nest) as bodyPartsData, $pub_exerciseTypes(uid, name, deleted :nest) as exerciseTypes"
s
didnt try that, hang on
Duh!!
of course haha!
i dont know why i didnt see that
but thanks
k
๐Ÿ˜„ welcome
s
hence why i threw the problem out for discussion as way to accomplish
as i clearly didnt think that through
k
This might still be useful in another context. Happy to consider if something like that shows up.
s
yeah exactly
hasnt as of yet
but we are only gradually exploring all the use cases for the joins
so as diff schemas come up, trying out as needed
must be close to pub release by now yea?
k
Yes, just documenting stuff. Pending one final change to syntax for join to make it conform to our other syntax changes. After that we will release.
s
nice work
not long now!
k
Yes, hopefully end of this week or early next.
โค๏ธ 1
s
I will test the other scenario with parent pub key and confirm all is working when the relevant dev is online shortly
๐Ÿ‘ 1
ok so update
it works to query
pub_food
when changing the scoped keys to be created from a parent key which allows collections
[ "pub_.+" ]
to be able to reference other collections like
user_userFood
however, this seems wrong from what you described - you said that the reason it should allow access is because the collection being queried has already explicitly made reference to another in schema, there should be access - which made sense
but when i check the food schema, it doesnt reference userFood at all
the userFood schema references the food collection
thus, is it allowed bi-directionally
k
Yes the reference is a bi directional link.
s
so as long as there is a reference in place
gotcha ok cool
yep makes sense
well that is weorking now then and clear on how it should be secured thanks
๐Ÿ‘ 1
b
Bit of a long thread, it seems I'm seeing very similar issues with scoped key not working at all. Tried 0.25.2 and 0.26.0rc68. @Shaun Byrnes did you actually get this working?
s
Ah yep
Scoped keys work for me
๐Ÿ‘ 1
Provided that the parent key has correct permission to access the collection being queried
Most this thread is related to join implementations, but we also use the scoped keys with it
b
Yeah, you just had exactly the same error. I can't get scoped keys to work at all, even with collections as "*", so can't see that being a blocker.
s
hmmm not entirely sure without more details, but it does work (and with joins too). The security error will be the same for possibly lots of things as it hides the real reason for security reasons, so a little hard to debug
but you could try the same query with the parent key and confirm that works first, then look at the included/filter by conditions in the scoped key, as well as check the expiry date
k
@Shaun Byrnes We have published
0.26.0.rc68
which contains a syntax change for join. The nest/merge of fields syntax is now through the
strategy
keyword:
Copy code
"include_fields": "$authors(*, strategy: merge)"
s
Ok got it thanks. I presume you mean rc69 tho? As iirc we are already on 68
Also, will it appear in TS cloud for us to update?
k
No you are on
0.26.0.rc67
Yes, it's now available on TS Cloud.
s
ah yes youre correct
ok updatng now
does it look for that keyword
strategy
after last comma yea? That way
$coll(id, strategy, strategy: nest)
would work still correct ?
as there might be a prop in doc called
strategy
h
strategy:
is what the parser looks for. So having
strategy
porp would not interfere.
s
yep makes sense
also as i checked its happy with
strategy:nest
as in no space
โœ… 1
i guess the regex just check
strategy\:\s*(nest|merge)
or somethink like that