https://typesense.org logo
Join Slack
Powered by
Question:  Hello everyone , I have an issue with t...
# community-help
s
Question:  Hello everyone , I have an issue with the scoped API key which can access data from collections out of the given scope. As I want to use search query on schema using scoped api key. Firstly I get the typesense server using wget . I used a typesense-go client for that server . For configuration , I used the default API key given the typesense document  to generate the search api key for a specific schema by creating a new client . Then I passed this search api key to the typesense server and created a schema. After that I used that API key to generate a scoped api key. Schema structure : func generateSchema(client *typesense.Client) *api.CollectionSchema {    schema := &api.CollectionSchema{        Name: "DemoCollection",        Fields: []api.Field{            {                Name: "id",                Type: "string",            },            {                Name: "name",                Type: "string",            },              {                Name: "accessible_to_user_ids",                Type: "int32",            },        },    }    return schema   }   As in this schema I used field accessible_to_user_ids ,on which I used filter_by parameter to generate scoped api key.So it can access only that ids for search query. Scoped key generation using search api key for schema as follow: func generateScopedKey(key string) string {    params := url.Values{}      //params.Add("filter_by", "accessible_to_user_ids:[\"1\"] ")    params.Add("filter_by", "accessible_to_user_ids:1")    paramsEncode := params.Encode()      hash := hmac.New(sha256.New, []byte(key))    hash.Write([]byte(paramsEncode))    KeyInBase64 := base64.StdEncoding.EncodeToString(hash.Sum(nil))      return KeyInBase64   }   Here I used filter by parameter to apply a search query on those ids that are included in the field of accessible_to_user_ids of id:1, but it’s not happening. This is how I generated a scoped key and started a search server with a scoped api key.But I can still access all documents.What am I doing wrong?
✅ 1
k
👋 Can you post a fully reproduceable Golang example as a Github gist? Also, note that the Golang client has an issue dealing with encoding of special characters in the filter and other params: https://github.com/typesense/typesense-go/issues/71
s
okay thanks ..I will send github link here
k
One quick way to rule out the encoding issue is to generate the API key in Go but trying making a CURL request with the API key.
s
I also tried using CURL request to produce scoped api key but it's now working..
Copy code
KEY_WITH_SEARCH_PERMISSIONS="RN23GFr1s6jQ9kgSNg2O7fYcAUXU7127"
EMBEDDED_SEARCH_PARAMETERS_JSON='{"filter_by":"accessible_user_id:1","expires_at":3006054106}'

digest=$(echo -n $EMBEDDED_SEARCH_PARAMETERS_JSON | openssl dgst -sha256 -hmac $KEY_WITH_SEARCH_PERMISSIONS -binary | base64 -w0)

scoped_api_key=$(echo -n "${digest}${KEY_WITH_SEARCH_PERMISSIONS:0:4}${EMBEDDED_SEARCH_PARAMETERS_JSON}" | base64 -w0)

echo $scoped_api_key
means that key will access all documents from the schema
k
Okay please post a full gist in either golang or with curl and I can look into it. Everything from collection creation, sample record insertion (1 should be enough), API key generation and then finally querying.
s
okay
k
ty I will check.
s
okay
one more question , if I am using scoped api key for search query , that also need to passed to typesense server.Otherwise it gave error as status: 401 response: {"message": "Forbidden - a valid 
x-typesense-api-key
header must be sent."}
k
that also need to passed to typesense server.
I don't follow what you mean by this.
s
when I want to call api using search api key then that api key needs to pass typesense server so that api will be called
But after I generated scoped api key from that search api key , and want to call api then there is need to pass newly generated scoped key pass to typesense server
means I asked that every time when scoped api key generated from parent key , then needs to pass to typesense server?
k
Yes, once you generated scoped API key, that's what you should be sending to Typesense.
s
ok
Is there any another way for that?
k
Why do you not want to send the scoped API key to the server?
Scoped API key == search api key + filters that cannot be changed. So it's a replacement for your search API key.
s
as I want apply filter on user ids to access documents so want scoped api key
k
Let me first look into the other golang issue. I will get back to you shortly on that. I can then answer this other question.
s
okay thanks
k
@Suvarna Chavan The 
generateScopedKey
function is not correct. Here's a fixed version: https://gist.github.com/kishorenc/853bedde59c73d4e2de8a031778be3a7 Also, the 
key
that you pass into this function MUST be a search-only API key: https://typesense.org/docs/0.21.0/api/api-keys.html#search-only-api-key
And, once you generate the scoped API key, you will have to create a new client object that uses this key:
Copy code
scoped_client := typesense.NewClient(
		typesense.WithServer("<http://localhost:8108>"),
		typesense.WithAPIKey("<SCOPED API KEY HERE>"))
s
okay..thanks
I used search only api key to generate scoped key..but as function incorrect so...thank you so much for help😊
k
yw
s
can you please tell me , when you are going to create search only api key before that needs to create client with api key..so which key tou used there
k
Use admin key for the client used to create search only api key.
Once you generate that you can just store it separately and use that to generate the scoped API key. Then use this scoped API key for the actual client used to make requests.
s
okay..I got it..but for client creation there is need of server url and api key..at that time which key used there
k
There will be a key you pass to start the Typesense server. That's the admin key.
s
yes..means I want to ask that for to generate admin api key need to be client.Foe client creation needs api key and server url...so which key used at first client creation...
k
Check this: https://typesense.org/docs/0.21.0/guide/install-typesense.html#%F0%9F%8E%AC-start Look how the server is started by passing an API key: 
./typesense-server --data-dir=/tmp/typesense-data --api-key=$TYPESENSE_API_KEY
Whatever API key you use to start the server is what you use initially -- that's the admin/master API key.
s
yes I know it..my question is different ..for creating first client need to give api key as argument and that's api key need to pass to server to initiate....so that's which will be first api key to create client and to initiate server?
or we can take random key there to start
k
For starting Typesense server?
s
yes
k
You can use a long, random key to start the server.
Then use that key in the client.
On Typesense Cloud, this key will be generated for you on the UI and you can download directly.
s
okay
Got it..thanks😊
👍 1
It's working now..as I took admin key from cloud
k
👍 Please delete the gist if it contains any actual keys.
s
no..it's random keys used..although I will delete it..one last query..every time I need to generate key from cloud or can I used it for long time?
k
The key that you download from the Cloud does not have an expiry. You can use it but I suggest generating actual secondary keys from that and using those instead.
That way, the key from the Cloud is the "master key" which can be used to manage these other secondary keys.
s
okay ..thanks👍
hello..can you please tell me , if we generate api key from typesense cloud and generate secondary keys from that key ..As you said no expiry of cloud generate key , I have query that , is there any expiry for these secondary keys
k
By default, none unless you specify it during key creation.
s
okay..as we passed data directory as /tmp/typesense-data for logs ,db for storing collections....but how can I check data saved in database..
through api calls I am able to get data stored in collection.....but which db used to store data..or how it's working..can you please explain shortly
k
The data is stored in 
/tmp/typesense-data
-- indices are in-memory.
s
Can I connect psql server to store data into our database?
k
No
s
okay , thank you!
hello..how can we update schema..means if I want to add custom fields into schema?
k
We don't have a way to update schema at the moment, but we plan to ship this feature in the next couple of months.
You have to create a new collection and use the alias feature to swap over.
s
ok thanks for help😊
👍 1
hello..while updating document in typesense , I have 201 status code in response.Can you please tell me why 201 ?
k
That's the standard status code to return when a new resource is created as per REST API conventions.
s
ok..thanks for reply.
👍 1
Hey , when I created API key with specific actions and collections and using that API key going to create a collection with name "xyz" but I have error of 401-\"Forbidden - a valid 
x-typesense-api-key
header must be sent..As I gave actions : "collections:create" for "xyz" collection.
This key generated using admin key which was generated by the typesense cloud. And I pass the admin key to typesense server. Is there need to pass that scoped api key to typesense server?
k
So you are saying that you have created a new API key with the 
collections:create
permission by using the admin API key, and then using this new API key to create a collection, but running into that error when you do that?
s
yes
k
Give me some time, I will check and get back to you.
s
sure
k
It works for me, check here: https://gist.github.com/kishorenc/825cc8f838db86030de6fa958da07749 If you can provide me a reproduceable example like that, I can take a look at why it's failing for you.
s
ok..I will check it and give you example
I am also trying to do same thing..with specific collections..but not able to create the schema using that new api key
k
Scoped API key can be created only from search only API key which restricts it to only search operations, so you cannot create a collection using that.
Even though the function 
generateScopedBasedKey
is named that way, it seems like it is generating only a normal key?
s
yes
k
Can you hit the list key end-point and post the snippet for the key that you are using? https://typesense.org/docs/0.21.0/api/api-keys.html#list-all-keys
s
means I have to create normal key based on some actions and collections
ok
message has been deleted
k
Can you double check that the client that is used to generate schema is indeed using a key that starts with 
x68v
?
In the mean time I will try the go example.
@Suvarna Chavan What version of Typesense are you using? Your Go example works for me perfectly.
s
I am using this version 0.21.0.
I will check that client using that key or not
k
I'm also running 0.21.0 locally. Are you able to reproduce the failure locally on the go example you gave me?
I started a 0.21.0 server and ran your code, and the test collection was created successfully.
s
But I have an error..as I took like this..client := tsClient("x68vIkmZpM6OjnibSKXgPeunWss51HtJ") fmt.Println("client info ...", client.Key(76)) schema := generateSchema(client)
k
Does the example go program produce the error locally?
s
yes
k
There must be something different. Because it runs fine. Maybe try doing the same thing with curl like my example earlier? It might help in finding out the issue.
s
ok..I will try it
Using curl it works for another key newly generated..As I specified only one collection while creating that key but using that key , I can create another collection also ,.... -d '{"description":"scoped key.","actions": ["collections:create"], "collections": ["openings"]}'
k
I don't follow you fully, but the best way for me to debug is if I can reproduce the issue somehow. Maybe you can get a curl snippet that fails?
s
I will share with you that curl here..
k
I don't see any errors in the screenshot.
s
there is no errors..as I created api key with actions[collectionscreate] and collections:["openings"]..but using that api key I can generate "open" collection also
this is I want to show you in that screenshot
k
Ah I see, what you mean.
We don't check the API key's collection whitelist while creating the collection at the moment.
I will add a todo for us to address this.
s
Ok thanks..
k
ty for reporting this.
s
It's my pleasure!
when I am going to create secondary API key from admin key with thie actions-> "actions":["collections:*","documents:*"], on collections -> "collections":["abc"],.Bu that new key will not able to create schema with name "abc"
k
It must be an array:
Copy code
"collections": ["*"]
s
yes , it's an array..as like this "actions":["collections:*","documents:*"],
sorry..got it..what you want to say
I want API key with all actions on specific collections..So I passed as like this:
Copy code
"actions":["*"],
    "collections":["openings"],
k
Okay, and what problem are you facing with that?
s
when I am going to create openings schema with the newly generated api key..then it's not able to create it
k
Please provide curl snippet like yesterday.
Wait, this is the same problem you faced yesterday right? We don't check the API key's collection whitelist while creating the collection at the moment.
s
yes..it's the same problem..
k
Yeah we have to fix that. For now you have to use 
*
s
ok
thanks for helping!
k
👍
s
Hey , today I tried curl..firstly I generate api key with scope of specific actions and collections from admin key..Then using newly generated key I created that collections specifying in collections of key generation params. Also able to create documents using this new key.And if I am going to access the action which was not mention in the list of actions then it throws an error
But same thing when I tried with go program ..It creates new api key but using that api key , I am not able to create schema..can you please tell me why this is happening?
I have an error like this when trying to create schema using new key : status: 401 response:
Copy code
{
  "message": "Forbidden - a valid `x-typesense-api-key` header must be sent."
}
k
If it works in curl but not in your go program, it must be some mistake in the Go program.
s
ok..But when I am using new api key in header of postman and call to create schema API through the postman then also give me above error of 401
k
If you can give me the set of CURL snippets to run that show cases the problem I can check.
s
ok ..I will send it
Using curl
k
What is the error here?
s
There is no error..Using CURL it works fine but when using go program, when I am creating new scoped based key from the admin key.And trying to create schema using new key gives error..
Copy code
status: 401 response:

 {"message": "Forbidden - a valid `x-typesense-api-key` header must be sent."}
When using that new api key and call create schema api through the postman also give same error
k
Postman should work exactly the same way as Curl. Are you sure the header is being set correctly in postman to the new key?
s
Yes ..I set header as like : x-typesense-api-key
Hello ..the problem of scoped API key getting solved..It caused due to old version of go client.But now it is working as per the scope given in action and collections. Sorry for inconvenience.....
thanks for help!
k
Glad to hear 🎉 You're welcome.
😊 1
s
Hey ..as I discussed when I created scoped API key for specific action and collections.Actions:["collections:create] and collections:["xyz"].....Then using newly generated key I am able to create collection with name:"xyz" and also able to create any other collection which is not in the above list.
k
Yes, as I said, this is something we need to prioritize and add support for. I will look into it.
@Suvarna Chavan I have a fix for this issue. Do you use Docker for your local development? If so, I can give a build for you to test it locally to see if the issue is fixed.
s
ok thanks for helping..Now I am not using docker..but I have docker in my machine and used it.. So please share a build so I will try on my machine.
As I used typesense locally now..but in upcoming days , I will use docker for it.
Can you please tell me , using admin key we can generate only search key..Is it the only a way to create search key?
k
Yes, use the admin key to create search keys. I will create and share the built soon.
s
ok thank you so much!!
k
@Suvarna Chavan Check against this Docker image:
Copy code
typesense/typesense:0.22.0.rcs8
s
yes sure..
hello..It's working as per the access..thank you so much for helping!!
k
Glad to hear. Do you use Typesense Cloud? If you do, we can also upgrade that to the same RC build.
s
No , I am not using Typesense cloud ..
👍 1
hello..as I wrote wrapper on typesense to store data into database so still no required cloud.But in future cloud will be needed..At that time I will let you know..
k
Sure!
s
Hey, good evening..I generated two keys from the admin key with permission of create collections...So can we create same schemas with these both keys?..because I am not able to do that..as 409 error..
or when we can generate same schemas?
k
Can you please post the curl examples? Easiest way for me check 🙂
s
yes sure
Open in Slack
PreviousNext
Powered by