Typesense

Typesense

#community-help

Issues with Indexing Data using Typesense in Go

TLDR Konrad was having trouble with a "Forbidden" error using Typesense in Go when indexing data remotely. David offered multiple troubleshooting steps and suggestions. Ultimately, Konrad discovered the issue lied within the application code.

Powered by Struct AI
49
3mo
Join the chat
Sep 03, 2023 (3 months ago)
Konrad
Photo of md5-756d5da34cc5127c88730a39db749024
Konrad
08:09 PM
Hi! I've been starting to use Typesense, but now I keep getting a not really descriptive error "Forbidden" when trying to index data. I'm using the Go library. Any ideas why it does this? The same code worked fine locally but now doesn't when I throw real data at it.

Any ideas how to solve this or figure out what's wrong?
08:12
Konrad
08:12 PM
I was thinking the problem might be caused by a wrong api key but that seems like it's not the problem.
08:18
Konrad
08:18 PM
I was able to connect to it via https://bfritscher.github.io/typesense-dashboard/ so it looks like this is not an api key issue.
08:19
Konrad
08:19 PM
also I now get errors when I search for the data from my application, looks like the api key can connect to Typesense and it is not the problem
David
Photo of md5-5c5edeceeb0deef59cc5dcc791ce7045
David
08:32 PM
can you give a full error? is it just a 403
08:36
David
08:36 PM
if it were api key related the error would be {"message": "Forbidden - a valid x-typesense-api-key header must be sent."}
Konrad
Photo of md5-756d5da34cc5127c88730a39db749024
Konrad
08:37 PM
It's literally only the message "Forbidden"
08:38
Konrad
08:38 PM
I have code like this:

    _, err = typesenseClient.Collection("tasks").
        Documents().
        Import(typesenseTasks, &api.ImportDocumentsParams{
            Action:    pointer.String("upsert"),
            BatchSize: (100),
        })
08:38
Konrad
08:38 PM
and that seems to return the error (actually I'm only 90% sure this is the part that's throwing the error but I'm about to find out for sure)
David
Photo of md5-5c5edeceeb0deef59cc5dcc791ce7045
David
08:39 PM
if it's only happening when you're accessing it remotely you might check your webserver configuration
Konrad
Photo of md5-756d5da34cc5127c88730a39db749024
Konrad
08:40 PM
No, I run it via docker compose and am accessing it from the container
08:40
Konrad
08:40 PM
08:40
Konrad
08:40 PM
It seems to work fine when I access it remotely, just not when I access it via the cli
08:41
Konrad
08:41 PM
Looks like the Schema was created successfully, it's the indexing that fails... does not look like a connection problem
David
Photo of md5-5c5edeceeb0deef59cc5dcc791ce7045
David
08:42 PM
your example does not follow example given in the typesense-go repo
08:43
David
08:43 PM
that pointer is not what they show
08:43
David
08:43 PM
pointer.String("upsert")
08:44
David
08:44 PM
does it work if you do one document at a time like they show here https://github.com/typesense/typesense-go#upserting-a-document
08:46
David
08:46 PM
08:46
David
08:46 PM
their Import call does not use a pointer.string to define the action
08:46
David
08:46 PM
try using just Action: "upsert" instead
Konrad
Photo of md5-756d5da34cc5127c88730a39db749024
Konrad
08:48 PM
That does not work because the type is defined as *string (not string) in the params struct
08:49
Konrad
08:49 PM
I'll try one document at a time
David
Photo of md5-5c5edeceeb0deef59cc5dcc791ce7045
David
08:52 PM
i'm not sure you can do an upsert with an import
08:53
David
08:53 PM
hm no you can
08:55
David
08:55 PM
you might try writing the data to a jsonl and execute the upsert import request from the shell
08:56
David
08:56 PM
curl -H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}" -X POST --data-binary @documents.jsonl \
''

Konrad
Photo of md5-756d5da34cc5127c88730a39db749024
Konrad
08:57 PM
jsonl is just one json document per line right?
David
Photo of md5-5c5edeceeb0deef59cc5dcc791ce7045
David
08:57 PM
ya, i found it easiest to write to a regular json and then use jq to convert to jsonl like their docs show
08:57
David
08:57 PM
jq -c '.[]' documents.json > documents.jsonl
Konrad
Photo of md5-756d5da34cc5127c88730a39db749024
Konrad
09:01 PM
Doing this with curl works
David
Photo of md5-5c5edeceeb0deef59cc5dcc791ce7045
David
09:01 PM
i wonder if it's how you're defining the params inline
09:01
David
09:01 PM
in their examples they define params separately and pass it to the import method
09:01
David
09:01 PM
params := &api.ImportDocumentsParams{Action: pointer.String("create")}
    responses, err := typesenseClient.Collection(collectionName).Documents().Import(documents, params)
Konrad
Photo of md5-756d5da34cc5127c88730a39db749024
Konrad
09:01 PM
it shouldn't really make a difference
David
Photo of md5-5c5edeceeb0deef59cc5dcc791ce7045
David
09:02 PM
_, err =
should that be _, err :=?
Konrad
Photo of md5-756d5da34cc5127c88730a39db749024
Konrad
09:02 PM
no, err is already defined in my case
09:03
Konrad
09:03 PM
:= defines a new variable on the fly but it will give you a compile error when you try to redeclare an existing variable
David
Photo of md5-5c5edeceeb0deef59cc5dcc791ce7045
David
09:07 PM
the import function just builds a jsonl and calls importjsonl
09:08
David
09:08 PM
09:09
David
09:09 PM
i don't see anywhere that typesense would reply with 403 forbidden, unauthorized (bad api key) would reply with 401 unauthorized
09:09
David
09:09 PM
i'm wondering if the error is occuring in the jsonDecoder logic
09:09
David
09:09 PM
or encoder
Konrad
Photo of md5-756d5da34cc5127c88730a39db749024
Konrad
09:10 PM
That would surprise me
David
Photo of md5-5c5edeceeb0deef59cc5dcc791ce7045
David
09:11 PM
either it's an http 403 forbidden, or something in the encoding is giving a "forbidden" error
Konrad
Photo of md5-756d5da34cc5127c88730a39db749024
Konrad
09:11 PM
But I guess we'll find out once I deployed the Upsert per document
Sep 04, 2023 (3 months ago)
Konrad
Photo of md5-756d5da34cc5127c88730a39db749024
Konrad
08:36 AM
Upserting each document individually did add a few records but stopped after ~27
08:36
Konrad
08:36 AM
I start to suspect this is not an issue of Typesense
09:13
Konrad
09:13 AM
jup, the issue was in my application code :man-facepalming:

Typesense

Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI

Indexed 3015 threads (79% resolved)

Join Our Community

Similar Threads

Issues with Importing Typesense Collection to Different Server

Kevin had problems migrating a Typesense collection between Docusaurus sites on different machines. Jason advised them on JSONL format, handling server hosting, and creating a collection schema before importing documents, leading to successful import.

3

35
3mo

Typesense Server Bulk Import/Upsert Issue Resolved

Adam was confused about the discrepancy between the successful responses and the actual indexed data while working with a custom WP plugin integrating with Typesense. The issue was a bug related to fetching documents in the wrong order, not a Typesense problem.

2

22
7mo

Troubleshooting Indexing Duration in Typesense Import

Alan asked about lengthy indexing times for importing documents to Typesense. Jason suggested various potential causes, including network connectivity and system resources. They later identified the problem to be an error in Alan's code.

5

43
15mo

Resolving Issues with Scoped API Keys in Typesense with Golang

Suvarna had problems with generating and using scoped API keys in Typesense with Golang. Several bugs misleading the user were found and fixed by Kishore Nallan.

6

158
28mo

Troubleshooting Typesense Document Import Error

Christopher had trouble importing 2.1M documents into Typesense due to memory errors. Jason clarified the system requirements, explaining the correlation between RAM and dataset size, and ways to tackle the issue. They both also discussed database-like query options.

3

30
11mo
Powered By