#community-help

Troubleshooting Indexing Duration in Typesense Import

TLDR Alan asked about lengthy indexing times for importing documents to Typesense. Jason suggested various potential causes, including network connectivity and system resources. They later identified the problem to be an error in Alan's code.

Powered by Struct AI

3

2

43
12mo
Solved
Join the chat
Oct 05, 2022 (12 months ago)
Alan
Photo of md5-6d168f201743aec43607f72d2864612d
Alan
06:33 PM
How long should 10k documents take to index in an import?
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
06:34 PM
It would vary depending on the number of fields in the document
Alan
Photo of md5-6d168f201743aec43607f72d2864612d
Alan
06:34 PM
4 -5
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
06:34 PM
That should take less than 1 minute
Alan
Photo of md5-6d168f201743aec43607f72d2864612d
Alan
06:34 PM
Any reason why it would be taking much longer?
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
06:35 PM
Potentially network connectivity from your machine to the typesense cluster
06:35
Jason
06:35 PM
Also you’re using the documents/import endpoint correct?
Alan
Photo of md5-6d168f201743aec43607f72d2864612d
Alan
06:35 PM
document().import

1

Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
06:35 PM
Another potential reason could be if your cluster is out of RAM / CPU
Alan
Photo of md5-6d168f201743aec43607f72d2864612d
Alan
06:36 PM
We're using Typesense cloud.
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
06:36 PM
Could you DM me your cluster ID, I can take a closer look
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
06:40 PM
Metrics look good, you have sufficient RAM and CPU. Could you share a curl snippet and a small dataset that replicates the issue? Feel free to DM me the snippet if the data set can’t be shared publicly
Alan
Photo of md5-6d168f201743aec43607f72d2864612d
Alan
06:41 PM
+return await client.collections(wid + -${ collection }).documents().import(documents, {action: isUpdate ? 'update' : 'upsert'})
.catch( err => {
console.log(err);
return { error: true, results: err.importResults }
});
06:43
Alan
06:43 PM
contacts: {
'name': null,
'fields': [
{
'name': 'id',
'type': 'string',
'facet': false
},
{
'name': 'number',
'type': 'string',
'facet': false
},
{
'name': 'number2',
'type': 'string',
'facet': false
},
{
'name': 'assigned_to',
'type': 'int64',
'facet': false
},
{
'name': 'channel_id',
'type': 'int64',
'facet': false
},
{
'name': 'meta',
'type': 'string',
'facet': false
},
{
'name': 'timestamp',
'type': 'int64',
'facet': false
}
],
'default_sorting_field': 'assigned_to'
},
06:43
Alan
06:43 PM
data gets formatted to match that schema
06:43
Alan
06:43 PM
No added fields
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
06:43 PM
I don’t see any significant inbound network traffic, so I wonder if the client connection settings are off. Could you share the snippet where client is instantiated?
Alan
Photo of md5-6d168f201743aec43607f72d2864612d
Alan
06:44 PM
const Typesense = require('typesense');

const config = {
nodes: [{
host: process.env.TYPESENSE_HOST,
port: process.env.TYPESENSE_PORT,
protocol: process.env.TYPESENSE_PROTOCOL
}],
apiKey: process.env.TYPESENSE_ADMIN_API_KEY,
connectionTimeoutSeconds: 5
};

const client = new Typesense.Client(config);

module.exports = client;
06:45
Alan
06:45 PM
Btw, small chunks get imported no problem
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
06:46 PM
Ahhh, I wonder if you might be running into a variation of this issue: https://github.com/typesense/typesense-js/issues/81
06:47
Jason
06:47 PM
Are these 10K documents on big JS array?
Alan
Photo of md5-6d168f201743aec43607f72d2864612d
Alan
06:48 PM
Yes
06:48
Alan
06:48 PM
Also, if it's 10k it doesn't actually finish the import
06:48
Alan
06:48 PM
If it's like 15 or something it does, but htat takes a while too.
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
06:49 PM
15 should be near instantaneous :thinking_face:
06:49
Jason
06:49 PM
Are you running this script locally or from a cloud server?
Alan
Photo of md5-6d168f201743aec43607f72d2864612d
Alan
06:50 PM
This is ran from my computer atm.
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
06:52 PM
Could you run this from your terminal and share the output: curl -v --trace-time <https://s6cl45z70rpan18ep-1.a1.typesense.net/health>
Alan
Photo of md5-6d168f201743aec43607f72d2864612d
Alan
06:53 PM
mac:EntranceTools alanboody$ curl -v --trace-time https://s6cl45z70rpan18ep-1.a1.typesense.net/health
'14:52:47.425229 * Trying 18.224.141.232...
14:52:47.425756 * TCP_NODELAY set
14:52:47.494845 * Connected to s6cl45z70rpan18ep-1.a1.typesense.net (18.224.141.232) port 443 (#0)
14:52:47.495083 * ALPN, offering h2
14:52:47.495116 * ALPN, offering http/1.1
14:52:47.507533 * successfully set certificate verify locations:
14:52:47.507563 * CAfile: /etc/ssl/cert.pem
CApath: none
14:52:47.507677 * TLSv1.2 (OUT), TLS handshake, Client hello (1):
14:52:47.575261 * TLSv1.2 (IN), TLS handshake, Server hello (2):
14:52:47.575632 * TLSv1.2 (IN), TLS handshake, Certificate (11):
14:52:47.577586 * TLSv1.2 (IN), TLS handshake, Server key exchange (12):
14:52:47.578544 * TLSv1.2 (IN), TLS handshake, Server finished (14):
14:52:47.585250 * TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
14:52:47.585288 * TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
14:52:47.585338 * TLSv1.2 (OUT), TLS handshake, Finished (20):
14:52:47.652901 * TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
14:52:47.653002 * TLSv1.2 (IN), TLS handshake, Finished (20):
14:52:47.653044 * SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
14:52:47.653073 * ALPN, server accepted to use h2
14:52:47.653101 * Server certificate:
14:52:47.653137 * subject: CN=.a1.typesense.net
14:52:47.653170 * start date: Jun 6 00:00:00 2022 GMT
14:52:47.653199 * expire date: May 27 23:59:59 2023 GMT
14:52:47.653239 * subjectAltName: host "s6cl45z70rpan18ep-1.a1.typesense.net" matched cert's "
.a1.typesense.net"
14:52:47.653281 * issuer: C=GB; ST=Greater Manchester; L=Salford; O=Sectigo Limited; CN=Sectigo RSA Domain Validation Secure Server CA
14:52:47.653309 * SSL certificate verify ok.
14:52:47.653361 * Using HTTP2, server supports multi-use
14:52:47.653390 * Connection state changed (HTTP/2 confirmed)
14:52:47.653419 * Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
14:52:47.653506 * Using Stream ID: 1 (easy handle 0x7fcf60808200)
14:52:47.653561 &gt; GET /health HTTP/2
14:52:47.653561 &gt; Host: s6cl45z70rpan18ep-1.a1.typesense.net
14:52:47.653561 &gt; User-Agent: curl/7.64.1
14:52:47.653561 &gt; Accept: /*
14:52:47.653561 &gt;
14:52:47.726042 * Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
14:52:47.726159 &lt; HTTP/2 200
14:52:47.726191 &lt; access-control-allow-origin: *
14:52:47.726219 &lt; content-type: application/json; charset=utf-8
14:52:47.726247 &lt;
14:52:47.726341 * Connection #0 to host s6cl45z70rpan18ep-1.a1.typesense.net left intact
{"ok":true}
Closing connection 0
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
06:56 PM
That looks fine
Alan
Photo of md5-6d168f201743aec43607f72d2864612d
Alan
06:57 PM
That wont' be the name when it gets created
06:58
Alan
06:58 PM
Does it have anything to do with that timeout at 5?
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
06:59 PM
If there was a timeout it should throw an error telling you it timed out, but yeah you definitely want to increase the client-side timeout to as high as even 30 minutes, to make sure the import call never times out
06:59
Jason
06:59 PM
For good measure, could you try increasing the timeout and trying the import again?
Alan
Photo of md5-6d168f201743aec43607f72d2864612d
Alan
07:01 PM
Running now.
07:05
Alan
07:05 PM
One sec may have spotted an issue.
07:08
Alan
07:08 PM
Found the issue. An edge case where it was checking if collection exists. Bug in my code. Thank you for your help nd sorry for wasting your time.

1

Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
07:08 PM
Phew! 😅
Alan
Photo of md5-6d168f201743aec43607f72d2864612d
Alan
07:09 PM
Yeah. lol
07:09
Alan
07:09 PM
That imported in like 1 sec too.

2

07:11
Alan
07:11 PM
Take care . Again, ty for the quick response as usual from your team.

1