Troubleshooting Indexing Duration in Typesense Import

How long should 10k documents take to index in an import?

It would vary depending on the number of fields in the document

4 -5

That should take less than 1 minute

Any reason why it would be taking much longer?

Potentially network connectivity from your machine to the typesense cluster

document().import

Another potential reason could be if your cluster is out of RAM / CPU

We're using Typesense cloud.

Could you DM me your cluster ID, I can take a closer look

Metrics look good, you have sufficient RAM and CPU. Could you share a curl snippet and a small dataset that replicates the issue? Feel free to DM me the snippet if the data set can’t be shared publicly

+return await client.collections(wid + -${ collection }).documents().import(documents, {action: isUpdate ? 'update' : 'upsert'})
.catch( err => {
console.log(err);
return { error: true, results: err.importResults }
});

I don’t see any significant inbound network traffic, so I wonder if the client connection settings are off. Could you share the snippet where client is instantiated?

const Typesense = require('typesense');

const config = {
nodes: [{
host: process.env.TYPESENSE_HOST,
port: process.env.TYPESENSE_PORT,
protocol: process.env.TYPESENSE_PROTOCOL
}],
apiKey: process.env.TYPESENSE_ADMIN_API_KEY,
connectionTimeoutSeconds: 5
};

const client = new Typesense.Client(config);

module.exports = client;

Ahhh, I wonder if you might be running into a variation of this issue: https://github.com/typesense/typesense-js/issues/81

Yes

15 should be near instantaneous :thinking_face:

This is ran from my computer atm.

Could you run this from your terminal and share the output: curl -v --trace-time <https://s6cl45z70rpan18ep-1.a1.typesense.net/health>

mac:EntranceTools alanboody$ curl -v --trace-time https://s6cl45z70rpan18ep-1.a1.typesense.net/health
'14:52:47.425229 * Trying 18.224.141.232...
14:52:47.425756 * TCP_NODELAY set
14:52:47.494845 * Connected to s6cl45z70rpan18ep-1.a1.typesense.net (18.224.141.232) port 443 (#0)
14:52:47.495083 * ALPN, offering h2
14:52:47.495116 * ALPN, offering http/1.1
14:52:47.507533 * successfully set certificate verify locations:
14:52:47.507563 * CAfile: /etc/ssl/cert.pem
CApath: none
14:52:47.507677 * TLSv1.2 (OUT), TLS handshake, Client hello (1):
14:52:47.575261 * TLSv1.2 (IN), TLS handshake, Server hello (2):
14:52:47.575632 * TLSv1.2 (IN), TLS handshake, Certificate (11):
14:52:47.577586 * TLSv1.2 (IN), TLS handshake, Server key exchange (12):
14:52:47.578544 * TLSv1.2 (IN), TLS handshake, Server finished (14):
14:52:47.585250 * TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
14:52:47.585288 * TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
14:52:47.585338 * TLSv1.2 (OUT), TLS handshake, Finished (20):
14:52:47.652901 * TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
14:52:47.653002 * TLSv1.2 (IN), TLS handshake, Finished (20):
14:52:47.653044 * SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
14:52:47.653073 * ALPN, server accepted to use h2
14:52:47.653101 * Server certificate:
14:52:47.653137 * subject: CN=.a1.typesense.net
14:52:47.653170 * start date: Jun 6 00:00:00 2022 GMT
14:52:47.653199 * expire date: May 27 23:59:59 2023 GMT
14:52:47.653239 * subjectAltName: host "s6cl45z70rpan18ep-1.a1.typesense.net" matched cert's ".a1.typesense.net"
14:52:47.653281 * issuer: C=GB; ST=Greater Manchester; L=Salford; O=Sectigo Limited; CN=Sectigo RSA Domain Validation Secure Server CA
14:52:47.653309 * SSL certificate verify ok.
14:52:47.653361 * Using HTTP2, server supports multi-use
14:52:47.653390 * Connection state changed (HTTP/2 confirmed)
14:52:47.653419 * Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
14:52:47.653506 * Using Stream ID: 1 (easy handle 0x7fcf60808200)
14:52:47.653561 > GET /health HTTP/2
14:52:47.653561 > Host: s6cl45z70rpan18ep-1.a1.typesense.net
14:52:47.653561 > User-Agent: curl/7.64.1
14:52:47.653561 > Accept: /*
14:52:47.653561 >
14:52:47.726042 * Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
14:52:47.726159 < HTTP/2 200
14:52:47.726191 < access-control-allow-origin: *
14:52:47.726219 < content-type: application/json; charset=utf-8
14:52:47.726247 <
14:52:47.726341 * Connection #0 to host s6cl45z70rpan18ep-1.a1.typesense.net left intact
{"ok":true} Closing connection 0

That looks fine

That wont' be the name when it gets created

If there was a timeout it should throw an error telling you it timed out, but yeah you definitely want to increase the client-side timeout to as high as even 30 minutes, to make sure the import call never times out

Running now.

Phew! 😅

Yeah. lol