Troubleshooting Indexing Duration in Typesense Import
TLDR Alan asked about lengthy indexing times for importing documents to Typesense. Jason suggested various potential causes, including network connectivity and system resources. They later identified the problem to be an error in Alan's code.
3
2
Oct 05, 2022 (15 months ago)
Alan
06:33 PMJason
06:34 PMAlan
06:34 PMJason
06:34 PMAlan
06:34 PMJason
06:35 PMJason
06:35 PMAlan
06:35 PM1
Jason
06:35 PMAlan
06:36 PMJason
06:36 PMAlan
06:37 PMJason
06:40 PMAlan
06:41 PM-${ collection }
).documents().import(documents, {action: isUpdate ? 'update' : 'upsert'}).catch( err => {
console.log(err);
return { error: true, results: err.importResults }
});
Alan
06:43 PM'name': null,
'fields': [
{
'name': 'id',
'type': 'string',
'facet': false
},
{
'name': 'number',
'type': 'string',
'facet': false
},
{
'name': 'number2',
'type': 'string',
'facet': false
},
{
'name': 'assigned_to',
'type': 'int64',
'facet': false
},
{
'name': 'channel_id',
'type': 'int64',
'facet': false
},
{
'name': 'meta',
'type': 'string',
'facet': false
},
{
'name': 'timestamp',
'type': 'int64',
'facet': false
}
],
'default_sorting_field': 'assigned_to'
},
Alan
06:43 PMAlan
06:43 PMJason
06:43 PMclient
is instantiated?Alan
06:44 PMconst config = {
nodes: [{
host: process.env.TYPESENSE_HOST,
port: process.env.TYPESENSE_PORT,
protocol: process.env.TYPESENSE_PROTOCOL
}],
apiKey: process.env.TYPESENSE_ADMIN_API_KEY,
connectionTimeoutSeconds: 5
};
const client = new Typesense.Client(config);
module.exports = client;
Alan
06:45 PMJason
06:46 PMJason
06:47 PMAlan
06:48 PMAlan
06:48 PMAlan
06:48 PMJason
06:49 PMJason
06:49 PMAlan
06:50 PMJason
06:52 PMcurl -v --trace-time <https://s6cl45z70rpan18ep-1.a1.typesense.net/health>
Alan
06:53 PM'14:52:47.425229 * Trying 18.224.141.232...
14:52:47.425756 * TCP_NODELAY set
14:52:47.494845 * Connected to s6cl45z70rpan18ep-1.a1.typesense.net (18.224.141.232) port 443 (#0)
14:52:47.495083 * ALPN, offering h2
14:52:47.495116 * ALPN, offering http/1.1
14:52:47.507533 * successfully set certificate verify locations:
14:52:47.507563 * CAfile: /etc/ssl/cert.pem
CApath: none
14:52:47.507677 * TLSv1.2 (OUT), TLS handshake, Client hello (1):
14:52:47.575261 * TLSv1.2 (IN), TLS handshake, Server hello (2):
14:52:47.575632 * TLSv1.2 (IN), TLS handshake, Certificate (11):
14:52:47.577586 * TLSv1.2 (IN), TLS handshake, Server key exchange (12):
14:52:47.578544 * TLSv1.2 (IN), TLS handshake, Server finished (14):
14:52:47.585250 * TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
14:52:47.585288 * TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
14:52:47.585338 * TLSv1.2 (OUT), TLS handshake, Finished (20):
14:52:47.652901 * TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
14:52:47.653002 * TLSv1.2 (IN), TLS handshake, Finished (20):
14:52:47.653044 * SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
14:52:47.653073 * ALPN, server accepted to use h2
14:52:47.653101 * Server certificate:
14:52:47.653137 * subject: CN=.a1.typesense.net
14:52:47.653170 * start date: Jun 6 00:00:00 2022 GMT
14:52:47.653199 * expire date: May 27 23:59:59 2023 GMT
14:52:47.653239 * subjectAltName: host "s6cl45z70rpan18ep-1.a1.typesense.net" matched cert's ".a1.typesense.net"
14:52:47.653281 * issuer: C=GB; ST=Greater Manchester; L=Salford; O=Sectigo Limited; CN=Sectigo RSA Domain Validation Secure Server CA
14:52:47.653309 * SSL certificate verify ok.
14:52:47.653361 * Using HTTP2, server supports multi-use
14:52:47.653390 * Connection state changed (HTTP/2 confirmed)
14:52:47.653419 * Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
14:52:47.653506 * Using Stream ID: 1 (easy handle 0x7fcf60808200)
14:52:47.653561 > GET /health HTTP/2
14:52:47.653561 > Host: s6cl45z70rpan18ep-1.a1.typesense.net
14:52:47.653561 > User-Agent: curl/7.64.1
14:52:47.653561 > Accept: /*
14:52:47.653561 >
14:52:47.726042 * Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
14:52:47.726159 < HTTP/2 200
14:52:47.726191 < access-control-allow-origin: *
14:52:47.726219 < content-type: application/json; charset=utf-8
14:52:47.726247 <
14:52:47.726341 * Connection #0 to host s6cl45z70rpan18ep-1.a1.typesense.net left intact
{"ok":true} Closing connection 0
Jason
06:56 PMJason
06:56 PMAlan
06:57 PMAlan
06:58 PMJason
06:59 PMJason
06:59 PMAlan
07:01 PMAlan
07:05 PMAlan
07:08 PM1
Jason
07:08 PMAlan
07:09 PMAlan
07:09 PM2
Alan
07:11 PM1
Typesense
Indexed 3015 threads (79% resolved)
Similar Threads
Typesense Server Bulk Import/Upsert Issue Resolved
Adam was confused about the discrepancy between the successful responses and the actual indexed data while working with a custom WP plugin integrating with Typesense. The issue was a bug related to fetching documents in the wrong order, not a Typesense problem.
Troubleshooting Write Timeouts in Typesense with Large CSVs
Agustin had issues with Typesense getting write timeouts while loading large CSV files. Kishore Nallan suggested chunking data or converting to JSONL before loading. Through troubleshooting, they identified a possible network problem at AWS and found a workaround.
Handling Kinesis Stream Event Batching with Typesense
Dui had questions about how to handle Kinesis stream events with Typesense. Kishore Nallan suggested using upsert mode for creation/update and differentiating with logical deletion. After various discussions including identifying and resolving a bug, they finalized to introduce an `emplace` action in Typesense v0.23.
Revisiting Typesense for Efficient DB Indexing and Querying
kopach experienced slow indexing and crashes with Typesense. The community suggested to use batch import and check the server's resources. Improvements were made but additional support was needed for special characters and multi-search queries.
Bulk Import 50MB JSON Files Error - Timeout and Solutions
madhweep encounters an error while bulk importing JSON files. Kishore Nallan provided help, but the issue persists. Jason intervenes and after troubleshooting, they concluded the cluster had run out of memory causing the issue. The problem was resolved by using a cluster with sufficient memory. Daniel also experienced a similar issue, resolved by increasing the timeout.