Hi everyone, I am a newbie in Typesense, implementing a search-as-you-type experience with a catalog of about 20.000 items. I tried to find everywhere some recommendations about creating my catalog (I have a valid catalog.jsonl file), but could not find any information about it... Could someone give me some hints? I am using a PHP library, launched the $client->collections['mycollectionname']->documents->import('catalog.jsonl') - but it never completed - took forever and got killed by the server since it used too many resources... I had tried the catalog creation with a file with only 100 items and it worked fine... So what should I do? I guess I'll need to split my catalog into chunks, but what would be a good chunk size?

Hi @John Doisneau I'm surprised

$client->collections['mycollectionname']->documents->import('catalog.jsonl')

worked, because you'd need to pass the actual contents of the file and not the file name to that method. You'd need to read the contents of the file separately into a variable, and then pass those contents into the import method.

Thanks Jason, I am actually using a non-oficial PHP library since it seems the oficial one is not compatible anymore with PHP 7.1 - do you confirm?

Oh I see. May I know what happens when you use the official PHP library with php7.1? Wonder if there's some new language feature we can sacrifice to make it work in earlier versions of PHP

Your requirements could not be resolved to an installable set of packages. Problem 1 - symfony/http-client[v5.2.0, ..., v5.2.8] require php >=7.2.5 -> your php version (7.1.33) does not satisfy that requirement. - Root composer.json requires psr/http-client-implementation ^1.0 -> satisfiable by symfony/http-client[v5.2.0, ..., v5.2.8].

@Abdullah Al-Faqeir Any thoughts on this?

Well, on my side I think I will keep using the other library - it's not too bad...

Hello @John Doisneau, I guess you need to take a different approach regarding this matter, loading a jsonl file containing 20K records at once is a total exhaustion on the server/device resources, another way would be using fopen to open the file, read the file lines in appropriate chunks then importing them using the bulk import method of the client considering setting the maximum execution time to 0 to avoid the process from being killed.

Fore sure! I have had this resolved for quite a while now. Just adding one by one...