#community-help

Creating Index for PDF Files Using Typesense

TLDR Greg asked for advice on using TypeSense to index PDFs. Jason suggested using Tika for text extraction before using Typesense. After discussing the details, Greg decided to extract and parse data directly into Typesense using Tika.

13mo

Join the chat

Sep 26, 2022 (13 months ago)

Photo of md5-681af81b94d6f3814dfc75a070b04432

Greg

04:45 PM

Looking to use TypeSense to create an index of 4000+ PDF files and was planning to use this approach but TypeSense instead of Algolia: https://stories.algolia.com/indexing-pdf-or-other-file-contents-for-searching-b2499c23568f

Thoughts on an easier way to approach the problem? I don’t think the docsearch scraper will get me what I want but perhaps it will.

Thanks for your feedback!

Photo of md5-8813087cccc512313602b6d9f9ece19f

Jason

05:41 PM

Greg You’d definitely want to use something like Tika to extract the text out of the PDF files. Looks like the output of Tika can be HTML, in which case if you’re able to output say one html file per chapter, serve that under a localhost domain as a static site, you can then use typesense-docsearch-scraper to create the actual records for you.

A couple of things to keep in mind with docsearch:

• The record structure created by docsearch is good, but is opinionated and built to work with Docsearch.js.
• docsearch-scraper does not work with web-servers running on non-standard port numbers (anything besides 80 and 443) due to an issue with the underlying Python scraping library used

Greg

06:26 PM

That’s great info… feels like more work than is worth the effort. I’m thinking we’ll just use Tika to extract it and parse it then pump it directly into TypeSense. Here’s one of the actual files and we’re really only interested in cross referencing the table data starting on Page 5.

https://public.kimballincslc.com/vikingschematics/new/BQC300T.pdf

Jason

07:33 PM

Looks like it’s already semi-structured data, especially if it’s in a table. So yeah, docsearch-scraper might be overkill

07:36

Jason

07:36 PM

There are also a couple of SaaS services that seem to do this btw - extract parseable data from PDF files

Greg

08:20 PM

Thankfully the PDF docs are pretty static so I think Tika will get us where we need to go.

08:20

Greg

08:20 PM

Appreciate the guidance and feedback! Glad we’re going the Typesense route!

Typesense

Lightning-fast, open source search engine for everyone | Knowledge Base powered by Struct.AI

Indexed 2776 threads (79% resolved)

Join Our Community Manage subscription

Similar Threads

Solving Typesense Docsearch Scraper Issues

Sandeep was having issues with Typesense's docsearch scraper and getting fewer results than with Algolia's scraper. Jason helped by sharing the query they use and advised checking the running version of the scraper. The issue was resolved when Sandeep ran the non-base regular docker image.

22mo

Trouble with DocSearch Scraper and Pipenv Across Multiple OSs

James ran into errors when trying to build Typesense DocSearch Scraper from scratch, and believes it’s because of a bad Pipfile.lock. Jason attempted to replicate the error, and spent hours trying to isolate the issue but ultimately fixed the problem and copied his bash history for future reference. The conversation touches briefly on the subject of using a virtual machine for testing.

161

8mo

Docsearch Scrapper Metadata Configuration and Filter Problem

Marcos faced issues with Docsearch scrapper not adding metadata attributes and filtering out documents without content. Jason helped fix the issue by updating the scraper and providing filtering instructions.

6mo