#community-help

New to TypeSense - Scrapping Websites & Accessing Data

TLDR Raj was unsure about locating and accessing their scrapped data, and Jason guided them to find the collection and use the search endpoint to query it.

Powered by Struct AI
+11
4
3mo
Solved
Join the chat
May 31, 2023 (4 months ago)
Raj
Photo of md5-fd3186ce27b535ef0b49080da4844d31
Raj
03:54 PM
Hello, I am new to TypeSense.

I installed typesense in my mac and able to search books catalogue from localhost UI portal. Then I scrapped the website using docSearch using docker image. Looks like scrapping is also successful. However I am not able to find following

1. Where is the scrapped documents stored? I looked at data-dir = /opt/homebrew/var/lib/typesense but I dont see data over there.
2. How can I point my scrapped data to above UI portal so that I can query my scrapped data?
3. I scrapped with index name as support. curl <https://localhost:8108/collections/support/documents/search?q=gift> is failing with no matches found
Jason
Photo of md5-8813087cccc512313602b6d9f9ece19f
Jason
06:53 PM
The scraped documents are stored in a Typesense collection. So if you do a GET /collections you’ll see the collection name, and then you can use the search endpoint to query this collection
07:07
Jason
07:07 PM
If you’re using docsearch, you want to use the docsearch UI for it: https://typesense.org/docs/guide/docsearch.html#step-2-add-a-search-bar-to-your-documentation-site
Jun 01, 2023 (3 months ago)
Raj
Photo of md5-fd3186ce27b535ef0b49080da4844d31
Raj
12:07 AM
Thank you Jason. I am able to see all available collections by running below command.

curl -X GET '<http://localhost:8108/collections>' -H 'Content-Type: application/json' -H 'X-TYPESENSE-API-KEY: xyz'
+11