can anyone help me out why the hits structure like that I th typesense #community-help

can anyone help me out why the hits structure like...

Rubai Mandal

03/19/2023, 8:24 PM

can anyone help me out why the hits structure like that ? I think

add sdk implementation

have to be on top .

Jason Bosco

03/19/2023, 9:05 PM

Could you open the browser’s network inspector, do a search, then look for a request to multi_search, then copy-as-curl the request and paste it here? Could you also copy the response for the API call and paste it here?

Rubai Mandal

03/19/2023, 10:19 PM

curl 'http://localhost:8108/multi_search?x-typesense-api-key=xyz' \ -H 'Accept: application/json, text/plain, */*' \ -H 'Accept-Language: en-GB,en-US;q=0.9,en;q=0.8' \ -H 'Cache-Control: no-cache' \ -H 'Connection: keep-alive' \ -H 'Content-Type: text/plain' \ -H 'Origin: https://3238-180-151-109-60.in.ngrok.io' \ -H 'Pragma: no-cache' \ -H 'Sec-Fetch-Dest: empty' \ -H 'Sec-Fetch-Mode: cors' \ -H 'Sec-Fetch-Site: cross-site' \ -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36' \ -H 'sec-ch-ua: "Chromium";v="110", "Not A(Brand";v="24", "Google Chrome";v="110"' \ -H 'sec-ch-ua-mobile: ?0' \ -H 'sec-ch-ua-platform: "macOS"' \ --data-raw '{"searches":[{"collection":"Developer_Docs","q":"add sdk impl","query_by":"hierarchy.lvl0,hierarchy.lvl1,hierarchy.lvl2,hierarchy.lvl3,hierarchy.lvl4,hierarchy.lvl5,hierarchy.lvl6,content","include_fields":"hierarchy.lvl0,hierarchy.lvl1,hierarchy.lvl2,hierarchy.lvl3,hierarchy.lvl4,hierarchy.lvl5,hierarchy.lvl6,content,anchor,url,type,id","highlight_full_fields":"hierarchy.lvl0,hierarchy.lvl1,hierarchy.lvl2,hierarchy.lvl3,hierarchy.lvl4,hierarchy.lvl5,hierarchy.lvl6,content","group_by":"url","group_limit":3,"filter_by":"product_tag:=payment-page_android"}]}' \ --compressed

Rubai Mandal

03/19/2023, 10:28 PM

Copy code

{
  "results": [
    {
      "facet_counts": [],
      "found": 1,
      "grouped_hits": [
        {
          "group_key": [
            "<https://3238-180-151-109-60.in.ngrok.io/payment-page/android/base-sdk-integration/getting-sdk#add-sdk-implementation>"
          ],
          "hits": [
            {
              "document": {
                "anchor": "add-sdk-implementation",
                "content": "After adding Assets Plugin and Hyper SDK to your project, please ensure you do a gradle sync and clean build",
                "hierarchy": {
                  "lvl0": "1. Getting the SDK",
                  "lvl1": "Add SDK implementation",
                  "lvl2": null,
                  "lvl3": null,
                  "lvl4": null,
                  "lvl5": null,
                  "lvl6": null
                },
                "hierarchy.lvl0": "1. Getting the SDK",
                "hierarchy.lvl1": "Add SDK implementation",
                "id": "158",
                "type": "content",
                "url": "<https://3238-180-151-109-60.in.ngrok.io/payment-page/android/base-sdk-integration/getting-sdk#add-sdk-implementation>"
              },
              "highlight": {
                "content": {
                  "matched_tokens": [
                    "SDK"
                  ],
                  "snippet": "After adding Assets Plugin and Hyper <mark>SDK</mark> to your project, please ensure you do a gradle sync and clean build",
                  "value": "After adding Assets Plugin and Hyper <mark>SDK</mark> to your project, please ensure you do a gradle sync and clean build"
                }
              },
              "highlights": [
                {
                  "field": "content",
                  "matched_tokens": [
                    "SDK"
                  ],
                  "snippet": "After adding Assets Plugin and Hyper <mark>SDK</mark> to your project, please ensure you do a gradle sync and clean build",
                  "value": "After adding Assets Plugin and Hyper <mark>SDK</mark> to your project, please ensure you do a gradle sync and clean build"
                }
              ],
              "text_match": 1736172785157800000,
              "text_match_info": {
                "best_field_score": "3315687620864",
                "best_field_weight": 14,
                "fields_matched": 3,
                "score": "1736172785157800051",
                "tokens_matched": 3
              }
            },
            {
              "document": {
                "anchor": "add-sdk-implementation",
                "content": "To inject SDK as dependency in your application, include it in your  application build.gradle  dependencies",
                "hierarchy": {
                  "lvl0": "1. Getting the SDK",
                  "lvl1": "Add SDK implementation",
                  "lvl2": null,
                  "lvl3": null,
                  "lvl4": null,
                  "lvl5": null,
                  "lvl6": null
                },
                "hierarchy.lvl0": "1. Getting the SDK",
                "hierarchy.lvl1": "Add SDK implementation",
                "id": "157",
                "type": "content",
                "url": "<https://3238-180-151-109-60.in.ngrok.io/payment-page/android/base-sdk-integration/getting-sdk#add-sdk-implementation>"
              },
              "highlight": {
                "content": {
                  "matched_tokens": [
                    "SDK"
                  ],
                  "snippet": "To inject <mark>SDK</mark> as dependency in your application, include it in your  application build.gradle  dependencies",
                  "value": "To inject <mark>SDK</mark> as dependency in your application, include it in your  application build.gradle  dependencies"
                }
              },
              "highlights": [
                {
                  "field": "content",
                  "matched_tokens": [
                    "SDK"
                  ],
                  "snippet": "To inject <mark>SDK</mark> as dependency in your application, include it in your  application build.gradle  dependencies",
                  "value": "To inject <mark>SDK</mark> as dependency in your application, include it in your  application build.gradle  dependencies"
                }
              ],
              "text_match": 1736172785157800000,
              "text_match_info": {
                "best_field_score": "3315687620864",
                "best_field_weight": 14,
                "fields_matched": 3,
                "score": "1736172785157800051",
                "tokens_matched": 3
              }
            },
            {
              "document": {
                "anchor": "add-sdk-implementation",
                "hierarchy": {
                  "lvl0": "1. Getting the SDK",
                  "lvl1": "Add SDK implementation",
                  "lvl2": null,
                  "lvl3": null,
                  "lvl4": null,
                  "lvl5": null,
                  "lvl6": null
                },
                "hierarchy.lvl0": "1. Getting the SDK",
                "hierarchy.lvl1": "Add SDK implementation",
                "id": "156",
                "type": "lvl1",
                "url": "<https://3238-180-151-109-60.in.ngrok.io/payment-page/android/base-sdk-integration/getting-sdk#add-sdk-implementation>"
              },
              "highlight": {
                "hierarchy": {
                  "lvl0": "1. Getting the SDK",
                  "lvl1": "Add SDK implementation",
                  "lvl2": null,
                  "lvl3": null,
                  "lvl4": null,
                  "lvl5": null,
                  "lvl6": null
                }
              },
              "highlights": [],
              "text_match": 1736172785157800000,
              "text_match_info": {
                "best_field_score": "3315687620864",
                "best_field_weight": 14,
                "fields_matched": 2,
                "score": "1736172785157800050",
                "tokens_matched": 3
              }
            }
          ]
        }
      ],
      "out_of": 346,
      "page": 1,
      "request_params": {
        "collection_name": "Developer_Docs_1679258843",
        "per_page": 10,
        "q": "add sdk impl"
      },
      "search_cutoff": false,
      "search_time_ms": 1
    }
  ]
}

Jason Bosco

03/20/2023, 4:52 PM

In your docsearch.js initialization code, could you add:

Copy code

typesenseSearchParameters: {
  filter_by: '...',
  sort_by: 'item_priority:desc',
},

And see if that helps?

Jason Bosco

03/20/2023, 4:52 PM

If it does, then I can update the docsearch.js defaults to set this automatically… Let me know

Rubai Mandal

03/20/2023, 6:38 PM

great that's worked . thanks a lot @Jason Bosco 🙏

Jason Bosco

03/20/2023, 6:39 PM

Ok great! I’m going to push out an update to docsearch.js that sets that by default, so you don’t have to do this manually. I’ll keep you posted shortly.

Rubai Mandal

03/20/2023, 6:40 PM

thanks . and anything about the blank space ? here you can see I have some data this page . still get blank . It happen when we search as lvl0

Jason Bosco

03/20/2023, 6:41 PM

Could you share the response from Typesense for that?

Rubai Mandal

03/20/2023, 6:41 PM

sure

Rubai Mandal

03/20/2023, 6:44 PM

can you please tell once the github link to send the large file

Jason Bosco

03/20/2023, 6:46 PM

gist.github.com

Rubai Mandal

03/20/2023, 6:47 PM

https://gist.github.com/rubai99/1619b4fa3d879b72cf6d44923fff4d7d

Jason Bosco

03/20/2023, 6:52 PM

May I know if you’re using docsearch.js or docsearch-react?

Rubai Mandal

03/20/2023, 6:52 PM

docsearch.js

Jason Bosco

03/20/2023, 6:53 PM

Could you upgrade to

3.4.0-0

typesense-docsearch.js

and then try again?

Jason Bosco

03/20/2023, 6:53 PM

Could you also remove the sort_by parameter that you added manually?

Rubai Mandal

03/20/2023, 6:54 PM

sure

Rubai Mandal

03/20/2023, 6:58 PM

can you please help me out how to upgrade docsearch.js

Jason Bosco

03/20/2023, 6:59 PM

in package.json

Jason Bosco

03/20/2023, 6:59 PM

Oh wait, are you using the script tag directly?

Rubai Mandal

03/20/2023, 7:00 PM

yes

Jason Bosco

03/20/2023, 7:00 PM

You configured it like this right: https://typesense.org/docs/guide/docsearch.html#option-c-custom-docs-framework-with-docsearch-js-v3-modal-layout

Rubai Mandal

03/20/2023, 7:00 PM

yes

Jason Bosco

03/20/2023, 7:01 PM

You want to change this line:

Copy code

<!-- Before the closing body -->
<script src="<https://cdn.jsdelivr.net/npm/typesense-docsearch.js@3.4.0-0>"></script>

Jason Bosco

03/20/2023, 7:01 PM

In the docs it currently says

3.0.1

for the script tag

Jason Bosco

03/20/2023, 7:01 PM

Change that to

3.4.0-0

Rubai Mandal

03/20/2023, 7:02 PM

awesome that works

Jason Bosco

03/20/2023, 7:03 PM

The blank result is gone too?

Jason Bosco

03/20/2023, 7:03 PM

Could you show me a screenshot of how the results looks now?

Rubai Mandal

03/20/2023, 7:03 PM

here is the result

Jason Bosco

03/20/2023, 7:03 PM

Awesome 🙌

Jason Bosco

03/20/2023, 7:04 PM

CC: @Abhishek Raj Thank you for that PR! ^ I’ll publish this change for you in the docusaurus theme as well by EOD

typesense 1

🙂 1

Rubai Mandal

03/20/2023, 7:05 PM

thanks a lot , convert the blank space to lvl0

👍 1

Rubai Mandal

03/20/2023, 7:13 PM

and one thing is it behaves like as expected ? because I searched a key & got the result on the top but I also got the example or text which is not required . you can see the document under backdrop . and can we add

(...)

at the start of hits if the match result are on a long text . so it's easy to understand such that there have some text before that

Jason Bosco

03/20/2023, 7:40 PM

because I searched a key & got the result on the top but I also got the example or text which is not required

The scraper just shows all content that is on the page, as specified by the css selectors. If you don’t want examples to show you, you want to exclude that via css selectors

Jason Bosco

03/20/2023, 7:42 PM

and can we add (...) at the start of hits if the match result are on a long text . so it’s easy to understand such that there have some text before that

…

should be shown at the end of the hits technically… looks like that’s hidden in the UI. In your docsearch initialization code, could you try adding this:

Copy code

typesenseSearchParameters: {
  filter_by: '...',
  highlight_affix_num_tokens: 3,
},

Rubai Mandal

03/20/2023, 7:46 PM

nothing changed after adding this

Jason Bosco

03/20/2023, 7:47 PM

Could you share the response from Typesense?

Rubai Mandal

03/20/2023, 7:49 PM

https://gist.github.com/rubai99/ed9406ec2f380f7d9c17b16bdfac9923

Rubai Mandal

03/20/2023, 7:50 PM

message has been deleted

Jason Bosco

03/20/2023, 7:51 PM

Hmm that doesn’t seem like the same API response for the search query in your screenshot

Jason Bosco

03/20/2023, 7:52 PM

Could you open the network inspector first, then type in that search query and then send me the api response of the last call to multi_search?

Rubai Mandal

03/20/2023, 7:52 PM

sure

Rubai Mandal

03/20/2023, 7:54 PM

response for this screenshot https://gist.github.com/rubai99/5964d067547bb21ead934154b088b18c

Jason Bosco

03/20/2023, 8:42 PM

Ah, could you also set

snippet_threshold: 5

Rubai Mandal

03/20/2023, 8:47 PM

in configJS?

Jason Bosco

03/20/2023, 8:47 PM

Copy code

typesenseSearchParameters: {
  filter_by: '...',
  snippet_threshold: 5,
},

Jason Bosco

03/20/2023, 8:48 PM

in the docsearch initialization code

Rubai Mandal

03/20/2023, 8:49 PM

nothing changed . can't see anything like

...

on hits

Jason Bosco

03/20/2023, 8:49 PM

Hmm, could you share the curl request to Typesense and the response once again?

Jason Bosco

03/20/2023, 8:50 PM

Did anything change in the UI at all or does it still look the same?

Rubai Mandal

03/20/2023, 8:50 PM

yes

Rubai Mandal

03/20/2023, 8:51 PM

I added

snippet_threshold: 5,

but still getting same result like previous one

Rubai Mandal

03/20/2023, 8:52 PM

Copy code

curl '<http://localhost:8108/multi_search?x-typesense-api-key=xyz>' \
  -H 'Accept: application/json, text/plain, */*' \
  -H 'Accept-Language: en-GB,en-US;q=0.9,en;q=0.8' \
  -H 'Cache-Control: no-cache' \
  -H 'Connection: keep-alive' \
  -H 'Content-Type: text/plain' \
  -H 'Origin: <http://localhost>' \
  -H 'Pragma: no-cache' \
  -H 'Referer: <http://localhost/>' \
  -H 'Sec-Fetch-Dest: empty' \
  -H 'Sec-Fetch-Mode: cors' \
  -H 'Sec-Fetch-Site: same-site' \
  -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36' \
  -H 'sec-ch-ua: "Chromium";v="110", "Not A(Brand";v="24", "Google Chrome";v="110"' \
  -H 'sec-ch-ua-mobile: ?0' \
  -H 'sec-ch-ua-platform: "macOS"' \
  --data-raw '{"searches":[{"collection":"Developer_Docs","q":"to be present","query_by":"hierarchy.lvl0,hierarchy.lvl1,hierarchy.lvl2,hierarchy.lvl3,hierarchy.lvl4,hierarchy.lvl5,hierarchy.lvl6,content","include_fields":"hierarchy.lvl0,hierarchy.lvl1,hierarchy.lvl2,hierarchy.lvl3,hierarchy.lvl4,hierarchy.lvl5,hierarchy.lvl6,content,anchor,url,type,id","highlight_full_fields":"hierarchy.lvl0,hierarchy.lvl1,hierarchy.lvl2,hierarchy.lvl3,hierarchy.lvl4,hierarchy.lvl5,hierarchy.lvl6,content","group_by":"url","group_limit":3,"sort_by":"item_priority:desc","filter_by":"product_tag:=payment-page_android","snippet_threshold":5}]}' \
  --compressed

Jason Bosco

03/20/2023, 8:53 PM

Could you run ngrok for port 8108 temporarily? I’d like to be able to reach that Typesense server

Rubai Mandal

03/20/2023, 8:53 PM

https://gist.github.com/rubai99/ee6bef7d38cf8474b2c7de52ccec9bc3

Rubai Mandal

03/20/2023, 8:54 PM

I will run it for 8108

Jason Bosco

03/20/2023, 8:54 PM

ok, could you share that url with me?

Rubai Mandal

03/20/2023, 8:55 PM

sure give me a moment

Rubai Mandal

03/20/2023, 9:05 PM

https://c26c-180-151-120-184.in.ngrok.io/

Jason Bosco

03/20/2023, 9:09 PM

Let’s try this:

Copy code

"snippet_threshold": 5,
      "highlight_affix_num_tokens": 3

Rubai Mandal

03/20/2023, 9:12 PM

getting same results

Jason Bosco

03/20/2023, 9:12 PM

Hmm the response is definitely different from the API

Jason Bosco

03/20/2023, 9:12 PM

Could you share a screenshot?

Jason Bosco

03/20/2023, 9:12 PM

of the UI

Rubai Mandal

03/20/2023, 9:14 PM

message has been deleted

Jason Bosco

03/20/2023, 9:14 PM

That is different from this right:

https://typesense-community.slack.com/files/U04RHF46W5B/F05044HG4M6/screenshot_2023-03-21_at_1.23.31_am.png▾

Rubai Mandal

03/20/2023, 9:15 PM

yes

Jason Bosco

03/20/2023, 9:16 PM

So it’s working now yeah?

Jason Bosco

03/20/2023, 9:16 PM

Or did I misunderstand the issue

Rubai Mandal

03/20/2023, 9:17 PM

no . you can see there is no

...

for the text . actually I am trying to say that the screenshot are taken different time but getting same result that's why it's look like same

Rubai Mandal

03/20/2023, 9:19 PM

see on

to be present

,the 2nd hits . it's a long text that's why I want to add

...

at start

Rubai Mandal

03/20/2023, 9:21 PM

and 2nd one is more matching then 1st one . but it not in the top

Jason Bosco

03/20/2023, 9:36 PM

Ahh got it, you’re talking about the ellipsis specifically…

Jason Bosco

03/20/2023, 9:36 PM

Looking into it

Jason Bosco

03/20/2023, 9:39 PM

To debug the order of the results, could you upgrade your Typesense server to 0.24.1.rc10 and let me know?

Rubai Mandal

03/20/2023, 9:48 PM

as of now the order of result are fine . I want to say for text hits like this ,start or end with

...

. I got this from https://docusaurus.io/ site

Jason Bosco

03/20/2023, 9:55 PM

Could you upgrade to

3.4.0-1

and check now?

Rubai Mandal

03/20/2023, 10:04 PM

getting error on the given version

Jason Bosco

03/21/2023, 3:17 AM

Could you try with

3.4.0-8

Rubai Mandal

03/21/2023, 8:54 AM

yea it's working

Rubai Mandal

03/21/2023, 9:12 AM

sometime before it's working fine but now facing this error

Rubai Mandal

03/21/2023, 9:13 AM

searchbar braking while opening the popup of searchbar

Jason Bosco

03/21/2023, 1:39 PM

Could you share curl request and response for that screenshot?

Rubai Mandal

03/21/2023, 2:10 PM

Copy code

curl '<http://localhost:8108/multi_search?x-typesense-api-key=xyz>' \
  -H 'Accept: application/json, text/plain, */*' \
  -H 'Accept-Language: en-GB,en-US;q=0.9,en;q=0.8' \
  -H 'Cache-Control: no-cache' \
  -H 'Connection: keep-alive' \
  -H 'Content-Type: text/plain' \
  -H 'Origin: <http://localhost:3000>' \
  -H 'Pragma: no-cache' \
  -H 'Referer: <http://localhost:3000/>' \
  -H 'Sec-Fetch-Dest: empty' \
  -H 'Sec-Fetch-Mode: cors' \
  -H 'Sec-Fetch-Site: same-site' \
  -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36' \
  -H 'sec-ch-ua: "Chromium";v="110", "Not A(Brand";v="24", "Google Chrome";v="110"' \
  -H 'sec-ch-ua-mobile: ?0' \
  -H 'sec-ch-ua-platform: "macOS"' \
  --data-raw '{"searches":[{"collection":"Developer_Docs","q":"session api","query_by":"hierarchy.lvl0,hierarchy.lvl1,hierarchy.lvl2,hierarchy.lvl3,hierarchy.lvl4,hierarchy.lvl5,hierarchy.lvl6,content","include_fields":"hierarchy.lvl0,hierarchy.lvl1,hierarchy.lvl2,hierarchy.lvl3,hierarchy.lvl4,hierarchy.lvl5,hierarchy.lvl6,content,anchor,url,type,id","highlight_full_fields":"hierarchy.lvl0,hierarchy.lvl1,hierarchy.lvl2,hierarchy.lvl3,hierarchy.lvl4,hierarchy.lvl5,hierarchy.lvl6,content","group_by":"url","group_limit":3,"sort_by":"item_priority:desc","snippet_threshold":5,"highlight_affix_num_tokens":3,"filter_by":"product_tag:=payment-page_android"}]}' \
  --compressed

https://gist.github.com/rubai99/8f21fb34638fd68f0683137d4e6ee810 . sometimes works ,sometimes braking . now it's breaking

Jason Bosco

03/21/2023, 4:07 PM

Could you update your script tag to this:

Copy code

<script src="<https://cdn.jsdelivr.net/npm/typesense-docsearch.js@3.4.0-8/dist/umd/index.js>"></script>

And then try replicating the same error in that screenshot and post a stack trace? (This is hopefully pulls in the source-map and shows a proper stack trace)

Rubai Mandal

03/21/2023, 4:28 PM

this error getting when I click on searchbar , it's happens for this version only . at first when I add the version it's working fine . after 5-6 min I got an error . then after 1 hr I add this version again but also get same thing , some time worked but suddenly getting this error . and now I am getting the error also

Jason Bosco

03/21/2023, 4:30 PM

Could you click on “Snippet.js1452” in that stack trace and post a screenshot?

Rubai Mandal

03/21/2023, 4:30 PM

you can check this problem via my site https://7a03-103-159-11-202.in.ngrok.io/payment-page/android/base-sdk-integration/session

Jason Bosco

03/21/2023, 4:31 PM

Looks like the Typesense URL is pointing to localhost, so I can’t see any search results because of that

Jason Bosco

03/21/2023, 4:31 PM

https://typesense-community.slack.com/archives/C01P749MET0/p1679416218527429?thread_ts=1679257476.885059&cid=C01P749MET0

Rubai Mandal

03/21/2023, 4:32 PM

message has been deleted

Rubai Mandal

03/21/2023, 4:34 PM

I think localhost URL is not an issue . because it's working for some times suddenly facing this things

Jason Bosco

03/21/2023, 4:34 PM

Yup yup, that’s a different issue, just with the ngrok URL you shared earlier

Jason Bosco

03/21/2023, 4:34 PM

I’m pushing a potential fix for the root cause.. let’s see

Rubai Mandal

03/21/2023, 4:34 PM

sure

Jason Bosco

03/21/2023, 4:37 PM

Could you try with

3.4.0-9

Rubai Mandal

03/21/2023, 4:37 PM

sure

Rubai Mandal

03/21/2023, 4:38 PM

now it's working fine you can check with my ngrock url

Jason Bosco

03/21/2023, 4:39 PM

The ngrok URL still doesn’t work for me, because it’s trying to connect to localhost:8108 to talk to Typesense

Jason Bosco

03/21/2023, 4:40 PM

To get it to work, you would have to start a separate ngrok tunnel for port 8108, then use that ngrok URL in the docsearch init code as the typesense hostname… But that’s too much effort, so that’s fine. Happy to hear that it works now!

Rubai Mandal

03/21/2023, 4:44 PM

message has been deleted

❤️ 1

Rubai Mandal

03/21/2023, 4:47 PM

thanks @Jason Bosco. till now I am asked for lot of issues , sorry for that . now it's working fine . great work👏

🙌 1

Jason Bosco

03/21/2023, 4:48 PM

That’s great to hear! Thank you for helping catch all these issues!

❤️ 1

Rubai Mandal

03/22/2023, 12:37 PM

hi @Jason Bosco is this possible to run multiple collection ? we have two product

payment-page

upi-inapp

in our documentation , suppose 1st time we run the scraper for collection

Developer_Docs_upi-inapp

and again run the scraper for other collection of

Developer_Docs_payment-page

, so can we access both collection in a single documentation , the benefit of this is when anything change happens for a product then we can scrape again for this particular product 's collection only . so here we don't need to run the scraper every product's collection . for reference you can check our documentation https://docs.juspay.in/

Jason Bosco

03/22/2023, 3:40 PM

Unfortunately this is not possible to do with the scraper - it creates a whole new collection each time. So you would have to fork the scraper and update it appropriately, if you want to do partial scraping into the same collection

👍 1

Rubai Mandal

03/23/2023, 7:31 AM

@Jason Bosco is there any API to run the scraper in production , and what are changes needed in .env for production

Copy code

TYPESENSE_API_KEY=xyz
TYPESENSE_HOST=host.docker.internal
TYPESENSE_PORT=8108
TYPESENSE_PROTOCOL=http

Rubai Mandal

03/23/2023, 7:05 PM

hey @Jason Bosco anything about that ?

Jason Bosco

03/23/2023, 7:29 PM

You would have to use something like say AWS Fargate (or any docker-based runtime even in your CI pipeline) to run the scraper using the docker image

Jason Bosco

03/23/2023, 7:30 PM

On Typesense Cloud, we only host the Typesense cluster itself - you still need to run the scraper in your infrastructure

Jason Bosco

03/23/2023, 7:30 PM

If you’re using Typesense Cloud, the .env file would look something like this:

Copy code

TYPESENSE_API_KEY=<GENERATED_FROM_DASHBOARD>
TYPESENSE_HOST=<http://xxxx.a1.typesense.net|xxxx.a1.typesense.net>
TYPESENSE_PORT=443
TYPESENSE_PROTOCOL=https

Jason Bosco

03/23/2023, 7:31 PM

The host and api key will be generated once you provision a cluster

Rubai Mandal

03/23/2023, 7:36 PM

and anything about chromedriver path ?

Jason Bosco

03/23/2023, 7:37 PM

You can leave that as the default, there’s a chrome executable inside the docker image we publish

Rubai Mandal

03/23/2023, 7:38 PM

so we have to build an api to run the scraper for production

Jason Bosco

03/23/2023, 7:49 PM

You can just run the docker command directly

Jason Bosco

03/23/2023, 7:49 PM

For eg, here’s how we call AWS Fargate that runs the docker command for us for the Typesense docs website: https://github.com/typesense/typesense-website/blob/cca16595a480dc880145bf8b01b8464476ba051e/docs-site/package.json#L12

Rubai Mandal

03/25/2023, 10:21 PM

can you please have a look why getting the error ?? the error getting while build dockerfile:base from typesense-docsearch-scraper can i change the version to 111.0.5563.110-1 . after changing the version can it be effects anything

Rubai Mandal

03/25/2023, 10:37 PM

and getting this error while run the scraper . what is the issue here ?

Jason Bosco

03/26/2023, 1:52 AM

You need to build the scraper on a Linux machine with intel cpu. Building it on a mac, especially an M1 doesn’t seem to work

Jason Bosco

03/26/2023, 1:53 AM

Any reason you’re not using the prebuilt Docker image we’ve published?

Rubai Mandal

03/26/2023, 11:11 AM

we made some changes that's why not using pre built docker image. we used dynamic config

Rubai Mandal

03/26/2023, 6:51 PM

then may I have change anything in the code for pod deployment for build the scraper

Jason Bosco

03/26/2023, 7:59 PM

I haven’t been able to get the scraper to build on M1. So you have to spin up a Linux VM and build it from there

Rubai Mandal

03/26/2023, 11:32 PM

that's for the 1st time . after that if anything change on docs we have to scrape the docs again . what should we do for that case ?

Jason Bosco

03/27/2023, 12:45 AM

You have to push the docker image you build to a docker registry, and then pull the pre-built image from there anytime you want to scrape

Rubai Mandal

03/27/2023, 9:45 AM

for typesense can we change

host='host.docker.internal'

host='localhost'

, cause we don't use docker as of now to run the scraper . we run it from VS code via an API and getting this error

Jason Bosco

03/27/2023, 3:54 PM

Yes, you can change that to any hostname, including localhost in the

.env

file you’re using

Rubai Mandal

03/28/2023, 7:44 AM

sitemap not working for me

Open in Slack

Previous Next