hi , I am trying to run the scraper in local . I tried with ngrock & 80 port but not scrapping my docs . but when I put

<https://example.com>

instead of

<http://host.docker.internal/>..

, it’s worked perfectly . can anyone please help me out . here is config file

{
  "index_name": "payment-page",
  "js_render": true,
  "js_wait": 10,
  "use_anchors": false,
  "user_agent": "Custom Bot",
  "start_urls": [
    "<http://host.docker.internal/payment-page/android/base-sdk-integration/session>"
  ],
  "sitemap_alternate_links": false,
  "selectors": {
    "lvl0": "h1,  h2 , .heading-text",
    "lvl1": "h3, .label",
    "lvl2": ".key-header, .step-card-header-text, .th-row",
    "text": ".screen2 p:not(:empty), .hero-welcome, .screen2 li, .main-screen, .only-steps p:not(:empty),td"
  },
  "strip_chars": " .,;:#",
  "scrap_start_urls": true,
  "custom_settings": {
    "synonyms": {
      "relevancy": [
        "relevant",
        "relevance"
      ],
      "relevant": [
        "relevancy",
        "relevance"
      ],
      "relevance": [
        "relevancy",
        "relevant"
      ]
    }
  }
}

output : > DocSearch: http://host.docker.internal/payment-page/android/base-sdk-integration/session 0 records)

Jason will be able to help when he is online later.

What link do you use to access your documentation site from your browser?

this issue is resolved , is typesense able to scrape all the links if the config have multiple index ? exaple config

[
 {
    "index_name": "payment-page",
    "js_render": true,
    "js_wait": 5,
    "use_anchors": false,
    "user_agent": "Custom Bot",
    "start_urls": [
      "<https://9491-103-159-11-202.in.ngrok.io/payment-page/android/overview/integration-architecture>",
      "<https://9491-103-159-11-202.in.ngrok.io/payment-page/android/overview/pre-requisites>",
      "<https://9491-103-159-11-202.in.ngrok.io/payment-page/android/base-sdk-integration/session>",
      "<https://9491-103-159-11-202.in.ngrok.io/payment-page/android/base-sdk-integration/order-status-api>",
      "<https://9491-103-159-11-202.in.ngrok.io/payment-page/android/base-sdk-integration/getting-sdk>",
      "<https://9491-103-159-11-202.in.ngrok.io/payment-page/android/base-sdk-integration/initiating-sdk>",
      "<https://9491-103-159-11-202.in.ngrok.io/payment-page/android/base-sdk-integration/processing-sdk>",
      "<https://9491-103-159-11-202.in.ngrok.io/payment-page/android/base-sdk-integration/handle-payment-response>",
      "<https://9491-103-159-11-202.in.ngrok.io/payment-page/android/base-sdk-integration/life-cycle-events>",
      "<https://9491-103-159-11-202.in.ngrok.io/payment-page/android/resources/error-codes>",
      "<https://9491-103-159-11-202.in.ngrok.io/payment-page/android/resources/transaction-status>",
      "<https://9491-103-159-11-202.in.ngrok.io/payment-page/android/resources/sample-payloads>"
    ],
    "sitemap_alternate_links": false,
    "selectors": {
      "lvl0":"h1,  h2 , .heading-text" ,
      "lvl1": "h3, .label" ,
      "lvl2": ".key-header, .step-card-header-text, .th-row",
      "text":".screen2 p:not(:empty), .hero-welcome, .screen2 li, .main-screen, .only-steps p:not(:empty),td"
    },
    "strip_chars": " .,;:#",
    "scrap_start_urls": true,
    "custom_settings": {
      "synonyms": {
        "relevancy": ["relevant", "relevance"],
        "relevant": ["relevancy", "relevance"],
        "relevance": ["relevancy", "relevant"]
      }
    }
  },
  {
    "index_name": "payment-page2",
    "js_render": true,
    "js_wait": 5,
    "use_anchors": false,
    "user_agent": "Custom Bot",
    "start_urls": [
      "<https://9491-103-159-11-202.in.ngrok.io/payment-page/android/overview/integration-architecture>",
      "<https://9491-103-159-11-202.in.ngrok.io/payment-page/android/resources/transaction-status>",
      "<https://9491-103-159-11-202.in.ngrok.io/payment-page/android/resources/sample-payloads>"
    ],
    "sitemap_alternate_links": false,
    "selectors": {
      "lvl0":"h1,  h2 , .heading-text" ,
      "lvl1": "h3, .label" ,
      "lvl2": ".key-header, .step-card-header-text, .th-row",
      "text":".screen2 p:not(:empty), .hero-welcome, .screen2 li, .main-screen, .only-steps p:not(:empty),td"
    },
    "strip_chars": " .,;:#",
    "scrap_start_urls": true,
    "custom_settings": {
      "synonyms": {
        "relevancy": ["relevant", "relevance"],
        "relevant": ["relevancy", "relevance"],
        "relevance": ["relevancy", "relevant"]
      }
    }
  }
]

Yup shouldn’t be a problem

Can you please guide me ,how to do it in a single config file ? Previously I tried but only last index url's are scraped .

Oh I see what you’re trying to do… It can’t be an array of configs like that

then how to restrict the url hits , which we don't want to show in search result.

You can use stop_urls: https://docsearch.algolia.com/docs/legacy/config-file#stop_urls-optional

but stop url's stop the scraping for that url . actually in my docs we have more than one product . and we want that when we enter a product in search result it shows result for that product only .

I’m not sure I understand your use-case… Could you rephrase?

https://docs.juspay.in/ you can see in the given link there are two product

payment page

and

in-app-upi

so , we want to filter on search result such that one product's search hits not comes in others product search result . so how would we able to do that in a single config file .

Ah I see… You want to add a meta HTML tag to each of your docs pages like this: https://typesense.org/docs/guide/docsearch.html#add-docsearch-meta-tags-optional So in your

payment page

docs pages, you’d add a meta tag to all pages called say:

<meta name="docsearch:product_tag" content="payment_page" />

In your

in-app-upi

docs pages, you’d add a meta tag to all pages called say:

<meta name="docsearch:product_tag" content="in_app_upi" />

Then on the front-end, depending on the product docs the user is visiting right now, you would pass in a

filter_by

parameter in the docsearch.js config like this: https://typesense.org/docs/guide/docsearch.html#option-c-custom-docs-framework-with-docsearch-js-v3-modal-layout See

typesenseSearchParameters.filter_by

. In your case you’d set that like this:

docsearch({
    container: '#searchbar',
    typesenseCollectionName: 'docs', 
    typesenseServerConfig: { ... },
    typesenseSearchParameters: { 
      filter_by: 'product:=in_app_upi' 
    },
  });

so you trying to say that , we would have to put all the urls of our docs in a single config file (every product urls ) . and add different meta tag on every pages for different product .

Could you open the network inspector in your browser’s dev console, then do a search in the search bar and then right-click on the network request to multi_search, copy-as-curl and paste that here?

curl 'http://localhost:8108/multi_search?x-typesense-api-key=xyz' \ -H 'Accept: application/json, text/plain, */*' \ -H 'Accept-Language: en-GB,en-US;q=0.9,en;q=0.8' \ -H 'Cache-Control: no-cache' \ -H 'Connection: keep-alive' \ -H 'Content-Type: text/plain' \ -H 'Origin: http://localhost:3000' \ -H 'Pragma: no-cache' \ -H 'Referer: http://localhost:3000/' \ -H 'Sec-Fetch-Dest: empty' \ -H 'Sec-Fetch-Mode: cors' \ -H 'Sec-Fetch-Site: same-site' \ -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36' \ -H 'sec-ch-ua: "Chromium";v="110", "Not A(Brand";v="24", "Google Chrome";v="110"' \ -H 'sec-ch-ua-mobile: ?0' \ -H 'sec-ch-ua-platform: "macOS"' \ --data-raw '{"searches":[{"collection":"Developer_Docs","q":"session ","query_by":"hierarchy.lvl0,hierarchy.lvl1,hierarchy.lvl2,hierarchy.lvl3,hierarchy.lvl4,hierarchy.lvl5,hierarchy.lvl6,content","include_fields":"hierarchy.lvl0,hierarchy.lvl1,hierarchy.lvl2,hierarchy.lvl3,hierarchy.lvl4,hierarchy.lvl5,hierarchy.lvl6,content,anchor,url,type,id","highlight_full_fields":"hierarchy.lvl0,hierarchy.lvl1,hierarchy.lvl2,hierarchy.lvl3,hierarchy.lvl4,hierarchy.lvl5,hierarchy.lvl6,content","group_by":"url","group_limit":3,"filter_by":"product:=payment-page_android"}]}' \ --compressed

Could you change

filter_by: `product:=${documentationJSON.documentation.productId}`

to

filter_by: `product_tag:=${documentationJSON.documentation.productId}`

and try again?

Thanks it's working now

Could you elaborate on what you mean by “partial scraping”?

sure , we have a call tonight from 1 PM if you are available . want to clear some doubt and some features or issue

The crawler will only crawl links that start with the urls mentioned in

start_urls

, regardless of it being mentioned in the sitemap or not. So for eg, in the above config, it will only crawl pages like this:

<https://docs.juspay.in/payment-page/android/overview/integration-architecture/*>

it's wont work for me . can you please check once is anything problem on my sitemap.xml

Your sitemap looks fine. I wonder if it also expect the sitemap to be in the same location as the start_urls…

may be that is the case