guess_domain

<p>path aka URL from which to infer domain</p>

<p>function guessing domain from path</p>

Provides functions to download and parse 'robots.txt' files.
Ultimately the package makes it easy to check if bots
(spiders, crawler, scrapers, ...) are allowed to access specific
resources on a domain.

Peter Meissner

robotstxt

guess_domain: function guessing domain from path

Description

Usage

Arguments