sanitize_permissions

permissions

<p>transforming permissions into regular expressions (whole permission)</p>

internal

Provides functions to download and parse 'robots.txt' files.
Ultimately the package makes it easy to check if bots
(spiders, crawler, scrapers, ...) are allowed to access specific
resources on a domain.

Peter Meissner

robotstxt

A 'robots.txt' Parser and 'Webbot'/'Spider'/'Crawler'
Permissions Checker

Oliver Keys

Rich Fitz John

sanitize_permissions function

transforming permissions into regular expressions (whole permission) — sanitize_permissions

transforming permissions into regular expressions (whole permission)

sanitize_permissions: transforming permissions into regular expressions (whole permission)

Description

Usage

Arguments