robotstxt (version 0.1.2)

get_robotstxt: downloading robots.txt file

Description

downloading robots.txt file

Usage

get_robotstxt(domain)

Arguments

domain

domain from which to download robots.txt file