robotstxt (version 0.1.2)

rt_get_comments: extrcting comments from robots.txt

Description

extrcting comments from robots.txt

Usage

rt_get_comments(txt)

Arguments

txt

content of the robots.txt file