Parse and Test Robots Exclusion Protocol Files and Rules
Description
The 'Robots Exclusion Protocol' documents
a set of standards for allowing or excluding robot/spider crawling of different areas of
site content. Tools are provided which wrap The 'rep-cpp'
C++ library for processing these 'robots.txt' files.