paths_allowed_worker for robotstxt flavor
paths_allowed_worker_robotstxt(domain, bot, paths, permissions_list)Domain for which paths should be checked. Defaults to "auto". If set to "auto" function will try to guess the domain by parsing the paths argument. Note however, that these are educated guesses which might utterly fail. To be on the save side, provide appropriate domains manually.
name of the bot, defaults to "*"
paths for which to check bot's permission, defaults to "/"
list of permissions from robotstxt objects