seoR (version 0.1.0)

allowedByRobots: Function to check if URL is blocked by robots.txt

Description

This function checks if a given URL is blocked by the robots.txt file of the website.

Usage

allowedByRobots(url, bot = "googlebot")

Arguments

url

The url you want to check

bot

The bot you want to check the indexability with. Default is googlebot allowedByRobots()

Examples

# NOT RUN {
allowedByRobots("https://www.r-project.org/", bot = "googlebot")
# }