⚠️There's a newer version (0.1.9-1) of this package.Take me there.

Rcrawler (version 0.1.1)

Web Crawler and Scraper

Description

Performs parallel web crawling and web scraping. It is designed to crawl, parse and store web pages to produce data that can be directly used for analysis application. For details see Khalil and Fakir (2017) .

Copy Link

Version

Install

install.packages('Rcrawler')

Monthly Downloads

Version

0.1.1

License

MIT + file LICENSE

Issues

Pull Requests

Stars

363

Forks

Repository

https://github.com/salimk/Rcrawler/

Maintainer

Salim Khalil

Last Published

June 17th, 2017

Functions in Rcrawler (0.1.1)

Get the list of parameters and values from an URL

Linkparamsfilter

Link parameters filter

RobotParser

RobotParser fetch and parse robots.txt

getDistance

Calculate Distance between two SimHash fingerprint

getsimHash

Calculate SimHash fingerprint in R

LinkExtractor

LinkNormalization

Link Normalization