rvest html_table

Question

I am trying to extract a table using html_table and the rvest package

library(rvest)
test <- html("http://www.privacyrights.org/data-breach/new?title=")
test %>% html_table(html_nodes("table.data-breach-table")[[1]])

however, I keep getting an error

Error in UseMethod("html_nodes"): no applicable method for 'html_nodes' applied to an object of class "character"

hrbrmstr hrbrmstr · Accepted Answer · 2015-10-15T11:00:34

If you are going to nest parenthesized calls anyway, why bother with piping?

html_table(html_nodes(test, "table.data-breach-table")[[1]])

Otherwise go full pipe and use magrittr as well:

library(magrittr)

test %>% 
  html_nodes("table.data-breach-table") %>% 
  extract2(1) %>%
  html_table()

NOTE:

the URL you are using does not have the table you want anyway
you should be using the newest rvest and using read_html

As far as why it wasn't working, you were piping test incorrectly and html_nodes was operating on the table… string instead of the parsed HTML document it expects.

Since you're trying to scrape breaches, this may be of help:

library(rvest)
library(dplyr)
library(pbapply)

urls <- sprintf("http://www.privacyrights.org/data-breach?title=&page=%d", 1:94)

pblapply(urls, function(URL) {

  pg <- read_html(URL)

  tab <- html_nodes(pg, "table")[3]
  rows <- html_nodes(tab, "tr:not(.data-breach-bottom)")

  bind_rows(lapply(seq(2, length(rows)-2, by=2), function(i) {

    tds_1 <- html_nodes(rows[i], "td")
    tds_2 <- html_text(html_nodes(rows[i+1], "td"), trim=TRUE)

    data_frame(date_public=html_text(tds_1[1], TRUE),
               name_loc=html_text(tds_1[2], TRUE),
               entity=html_text(tds_1[3], TRUE),
               type=html_text(tds_1[4], TRUE),
               recs=html_text(tds_1[5], TRUE),
               descr=tds_2[1])

  }))

}) -> things

It's from an older gitst of mine. You'll need to add a randomized sleep delay to that if you do plan on scraping all their breaches.

NOTE also that it's skewed data and be very aware of it's limitations as you attempt to use it (I do data breach research for a living).

rvest html_table

1 Answers