R web scraping by class

Question

I have a finance assignment for collecting the beta value for calculation, I am new in R, I would like to web scraping the beta value by package rvest or httr. However, output is character(0).

xpath:
//*[@id="StkList"]/ul/li[48]

library(rvest)
library(dplyr)

sym <- "1212"
url.3 < paste("http://www.etnet.com.hk/www/eng/stocks/realtime/quote.php?code=",sym,sep="")

beta.value <- url.3 %>% read_html() %>% html_nodes(xpath = "//*[@id='StkList']/ul/li[48]")

output:
character(0)

desired output:
0.270

I tried not using xpath but, the html_nodes("div.value.highlight") but don't work as well. Is anyone can help or advice ? Thank you.

Btw I couldn't see 0.27 in that page. Could you please let me know where it is? — amrrs
@amrrs Beta +0.270. sorry for any inconvenience. ctrl+F and type beta that row — Peter Chung
In fact I couldn't see Beta, Could you please update the screenshot? — amrrs
Thanks @Peter but for some reason I couldn't even see this in the table. Maybe because i'm accessing from a different country? — amrrs

hrbrmstr hrbrmstr · Accepted Answer · 2018-01-25T14:29:57

They check referer before displaying the page, so you have to add some headers:

library(magrittr)
library(httr)
library(rvest)

httr::GET(
  url = "http://www.etnet.com.hk/www/eng/stocks/realtime/quote.php?code=1212", 
  httr::add_headers(
    Host = "www.etnet.com.hk",
    Referer = "http://www.etnet.com.hk/www/eng/stocks/realtime/quote.php?code=1212"
  )
) -> res

res <- content(res, encoding="UTF-8")

html_node(res, xpath=".//li[contains(., 'Beta')]/following-sibling::li[1]") %>% 
  html_text()
## [1] "+0.270"

R web scraping by class

1 Answers