How to use R to scrape financials from Yahoo Finance

Question

I am interested in analyzing the balance, income and cash flow statements from Yahoo Finance for multiple tickers using R.

I have seen that there are R packages that pull information from Yahoo Finance, but all the examples I have seen concern historical stock price information. Is there a way I can pull historical information from these statements using R?

For example, for Apple (AAPL) the retrievable links are as follows:

In essence, the goal is to create three data-frames (AAPL_cashflow, AAPL_income & AAPL_balance) that consists the same pattern as on the website. Each row is identified by the type of financial and the columns are the date.

Does anybody have any experience with parsing and scraping tables? I think that rvest can help with this, right?

Thanks in advance!

What have you tried so far? Please add the code you've written to your question. — cmaher

tyluRp tyluRp · Accepted Answer · 2018-04-15T18:34:20

With a handful of packages from the tidyverse, this should get you started:

library(tidyverse)
library(rvest)

"https://finance.yahoo.com/quote/AAPL/financials?p=AAPL" %>% 
  read_html() %>% 
  html_table() %>% 
  map_df(bind_cols) %>% 
  as_tibble()

# A tibble: 28 x 5
   X1                                 X2                 X3                 X4                 X5      
   <chr>                              <chr>              <chr>              <chr>              <chr>   
 1 Revenue                            9/30/2017          9/24/2016          9/26/2015          9/27/20…
 2 Total Revenue                      229,234,000        215,639,000        233,715,000        182,795…
 3 Cost of Revenue                    141,048,000        131,376,000        140,089,000        112,258…
 4 Gross Profit                       88,186,000         84,263,000         93,626,000         70,537,…
 5 Operating Expenses                 Operating Expenses Operating Expenses Operating Expenses Operati…
 6 Research Development               11,581,000         10,045,000         8,067,000          6,041,0…
 7 Selling General and Administrative 15,261,000         14,194,000         14,329,000         11,993,…
 8 Non Recurring                      -                  -                  -                  -       
 9 Others                             -                  -                  -                  -       
10 Total Operating Expenses           167,890,000        155,615,000        162,485,000        130,292…
# ... with 18 more rows

Note that if you want to take the first row and treat it as a column name, add header = TRUE to the html_table call. This will give you the dates as column names in the finances data frame for example.

Additionally, there are multiple tables inside this data frame so you will need to reshape it in order to play with the data. For example, var X2 through X5 are currently character when they should be numeric type.

One example might be:

finances <- "https://finance.yahoo.com/quote/AAPL/financials?p=AAPL" %>% 
  read_html() %>% 
  html_table(header = TRUE) %>% 
  map_df(bind_cols) %>% 
  as_tibble()

finances %>% 
  mutate_all(funs(str_replace_all(., ",", ""))) %>% 
  mutate_all(funs(str_replace(., "-", NA_character_))) %>%
  mutate_at(vars(-Revenue), funs(str_remove_all(., "[a-zA-Z]"))) %>% 
  mutate_at(vars(-Revenue), funs(as.numeric)) %>% 
  drop_na()

# A tibble: 14 x 5
   Revenue                                `9/30/2017` `9/24/2016` `9/26/2015` `9/27/2014`
   <chr>                                        <dbl>       <dbl>       <dbl>       <dbl>
 1 Total Revenue                           229234000.  215639000.  233715000.  182795000.
 2 Cost of Revenue                         141048000.  131376000.  140089000.  112258000.
 3 Gross Profit                             88186000.   84263000.   93626000.   70537000.
 4 Research Development                     11581000.   10045000.    8067000.    6041000.
 5 Selling General and Administrative       15261000.   14194000.   14329000.   11993000.
 6 Total Operating Expenses                167890000.  155615000.  162485000.  130292000.
 7 Operating Income or Loss                 61344000.   60024000.   71230000.   52503000.
 8 Total Other Income/Expenses Net           2745000.    1348000.    1285000.     980000.
 9 Earnings Before Interest and Taxes       61344000.   60024000.   71230000.   52503000.
10 Income Before Tax                        64089000.   61372000.   72515000.   53483000.
11 Income Tax Expense                       15738000.   15685000.   19121000.   13973000.
12 Net Income From Continuing Ops           48351000.   45687000.   53394000.   39510000.
13 Net Income                               48351000.   45687000.   53394000.   39510000.
14 Net Income Applicable To Common Shares   48351000.   45687000.   53394000.   39510000.

We could go a step further and make the data frame more "tidy" using gather:

finances %>% 
  mutate_all(funs(str_replace_all(., ",", ""))) %>% 
  mutate_all(funs(str_replace(., "-", NA_character_))) %>%
  mutate_at(vars(-Revenue), funs(str_remove_all(., "[a-zA-Z]"))) %>% 
  mutate_at(vars(-Revenue), funs(as.numeric)) %>% 
  drop_na() %>% 
  gather(key = "date", value, -Revenue) %>% 
  mutate(date = lubridate::mdy(date)) %>% 
  rename("var" = Revenue) %>% 
  as_tibble()

# A tibble: 56 x 3
   var                                date            value
   <chr>                              <date>          <dbl>
 1 Total Revenue                      2017-09-30 229234000.
 2 Cost of Revenue                    2017-09-30 141048000.
 3 Gross Profit                       2017-09-30  88186000.
 4 Research Development               2017-09-30  11581000.
 5 Selling General and Administrative 2017-09-30  15261000.
 6 Total Operating Expenses           2017-09-30 167890000.
 7 Operating Income or Loss           2017-09-30  61344000.
 8 Total Other Income/Expenses Net    2017-09-30   2745000.
 9 Earnings Before Interest and Taxes 2017-09-30  61344000.
10 Income Before Tax                  2017-09-30  64089000.
# ... with 46 more rows

How to use R to scrape financials from Yahoo Finance

2 Answers