Yahoo Finance API Java Download CSV

Question

I have been using the Yahoo Finance "API" with the code below for a few weeks. Since about a week ago, it stopped working. If you type in this address: http://ichart.yahoo.com/table.csv?s=MSFT, you'll get some historical data for Microsoft (my browser automatically downloads it). However, when I try to read it through Java, the stream is apparently opened (I don't get an exception), however, the stream contains no data. Anyone know why the buffered reader below is not able to stream the object as it had been able to do in previous weeks? I suspect that maybe Yahoo added some java script to block automated downloading.

URL url = new URL("http://ichart.yahoo.com/table.csv?s=" + symbol);
URLConnection hc = url.openConnection();
hc.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 4.10; rv:52.0) Gecko/20100101 Firefox/52.0");
BufferedReader in = new BufferedReader(new InputStreamReader(hc.getInputStream()));
    br.readLine();
    //Read File Line By Line
    String strLine;
    while ((strLine = br.readLine()) != null) {
        System.out.println(strLine);
    }
    br.close();

Answer 1

You can change the URL from "http" to "https" which works without any issues.

Answer 2

Many Websites are checking for a User Agent String and block acess if no String has been sent to block automated access.

This is an example how you can add the user agent string:

    URL url = new URL("http://ichart.yahoo.com/table.csv?s=MSFT");
    URLConnection hc = url.openConnection();
    hc.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 4.10; rv:52.0) Gecko/20100101 Firefox/52.0");

Answer 3

To understand the reason the java communication doesn't work, track the html communication. I used Chrome browser, with HTTP trace extension enabled. The trace output shows that the link http://ichart.yahoo.com/table.csv?s=^GSPC redirects to https://ichart.yahoo.com/table.csv?s=^GSPC.

In Java, you need to implement a URL connection to follow the redirect. An example of following the redirect is at http://www.mkyong.com/java/java-httpurlconnection-follow-redirect-example/. Modify the example by passing your url string, and you should get the expected output.

Answer 4

1. This is a java code to download the historical stocks data from yahoo finance.

2. here I have taken the required date parameters and set them in a Calender Object, then this object is converted to period object this object is passed to the function where it will be converted to UNIX timestamp and then that value is appended to the link

3. I have created a url object and that will get the csv file of the historical data for the required ticker value which is read using the input stream reader and them it is displayed on the console.

    public class demo {
    public static void main(String[] args) throws IOException, ParseException {
   
    final String SYM = "TSLA";
    Calendar cal = Calendar.getInstance();
    Calendar cal1 = Calendar.getInstance();
    cal.set(Calendar.YEAR, 2019);
    cal.set(Calendar.MONTH, Calendar.JULY);
    cal.set(Calendar.DAY_OF_MONTH, 16);
   
    Date period1 = datechange(cal);
    cal1.set(Calendar.YEAR, 2019);
    cal1.set(Calendar.MONTH, Calendar.JULY);
    cal1.set(Calendar.DAY_OF_MONTH, 23);
    Date period2 = datechange(cal1);
   
    String interval="1d";
   
    long strDate = (period1.getTime());
    strDate= strDate/1000;
    long strDate1 = (period2.getTime());
    strDate1= strDate1/1000;
    //System.out.println(strDate+" "+strDate1+"     ans");
    String link="https://query1.finance.yahoo.com/v7/finance/download/"+SYM+"?period1="+strDate+"&period2="+strDate1+"&interval="+interval+"&events=history";
   
    URL url = new URL(link);
    URLConnection urlConn = url.openConnection();
    InputStreamReader inStream = new InputStreamReader(urlConn.getInputStream());
    BufferedReader buf = new BufferedReader(inStream);
   
   
    String line =buf.readLine();
    while(line != null) {
        System.out.println(line);
        line=buf.readLine();
    }
    }
   
    public static Date datechange(Calendar cal) throws ParseException {
    Date dateOne =cal.getTime();
    //boolean date1904 = true;
    //double ans =DateUtil.getExcelDate(cal,date1904);
   
    String a = dateOne.toString();
    String b[] = a.split(" ");
    String c = b[1]+" "+b[2]+" "+b[5];
    SimpleDateFormat sdf = new SimpleDateFormat("MMM dd yyyy", 
    Locale.ENGLISH);
    sdf.setTimeZone(TimeZone.getTimeZone("GMT"));
    cal.setTime(sdf.parse(c));
    dateOne=cal.getTime();
    sdf.format(dateOne);
    return dateOne;
   
    }
   
   
    }
   

Answer 5

Try executing this code from some other network. That'll give you clarity whether Yahoo has blocked automated downloading or only blacklisted the network you're using.