Web Scraping Weather Data 400 error

Question

I have tried using get requests with params, post requests with and without json.dumps but I can not seem to get the right response from the webpage.

I want to get back a text file with all the weather data. My code, the data I want, and the headers for the post request using the inspect on chrome is attached.

Thank you for any help!

You can also reach the data by

going to http://climod2.nrcc.cornell.edu
click on daily data listing
input dates from 6/1/18 to current date
check csv instead of html
then type NY City Central Park for the station selection which will return search results with that station then click go.

Below are the screenshots:
1. Attempt
2. Had to delete due to not being able to post more than 8 links.
3. Headers
4. Desired data
5. Response Data `
from urllib.request import urlopen as uReq from bs4 import BeautifulSoup as soup import requests import datetime import json

payload = {'method': 'POST','params':{"elems":[{"name":"maxt","add":"t"}, {"name":"mint","add":"t"},{"name":"pcpn","add":"t"}, {"name":"snow","add":"t"},{"name":"snwd","add":"t"}],"sId":"94728 1","sDate":"2018-06-01","eDate":"2018-06-28"},'output':'json'}

parameters = {"elems":[{"name":"maxt","add":"t"},{"name":"mint","add":"t"}, {"name":"pcpn","add":"t"},{"name":"snow","add":"t"}, {"name":"snwd","add":"t"}],"sid":"94728 1","sDate":"2018-06- 01","eDate":"2018-06-28"}

headers = {'Accept': 'application/json, text/javascript, /; q=0.01','Content-Type': 'application/x-www-form-urlencoded; charset=UTF- 8','Origin': 'http://climod2.nrcc.cornell.edu','Referer':'http://climod2.nrcc.cornell.edu/elems=name&elems=add&elems=name&elems=add&elems=name&elems=add&elems=name&elems=add&elems=name&elems=add&sDate=2018-06-01&eDate=2018-06-28&meta=name&meta=state&meta=ll&meta=sids&bbox=-74.44262803978918&bbox=40.4207924883181&bbox=-73.4880821602','User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Safari/537.36'}

dataurl='https://data.rcc-acis.org/StnData'

r = requests.post(dataurl, data = json.dumps(payload), headers = headers)

print(r.status_code) print(r.elapsed) print(r.json) print(r.text)`

Thank you! I couldn't figure out how to get the imports in the block though. — Jack
select the code and enter Ctrl + K, refer stackoverflow.com/editing-help#code if you need more help. — Abhilash

jlaur jlaur · Accepted Answer · 2018-06-29T16:56:06

You just have to set the payload right.

This will do the trick.

import requests
from pprint import pprint

url = "https://data.rcc-acis.org/StnMeta"

payload = {
    "output": "json",
    "params": {"elems":[{"name":"maxt","add":"t"},{"name":"mint","add":"t"},{"name":"pcpn","add":"t"},{"name":"snow","add":"t"},{"name":"snwd","add":"t"}],"sDate":"2018-06-01","eDate":"2018-06-29","meta":["name","state","ll","sids"],"bbox":[-74.44262803978918,40.4207924883181,-73.48808216021084,41.144936911681896]}
}
r = requests.post(url, json=payload)
data = r.json()

The object named data will now consist of list of dicts for each place that has weather data. Each dict contain amongst other an sid, that you need to feed the endpoint that return the actual weather data.

Each dict will look like this:

{'ll': [-74.42259, 40.47282],
  'name': 'NEW BRUNSWICK 3 SE',
  'sids': ['286055 2', 'USC00286055 6', 'NBRN4 7'],
  'state': 'NJ'}

Assuming that you want to loop through all these and return the weather data for each you first need to grab the sid's and store them in a list

Some of them, tho, does not contain weather data. They will look like this:

"US1NJES0018 6"

... while those that has weather data does not have letters in them. This can be exploited with a regex to filter away all sid's with letters in them.

So append this to the code:

import re # the regex library
sids_list = []

is_garbage = re.compile('[a-zA-z]+') # will match everything with a letter in
meta = data['meta']
for m in meta:
    name = m['name']
    sid = m['sids'][0]
    if not is_garbage.search(sid):
        sids_list.append([name,sid])

Now that you have a list of sid's, all you need to do is use the right payload to the final endpoint, set the dates you want data for - and start a loop.

start_date = "2018-06-01"
end_date = "2018-07-01"
for name, sid in sids_list:
    print("------------- DATA FOR {} (sid: {}) -------------".format(name,sid))
    print()
    payload2 = {
        "params": {
            "elems":
            [
                {"name":"maxt","add":"t"},
                {"name":"mint","add":"t"},
                {"name":"pcpn","add":"t"},
                {"name":"snow","add":"t"},
                {"name":"snwd","add":"t"}
            ],
            "sid": sid,
            "sDate":start_date,
            "eDate":end_date},
        "output": "json"
    }

    weather_url = "https://data.rcc-acis.org/StnData"
    r = requests.post(weather_url, json=payload2)
    weather_data = r.json()['data']
    print("{:15}{:15}{:15}{:15}{:15}{:15}".format("Date", "MaxTemperature", "MinTemperature", "Precipitation", "Snowfall", "SnowDepth"))
    for Date, MaxTemperature, MinTemperature, Precipitation, Snowfall, SnowDepth in weather_data:
        print("{:15}{:15}{:15}{:15}{:15}{:15}".format(Date, MaxTemperature[0], MinTemperature[0], Precipitation[0], Snowfall[0], SnowDepth[0]))
    print()

Here I'm just printing the data to screen in a nicely formatted view. Hope this solves your issue. Cheers.

Web Scraping Weather Data 400 error

1 Answers