Web Scraping Asp.NET site using Beautiful Soup and Python

Question

I have the following code, but its gives be 200 OK with first page (state of default drop down) response. Please note that the Drop Down lists are dymanic and progressive until final search button appears , Can someone correct me as to what is wrong with my code?

def process(ghatno):
    home_url = 'http://igrmaharashtra.gov.in/eASR/eASRCommon.aspx?hDistName=Nashik'
    post_url = 'http://igrmaharashtra.gov.in/eASR/eASRCommon.aspx?hDistName=Nashik'

    print "Please wait...getting details of :" + ghatno

    with requests.Session() as session:
        r = session.get(url=post_url)
        cookies = r.cookies
        pprint.pprint(r.headers)
        gethead = r.headers
        soup = BeautifulSoup(r.text, 'html.parser')
        viewstate = soup.select('input[name="__VIEWSTATE"]')[0]['value']
        csrftoken = soup.select('input[name="__CSRFTOKEN"]')[0]['value']
        eventvalidation = soup.select('input[name="__EVENTVALIDATION"]')[0]['value']
        viewgen = soup.select('input[name="__VIEWSTATEGENERATOR"]')[0]['value']

        data = {
            '__CSRFTOKEN':csrftoken,
            '__EVENTARGUMENT':'',
            '__EVENTTARGET':'',
            '__LASTFOCUS':'',
            '__SCROLLPOSITION':'0',
            '__SCROLLPOSITIONY':'0',
            '__EVENTVALIDATION': eventvalidation,
            '__VIEWSTATE':viewstate,
            '__VIEWSTATEGENERATOR': viewgen,
            'ctl00$ContentPlaceHolder5$ddlLanguage' : 'en-US',
            'ctl00$ContentPlaceHolder5$btnSearchCommonSr':'Search',
            'ctl00$ContentPlaceHolder5$ddlTaluka': '2',
            'ctl00$ContentPlaceHolder5$ddlVillage': '25',
            'ctl00$ContentPlaceHolder5$ddlYear': '20192020',
            'ctl00$ContentPlaceHolder5$grpSurveyLocation': 'rdbSurveyNo',
            'ctl00$ContentPlaceHolder5$txtCommonSurvey': 363
        }


        headers = {
        'Host': 'igrmaharashtra.gov.in',
        'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:60.0) Gecko/20100101 Firefox/60.0',
        'Referer': 'http://igrmaharashtra.gov.in/eASR/eASRCommon.aspx?hDistName=Nashik',
        'Host': 'igrmaharashtra.gov.in',
        'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
        'Accept-Language': 'en-US,en;q=0.5',

        }


        r = requests.post(url=post_url, data=json.dumps(data), cookies=cookies, headers = headers)
        soup = BeautifulSoup(r.text, 'html.parser')
        table = SoupStrainer('tr')
        soup = BeautifulSoup(soup.get_text(), 'html.parser', parse_only=table)
        print(soup.get_text())
        pprint.pprint(r.headers)
        print r.text
        getpost = r.headers
        getpostrequest = r.request.headers
        getresponsebody = r.request.body

        f = open('/var/www/html/nashik/hiren.txt', 'w')
        f.write(str(gethead))
        f.write(str(getpostrequest))
        f.write(str(getresponsebody))
        f.write(str(getpost))

My response is as below :

Response header - (GET Request)

{'Content-Length': '5994', 'X-AspNet-Version': '4.0.30319', 'Set-Cookie': 'ASP.NET_SessionId=24wwh11lwvzy5gf0xlzi1we4; path=/; HttpOnly, __CSRFCOOKIE=d7b10286-fc9f-4ed2-863d-304737df8758; path=/; HttpOnly', 'Content-Encoding': 'gzip', 'Vary': 'Accept-Encoding', 'X-Powered-By': 'ASP.NET', 'Server': 'Microsoft-IIS/8.0', 'Cache-Control': 'private', 'Date': 'Thu, 02 May 2019 08:21:48 GMT', 'Content-Type': 'text/html; charset=utf-8'}

Response header - (GET Request)

{'Content-Length': '3726', 'Accept-Language': 'en-US,en;q=0.5', 'Accept-Encoding': 'gzip, deflate', 'Host': 'igrmaharashtra.gov.in', 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', 'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:60.0) Gecko/20100101 Firefox/60.0', 'Connection': 'keep-alive', 'Referer': 'http://igrmaharashtra.gov.in/eASR/eASRCommon.aspx?hDistName=Nashik', 'Cookie': '__CSRFCOOKIE=d7b10286-fc9f-4ed2-863d-304737df8758; ASP.NET_SessionId=24wwh11lwvzy5gf0xlzi1we4', 'Content-Type': 'application/x-www-form-urlencoded'}

Response header - (POST Request)

{'Content-Length': '7834', 'X-AspNet-Version': '4.0.30319', 'Content-Encoding': 'gzip', 'Vary': 'Accept-Encoding', 'X-Powered-By': 'ASP.NET', 'Server': 'Microsoft-IIS/8.0', 'Cache-Control': 'private', 'Date': 'Fri, 03 May 2019 10:21:45 GMT', 'Content-Type': 'text/html; charset=utf-8'}

**Default Page Selected Drop Down is returned **

नाशिक and - - Select Taluka - - INSTEAD of option value "2" i.e इगतपुरी once option "2" is selected I want value "25" in next drop down before I put my final survey "363" for results.

Please note I tried Mechanize browser too, but no luck !!

What is your code supposed to do? And can you supply test ghatno value with expected output? — QHarr
@QHarr , the number is 363 and i just reliazed that the drop downs are dymanically loading on each POST request, i.e once you select 1st drop down, there is a POST requests, then you select second drop down and so on.. — DaVinci007
So the post request is responsible for populating next dropdown? — QHarr
yes, so inorder to get the result i need to post form 3 times , in the code i am passing the values for the last post, now i am getting response but that of 1 post state i.e only 1st drop down selected state rest drop downs not visible in response — DaVinci007
@DaVinci007 can you post the correct answer in the answer section below? — Praveen Talavar

DaVinci007 DaVinci007 · Accepted Answer · 2019-06-14T04:50:06

Finally the solution is to do post requests multiple times in same "session" with same "cookie" and iterate through them. It works now !

Web Scraping Asp.NET site using Beautiful Soup and Python

1 Answers