post - Submitting to a web form using python

Question

I have seen questions like this asked many many times but none are helpful

Im trying to submit data to a form on the web ive tried requests, and urllib and none have worked

for example here is code that should search for the [python] tag on SO:

import urllib
import urllib2

url = 'http://stackoverflow.com/'

# Prepare the data
values = {'q' : '[python]'}
data = urllib.urlencode(values)

# Send HTTP POST request
req = urllib2.Request(url, data)
response = urllib2.urlopen(req)

html = response.read()

# Print the result
print html

yet when i run it i get the html soure of the home page

here is an example of using requests:

import requests

data= {
    'q': '[python]'
    }
r = requests.get('http://stackoverflow.com', data=data)

print r.text

same result! i dont understand why these methods arent working i've tried them on various sites with no success so if anyone has successfully done this please show me how!

Thanks so much!

Answer 1

If you want to pass q as a parameter in the URL using requests, use the params argument, not data (see Passing Parameters In URLs):

r = requests.get('http://stackoverflow.com', params=data)

This will request https://stackoverflow.com/?q=%5Bpython%5D , which isn't what you are looking for.

You really want to POST to a form. Try this:

r = requests.post('https://stackoverflow.com/search', data=data)

This is essentially the same as GET-ting https://stackguides.com/questions/tagged/python , but I think you'll get the idea from this.

Answer 2

import urllib
import urllib2

url = 'http://www.someserver.com/cgi-bin/register.cgi'
values = {'name' : 'Michael Foord',
      'location' : 'Northampton',
      'language' : 'Python' }

data = urllib.urlencode(values)
req = urllib2.Request(url, data)
response = urllib2.urlopen(req) 
the_page = response.read()

This makes a POST request with the data specified in the values. we need urllib to encode the url and then urllib2 to send a request.

Answer 3

Mechanize library from python is also great allowing you to even submit forms. You can use the following code to create a browser object and create requests.

import mechanize,re
br = mechanize.Browser()
br.set_handle_robots(False)   # ignore robots
br.set_handle_refresh(False)  # can sometimes hang without this
br.addheaders = [('User-agent', 'Firefox')]             
br.open( "http://google.com" )
br.select_form( 'f' )
br.form[ 'q' ] = 'foo'
br.submit()
resp = None

for link in br.links():
    siteMatch = re.compile( 'www.foofighters.com' ).search( link.url )

    if siteMatch:
        resp = br.follow_link( link )
        break

content = resp.get_data()
print content