Get form url without submit - using Python

Question

I have a webpage which contains one or more forms. What i want to do is:

Identify the forms
Send post requests and catch the response.

I'm over point 1, i'm using requests.get and Beautifulsoup to identify the forms from the webpage. My question is, how can i get the form url without submitting the form?

Example: I'll search for "test" on https://stackoverflow.com/

The url looks like this: https://stackoverflow.com/search?q=test

I'm interested in getting this part: /search?q because other sites have more complicated urls in these cases and i want to build a scraper that is not website-dependent.

The full code which i tried:

from bs4 import BeautifulSoup
import urllib.request
import requests
import mechanicalsoup

#### What?
search_words=['search1','search2']
website='http://www.website.com/'
####
s=requests.Session()
r=s.get(website)
soup_main = BeautifulSoup(r.content,'lxml')

form=soup_main.find('form')
print(form)
param={'searchword':search_words[0]}

method = str(form.get("method"))
print(method)
action =form.get("action")
url = urllib.parse.urljoin(website, action)
print(action)

request1=requests.Request(method,url,params=param)

It is the action of the form (can be relative) plus the url-encoded query (the form data). BTW this only applies to GET queries. — Klaus D.

Matthieu Moy Matthieu Moy · Accepted Answer · 2017-10-18T11:56:10

Here's a working example doing this:

>>> import mechanicalsoup                                                                                                         
>>> browser = mechanicalsoup.StatefulBrowser()                                                                                  
>>> browser.open('https://stackoverflow.com/')                                                                                      
<Response [200]>
>>> form = browser.select_form("form.searchbar")  # Get a form with class 'searchbar'
>>> action = form.form.attrs['action']  # Get the action="" field
>>> browser.absolute_url(action)  # Make the URL absolute
'https://stackoverflow.com/search'

Note that the q= is not part of the submission URL, it is actually part of the arguments given to the URL.

Depending on what you want to do with this URL, you may also want to let MechanicalSoup do the form submission for you:

>>> browser.select_form("form.searchbar")                                                                                     
<mechanicalsoup.form.Form object at 0x7fb5ae5c3eb8>
>>> browser["q"] = 'How to use MechanicalSoup?'                                                                                       
>>> browser.submit_selected()
<Response [200]>
>>> browser.get_url()
'https://stackoverflow.com/search?q=How+to+use+MechanicalSoup%3F'

You can check the state of the browser at any time with:

>>> browser.launch_browser()

Get form url without submit - using Python

1 Answers