list spiders from scrapy shell and run individual spider

Question

I want to access scrapy shell from within "scrapy project folder" in terminal. And want to list all the available spiders in my project. I also like to run individual spider and play with the response.

Once i enter into scrapy shell i get following object:

[s] Available Scrapy objects:
[s]   crawler    <scrapy.crawler.Crawler object at 0x10b75cbd0>
[s]   item       {}
[s]   settings   <scrapy.settings.Settings object at 0x10cba1b90>
[s] Useful shortcuts:
[s]   shelp()           Shell help (print this help)
[s]   fetch(req_or_url) Fetch request (or URL) and update local objects
[s]   view(response)    View response in a browser

My best guess is, i use methods from "crawler" object to list available spider. But i have no luck. Also please do let me know how to run spider once i list it.

alecxe alecxe · Accepted Answer · 2016-02-25T04:29:41

list all the available spiders in my project

Use crawler.spiders.list():

>>> for spider_name in crawler.spiders.list():
...     print(spider_name)

list spiders from scrapy shell and run individual spider

1 Answers