In v2.2.1 there is a nutch crawl command in which you can give depth to crawl
bin/nutch crawl [-solr ] [-dir d] [-threads n] [-depth i] [-topN N]
How to do the same for nutch v2.3 ?
0
votes
2 Answers
1
votes
The documentation for the script is not updated, I think the script there is relevant for Nutch 1.4.
You can always check Nutch formal repository at github, of course make sure that you're looking at the right branch.
Anyway, Nutch 2.X crawl script is as follows (and you can see the 2.3 source here):
crawl <seedDir> <crawlId> <solrURL> <numberOfRounds>
Where seedDir is the directory where your seed urls' files are, crawlId is a name you wish to call your crawl job, solrURL is self-explantory and numberOfRounds is what you are looking for.
Notice however that the script also has parameters that you can change (within the script), like the number of fetched pages per level (i.e. the breadth of crawl).
