How to get links using mechanize and nokogiri ruby

Question

Given the sample below, can anyone show me how I could use Nokogiri and Mechanize to get all the links under each of the <h4> tags in separate groups, I.E. all the links under:

1. "some text"
2. "some more text"
3. "some additional text"

<div id="right_holder">
    <h3><a href="#"><img src="http://example.com" width="11" height="11"></a></h3>
    <br />
    <br />
    <h4><a href="#">Some text</a></h4>
    <a href="#" alt="name of item"><img src="http://some.image.com" class="class1"></a>
    <a href="#" alt="name of item"><img src="http://some.image.com" class="class1"></a>
    <a href="#" alt="name of item"><img src="http://some.image.com" class="class1"></a>
    <a href="#" alt="name of item"><img src="http://some.image.com" class="class1"></a>
    <a href="#" alt="name of item"><img src="http://some.image.com" class="class1"></a>
    <a href="#" alt="name of item"><img src="http://some.image.com" class="class1"></a>
    <a href="#" alt="name of item"><img src="http://some.image.com" class="class1"></a>
    <br />
    <br />
    <h4><a href="#">Some more text</a></h4>
    <a href="#" alt="name of item"><img src="http://some.image.com" class="class1"></a>
    <a href="#" alt="name of item"><img src="http://some.image.com" class="class1"></a>
    <a href="#" alt="name of item"><img src="http://some.image.com" class="class1"></a>
    <a href="#" alt="name of item"><img src="http://some.image.com" class="class1"></a>
    <a href="#" alt="name of item"><img src="http://some.image.com" class="class1"></a>
    <a href="#" alt="name of item"><img src="http://some.image.com" class="class1"></a>
    <a href="#" alt="name of item"><img src="http://some.image.com" class="class1"></a>
    <br />
    <br />
    <h4><a href="#">Some additional text</a></h4>
    <a href="#" alt="name of item"><img src="http://some.image.com" class="class1"></a>
    <a href="#" alt="name of item"><img src="http://some.image.com" class="class1"></a>
    <a href="#" alt="name of item"><img src="http://some.image.com" class="class1"></a>
    <a href="#" alt="name of item"><img src="http://some.image.com" class="class1"></a>
    <a href="#" alt="name of item"><img src="http://some.image.com" class="class1"></a>
    <a href="#" alt="name of item"><img src="http://some.image.com" class="class1"></a>
    <a href="#" alt="name of item"><img src="http://some.image.com" class="class1"></a>
</div>

Answer 1

In general you would do:

page.search('h4 a').each do |a|
  puts a[:href]
end

But I'm sure you already noticed that none of those links actually go anywhere.

Update:

To group them how about some nodeset math:

page.search('h4').each do |h4|
  puts h4.text
  (h4.search('~ a') - h4.search('~ h4 ~ a')).each do |a|
    puts a.text
  end
end

That means every a that follows the h4 and doesn't also follow another h4

Answer 2

You could go through and separate the data like "How to split a HTML document using Nokogiri?" but if you know what the tag is going to be you could just split it:

# html is the raw html string
html.split('<h4').map{|g| Nokogiri::HTML::DocumentFragment.parse(g).css('a') }

---

page = Nokogiri::HTML(html).css("#right_holder")
links = page.children.inject([]) do |link_hash, child|
  if child.name == 'h4'
    name = child.text
    link_hash << { :name => name, :content => ""}
  end

  next link_hash if link_hash.empty?
  link_hash.last[:content] << child.to_xhtml
  link_hash
end

grouped_hsh = links.inject({}) do |hsh, link|
  hsh[link[:name]] = Nokogiri::HTML::DocumentFragment.parse(link[:content]).css('a')
  hsh
end

# {"Some text"=>[#<Nokogiri::XML::Element:0x3ff4860d6c30,
#  "Some more text"=>[#<Nokogiri::XML::Element:0x3ff486096c20...,
#  "Some additional text"=>[#<Nokogiri::XML::Element:0x3ff486f2de78...}