Extract URL from href-tag in groovy

Question

Extract URL from href-tag in groovy

3.7k views Asked by Jakunar At 17 March 2013 at 16:01

I need to parse a malformed HTML-page and extract certain URLs from it as any kind of Collection. I don't really care what kind of Collection, I just need to be able to iterate over it.

Let's say we have a structure like this:

<html>
  <body>
    <div class="outer">
      <div class="inner">
        <a href="http://www.google.com" title="Google">Google-Link</a>
        <a href="http://www.useless.com" title="I don't need this">Blah blah</a>
      </div>
      <div class="inner">
        <a href="http://www.youtube.com" title="Youtube">Youtube-Link</a>
        <a href="http://www.useless2.com" title="I don't need this2">Blah blah2</a>
      </div>
    </div>
  </body>
</html>

And here is what I do so far:

// tagsoup version 1.2 is under apache license 2.0
@Grab(group='org.ccil.cowan.tagsoup', module='tagsoup', version='1.2' )
XmlSlurper slurper = new XmlSlurper(new org.ccil.cowan.tagsoup.Parser());

GPathResult nodes = slurper.parse("test.html"); 
def links = nodes."**".findAll { it.@class == "inner" }
println links

I want something like

["http://google.com", "http://youtube.com"]

but all I get is:

["Google-LinkBlah blah", "Youtube-LinkBlah blah2"]

To be more precise I can't use all URLs, because the HTML-document, that I need parse is about 15-thousand lines long and has alot of URLs that I don't need. So I need the first URL in each "inner" block.

Original Q&A

There are 2 answers

The Trav On 18 March 2013 at 02:18

you're looking for @href on each of your nodes

**tim_yates** · Accepted Answer · 2013-03-18T09:43:45+00:00

As The Trav says, you need to grab the href attribute from each matching a tag.

You've edited your question so the class bit in the findAll makes no sense, but with the current HTML example, this should work:

def links = nodes.'**'.findAll { it.name() == 'a' }*.@href*.text()

Edit

If (as you say after the edit) you just want the first a inside anything marked with class="inner", then try:

def links = nodes.'**'.findAll { it.@class?.text() == 'inner' }
                 .collect { d -> d.'**'.find { it.name() == 'a' }?.@href }
                 .findAll() // remove nulls if there are any

TechQA.

Extract URL from href-tag in groovy

There are 2 answers

Edit

Related Questions in GROOVY

Related Questions in XMLSLURPER

Related Questions in TAG-SOUP

Popular Questions

Popular Tags

Trending Questions