Read only lines that start with specific regular expression

Question

I want to read only line that start with a specific regular expression.

 val rawData = spark.read.textFile(file.path).filter(f => f.nonEmpty && f.length > 1 && f.startsWith("(")) 

is what I did until now.

Now I found out that I have entries starting with: (W);27536- or (W) 28325- (5 digits after seperator)
I only want to read lines that start with (W);1234- (4 digits after seperator)

The regular expression that would catch this look like: \(\D\)(;|\s)\d{4} for a boolean return or \(\D\)(;|\s)\d{4}-.* for a string match return

My problem now is that I don't know how to include the regular expression in my read.textFile command.
f.startswith only works with strings
f.matches also only works with strings

I also tried using http://www.scala-lang.org/api/2.12.3/scala/util/matching/Regex.html but this returns a string and not a boolean, which I can not use in the filter function

Any help would be appreciated.

Answer 1

Other answers are over-thinking this. Just use matches

val lineRegex = """\(\D\)(;|\s)\d{4}-.*"""
val ns = List ("(W);1234-something",
               "(W);12345-something",
               "(W);2345-something",
               "(W);23456-something",
               "(W);3456-something",
               "",
               "1" )
ns.filter(f=> f.matches(lineRegex))

results in

List("(W);1234-something", "(W);2345-something", "(W);3456-something")

Answer 2

I found the answer to my question.

The command needs to look like this.

 val lineregex = """\(\D\)(;|\s)\d{4}-.*""".r

 val rawData = spark.read.textFile(file.path)
  .filter(f => f.nonEmpty && f.length > 1 && lineregex.unapplySeq(f).isDefined )

Answer 3

You can try to find a match of the Regex using the findFirstMatchIn method, which returns an Option[Match]:

spark.read.textFile(file.path).filter { line =>
  line.nonEmpty &&
  line.length > 1 &&
  "regex".r.findFirstMatchIn(line).isDefined
}