Attoparsec /= version of stringCI

Question

I'm parsing robots.txt files and I've written the parser to successfully parse a "well-formed" robots.txt file. I've been able to adjust the parser to skip lines that start with a symbol (like # or / for comments) but only using inClass "#/".

One problem I've been unable to solve is skipping a line if it DOES NOT contain the string I want to match.

User-agent: *
Disallow: /includes/
Disallow: /misc/
Disallow: /modules/
Doesn't belong here
Disallow: /profiles/
Disallow: /scripts/
Disallow: /themes/

I first tried matching using:

satisfy (notInClass "DdUu") *> skipWhile (not . isEndOfLine)

And figured doing it that way would negate my need for the specific comment line parser as the hashes or slashes don't fall into the character class. The problem is that this doesn't work.

I also realize it wouldn't work ANYWAY if it did, because it wouldn't solve matching for something like "Disallow" vs. "Don't allow".

Here's the parsing code (without the comment skipping code, this only works for well-formed robots.txt):

{-# LANGUAGE OverloadedStrings, RankNTypes #-}

import           Prelude hiding (takeWhile)
import           Control.Applicative hiding (many)
import           Data.Char
import           Data.Text as T hiding (toLower)
import           Data.Text.Encoding as E
import           Control.Monad
import           Data.Attoparsec.ByteString
import qualified Data.Attoparsec.Char8 as AC
import           Data.Array.Unboxed
import           Data.ByteString as B hiding (takeWhile)
import qualified Data.ByteString.Internal as BI
import           Data.Word (Word8)

type RuleMap = [(ByteString, ByteString)]

-- newtype for indexable ua
newtype UserAgent = UserAgent { unUA :: ByteString }
    deriving (Eq, Ord, Show)

data RuleSet = RuleSet
    { userAgent :: UserAgent,
      rules     :: RuleMap }
     deriving (Eq, Ord, Show)

main = do
    r <- B.readFile "/Users/ixmatus/Desktop/robots.txt"
    print $ parse (many1 parseUABlock) r

stripper = E.encodeUtf8 . T.strip . E.decodeUtf8

isNotEnd = not . AC.isEndOfLine

-- | Catch all character matching, basically
matchALL :: Word8 -> Bool
matchALL = inClass ":/?#[]@!$&'()*%+,;=.~a-zA-Z0-9 _-"

-- | @doParse@ Run the parser, complete the partial if the end of the stream has
-- a newline with an empty string
doParse :: ByteString -> [RuleSet]
doParse cont =
    case parse (many1 parseUABlock) cont of
        Done _ set -> set
        Partial f -> handlePartial (f B.empty)
        Fail {} -> []

-- | @handlePartial@ Handle a partial with empty string by simply
-- returning the last completion
handlePartial :: forall t a. IResult t [a] -> [a]
handlePartial (Done _ r) = r
handlePartial (Fail {})  = []

-- | @parseUABlock@ Parse a user-agent and rules block
parseUABlock = do
    ua    <- parseUACol *> uA
    rulez <- many1 parseRules

    return RuleSet { userAgent = UserAgent ua,
                     rules = rulez }

-- | @matchUACol@ Parse the UA column and value taking into account
-- possible whitespace craziness
parseUACol = AC.skipSpace
          *> AC.stringCI "User-Agent"
          <* AC.skipSpace
          *> AC.char8 ':'
          *> AC.skipSpace

uA = do
    u <- takeWhile1 isNotEnd
    return (stripper u)

-- | @parseRules@ Parse the directives row
parseRules = (,) <$> parseTransLower
             <*> directiveRule

directiveRule = do
    rule <- takeWhile1 matchALL <* many1 AC.endOfLine

    return (stripper rule)

parseTransLower = do
    res <- parseDirectives <* AC.skipSpace
    return (lowercase res)

ctypeLower = listArray (0,255) (Prelude.map (BI.c2w . toLower) ['\0'..'\255']) :: UArray Word8 Word8
lowercase = B.map (\x -> ctypeLower!x)

directives = AC.stringCI "Disallow" <|> AC.stringCI "Allow"

-- | @parseDirectives@ Parse the directive column and any possibly
-- funny whitespace
parseDirectives = AC.skipSpace
                  *> directives -- <|> AC.stringCI "Crawl-delay" <|> AC.stringCI "Sitemap")
                  <* AC.skipSpace
                  <* AC.char8 ':'

ErikR ErikR · Accepted Answer · 2013-04-10T05:08:53

Consider this approach.

Define:

data RobotsDirective = RobotsDirective String String

This represents a parsed directive in a robots.txt file. The first string is the directive (i.e. UserAgent, Allow, Disallow, etc.) and the second string is the stuff after the colon.

Now write a parser for a RobotsDirective:

parseRD :: Parser RobotsDirective

parseRD will look for a directive name (which should only contain letters, digits and dashes and maybe underscores) followed by a colon followed by zero or more non-newline characters. Ignore white space as appropriate. If parseRD finds such a pattern it will create and return a RobotsDirective. Otherwise it will skip over one line of characters and try again.

Now that you have a parser for a RobotsDirective, you can create a parser for [RobotsDirective] in the standard way.

This parser simply skips over any line which doesn't look like a directive and this will include blank lines, comment lines and lines that begin with Don't allow.... However, it can return a RobotsDirective for lines which are not valid in a robots.txt file, i.e.:

foo: blah

will return RobotsDirective "foo" "blah". After you have parsed a robots.txt file and have gotten a list of RobotsDirective values, simply go through that list and ignore the ones you are not interested in.

Attoparsec /= version of stringCI

1 Answers