Haskell JSON parsing with Aeson

Question

I have a JSON data source which looks like this:

{ "fields": [
{ "type": "datetime",
  "name": "Observation Valid",
  "description": "Observation Valid Time"},
{ "type": "datetime",
  "name": "Observation Valid UTC",
  "description": "Observation Valid Time UTC"},
{ "type": "number",
  "name": "Air Temperature[F]",
  "description": "Air Temperature at 2m AGL"},
{ "type": "number",
  "name": "Wind Speed[kt]",
  "description": "Wind Speed"},
{ "type": "number",
  "name": "Wind Gust[kt]",
  "description": "Wind Gust"},
{ "type": "number", "name":
  "Wind Direction[deg]",
  "description": "Wind Direction"}
  ],
"rows": [
["2018-04-22T00:10:00", "2018-04-22T05:10:00Z", 50.0, 9.0, null, 50.0],
["2018-04-22T00:15:00", "2018-04-22T05:15:00Z", 50.0, 9.0, null, 60.0],
["2018-04-22T00:20:00", "2018-04-22T05:20:00Z", 50.0, 8.0, null, 60.0],
["2018-04-22T00:30:00", "2018-04-22T05:30:00Z", 50.0, 9.0, null, 60.0]
]
}

            ( https://mesonet.agron.iastate.edu/json/obhistory.py?station=TVK&network=AWOS&date=2018-04-22 )

And tried several data descriptions, lastly this:

data Entry =             -- Data entries
  Entry {  time      ::  Text     -- Observation Valid Time
         , timeUTC   ::  Text     -- Observation Valid Time UTC
         , airTemp   ::  Float    -- Air Temperature[F] at 2m AGL
         , wind      ::  Float    -- Wind Speed [kt]
         , gust      ::  Float    -- Wind Gust [kt]
         , direction ::  Int      -- Wind Direction[deg]
           } deriving (Show,Generic)

data Field =             -- Schema Definition
  Field {  ftype       :: String     -- 
         , name        :: String     -- 
         , description :: String    -- 
           } deriving (Show,Generic)

data Record =
  Record {  fields  :: [Field]     -- 
          , rows    :: [Entry]     -- data
           } deriving (Show,Generic)

-- Instances to convert our type to/from JSON.
instance FromJSON Entry
instance FromJSON Field
instance FromJSON Record

-- Get JSON data and decode it
dat <- (eitherDecode <$> getJSON) :: IO (Either String Record)

which gives this error: Error in $.fields[0]: key "ftype" not present

The (first) error comes from the field definitions (which I don’t use). In the JSON the Entry’s are arrays of mixed types, but in the Haskell it is just a data structure, not an array – not sure how to reconcile these.

No doubt a beginner error – but I haven’t found any examples which seem to have this structure. Do I need to write a custom parser for this?

Answer 1

Field has an ftype field, so AESON is trying to find ftype in the JSON but can't (as it contains ftype). I understand you can't name a field type in Haskell so you need to find a way make AESON use a different name. You need to use template Haskell and set fieldLabelModifier accordingly. Alternatively, writing the insistence manually might be simpler.

Answer 2

Three things prevent this from working as intended:

• The JSON data contains a field named "type" . A custom FromJson instance for the Field record type can handle this.
• The data in the Entry type is unnamed so it is better represented as either a data record without field names or a tuple.
• The Float representing wind gust is sometimes null so it should be a Maybe Float

The code below contains all of these modifications and parses your example JSON data :

{-# LANGUAGE DeriveGeneric #-}
{-# LANGUAGE OverloadedStrings #-}

import Data.ByteString.Lazy as BSL
import Data.Text (Text)
import Data.Aeson 
import GHC.Generics

-- Either this tuple definition of Entry or the data definition without
-- names (commented out) will work.  

type Entry =             -- Data entries
  ( Text     -- Observation Valid Time
  , Text     -- Observation Valid Time UTC
  , Float    -- Air Temperature[F] at 2m AGL
  , Float    -- Wind Speed [kt]
  , Maybe Float    -- Wind Gust [kt]
  , Int      -- Wind Direction[deg]
  ) 

-- data Entry =             -- Data entries
--   Entry Text     -- Observation Valid Time
--         Text     -- Observation Valid Time UTC
--         Float    -- Air Temperature[F] at 2m AGL
--         Float    -- Wind Speed [kt]
--         (Maybe Float)    -- Wind Gust [kt]
--         Int      -- Wind Direction[deg]
--         deriving (Show,Generic)

-- instance FromJSON Entry

data Field =             -- Schema Definition
  Field {  ftype       :: String    -- 
        ,  name        :: String    -- 
        ,  description :: String    -- 
        } deriving (Show,Generic)

instance FromJSON Field where
  parseJSON = withObject "Field" $ \v -> Field
    <$> v .: "type"
    <*> v .: "name"
    <*> v .: "description"

data Record =
  Record {  fields  :: [Field]     -- 
         ,  rows    :: [Entry]     -- data
         } deriving (Show,Generic)

instance FromJSON Record

getJSON :: IO ByteString
getJSON = BSL.readFile "json.txt" 

main :: IO()
main = do
  -- Get JSON data and decode it
  dat <- (eitherDecode <$> getJSON) :: IO (Either String Record)
  case dat of
    Right parsed -> print parsed
    Left err -> print err