I am using spark metrics feature in my spark streaming application i am already adding two custom metric to spark metric system those are
- Incoming Events per second : using Spark Meter
- Number events successfully processed: using Spark counter
The above metrics are written to CSV file by spark metric system as per configuration is metrics.properties
Now my requirement is to add a json String at specified interval to spark metrics system.
The output i am expecting is CSV file which will have data some thing like below
1,jsonString1
2,jsonString2
or
jsonString1
jsonString2
Please suggest a way to do this i searched a lot but could not get the answer i am expecting
Thanks in advance!