logstash parse complex message from Telegram

Question

I'm processing through Telegram history (txt file) and I need to extract & process quite complex (nested) multiline pattern. Here's the whole pattern

Free_Trade_Calls__AltSignals:IOC/ BTC (bittrex)

BUY :  0.00164

SELL :

TARGET 1 : 0.00180
TARGET 2 : 0.00205
TARGET 3 : 0.00240

STOP LOS : 0.000120
2018-04-19 15:46:57 Free_Trade_Calls__AltSignals:TARGET

basically I am looking for a pattern starting with

Free_Trade_Calls__AltSignals: ^%(

and ending with a timestamp. Inside that pattern (telegram message)

 - exchange - in brackets in the 1st line 
 - extract value after BUY 
 - SELL values in a array of 3 SELL[3] : target 1-3
 - STOP loss value (it can be either STOP, STOP LOSS, STOP LOS)....

I've found this Logstash grok multiline message but I am very new to logstash firend advised it to me. I was trying to parse this text in NodeJS but it really is pain in the ass and mad about it.

Thanks Rob :)

Sufiyan Ghori Sufiyan Ghori · Accepted Answer · 2018-05-23T23:19:25

Since you need to grab values from each line, you don't need to use multi-line modifier. You can skip empty line with %{SPACE} character.

For your given log, this pattern can be used,

Free_Trade_Calls__AltSignals:.*\(%{WORD:exchange}\)\s*BUY\s*:\s*%{NUMBER:BUY}\s*SELL :\s*TARGET 1\s*:\s*%{NUMBER:TARGET_1}\s*TARGET 2\s*:\s*%{NUMBER:TARGET_2}\s*TARGET 3\s*:\s*%{NUMBER:TARGET_3}\s*.*:\s*%{NUMBER:StopLoss}

please note that \s* equals to %{SPACE} It will output,

{
  "exchange": [
    [
      "bittrex"
    ]
  ],
  "BUY": [
    [
      "0.00164"
    ]
  ],
  "BASE10NUM": [
    [
      "0.00164",
      "0.00180",
      "0.00205",
      "0.00240",
      "0.000120"
    ]
  ],
  "TARGET_1": [
    [
      "0.00180"
    ]
  ],
  "TARGET_2": [
    [
      "0.00205"
    ]
  ],
  "TARGET_3": [
    [
      "0.00240"
    ]
  ],
  "StopLoss": [
    [
      "0.000120"
    ]
  ]
}

logstash parse complex message from Telegram

1 Answers