WSO2 CEP Sequence and external time

Question

I would like to improve my product and I would like to write a complex analyzer interface for it . I am using elasticsearch for storing logs, and more than 50 million logs are stored on daily basis . So the important thing would be that different queries could be created on the above mentioned interface which would significantly limit the number of logs to be analyzed. The wso2 cep component would get the logs to be tested then it would return back the result as json which would be displayed on my application. WSO2 cep library would be great if I could use the timestamp of logs instead of internal time ( System.currentTimeMillis() ) .

https://docs.wso2.com/display/CEP300/Windows

It would be nice if there were a way to use attribute as timestamp or use the timestamp parameter of send method .

public void send(long timeStamp, Object[] data) throws InterruptedException

In the meantime I found another problem : I need those events which followed by a specified events.

Example :

        query = " from every a1=LoginEvents[ip == '192.10.1.2']<2> -> " +
            "            b1=LoginEvents[ip == '192.10.1.1']<1> " +
            " within 10 sec " +
            " select a1[0].myTime, a1[0].ip, b1[0].ip as b1ip, b1[0].myTime as b2Time " +
            " insert into OutPut for all-events ";

    siddhiManager.addCallback(queryReference, new QueryCallback() {
        @Override
        public void receive(long timeStamp, Event[] inEvents, Event[] removeEvents) {
            if (inEvents != null) {
                for (Event e : inEvents) {
                    System.out.printf(" Ip1 : %s Time1 : %s Ip2: %s Time2 : %s \n", e.getData(1), new Date((Long) e.getData(0)).toString(), e.getData(2), new Date((Long) e.getData(3)).toString());
                }
            }
        }
    });


    InputHandler inputHandler = siddhiManager.getInputHandler("LoginEvents");
    Calendar c = Calendar.getInstance();
    c.add(Calendar.HOUR, 1);
    c.add(Calendar.SECOND, 1);
    int i = 0;
    String ip = null;
    for (i = 0; i <= 50; i++) {
        ip = "192.10.1.2";
        c.add(Calendar.SECOND, 2);
        if (i % 10 == 0) {
            ip = "192.10.1.1";
        }
        System.out.printf("Sending event : time %s , ip : %s \n", c.getTime().toString(), ip);
        inputHandler.send(c.getTime().getTime(), new Object[]{c.getTime().getTime(), ip, "1", new Random().nextInt(1000)});
    }

There are no results for the above query. Neverthless, it seems that it should be a hit because the following logs were sent to the siddhiManager. Output :

Sending event : time Wed Sep 23 11:06:24 CEST 2015 , ip : 192.10.1.2 
Sending event : time Wed Sep 23 11:06:26 CEST 2015 , ip : 192.10.1.2 
Sending event : time Wed Sep 23 11:06:28 CEST 2015 , ip : 192.10.1.2 
Sending event : time Wed Sep 23 11:06:30 CEST 2015 , ip : 192.10.1.2 
Sending event : time Wed Sep 23 11:06:32 CEST 2015 , ip : 192.10.1.2 
Sending event : time Wed Sep 23 11:06:34 CEST 2015 , ip : 192.10.1.2 
Sending event : time Wed Sep 23 11:06:36 CEST 2015 , ip : 192.10.1.2 
Sending event : time Wed Sep 23 11:06:38 CEST 2015 , ip : 192.10.1.2 
Sending event : time Wed Sep 23 11:06:40 CEST 2015 , ip : 192.10.1.2 
Sending event : time Wed Sep 23 11:06:42 CEST 2015 , ip : 192.10.1.1 
Sending event : time Wed Sep 23 11:06:44 CEST 2015 , ip : 192.10.1.2 
Sending event : time Wed Sep 23 11:06:46 CEST 2015 , ip : 192.10.1.2 
Sending event : time Wed Sep 23 11:06:48 CEST 2015 , ip : 192.10.1.2 
Sending event : time Wed Sep 23 11:06:50 CEST 2015 , ip : 192.10.1.2 
Sending event : time Wed Sep 23 11:06:52 CEST 2015 , ip : 192.10.1.2 
Sending event : time Wed Sep 23 11:06:54 CEST 2015 , ip : 192.10.1.2 
Sending event : time Wed Sep 23 11:06:56 CEST 2015 , ip : 192.10.1.2 
Sending event : time Wed Sep 23 11:06:58 CEST 2015 , ip : 192.10.1.2 
Sending event : time Wed Sep 23 11:07:00 CEST 2015 , ip : 192.10.1.2 
Sending event : time Wed Sep 23 11:07:02 CEST 2015 , ip : 192.10.1.1 
Sending event : time Wed Sep 23 11:07:04 CEST 2015 , ip : 192.10.1.2 
Sending event : time Wed Sep 23 11:07:06 CEST 2015 , ip : 192.10.1.2 
Sending event : time Wed Sep 23 11:07:08 CEST 2015 , ip : 192.10.1.2 
Sending event : time Wed Sep 23 11:07:10 CEST 2015 , ip : 192.10.1.2 
Sending event : time Wed Sep 23 11:07:12 CEST 2015 , ip : 192.10.1.2 
Sending event : time Wed Sep 23 11:07:14 CEST 2015 , ip : 192.10.1.2 
Sending event : time Wed Sep 23 11:07:16 CEST 2015 , ip : 192.10.1.2 
Sending event : time Wed Sep 23 11:07:18 CEST 2015 , ip : 192.10.1.2 
Sending event : time Wed Sep 23 11:07:20 CEST 2015 , ip : 192.10.1.2 
Sending event : time Wed Sep 23 11:07:22 CEST 2015 , ip : 192.10.1.1 
Sending event : time Wed Sep 23 11:07:24 CEST 2015 , ip : 192.10.1.2 
Sending event : time Wed Sep 23 11:07:26 CEST 2015 , ip : 192.10.1.2 
Sending event : time Wed Sep 23 11:07:28 CEST 2015 , ip : 192.10.1.2 
Sending event : time Wed Sep 23 11:07:30 CEST 2015 , ip : 192.10.1.2 
Sending event : time Wed Sep 23 11:07:32 CEST 2015 , ip : 192.10.1.2 
Sending event : time Wed Sep 23 11:07:34 CEST 2015 , ip : 192.10.1.2 
Sending event : time Wed Sep 23 11:07:36 CEST 2015 , ip : 192.10.1.2 
Sending event : time Wed Sep 23 11:07:38 CEST 2015 , ip : 192.10.1.2 
Sending event : time Wed Sep 23 11:07:40 CEST 2015 , ip : 192.10.1.2 
Sending event : time Wed Sep 23 11:07:42 CEST 2015 , ip : 192.10.1.1 
Sending event : time Wed Sep 23 11:07:44 CEST 2015 , ip : 192.10.1.2

The following events should have matched to the "query" but instaed gave me nothing. The folowing logs should have been the output :

Sending event : time Wed Sep 23 11:06:58 CEST 2015 , ip : 192.10.1.2 
Sending event : time Wed Sep 23 11:07:00 CEST 2015 , ip : 192.10.1.2 
Sending event : time Wed Sep 23 11:07:02 CEST 2015 , ip : 192.10.1.1

Furthermore, if the query type is sequence I want only one pair as a result. Suppose that the following logs will be sent to the queue:

    Sending event : time Mon Sep 28 14:49:01 CEST 2015 , ip : 192.10.1.1 
Sending event : time Mon Sep 28 14:49:03 CEST 2015 , ip : 192.10.1.2 
Sending event : time Mon Sep 28 14:49:05 CEST 2015 , ip : 192.10.1.2 
Sending event : time Mon Sep 28 14:49:07 CEST 2015 , ip : 192.10.1.2 
Sending event : time Mon Sep 28 14:49:09 CEST 2015 , ip : 192.10.1.2 
Sending event : time Mon Sep 28 14:49:11 CEST 2015 , ip : 192.10.1.2 
Sending event : time Mon Sep 28 14:49:13 CEST 2015 , ip : 192.10.1.2 
Sending event : time Mon Sep 28 14:49:15 CEST 2015 , ip : 192.10.1.2 
Sending event : time Mon Sep 28 14:49:17 CEST 2015 , ip : 192.10.1.2 
Sending event : time Mon Sep 28 14:49:19 CEST 2015 , ip : 192.10.1.2 
Sending event : time Mon Sep 28 14:49:21 CEST 2015 , ip : 192.10.1.1 
Sending event : time Mon Sep 28 14:49:23 CEST 2015 , ip : 192.10.1.2 
Sending event : time Mon Sep 28 14:49:25 CEST 2015 , ip : 192.10.1.2 
Sending event : time Mon Sep 28 14:49:27 CEST 2015 , ip : 192.10.1.2 
Sending event : time Mon Sep 28 14:49:29 CEST 2015 , ip : 192.10.1.2 
Sending event : time Mon Sep 28 14:49:31 CEST 2015 , ip : 192.10.1.2 
Sending event : time Mon Sep 28 14:49:33 CEST 2015 , ip : 192.10.1.2 
Sending event : time Mon Sep 28 14:49:35 CEST 2015 , ip : 192.10.1.2 
Sending event : time Mon Sep 28 14:49:37 CEST 2015 , ip : 192.10.1.2 
Sending event : time Mon Sep 28 14:49:39 CEST 2015 , ip : 192.10.1.2 
Sending event : time Mon Sep 28 14:49:41 CEST 2015 , ip : 192.10.1.1 
Sending event : time Mon Sep 28 14:49:43 CEST 2015 , ip : 192.10.1.2 
Sending event : time Mon Sep 28 14:49:45 CEST 2015 , ip : 192.10.1.2 
Sending event : time Mon Sep 28 14:49:47 CEST 2015 , ip : 192.10.1.2 
Sending event : time Mon Sep 28 14:49:49 CEST 2015 , ip : 192.10.1.2 
Sending event : time Mon Sep 28 14:49:51 CEST 2015 , ip : 192.10.1.2 
Sending event : time Mon Sep 28 14:49:53 CEST 2015 , ip : 192.10.1.2 
Sending event : time Mon Sep 28 14:49:55 CEST 2015 , ip : 192.10.1.2 
Sending event : time Mon Sep 28 14:49:57 CEST 2015 , ip : 192.10.1.2 
Sending event : time Mon Sep 28 14:49:59 CEST 2015 , ip : 192.10.1.2 
Sending event : time Mon Sep 28 14:50:01 CEST 2015 , ip : 192.10.1.1 
Sending event : time Mon Sep 28 14:50:03 CEST 2015 , ip : 192.10.1.2 
Sending event : time Mon Sep 28 14:50:05 CEST 2015 , ip : 192.10.1.2 
Sending event : time Mon Sep 28 14:50:07 CEST 2015 , ip : 192.10.1.2 
Sending event : time Mon Sep 28 14:50:09 CEST 2015 , ip : 192.10.1.2 
Sending event : time Mon Sep 28 14:50:11 CEST 2015 , ip : 192.10.1.2 
Sending event : time Mon Sep 28 14:50:13 CEST 2015 , ip : 192.10.1.2 
Sending event : time Mon Sep 28 14:50:15 CEST 2015 , ip : 192.10.1.2 
Sending event : time Mon Sep 28 14:50:17 CEST 2015 , ip : 192.10.1.2 
Sending event : time Mon Sep 28 14:50:19 CEST 2015 , ip : 192.10.1.2 
Sending event : time Mon Sep 28 14:50:21 CEST 2015 , ip : 192.10.1.1 
Sending event : time Mon Sep 28 14:50:23 CEST 2015 , ip : 192.10.1.2 
Sending event : time Mon Sep 28 14:50:25 CEST 2015 , ip : 192.10.1.2 
Sending event : time Mon Sep 28 14:50:27 CEST 2015 , ip : 192.10.1.2 
Sending event : time Mon Sep 28 14:50:29 CEST 2015 , ip : 192.10.1.2 
Sending event : time Mon Sep 28 14:50:31 CEST 2015 , ip : 192.10.1.2 
Sending event : time Mon Sep 28 14:50:33 CEST 2015 , ip : 192.10.1.2 
Sending event : time Mon Sep 28 14:50:35 CEST 2015 , ip : 192.10.1.2 
Sending event : time Mon Sep 28 14:50:37 CEST 2015 , ip : 192.10.1.2 
Sending event : time Mon Sep 28 14:50:39 CEST 2015 , ip : 192.10.1.2 
Sending event : time Mon Sep 28 14:50:41 CEST 2015 , ip : 192.10.1.1

I need only those events which are followed by a specified event. I just need those logs which are followed by log with ip 192.10.1.1 (if there was an event that was analyzed then it should not to be analyzed again ) . So it should have been 6-7 hit as a result but siddhi gave me all possible hits.

For example :

     query = " from  a1=LoginEvents[ip == '192.10.1.2']+, " +
        "            b1=LoginEvents[ip == '192.10.1.2']+," +
        "            c1=LoginEvents[ip == '192.10.1.1']  " +
        " within 10 sec " +
        " select a1[0].myTime, a1[0].ip, b1[0].ip as b1ip, b1[0].myTime as b2Time " +
        " insert into OutPut for current-events ";
;

and this is true for patterns too

   query = " from every a1=LoginEvents[ip == '192.10.1.2'] -> " +
                "            b1=LoginEvents[ip=='192.10.1.1'] " +
                " within 10 sec " +
                " select a1[0].myTime, a1[0].ip, b1[0].ip as b1ip, b1[0].myTime as b2Time " +
                " insert into OutPut for all-events ";
        ;

The output is the following :

Sending event : time Mon Sep 28 15:13:21 CEST 2015 , ip : 192.10.1.1 
log4j:WARN No appenders could be found for logger (org.wso2.siddhi.core.query.processor.handler.sequence.SequenceInnerHandlerProcessor).
log4j:WARN Please initialize the log4j system properly.
Sending event : time Mon Sep 28 15:13:23 CEST 2015 , ip : 192.10.1.2 
Sending event : time Mon Sep 28 15:13:25 CEST 2015 , ip : 192.10.1.2 
Sending event : time Mon Sep 28 15:13:27 CEST 2015 , ip : 192.10.1.2 
Sending event : time Mon Sep 28 15:13:29 CEST 2015 , ip : 192.10.1.2 
Sending event : time Mon Sep 28 15:13:31 CEST 2015 , ip : 192.10.1.2 
Sending event : time Mon Sep 28 15:13:33 CEST 2015 , ip : 192.10.1.2 
Sending event : time Mon Sep 28 15:13:35 CEST 2015 , ip : 192.10.1.2 
Sending event : time Mon Sep 28 15:13:37 CEST 2015 , ip : 192.10.1.2 
Sending event : time Mon Sep 28 15:13:39 CEST 2015 , ip : 192.10.1.2 
Sending event : time Mon Sep 28 15:13:41 CEST 2015 , ip : 192.10.1.1 
--------
event
 Ip1 : 192.10.1.2 Time1 : Mon Sep 28 15:13:23 CEST 2015 Ip2: 192.10.1.2 Time2 : Mon Sep 28 15:13:25 CEST 2015 
--------
--------
event
 Ip1 : 192.10.1.2 Time1 : Mon Sep 28 15:13:23 CEST 2015 Ip2: 192.10.1.2 Time2 : Mon Sep 28 15:13:27 CEST 2015 
--------
--------
event
 Ip1 : 192.10.1.2 Time1 : Mon Sep 28 15:13:25 CEST 2015 Ip2: 192.10.1.2 Time2 : Mon Sep 28 15:13:27 CEST 2015 
--------
--------
event
 Ip1 : 192.10.1.2 Time1 : Mon Sep 28 15:13:23 CEST 2015 Ip2: 192.10.1.2 Time2 : Mon Sep 28 15:13:29 CEST 2015 
--------
--------
event
 Ip1 : 192.10.1.2 Time1 : Mon Sep 28 15:13:25 CEST 2015 Ip2: 192.10.1.2 Time2 : Mon Sep 28 15:13:29 CEST 2015 
--------
--------
event
 Ip1 : 192.10.1.2 Time1 : Mon Sep 28 15:13:27 CEST 2015 Ip2: 192.10.1.2 Time2 : Mon Sep 28 15:13:29 CEST 2015 
--------
--------
event
 Ip1 : 192.10.1.2 Time1 : Mon Sep 28 15:13:23 CEST 2015 Ip2: 192.10.1.2 Time2 : Mon Sep 28 15:13:31 CEST 2015 
--------
--------
event
 Ip1 : 192.10.1.2 Time1 : Mon Sep 28 15:13:25 CEST 2015 Ip2: 192.10.1.2 Time2 : Mon Sep 28 15:13:31 CEST 2015 
--------
--------
event
 Ip1 : 192.10.1.2 Time1 : Mon Sep 28 15:13:27 CEST 2015 Ip2: 192.10.1.2 Time2 : Mon Sep 28 15:13:31 CEST 2015 
--------
--------
event
 Ip1 : 192.10.1.2 Time1 : Mon Sep 28 15:13:29 CEST 2015 Ip2: 192.10.1.2 Time2 : Mon Sep 28 15:13:31 CEST 2015 
--------
--------
event
 Ip1 : 192.10.1.2 Time1 : Mon Sep 28 15:13:23 CEST 2015 Ip2: 192.10.1.2 Time2 : Mon Sep 28 15:13:33 CEST 2015 
--------
--------
event
 Ip1 : 192.10.1.2 Time1 : Mon Sep 28 15:13:25 CEST 2015 Ip2: 192.10.1.2 Time2 : Mon Sep 28 15:13:33 CEST 2015 
--------
--------
event
 Ip1 : 192.10.1.2 Time1 : Mon Sep 28 15:13:27 CEST 2015 Ip2: 192.10.1.2 Time2 : Mon Sep 28 15:13:33 CEST 2015 
--------
--------
event
 Ip1 : 192.10.1.2 Time1 : Mon Sep 28 15:13:29 CEST 2015 Ip2: 192.10.1.2 Time2 : Mon Sep 28 15:13:33 CEST 2015 
--------
--------
event
 Ip1 : 192.10.1.2 Time1 : Mon Sep 28 15:13:31 CEST 2015 Ip2: 192.10.1.2 Time2 : Mon Sep 28 15:13:33 CEST 2015 
--------
--------
event
 Ip1 : 192.10.1.2 Time1 : Mon Sep 28 15:13:23 CEST 2015 Ip2: 192.10.1.2 Time2 : Mon Sep 28 15:13:35 CEST 2015 
--------
--------
event
 Ip1 : 192.10.1.2 Time1 : Mon Sep 28 15:13:25 CEST 2015 Ip2: 192.10.1.2 Time2 : Mon Sep 28 15:13:35 CEST 2015 
--------
--------
event
 Ip1 : 192.10.1.2 Time1 : Mon Sep 28 15:13:27 CEST 2015 Ip2: 192.10.1.2 Time2 : Mon Sep 28 15:13:35 CEST 2015 
--------
--------
event
 Ip1 : 192.10.1.2 Time1 : Mon Sep 28 15:13:29 CEST 2015 Ip2: 192.10.1.2 Time2 : Mon Sep 28 15:13:35 CEST 2015 
--------
--------
event
 Ip1 : 192.10.1.2 Time1 : Mon Sep 28 15:13:31 CEST 2015 Ip2: 192.10.1.2 Time2 : Mon Sep 28 15:13:35 CEST 2015 
--------
--------
event
 Ip1 : 192.10.1.2 Time1 : Mon Sep 28 15:13:33 CEST 2015 Ip2: 192.10.1.2 Time2 : Mon Sep 28 15:13:35 CEST 2015 
--------
--------
event
 Ip1 : 192.10.1.2 Time1 : Mon Sep 28 15:13:23 CEST 2015 Ip2: 192.10.1.2 Time2 : Mon Sep 28 15:13:37 CEST 2015 
--------
--------
event
 Ip1 : 192.10.1.2 Time1 : Mon Sep 28 15:13:25 CEST 2015 Ip2: 192.10.1.2 Time2 : Mon Sep 28 15:13:37 CEST 2015 
--------
--------
event
 Ip1 : 192.10.1.2 Time1 : Mon Sep 28 15:13:27 CEST 2015 Ip2: 192.10.1.2 Time2 : Mon Sep 28 15:13:37 CEST 2015 
--------
--------
event
 Ip1 : 192.10.1.2 Time1 : Mon Sep 28 15:13:29 CEST 2015 Ip2: 192.10.1.2 Time2 : Mon Sep 28 15:13:37 CEST 2015 
--------
--------
event
 Ip1 : 192.10.1.2 Time1 : Mon Sep 28 15:13:31 CEST 2015 Ip2: 192.10.1.2 Time2 : Mon Sep 28 15:13:37 CEST 2015 
--------
--------
event
 Ip1 : 192.10.1.2 Time1 : Mon Sep 28 15:13:33 CEST 2015 Ip2: 192.10.1.2 Time2 : Mon Sep 28 15:13:37 CEST 2015 
--------
--------
event
 Ip1 : 192.10.1.2 Time1 : Mon Sep 28 15:13:35 CEST 2015 Ip2: 192.10.1.2 Time2 : Mon Sep 28 15:13:37 CEST 2015 
--------
--------
event
 Ip1 : 192.10.1.2 Time1 : Mon Sep 28 15:13:23 CEST 2015 Ip2: 192.10.1.2 Time2 : Mon Sep 28 15:13:39 CEST 2015 
--------
--------
event
 Ip1 : 192.10.1.2 Time1 : Mon Sep 28 15:13:25 CEST 2015 Ip2: 192.10.1.2 Time2 : Mon Sep 28 15:13:39 CEST 2015 
--------
--------
event
 Ip1 : 192.10.1.2 Time1 : Mon Sep 28 15:13:27 CEST 2015 Ip2: 192.10.1.2 Time2 : Mon Sep 28 15:13:39 CEST 2015 
--------
--------
event
 Ip1 : 192.10.1.2 Time1 : Mon Sep 28 15:13:29 CEST 2015 Ip2: 192.10.1.2 Time2 : Mon Sep 28 15:13:39 CEST 2015 
--------
--------
event
 Ip1 : 192.10.1.2 Time1 : Mon Sep 28 15:13:31 CEST 2015 Ip2: 192.10.1.2 Time2 : Mon Sep 28 15:13:39 CEST 2015 
--------
--------
event
 Ip1 : 192.10.1.2 Time1 : Mon Sep 28 15:13:33 CEST 2015 Ip2: 192.10.1.2 Time2 : Mon Sep 28 15:13:39 CEST 2015 
--------
--------
event
 Ip1 : 192.10.1.2 Time1 : Mon Sep 28 15:13:35 CEST 2015 Ip2: 192.10.1.2 Time2 : Mon Sep 28 15:13:39 CEST 2015 
--------
--------
event
 Ip1 : 192.10.1.2 Time1 : Mon Sep 28 15:13:37 CEST 2015 Ip2: 192.10.1.2 Time2 : Mon Sep 28 15:13:39 CEST 2015 
-

Tharik Kanaka Tharik Kanaka · Accepted Answer · 2015-09-29T10:29:13

CEP 3.0.0 is the old version which came with Siddhi 2.0.0. I would recommend you to use latest WSO2 CEP 4.0.0 which has significant improvements with Siddhi 3.0.0 which has been re written. You can download latest CEP pack from here. On the other hand Siddhi repository is available in GitHub.

In Siddhi 3.0.0 you can use an external time window to specify attribute name as below.

from LoginEvents#window.externalTime(timeStamp,5 sec) 
select timeStamp, ip
insert all events into uniqueIps

Please refer WSO2 documentation on Time Windows for more details. For regarding Pattern issue i think its better you check with Siddhi 3.0 Within Test Case Syntax and implement your logic.

WSO2 CEP Sequence and external time

1 Answers