Override @timestamp to get correct correct %{+yyyy.MM.dd} in index name

Question

I'm let Filebeat reading line-by-line json files, in each json event, I already have timestamp field (format: 2021-03-02T04:08:35.241632)

After processing, there is a new field @timestamp (might meta field Filebeat added, equals to current time), and seems index pattern %{+yyyy.MM.dd} (https://www.elastic.co/guide/en/beats/filebeat/current/elasticsearch-output.html#index-option-es) was configured to that field. I want to override @timestamp with timestamp processor: https://www.elastic.co/guide/en/beats/filebeat/current/processor-timestamp.html but not work, might be the layout was not set correctly?

- timestamp:
      field: timestamp
      layouts:
        - '2021-03-02T03:29:29.787331'

2021-03-07T11:29:39.382+0700    DEBUG   [processor.timestamp]   timestamp/timestamp.go:173  Failure parsing time field. {"error": "failed parsing time field timestamp='2021-03-02T03:29:29.787331'", "errorCauses": [{"error": "failed using layout [2021-03-02T03:29:29.787331] cannot parse [-03-02T03:29:29.787331] as [1]"}]}

I also tried another approach to parse timestamp using Date.parse but not work, not sure if ECMA 5.1 implemented in Filebeat missing something:

var date = new Date.parse('2021-03-02T03:29:29.787331');
    event.Put('@metadata.index_suffix', date.getFullYear() + '-' + date.getMonth() + '-' + date.getDate());

  "error": {
    "message": "TypeError: Not a constructor at process (/Users/thoong/Projects/shodan/shodan-trends/transform.js:25:16(106))"
  },

So with my timestamp format is 2021-03-02T03:29:29.787331, I want to ask what is the correct layouts for the processor or to parse with Date.parse?

Solution, here is the correct layouts from @Val answer

- timestamp:
      field: timestamp
      layouts:
        - '2006-01-02T15:04:05.000000'

Val Val · Accepted Answer · 2021-03-07T06:17:23

You don't need to specify the layouts parameter if your timestamp field already has the ISO8601 format.

However, if your timestamp field has a different layout, you must specify a very specific reference date inside the layout section, which is Mon Jan 2 15:04:05 MST 2006 and you can also provide a test date. (more info)

So in your case it would be:

- timestamp:
    field: timestamp
    ignore_missing: true
    ignore_failure: true
    layouts:
      - '2006-01-02T15:04:05.000000'
    test:
      - '2021-03-02T03:29:29.787331'

Override @timestamp to get correct correct %{+yyyy.MM.dd} in index name

Solution, here is the correct layouts from @Val answer

1 Answers