Sending CSV to influxDB

Question

I have a decent PowerShell script for getting event logs and exporting them to CSV, for which I intend to then parse to influxDB and I have tried several means. of which is:

Streamwriting to Telegraf, via PowerShell. Would send packages but I don't think anything was in the packages, because of some text encoding way over my head.
POST queries via the API.
Third party programs (though they were Syslog focused, and also to no help).

I have now found a Python script on Github that will send a CSV file to my Influx server, which would be perfect and seem like it would work, other than the fact that it gives me error about the timestamps. And while this Python script does indeed connect and can create the database, there seems to be a fault with the timestamp from the eventlog and even if I manually change it to the default timestamp, within the CSV file itself, it still complains with:

python.exe : Traceback (most recent call last):
At line:1 char:1
+ python.exe .\csv-to-influx\csv-to-influxdb.py `
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (Traceback (most recent call last)::String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError

  File ".\csv-to-influx\csv-to-influxdb.py", line 175, in 
    args.timezone)
  File ".\csv-to-influx\csv-to-influxdb.py", line 68, in loadCsv
    for row in reader:
  File "C:\Users\alexander.hansen\AppData\Local\Programs\Python\Python37-32\lib\csv.py", line 111, in __next__
    self.fieldnames
  File "C:\Users\alexander.hansen\AppData\Local\Programs\Python\Python37-32\lib\csv.py", line 98, in fieldnames
    self._fieldnames = next(self.reader)
_csv.Error: line contains NULL byte

This comes up when trying to insert this 1 line of CSV with the Py script:

EntryType,"TimeGenerated","Source","EventID","MachineName","Message"
Warning,"2019-07-03 13:27:03","Group Policy Local Users and Groups","4098","Hostname","The computer 'admin' preference item in the ---' Group Policy Object did not apply because it failed with error code '0x8007052a This operation is disallowed as it could result in an administration account being disabled, deleted or unable to logon.' This error was suppressed."

I would love to be able to actually understand what the problem is, but I am just not sure. The CSV seems to be as it should be?

The documentation uses this example python csv-to-influxdb.py --dbname test --input data.csv --fieldcolumns value. Try specifying the parameters input, dbname — Lucas Wieloch

Ken Ken · Accepted Answer · 2019-07-19T07:14:50

I found an enhancement export csv to influx. https://github.com/Bugazelle/export-csv-to-influx

Install by: pip install ExportCsvToInflux

export_csv_to_influx \
--csv test.csv \
--dbname test \
--measurement sample \
--tag_columns EntryType,Source,MachineName \
--field_columns EntryType,Source,EventID,MachineName,Message \
--time_column TimeGenerated \
--user admin \
--password admin \
--server 127.0.0.1:8086

Here is the influx data:

> use test
Using database test
> select * from sample
name: sample
time                EntryType EntryType_1 EventID MachineName MachineName_1 Message                                                                                                                                                                                                                                                                               Source                              Source_1
----                --------- ----------- ------- ----------- ------------- -------                                                                                                                                                                                                                                                                               ------                              --------
1562160423000000000 Warning   Warning     4098    Hostname    Hostname      The computer 'admin' preference item in the ---' Group Policy Object did not apply because it failed with error code '0x8007052a This operation is disallowed as it could result in an administration account being disabled, deleted or unable to logon.' This error was suppressed. Group Policy Local Users and Groups Group Policy Local Users and Groups
>

Sending CSV to influxDB

1 Answers