Output limit when using API Requests to pull Splunk Data to HDFS

Question

I'm trying python requests to pull Splunk data into hdfs using an API Call. I don't know if this has anything to do with Splunk data itself or if it is a limitation of API Calls.

I am able to pull small amounts of data, but I tried to pull an hours worth of data and it only returned 100 records. In splunk, the same query returned 100K+ records.

Execute splunk query:

import os
import requests
import sys
import time
import xml.etree.ElementTree as ET

data = {
    'search': search
}

r = requests.post(ENDPOINT, 
                  data=data, 
                  verify=False, 
                  auth=(username, password))`enter code here`

response_xml_as_string = r.text
responseXml = ET.fromstring(response_xml_as_string)
sid= responseXml.find('sid')

Check to see if it is done. If you get 0, rerun this until you get a 1:

res = requests.get(ENDPOINT + '/{0}' .format(sid), 
               verify=False, 
               auth=(username, password))

root = ET.fromstring(res.text)
for child in root.iter():
    try:
        if child.attrib['name'] == 'isDone':
            is_done = child.text
    except:
        is_done=0
print(is_done)

Stream splunk data to hdfs:

data = {
  'output_mode': 'csv',
  'count': '5'
}


r = requests.get(ENDPOINT + '/{0}/results' .format(sid), 
                   data=data, 
                   verify=False, 
                   auth=(username, password))

os.system('echo "{0}" | hdfs dfs -put - {1}' .format(r.text,hdfs_path))

I'm not exactly sure what count:5 means in my last data dictionary. Can requests only pull in a certain number of records? The dataset is a very narrow (3 columns) and so I don't think that it is a MB issue. There could be, but that's not what is happening here. I have a much bigger query that I need to run later and so I'd appreciate any insight into size or record limits for API Requests. If I write this to a text file on linux instead of to hdfs I still only get 100 records so I don't think the streaming part is the bottle neck.

Simon Duff Simon Duff · Accepted Answer · 2018-10-12T01:36:10

The Splunk REST endpoint for getting results of a search can be found here

The count parameter is the maximum number of results to return. If value is set to 0, then all available results are returned, otherwise, the default of 100 is used.

Try your code again, this time with count set to 0

data = {
  'output_mode': 'csv',
  'count': '0'
}

Output limit when using API Requests to pull Splunk Data to HDFS

1 Answers