LWP::Simple & LWP::UserAgent doesn't return the complete webpage content

Question

I have an internal local machine, hosting analytics which I need to process. The analytics are returned in JSON format if you have the URL. for instance,

http://1.2.3.4:8081/analytics/mydata/myNodeData?flat

it would give the JSON file. Now I am getting the JSON file via LWP::Simple (also tried LWP::UserAgent) into a variable, and later parsing it. It works fine for most of my URLs. The problem is that with some of the URLs, it doesn't read the whole data into my variable, but only partial.

use LWP::Simple;                # From CPAN
use LWP::UserAgent;
use JSON qw( decode_json );     # From CPAN
use Data::Dumper;               # Perl core module
use Data::Diver qw{ Dive };         # for diving in the Hash
use strict;                     # Good practice
use warnings;                   # Good practice

#using LWP Simple    
my $trendsurl = 'http://1.2.3.4:8081/analytics/mydata/myNodeData?flat';
my $json = get( $trendsurl );
die "Could not get $trendsurl!" unless defined $json;
my $decoded_json = decode_json( $json );

#using LWP::UserAgent
my $ua = LWP::UserAgent->new();
my $req = new HTTP::Request GET => $trendsurl;
my $res = $ua->request($req);
my $content = $res->content;

In case its helpful here is the display of debug info of content (the decoded_json) is the same.

  DB<1> p $content
{"NodeStatus": {"deleted": false, "disk_usage_info": [{"partition_space_availabl
e_1k": 791475728, "partition_space_used_1k": 171611096, "partition_name": "/dev/
mapper/os-root", "partition_type": "ext4"}, {"partition_space_available_1k": 151
200, "partition_space_used_1k": 39244, "partition_name": "/dev/vda3", "partition
_type": "ext2"}], "process_info": [{"process_name": "XX-api:0", "process_s
tate": "PROCESS_STATE_RUNNING", "last_stop_time": null, "start_count": 2, "core_
file_list": [], "last_start_time": "1461822985334246", "stop_count": 0, "last_ex
it_time": null, "exit_count": 0}, {"process_name": "XX-config-nodemgr", "p
rocess_state": "PROCESS_STATE_RUNNING", "last_stop_time": null, "start_count": 2
, "core_file_list": [], "last_start_time": "1461822979324868", "stop_count": 0,
"last_exit_time": null, "exit_count": 0}, {"process_name": "XX-discovery:0
", "process_state": "PROCESS_STATE_RUNNING", "last_stop_time": null, "start_coun
t": 2, "core_file_list": [], "last_start_time": "1461822983332516", "stop_count"
: 0, "last_exit_time": null, "exit_count": 0}, {"process_name": "XX-svc-m

  DB<2>

You see the log file is abruptly terminated .. If I get this via my browser,its complete ..

EDIT:

I added the following eval statement while fetching URL and then decoding JSON .. the retreival is without error but JSON decoding fails because its not a complete JSON.

eval { $decoded_json = parse_json( $json ) };
if ($@) {
    warn "func raised an exception: $@";
}
func raised an exception: JSON error at line 1: Unexpected end of input parsing 
string starting from byte 1105 at Tester.pl line 28.

EDIT 2: I just tried another way ...

#!/usr/bin/perl  

 use LWP::UserAgent;  
 use HTTP::Request;  

 my $URL = 'http://1.2.3.4:8081/analytics/mydata/myNodeData?flat';  

 my $ua = LWP::UserAgent->new();  
 my $header = HTTP::Request->new(GET => $URL);  
 my $request = HTTP::Request->new('GET', $URL, $header);  
 my $response = $ua->request($request);  

 if ($response->is_success){  
     print "URL:$URL\nHeaders:\n";  
     print $response->headers_as_string;  
 }elsif ($response->is_error){  
     print "Error:$URL\n";  
     print $response->error_as_HTML;  
 }

and the debug shows that $response->is_success block is selected and here is the output.

URL:http://1.2.3.4:8081/analytics/mydata/myNodeData?flat
Headers:
Connection: close
Date: Tue, 10 May 2016 19:49:40 GMT
Content-Type: application/json
Client-Aborted: die
Client-Date: Tue, 10 May 2016 19:49:38 GMT
Client-Peer: 1.2.3.4:8081
Client-Response-Num: 1
Client-Transfer-Encoding: chunked
X-Died: read failed: An existing connection was forcibly closed by the remote ho
st. at C:/Perl64/lib/LWP/Protocol/http.pm line 465.

Since there's no eval and your decode_json didn't fail, didn't it actually download correctly and isn't it just debugging output that is truncated? — Oleg V. Volkov
Unless there's more to this than you have shown, this is a server bug. Your code using LWP is a little clunky but I can't see it causing the returned data to be truncated. Try this instead my $ua = LWP::UserAgent->new; my $res = $ua->get($trendsurl); print $res->status_line, "\n"; my $content = $res->decoded_content; — Borodin
its not truncated by debug .. thats the complete output .. And as far as server bug is concerned, I am able to get the complete data using the browser ... which is a legit json — Muzammil
What does the Content-Length header say? Does it match the length of the text? — Joe McMahon

Jose Bernalte Jose Bernalte · Accepted Answer · 2016-05-10T18:06:05

I use this code or something like that, this code use POST for params, i get a large JSON with this and its works well

use LWP::UserAgent;
use Data::Dumper;
use JSON;
use JSON::Parse;

my $params = 
{(
'flat' => 1,
)};

    my $server_endpoint = "http://1.2.3.4:8081/analytics/mydata/myNodeData";
    my $ua = LWP::UserAgent->new;
    my $req = HTTP::Request->new();
    my $response = $ua->post( $server_endpoint, $params );
    my $result  = $response->decoded_content();

    $result = from_json($result);
    print Dumper $result;

LWP::Simple & LWP::UserAgent doesn't return the complete webpage content

1 Answers