REST API returning HTML garbage on Standard HTTPWebRequest POST

Question

I have an Application making a Web Request to get back a data set to display to Users live quotes. I checked the Web API and not much has changed, my web request seems to be formatted correctly, with the correct request body Payload but I am not getting back the data.

HttpWebRequest httpRequest = (HttpWebRequest)WebRequest.Create(SearchHelper.SearchUrl);
httpRequest.Method = "POST"; 
httpRequest.ContentType = "application/json;charset=utf-8"; 
httpRequest.CookieContainer = new CookieContainer(); 
httpRequest.BeginGetRequestStream(new AsyncCallback(GetWebRequestBody), httpRequest); 

Does anyone know what is going on? I can make a similar request via a browser and I still get back my properly formatted data for my live quotes, however through a .Net HttpWebRequest that has always worked before is now returning the below Content.

" Loading site please wait... html{height: 100%}body{margin: 0;height: 100%;}header{background-color: #020202;height: 64px;width: 100%;display: none;}header img{margin: 3px 0 0 2px;}section{height: calc(100% - 64px);}.content{padding: 0;position: relative;top: calc(50% - 64px);text-align: center;margin: auto;transform: translateY(-50%);}.left-img{display: inline-block;margin-right: 23px;}#JSCookieMSG .left-img, #JSOffMSG .left-img{display: inline-block;margin: 0 23px 0 0;}.right-msg{display: inline-block;color: #4b4b4b;text-align: left;}.msg-head{vertical-align: top;margin: 0;display: inline-block;color: #4b4b4b;font-size: 35px;line-height: 33px;font-weight: bold;font-family: 'Arial Narrow', Arial, sans-serif;padding-top: 46px;}#loading-content{padding: 10px;}#loading-content .left-img{margin-right: 10px;height: 10px;}#loading-content .left-img img{height: 8px;}#loading-content .msg-head{vertical-align: inherit;color: #4b4b4b;font-size: 17px;font-weight: normal;padding-top: 0;line-height: 22px;}#JSCookieMSG .msg-head, #JSOffMSG .msg-head{font-size: 35px;line-height: 33px;font-weight: bold;font-family: 'Arial Narrow', Arial, sans-serif;}.divider{display: inline-block;margin: 0 21px 0 20px;}.msg-head-cookies{margin: 0;}.msg-head-js{margin: 0 0 0 2px;}.msg-main{margin-top: 14px;padding: 0;font-size: 13px;line-height: 16px;font-family: Arial, 'Helvetica Neue', Helvetica, sans-serif;}.icon{margin: -4px 9px 0 -3px;}.incident-id{display: inline-block;font-size: 15px;vertical-align: top;font-family: Arial, 'Helvetica Neue', Helvetica, sans-serif;}@media(max-width: 768px){header img{margin-top: 3px;min-width: 400px;width: 100%;max-width: 561px;height: 65px;}.content{top: 200px;}}@media(max-width: 556px){.left-img{display: none;}.content{top: 200px;}}@media(max-height: 580px){.content{top: 200px;}} Loading site please wait... #loading-content{display: none;} Cookies disabled This website is using a security service to protect itself from online attacks.
The service requires full cookie support in order to view the website.
Please enable Cookies on your browser and try again.

Reference ID: ff0bdf6aeb56b1845c11e93b11faecc9 JavaScript Disabled This website is using a security service to protect itself from online attacks.
The service requires full JavaScript support in order to view the website.
Please enable Javascript on your browser and try again. Reference ID: ff0bdf6aeb56b1845c11e93b11faecc9 function sbbloadmid(){if(window.sbrmp){var smbMode=\"frm\";if(smbMode==\"frm\"){var bodyObj=document.getElementsByTagName(\"body\")[0];var submitFrm=document.createElement(\"form\");bodyObj.appendChild(submitFrm);submitFrm.id=\"sbmtfrm\";submitFrm.method=\"post\";submitFrm.action=\"\";submitFrm.sbbSbmt=submitFrm.submit;submitFrm.sbbSbmt();}else window.location.reload(true);}else{setTimeout(\"sbbloadmid()\", 50);}}var cookieenabled=false;if(navigator.cookieEnabled){if(navigator.cookieEnabled==true){var exdate=new Date();exdate.setDate(exdate.getDate()+1);document.cookie=\"sbtsck=jav;path=/;expires=\"+exdate.toGMTString()+\"\";cookieenabled=(document.cookie.indexOf(\"sbtsck\")!=-1)? true : false;}}if(cookieenabled){setTimeout(\"sbbloadmid()\",50);}else{var oJSCookieMSGObj=document.getElementById('JSCookieMSG');var loadingContent=document.getElementById('loading-content');var header=document.getElementsByTagName('header');oJSCookieMSGObj.style.display='block';header.style.display='block';loadingContent.style.display='none';} \n
\n sbbvscc='';\n
sbbgscc='';\n function genPid() {return String.fromCharCode(119)+String.fromCharCode(70) ; };\n
\n \n (function(XHR){var open=XHR.prototype.open;var send=XHR.prototype.send;var parser=document.createElement('a');XHR.prototype.open=function(method, url, async, user, pass){if(typeof async=='undefined'){async=true;}parser.href=url;if(parser.host==''){parser.href=parser.href;}this.ajax_hostname=parser.hostname;open.call(this, method, url, async, user, pass);};XHR.prototype.send=function(data){if(location.hostname==this.ajax_hostname)this.setRequestHeader(\"X-MOD-SBB-CTYPE\", \"xhr\");send.call(this, data);}})(XMLHttpRequest);function sbbgc(check_name){var start=document.cookie.indexOf(check_name+\"=\");var oVal='';var len=start+check_name.length+1;if((!start)&&(document.cookie.substring(0,check_name.length)!=check_name)){oVal='';}else if(start==-1){oVal='';}else{var end=document.cookie.indexOf(';',len);if(end==-1)end=document.cookie.length;var oVal=document.cookie.substring(len,end);};return oVal;}function addmg(inm,ext){var primgobj=document.createElement('IMG');primgobj.src=window.location.protocol+\"//\"+window.location.hostname+(window.location.port && window.location.port!=80 ? ':'+window.location.port: '')+\"/sbbi/?sbbpg=\"+inm+(ext?\"&\"+ext:\"\");var sbbDiv=document.getElementById('sbbfrcc');sbbDiv.appendChild(primgobj);};function addprid(prid){var oldVal=sbbgc(\"PRLST\");if((oldVal.indexOf(prid)==-1)&&(oldVal.split('/').length<5)){if(oldVal!='')oldVal+='/';document.cookie='PRLST='+oldVal+escape(prid)+';path=/';}}var sbbeccf=function(){this.sp3=\"jass\";this.sf1=function(vd){return sf2(vd)+32;};var sf2=function(avd){return avd*12;};this.sf4=function(yavd){return yavd+2;};var strrp=function(str, key, value){if(str.indexOf('&'+key+'=')> -1 || str.indexOf(key+'=')==0){var idx=str.indexOf('&'+key+'=');if(idx==-1)idx=str.indexOf(key+'=');var end=str.indexOf('&', idx+1);var newstr;if(end !=-1)newstr=str.substr(0, idx)+str.substr(end+(idx ? 0 : 1))+'&'+key+'='+value;else newstr=str.substr(0, idx)+'&'+key+'='+value;return newstr;}else return str+'&'+key+'='+value;};var strgt=function(name, text){if(typeof text !='string')return \"\";var nameEQ=name+\"=\";var ca=text.split(/[;&]/);for(var i=0;i < ca.length;i++){var c=ca[i];while(c.charAt(0)==' ')c=c.substring(1, c.length);if(c.indexOf(nameEQ)==0)return c.substring(nameEQ.length, c.length);}return \"\";};this.sfecud={f:function(name, value){var fv=\"\";try{var elm=document.getElementById('sbbfrcc');elm.style.behavior=\"url(#default#userData)\";if(typeof(value)!=\"undefined\"){elm.setAttribute(name, value);elm.save(name);}else{elm.load(name);fv=elm.getAttribute(name);}}catch(e){}return fv;}, name:\"sbbrf\"};this.sfecgs={sbbgh:function(){var domain=document.location.host;if(domain.indexOf('www.')==0)domain=domain.replace('www.', '');return domain;}, f:function(name, value){var fv=\"\";if(window.globalStorage){var host=this.sbbgh();try{if(typeof(value)!=\"undefined\")globalStorage[host][name]=value;else{fv=globalStorage[host][name];if(typeof(fv.toString)!=\"undefined\")fv=fv.toString();}}catch(e){}}return fv;}, name:\"sbbrf\"};this.sfecls={f:function(name, value){var fv=\"\";try{if(window.localStorage){if(typeof(value)!=\"undefined\")localStorage.setItem(name, value);else{fv=localStorage.getItem(name);if(typeof(fv.toString)!=\"undefined\")fv=fv.toString();}}}catch(e){}return fv;}, name:\"sbbrf\"};this.sbbcv=function(invl){try{var invalArr=invl.split(\"-\");if(invalArr.length>1){if(invalArr[0]==\"A\"||invalArr[0]==\"D\"){invl=invalArr[1];}else invl=\"\";}if(invl==null||typeof(invl)==\"undefined\"||invl==\"falseImgUT\"||invl==\"undefined\"||invl==\"null\"||invl!=encodeURI(invl))invl=\"\";if(typeof(invl).toLowerCase()==\"string\")if(invl.length>20)if(invl.substr(0,2)!=\"h4\")invl=\"\";}catch(ex){invl=\"\";}return invl;};this.sbbsv=function(fv){for(var elm in this){if(this[elm].name==\"sbbrf\"){this[elm].f(\"altutgv2\",fv);}}document.cookie=\"UTGv2=\"+fv+';expires=Tue, 31 Dec 2030 00:00:00 UTC;path=/';};this.sbbgv=function(){var valArr=Array();var currVal=\"\";for(var elm in this){if(this[elm].name==\"sbbrf\"){currVal=this[elm].f(\"altutgv2\");currVal=this.sbbcv(currVal);if(currVal!=\"\")valArr[currVal]=(typeof(valArr[currVal])!=\"undefined\"?valArr[currVal]+1:1);}}var lb=0;var fv=\"\";for(var val in valArr){if(valArr[val]>lb){fv=val;lb=valArr[val]}}if(fv==\"\")fv=sbbgc(\"UTGv2\");fv=this.sbbcv(fv);if(fv!=\"\")this.sbbsv(fv);else this.sbbsv(\"D-h470427f615e9a3c2e0e8cb93383234e8d86\");return fv;};};function m2vr(m1,m2){var i=0;var rc=\"\";var est=\"ghijklmnopqrstuvwyz\";var rnum;var rpl;var charm1=m1.charAt(i);var charm2=m2.charAt(i);while(charm1!=\"\"||charm2!=\"\"){rnum=Math.floor(Math.random()* est.length);rpl=est.substring(rnum,rnum+1);rc+=(charm1==\"\"?rpl:charm1)+(charm2==\"\"?rpl:charm2);i++;charm1=m1.charAt(i);charm2=m2.charAt(i);}return rc;}function sbbls(prid){try{var eut=sbbgc(\"UTGv2\");sbbeccfi=new sbbeccf();sbbgs=sbbeccfi.sbbgv();if(eut!=sbbgs && sbbgs!=\"\" && typeof(sbbfcr)==\"undefined\"){addmg('utMedia',\"vii=\"+m2vr(\"15e90d11d95bce15d7faa0030fd4a92d\",sbbgs));}var sbbiframeObj=document.createElement('IFRAME');var dfx=new Date();sbbiframeObj.id='SBBCrossIframe';sbbiframeObj.style.border='0px';if(document.all){sbbiframeObj.style.position='absolute';sbbiframeObj.style.top='-1px';sbbiframeObj.style.height='1px';sbbiframeObj.style.width='28px';}else{sbbiframeObj.style.height='1px';sbbiframeObj.style.width='0px';}sbbiframeObj.scrolling=\"NO\";sbbiframeObj.src=window.location.protocol+\"//\"+window.location.hostname+(window.location.port && window.location.port!=80 ? ':'+window.location.port: '')+'/sbbi/?sbbpg=sbbShell&gprid='+prid + '&sbbgs='+sbbgs+'&ddl='+(Math.round(dfx.getTime()/1000)-1490112137)+'';var sbbDiv=document.getElementById('sbbfrcc');sbbDiv.appendChild(sbbiframeObj);}catch(ex){;}}try{y=unescape(sbbvscc.replace(/^<\!\-\-\s*|\s*\-\->$/g,''));document.getElementById('sbbhscc').innerHTML=y;x=unescape(sbbgscc.replace(/^<\!\-\-\s*|\s*\-\->$/g,''));}catch(e){x='function genPid(){return \"jser\";}';}try{if(window.gprid==undefined)document.write('<'+'script type=\"text/javascri'+'pt\">'+x+\"var gprid=genPid();addprid(gprid);sbbls(gprid);<\"+\"/script>\");}catch(e){addprid(\"dwer\");} "

Answer 1

I think the answer is to provide a UserAgent when making the request.

var wc = new System.Net.WebClient();
wc.Headers.Add ("user-agent", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; .NET CLR 1.0.3705;)");
var result = wc.DownloadString(new Uri("url"));

Result now contains the expected JSON

---

OP has clarified that he uses HttpWebRequest. Existing code would remain the same, just add this line with an appropriate UserAgent string:

httpRequest.UserAgent = 
    "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; .NET CLR 1.0.3705;)";

Quick example (could be improved, just to prove a point):

var httpRequest = (System.Net.HttpWebRequest)System.Net.WebRequest.Create("url"); 
httpRequest.Method = "POST"; 
httpRequest.ContentType = "application/json; charset=utf-8"; 
httpRequest.UserAgent = "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; .NET CLR 1.0.3705;)";
var response = (System.Net.HttpWebResponse)httpRequest.GetResponse();
Stream receiveStream = response.GetResponseStream();
StreamReader readStream = new StreamReader(receiveStream);
var result = readStream.ReadToEnd();

Result is the same as above.