Timeout occurred in java httpurlconnection only when user-agent request header is present

0
votes

I need to read a website response and parse it, I am using java HttpsURLConnection class. Timeout error occurs when I add user-agent header . without this header I can read the response. curl command with user-agent header is also working in terminal. I want to know if its related to something, Kindly let me where to look.

My sample code

String stringUrl = "https://www.example.com/";
        try {
            URL url=new URL(stringUrl);
            HttpURLConnection con=(HttpURLConnection)url.openConnection();  
            con.setRequestMethod("GET");
            System.out.println("Started");
            con.setRequestProperty("Referer", stringUrl);
            con.setRequestProperty("user-agent", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36");
            con.setReadTimeout(5000);
            con.setConnectTimeout(5000);
            
            BufferedReader in = new BufferedReader(new InputStreamReader(con.getInputStream()));
            String inputLine;
            StringBuffer response = new StringBuffer();

            while ((inputLine = in.readLine()) != null) {
                response.append(inputLine);
            }
            in.close();

            // print result
            System.out.println(response.toString());
            
                con.disconnect();   
        } catch (Exception e) {
            e.printStackTrace();
        }

hiding url for security reasons.

Note: this works for other website, its not working for a particular website. Posting this question to also know whether we can prevent hitting our website programmatically

javacurlhttpurlconnection