1
votes

Hello - I'm trying to download a file using Apache commons fileUtils but it always ends up getting a 400 error. The file's URL is valid because I successfully downloaded it many times using the browser. Any ideas?

java.io.IOException: Server returned HTTP response code: 400 for URL: http://www.nikaia-hosp.gr/img/ΤΕΛΙΚΕΣ ΠΡΟΔΙΑΓΡΑΦΕΣ ΓΙΑ ΥΠΕΡΗΧΟ ΓΥΝΑΙΚΟΛΟΓΙΚΟ ΜΑΙΕΥΤΙΚΟ ΠΡΟΓΕΝΝΗΤΙΚΟΥ ΕΛΕΓΧΟΥ.pdf at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492) at java.net.URL.openStream(URL.java:1045) at org.apache.commons.io.FileUtils.copyURLToFile(FileUtils.java:1478) at com.nikaia.main.NikaiaReader.Downloader.download(Downloader.java:17) at com.nikaia.main.NikaiaReader.Downloader.main(Downloader.java:32)

import java.io.File;
import java.io.IOException;
import java.net.MalformedURLException;
import java.net.URL;

import org.apache.commons.io.FileUtils;

public class Downloader {

    public static void download(String url,String filename){

        //System.out.println("filename is : "+filename);
        try {
        //  FileUtils.copyURLToFile(new URL(url), new File("C:/downloads/"+filename));
                FileUtils.copyURLToFile(new URL(url), new File(PropertyReader.readProperty("ExtractedFilesPath")+"/"+filename));
            try {
                Thread.sleep(Integer.parseInt(PropertyReader.readProperty("downloadTimeout"))*1000);
            } catch (InterruptedException e) {
                // TODO Auto-generated catch block
                e.printStackTrace();
            }
        } catch (MalformedURLException e) {
            e.printStackTrace();
        } catch (IOException e) {
            e.printStackTrace();
        }
    }

    public static void main(String ar[]){
        download("http://www.nikaia-hosp.gr/img/ΤΕΛΙΚΕΣ ΠΡΟΔΙΑΓΡΑΦΕΣ ΓΙΑ ΥΠΕΡΗΧΟ ΓΥΝΑΙΚΟΛΟΓΙΚΟ ΜΑΙΕΥΤΙΚΟ ΠΡΟΓΕΝΝΗΤΙΚΟΥ ΕΛΕΓΧΟΥ.pdf","stupid.pdf");
    }
}
2
I tried it and now I get java.net.MalformedURLException : no protocol .. http%3A%2F%2Fwww.nikaia-hosp.gr%2Fimg%2F%C .... a very log string - chris_cs
download("nikaia-hosp.gr/img/"+URLEncoder.encode("ΤΕΛΙΚΕΣ ΠΡΟΔΙΑΓΡΑΦΕΣ ΓΙΑ ΥΠΕΡΗΧΟ ΓΥΝΑΙΚΟΛΟΓΙΚΟ ΜΑΙΕΥΤΙΚΟ ΠΡΟΓΕΝΝΗΤΙΚΟΥ ΕΛΕΓΧΟΥ.pdf"),"stupid.pdf"); - vts
still no protocol ... - chris_cs
I think the bad request error is due to the whitespace in your url. Browser is able to handle this. in code we need to handle this - vts

2 Answers

0
votes

OK answer found , I checked the encoded browser url and the url that UTF-8 java returns and the difference was that browser had %20 in the url but java had + . I replaced all + with %20 in java and its working.

import java.io.File;
import java.io.IOException;
import java.net.MalformedURLException;
import java.net.URL;
import java.net.URLEncoder;

import org.apache.commons.io.FileUtils;

public class Downloader {

    public static void download(String url, String filename) {

        try {

            String base = "http://www.nikaia-hosp.gr/img/";
            if (url.contains("http://www.nikaia-hosp.gr/img/")) {
                FileUtils.copyURLToFile(
                        new URL(base + URLEncoder.encode(url.replace(base, ""), "UTF-8").replaceAll("\\+", "%20")),
                        new File(PropertyReader.readProperty("ExtractedFilesPath") + "/" + filename));
            } else {
                FileUtils.copyURLToFile(new URL(url),
                        new File(PropertyReader.readProperty("ExtractedFilesPath") + "/" + filename));
            }

            try {
                Thread.sleep(Integer.parseInt(PropertyReader.readProperty("downloadTimeout")) * 1000);
            } catch (InterruptedException e) {
                // TODO Auto-generated catch block
                e.printStackTrace();
            }
        } catch (MalformedURLException e) {
            e.printStackTrace();
        } catch (IOException e) {
            e.printStackTrace();
        }
    }

    public static void main(String ar[]) {
        download(
                "http://www.nikaia-hosp.gr/img/ΤΕΛΙΚΕΣ ΠΡΟΔΙΑΓΡΑΦΕΣ ΓΙΑ ΥΠΕΡΗΧΟ ΓΥΝΑΙΚΟΛΟΓΙΚΟ ΜΑΙΕΥΤΙΚΟ ΠΡΟΓΕΝΝΗΤΙΚΟΥ ΕΛΕΓΧΟΥ.pdf",
                "stupid.pdf");
    }
}
0
votes

this work for me, the problem was the encoding, you need encode only the path of the url

InputStream in = new URL(url).openStream();
FileUtils.copyToFile(in,new File(filename));

first open a Stream with the url and then copy this stream data in a file. using copyToFile method

your code will be

public static void download(String url,String filename){

        try {
               //changed this 2 lines    
               URL encodeUrl = new URL(UriUtils.encodePath(url, "UTF-8"));
               InputStream in = encodeUrl.openStream(); 
               FileUtils.copyToFile(in, new File(PropertyReader.readProperty("ExtractedFilesPath")+"/"+filename));

            try {
                Thread.sleep(Integer.parseInt(PropertyReader.readProperty("downloadTimeout"))*1000);
            } catch (InterruptedException e) {
                // TODO Auto-generated catch block
                e.printStackTrace();
            }
        } catch (MalformedURLException e) {
            e.printStackTrace();
        } catch (IOException e) {
            e.printStackTrace();
        }
    }

public static void main(String ar[]){
        download("http://www.nikaia-hosp.gr/img/ΤΕΛΙΚΕΣ ΠΡΟΔΙΑΓΡΑΦΕΣ ΓΙΑ ΥΠΕΡΗΧΟ ΓΥΝΑΙΚΟΛΟΓΙΚΟ ΜΑΙΕΥΤΙΚΟ ΠΡΟΓΕΝΝΗΤΙΚΟΥ ΕΛΕΓΧΟΥ.pdf","stupid.pdf");
}

and add this dependency to your pom.xml

<dependency>
    <groupId>org.springframework</groupId>
    <artifactId>spring-web</artifactId>
    <version>3.0.4.RELEASE</version>
</dependency>

this does the magic.

UriUtils.encodePath(host+path, "UTF-8");