PHP Curl return 404 from aws but works from other host and even my machine

Question

Been banging my head over this for awhile. I have written php code to search the website https:paytm.com which works from the other hosting site I have been with earlier but not from AWS.

My machine is running ubuntu with php7, curl is installed and running. the search using curl works for other sites, curl https also works. the same code works brilliantly on the other host (via ssh as well as frontend), it even works from the browser.

The url I am trying to access is curl -v 'https://search.paytm.com/search/?page_count=1&items_per_page=10&quality=high&curated=1&cat_tree=1&from=organic&channel=web&version=2&userQuery=iphone'

Things I have checked:

1. The certificates are up to date
2. The code works fine on my machine and other hosting provider
3. AWS setup is ubuntu 12.04, running php
4. Setting up the machine again
5. Running the script from browser
6. Checking https curl on google, twitter, linkedIn from my aws setup, they work fine.

The resulting response coming from my AWS setup is:

    ubuntu@ip-172-31-20-200:/usr/local/share$ curl -v https://search.paytm.com/search/?page_count=1&items_per_page=10&quality=high&curated=1&cat_tree=1&from=organic&channel=web&version=2&userQuery=iphone
    [1] 12595
    [2] 12596
    [3] 12597
    [4] 12598
    [5] 12599
    [6] 12600
    [7] 12601
    [8] 12602
    ubuntu@ip-172-31-20-200:/usr/local/share$ * Hostname was NOT found in DNS cache
    *   Trying 96.6.72.42...
    * Connected to search.paytm.com (96.6.72.42) port 443 (#0)
    * successfully set certificate verify locations:
    *   CAfile: none
      CApath: /etc/ssl/certs
    * SSLv3, TLS handshake, Client hello (1):
    * SSLv3, TLS handshake, Server hello (2):
    * SSLv3, TLS handshake, CERT (11):
    * SSLv3, TLS handshake, Server key exchange (12):
    * SSLv3, TLS handshake, Server finished (14):
    * SSLv3, TLS handshake, Client key exchange (16):
    * SSLv3, TLS change cipher, Client hello (1):
    * SSLv3, TLS handshake, Finished (20):
    * SSLv3, TLS change cipher, Client hello (1):
    * SSLv3, TLS handshake, Finished (20):
    * SSL connection using ECDHE-RSA-AES256-GCM-SHA384
    * Server certificate:
    *    subject: C=IN; ST=Uttar Pradesh; L=Noida; O=One 97 Communications Limited; CN=secure.paytm.in
    *    start date: 2015-10-29 00:00:00 GMT
    *    expire date: 2016-10-28 23:59:59 GMT
    *    subjectAltName: search.paytm.com matched
    *    issuer: C=US; O=GeoTrust Inc.; CN=GeoTrust SSL CA - G3
    *    SSL certificate verify ok.
    > GET /search/?page_count=1 HTTP/1.1
    > User-Agent: curl/7.35.0
    > Host: search.paytm.com
    > Accept: */*
    > 
    
    404 Not Found
    
    
404 Not Found
    
nginx
    
    
    * Connection #0 to host search.paytm.com left intact

The result coming from the different hosting provider (and from my own machine is):

    [ps527167]$ curl -v https://search.paytm.com/search/?page_count=1&items_per_page=10&quality=high&curated=1&cat_tree=1&from=organic&channel=web&version=2&userQuery=iphone
    [1] 26241
    [2] 26242
    [3] 26243
    [4] 26244
    [5] 26245
    [6] 26246
    [7] 26247
    [8] 26248
    [2]   Done                    items_per_page=10
    [3]   Done                    quality=high
    [4]   Done                    curated=1
    [5]   Done                    cat_tree=1
    [6]   Done                    from=organic
    [7]-  Done                    channel=web
    [ps527167]$ * Hostname was NOT found in DNS cache
    *   Trying 23.218.97.132...
    * Connected to search.paytm.com (23.218.97.132) port 443 (#0)
    * successfully set certificate verify locations:
    *   CAfile: none
      CApath: /etc/ssl/certs
    * SSLv3, TLS handshake, Client hello (1):
    * SSLv3, TLS handshake, Server hello (2):
    * SSLv3, TLS handshake, CERT (11):
    * SSLv3, TLS handshake, Server key exchange (12):
    * SSLv3, TLS handshake, Server finished (14):
    * SSLv3, TLS handshake, Client key exchange (16):
    * SSLv3, TLS change cipher, Client hello (1):
    * SSLv3, TLS handshake, Finished (20):
    * SSLv3, TLS change cipher, Client hello (1):
    * SSLv3, TLS handshake, Finished (20):
    * SSL connection using ECDHE-RSA-AES256-GCM-SHA384
    * Server certificate:
    *    subject: C=IN; ST=Uttar Pradesh; L=Noida; O=One 97 Communications Limited; CN=secure.paytm.in
    *    start date: 2015-10-29 00:00:00 GMT
    *    expire date: 2016-10-28 23:59:59 GMT
    *    subjectAltName: search.paytm.com matched
    *    issuer: C=US; O=GeoTrust Inc.; CN=GeoTrust SSL CA - G3
    *    SSL certificate verify ok.
    > GET /search/?page_count=1 HTTP/1.1
    > User-Agent: curl/7.35.0
    > Host: search.paytm.com
    > Accept: */*

     HTTP/1.1 200 OK
     Content-Type: application/json; charset=utf-8
     Server openresty is not blacklisted
     Server: openresty
     Strict-Transport-Security: max-age=31536000
     Strict-Transport-Security: max-age=31536000
     X-Frame-Options: SAMEORIGIN
     X-PAYTM-SRV-ID: pawslmktsearchapp04
     Date: Mon, 06 Jun 2016 15:22:18 GMT
     Content-Length: 907
     Connection: keep-alive
     Connection #0 to host search.paytm.com left intact

    {"result_type":"grid","default_sorting_param":"sort_relevance=1","meta":{"version":"1.0.0","query":null,"category":[],"mappedQuery":null},"sorting_keys":[{"name":"Relevance","urlParams":"sort_relevance","default":"sort_relevance=1"},{"name":"New","urlParams":"sort_new","default":"sort_new=1"},{"name":"Price","urlParams":"sort_price","default":"sort_price=0"}],"frontend_filters":[],"filters":[{"title":"In-Stock","values":[{"id":"0","name":"0"},{"id":"1","name":"In-Stock"}],"filter_param":"availability","type":"boolean"}],"search_suggestion":null,"has_more":false,"total_count":0,"grid_layout":[],"featured_products":[],"related_searches":[],"search_user_id":"eyJhbGciOiJIUzI1NiJ9.NjZmN2M3OWMtODllZC00OTRjLWI2MDYtNjNiODhlNWE4MTZi.7RvVFv4nDejWf4joXgy_TMdVaaQRTC2F-QPP6FvU_gQ","search_id":"eyJhbGciOiJIUzI1NiJ9.N2JhYzJkNjYtZWE4Ny00OTVmLWE2NzktMTlmM2QwNzY2OWI2.o9dbkUxVEWk8j8PdZnafMzxuAIzubTY9TJFiwyhEy8Q"}
    [1]-  Done                    curl -v https://search.paytm.com/search/?page_count=1
    [8]+  Done                    version=2

Really stumped as to what to do, tried searching stackoverflow to the core and found few solutions like setting cookies or user agent but both doesn't work, also if you see the user-agent string for the other hosting is also set to the same and yet it returns the correct response, heck it even runs the same version of ubuntu. Would be awesome if I could get any help regarding this from the community.

To wrap it up, my skill level in ubuntu, terminal, ssh, etc is just newfound so you might have to lay down the terms and help in basic way... sorry for that.

Answer 1

Your problem is you're not escaping your url. The &'s mean something to the shell.

Try the following:

curl -v 'https://search.paytm.com/search/?page_count=1&items_per_page=10&quality=high&curated=1&cat_tree=1&from=organic&channel=web&version=2&userQuery=iphone'

The clue is this output:

[1] 26241
[2] 26242
[3] 26243
[4] 26244
[5] 26245
[6] 26246
[7] 26247
[8] 26248
[2]   Done                    items_per_page=10
[3]   Done                    quality=high
[4]   Done                    curated=1
[5]   Done                    cat_tree=1
[6]   Done                    from=organic
[7]-  Done                    channel=web

Those are shell background jobs. It is literally trying to run the following commands and put them into the background:

$ curl -v 'https://search.paytm.com/search/?page_count=1 &
$ items_per_page=10 &
$ quality=high &
[... and so on ...]

Answer 2

search.paytm.com is being hosted on the Akamai CDN, so different clients are routed to different servers. Notice that your first connection went to 96.6.72.42 and the second one went to 23.218.97.132. When I look up the hostname I get different IPs, alternating between 104.97.19.53 and 23.203.115.39; I'm located in Boston using Comcast, and traceroute shows that those are located in Comcast datacenters in Boston and New York City.

For some reason, the page you want is only located on the second server. This is a problem with their configuration, you need to contact them.

Answer 3

The fact that the 2 systems resolve to 2 completely different IP ranges on the same host name suggests that the target IP was recently changed, yet your system that fails is still getting the old IP as a DNS response.

Suggest using Google's free DNS servers.. https://developers.google.com/speed/public-dns/docs/using#linux

Answer 4

Turns out that the AWS entire range of IP is blacklisted which is the reason for the error to pop up.