Angular Universal - Pre-render only for web crawlers?

Question

I am intending to user Angular Universal for server side rendering (SSR) but this should only be done for crawlers and bots from selected search engines.

What I want is the following schema:

source: https://dingyuliang.me/use-prerender-improve-angularjs-seo/

After following the official instructions to set up SSR I can now validate that Googlebot (finally) "sees" my website and should be able to index it.

However, at the moment all requests are rendered on the server. Is there a way to determine whether incoming requests are coming from search engines and pre-render the site only for them?

Answer 1

You can achieve that with Nginx.

In Nginx you can forward the request to the universal served angular application via..

        if ($http_user_agent ~* "googlebot|yahoo|bingbot") {
            proxy_pass 127.0.0.1:5000; 
            break;
        }
        root /var/www/html;

..assuming that you are serving angular universal via 127.0.0.1:5000.

In case a browser user agent comes along, we serve the page via root /var/www/html

So the complete config would be something like..

server {
    listen 80 default;
    server_name angular.local;

    location / {
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $remote_addr;
        proxy_set_header Host $http_host;

        if ($http_user_agent ~* "googlebot|yahoo|bingbot") {
            proxy_pass 127.0.0.1:5000; 
            break;
        }

        root /var/www/html;
    }

}

Answer 2

This is what I came up with IIS:

1. Add the Angular Universal to your project according to the official guide

2. In order to get rid of complex folder structures, change the following line in server.ts

   const distFolder = join(process.cwd(), 'dist/<Your Project>/browser');
   

   to this:

   const distFolder = process.cwd();
   

3. Run the npm run build:ssr command. You will end up with the browser and server folders inside the dist folder.

4. Create a folder for hosting in IIS and copy the files that are in the browser and server folders in to the created folder.

   iis\
     -assets\
     -favicon.ico
     -index.html
     -main.js => this is the server file
     -main-es2015.[...].js
     -polyfills-es2015.[...].js
     -runtime-es2015.[...].js
     -scripts.[...].js
     -...
   

5. Add a new file to this folder named web.config with this content:

   <?xml version="1.0" encoding="utf-8"?>
   <configuration>
     <system.webServer>
       <rewrite>
         <rules>
           <rule name="Angular Routes" stopProcessing="true">
             <match url=".*" />
             <conditions logicalGrouping="MatchAll">
               <add input="{REQUEST_FILENAME}" matchType="IsFile" negate="true" />
               <add input="{REQUEST_FILENAME}" matchType="IsDirectory" negate="true" />
               <add input="{HTTP_USER_AGENT}" pattern="(.*[Gg]ooglebot.*)|(.*[Bb]ingbot.*)" negate="true" />
             </conditions>
             <action type="Rewrite" url="/index.html" />
           </rule>
           <rule name="ReverseProxyInboundRule1" stopProcessing="true">
             <match url=".*" />
             <conditions>
               <add input="{HTTP_USER_AGENT}" pattern="(.*[Gg]ooglebot.*)|(.*[Bb]ingbot.*)" />
             </conditions>
             <action type="Rewrite" url="http://localhost:4000/{R:0}" />
           </rule>
         </rules>
       </rewrite>
       <directoryBrowse enabled="false" />
     </system.webServer>
   </configuration>
   

6. Inside this folder open a Command Prompt or PowerShell and run the following:

   > node main.js
   

   Now you should be able to view your Server-Side Rendered website with localhost:4000 (if you haven't changed the port)

7. Install the IIS Rewrite Module

8. Add the folder to your IIS for hosting

IIS will redirect requests that have googlebot or bingbot in them to localhost:4000 which is handled by Express and will return server-side rendered content.

You can test this with Google Chrome, open Developer Console, from the menu select "More tools>Network conditions". Then from the User Agent section disable "Select automatically" and choose Googlebot.