Prerender Application Level Middleware

In the previous post Use Prerender to improve AngularJS SEO, I have explained different solutions at 3 different levels to implement Prerender. In this post, I will explain how to implement a ASP.NET HttpModule as a application level middleware to implement prerender. Since we call it ASP.NET, so it is applicable for both ASP.NET WebForm and MVC.

Application Level Middleware Architecture

At first, let's review what's the appliaction level middleware solution architecture.

ASP.NET HttpModule - PrerenderHttpModule

From above diagram, the easest way to implement it in ASP.NET is to create a HttpModule. Here I named it as PrerenderHttpModule.

At first, let's create PrerenderHttpModule and register BeginRequest event.

#region Implement IHttpModule

/// <summary>

/// init

/// </summary>

/// <param name="context"></param>

public void Init(HttpApplication context)

{

    context.BeginRequest += context_BeginRequest;

}

/// <summary>

/// dispose

/// </summary>

public void Dispose()

{

}

#endregion

#region Begin Request

protected void context_BeginRequest(object sender, EventArgs e)

{

    try

    {

        Prerender(sender as HttpApplication);

    }

    catch (Exception exception)

    {

        Debug.Write(exception.ToString());

    }

}

#endregion

In PrerenderHttpModule, the major method is Prerender(HttpApplication)

private void Prerender(HttpApplication application)

{

    var httpContext = application.Context;

    var request = httpContext.Request;

    var response = httpContext.Response;

    if (IsValidForPrerenderPage(request))

    {

        // generate URL

        var requestUrl = request.Url.AbsoluteUri;

        // if traffic is forwarded from https://, we convert http:// to https://.

        if (string.Equals(request.Headers[Constants.HttpHeader_XForwardedProto], Constants.HttpsProtocol, StringComparison.InvariantCultureIgnoreCase)

         && requestUrl.StartsWith(Constants.HttpProtocol, StringComparison.InvariantCultureIgnoreCase))

        {

            requestUrl = Constants.HttpsProtocol + requestUrl.Substring(Constants.HttpProtocol.Length);

        }

        var prerenderUrl = $"{Configuration.ServiceUrl.Trim('/')}/{requestUrl}";

        // create request

        var webRequest = (HttpWebRequest)WebRequest.Create(prerenderUrl);

        webRequest.Method = "GET";

        webRequest.UserAgent = request.UserAgent;

        webRequest.AllowAutoRedirect = false;

        webRequest.Headers.Add("Cache-Control", "no-cache");

        webRequest.ContentType = "text/html";

        // Proxy Information

        if (!string.IsNullOrEmpty(Configuration.ProxyUrl) && Configuration.ProxyPort > 0)

            webRequest.Proxy = new WebProxy(Configuration.ProxyUrl, Configuration.ProxyPort);

        // Add token

        if (!string.IsNullOrEmpty(Configuration.Token))

            webRequest.Headers.Add(Constants.HttpHeader_XPrerenderToken, Configuration.Token);

        var webResponse = default(HttpWebResponse);

        try

        {

            // Get the web response and read content etc. if successful

            webResponse = (HttpWebResponse)webRequest.GetResponse();

        }

        catch (WebException e)

        {

            // Handle response WebExceptions for invalid renders (404s, 504s etc.) - but we still want the content

            webResponse = e.Response as HttpWebResponse;

        }

        // write response

        response.StatusCode = (int)webResponse.StatusCode;

        foreach (string key in webResponse.Headers.Keys)

        {

            response.Headers[key] = webResponse.Headers[key];

        }

        using (var reader = new StreamReader(webResponse.GetResponseStream(), DefaultEncoding))

        {

            response.Write(reader.ReadToEnd());

        }

        response.Flush();

        application.CompleteRequest();

    }

}

Also, in order to make the logic flexible and easy to configure, I have add a configuration class and IsValidForPrerenderPage() method

private bool IsValidForPrerenderPage(HttpRequest request)

{

    var userAgent = request.UserAgent;

    var url = request.Url;

    var rawUrl = request.RawUrl;

    var relativeUrl = request.AppRelativeCurrentExecutionFilePath;

    // check if follows google search engine suggestion

    if (request.QueryString.AllKeys.Any(a => a.Equals(Constants.EscapedFragment, StringComparison.InvariantCultureIgnoreCase)))

        return true;

    // check if has user agent

    if (string.IsNullOrEmpty(userAgent))

        return false;

    // check if it's crawler user agent.

    var crawlerUserAgentPattern = Configuration.CrawlerUserAgentPattern ?? Constants.CrawlerUserAgentPattern;

    if (string.IsNullOrEmpty(crawlerUserAgentPattern)

     || !Regex.IsMatch(userAgent, crawlerUserAgentPattern, RegexOptions.IgnorePatternWhitespace))

        return false;

    // check if the extenion matchs default extension

    if (Regex.IsMatch(relativeUrl, DefaultIgnoredExtensions, RegexOptions.IgnorePatternWhitespace))

        return false;

    if (!string.IsNullOrEmpty(Configuration.AdditionalExtensionPattern) && Regex.IsMatch(relativeUrl, Configuration.AdditionalExtensionPattern, RegexOptions.IgnorePatternWhitespace))

        return false;

    if (!string.IsNullOrEmpty(Configuration.BlackListPattern)

      && Regex.IsMatch(rawUrl, Configuration.BlackListPattern, RegexOptions.IgnorePatternWhitespace))

        return false;

    if (!string.IsNullOrEmpty(Configuration.WhiteListPattern)

      && Regex.IsMatch(rawUrl, Configuration.WhiteListPattern, RegexOptions.IgnorePatternWhitespace))

        return true;

    return false;

}

The priority of checking whether is valid for prerender page:

Constant -> EscapedFragment: _escaped_fragment_

User Agent: Setting -> CrawlerUserAgentPattern

(google)|(bing)|(Slurp)|(DuckDuckBot)|(YandexBot)|(baiduspider)|(Sogou)|(Exabot)|(ia_archiver)|(facebot)|(facebook)|(twitterbot)|(rogerbot)|(linkedinbot)|(embedly)|(quora)|(pinterest)|(slackbot)|(redditbot)|(Applebot)|(WhatsApp)|(flipboard)|(tumblr)|(bitlybot)|(Discordbot)

Constant -> DefaultIgnoredExtensions

\\.vxml|js|css|less|png|jpg|jpeg|gif|pdf|doc|txt|zip|mp3|rar|exe|wmv|doc|avi|ppt|mpg|mpeg|tif|wav|mov|psd|ai|xls|mp4|m4a|swf|dat|dmg|iso|flv|m4v|torrent

Setting -> AdditionalExtensionPattern
Setting -> BlackListPattern
Setting -> WhiteListPattern
At last, return false.

Register PrerenderHttpModule

Generally, we have two different ways to register a HttpModule in ASP.NET,

Use HttpModule configuration in Web.config
Use DynamicModuleUtility (Microsoft.Web.Infrastructure.dll)

Here, I have added the logic to support both, and I have added a app setting to control which one we want to use. By default, it will use DynamicModuleUtility, as we don't need to configure HttpModule in web.config, it's automatical.

Note, in order to use this HttpModule, please configure your IIS Application Pool to integrated mode.

<!--If it's false, please configure http module for UsePrestartForPrenderModule-->

<add key="UsePrestartForPrenderModule" value="true"/>

1. Use DynamicModuleUtility (Microsoft.Web.Infrastructure.dll)

This is the default configuration.

Configre UsePrestartForPrerenderModule to true or remove this setting

<!--If it's false, please configure http module for UsePrestartForPrenderModule-->

<add key="UsePrestartForPrenderModule" value="true"/>

Add reference for Microsoft.Web.Infrastructure.dll
Add a static class PrerenderPreApplicationStart for assembly prestart. We need to use static method in static class.

public static class PrerenderPreApplicationStart

{

    public const string StartMethodName = "Prestart";

    static bool UsePrestart = !bool.FalseString.Equals(ConfigurationManager.AppSettings[Constants.AppSetting_UsePrestartForPrenderModule], StringComparison.InvariantCultureIgnoreCase);

    /// <summary>

    /// used to configure for PreApplicationStart.

    /// i.e. [assembly: PreApplicationStartMethod(typeof(PrerenderPreApplicationStart), "Start")]

    /// </summary>

    public static void Prestart()

    {

        if (UsePrestart)

        {

            DynamicModuleUtility.RegisterModule(typeof(PrerenderHttpModule));

        }

    }

}

[assembly: PreApplicationStartMethodAttribute(typeof(PrerenderPreApplicationStart), PrerenderPreApplicationStart.StartMethodName)]

Once the ASP.NET application loads this assembly, it will trigger PrerenderPreApplicationStart.Prestart() method, then registers the PrerenderHttpModule.

2. Use HttpModule configuration in Web.config

This is a very common and easy way, what we need to do is to:

Configure UsePrestartForPrerenderModule to false

<!--If it's false, please configure http module for UsePrestartForPrenderModule-->

<add key="UsePrestartForPrenderModule" value="false"/>

Configure PrerenderHttpModule

<system.webServer>

  <validation validateIntegratedModeConfiguration="false"/>

  <modules runAllManagedModulesForAllRequests="false">

    <!--Configure PrerenderHttpModule when UsePrestartForPrenderModule is false; -->

    <add name="prerender" type="DotNetOpen.PrerenderModule.PrerenderHttpModule, DotNetOpen.PrerenderModule" />

    <remove name="FormsAuthentication"/>

  </modules>

</system.webServer>

Configuration Section in Web.config

I have added a configuration section PrerenderConfigurationSection for prerender options

Declare the configuration section in web.config section group.

  <configSections>

    <!-- For more information on Entity Framework configuration, visit http://go.microsoft.com/fwlink/?LinkID=237468 -->

    <section name="prerender" type="DotNetOpen.PrerenderModule.Configuration.PrerenderConfigurationSection, DotNetOpen.PrerenderModule"/>

    <section name="entityFramework" type="System.Data.Entity.Internal.ConfigFile.EntityFrameworkSection, EntityFramework, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" requirePermission="false"/>

  </configSections>

Configure options.

  <!--prerender settings-->

  <!--CrawlerUserAgentPattern: "(google)|(bing)|(Slurp)|(DuckDuckBot)|(YandexBot)|(baiduspider)|(Sogou)|(Exabot)|(ia_archiver)|(facebot)|(facebook)|(twitterbot)|(rogerbot)|(linkedinbot)|(embedly)|(quora)|(pinterest)|(slackbot)|(redditbot)|(Applebot)|(WhatsApp)|(flipboard)|(tumblr)|(bitlybot)|(Discordbot)"-->

  <!--WhiteListPattern, BlackListPattern: will check raw URL, which includes query string-->

  <!--AdditionalExtensionPattern: will only check extension-->

  <prerender ServiceUrl="http://localhost:3000"

             Token=""

             WhiteListPattern=""

             BlackListPattern=""

             AdditionalExtensionPattern=""

             ProxyUrl=""

             ProxyPort="80"/>

You can go to my github wiki page to get more details about each option: Configuration & Check Priority

Nuget Package

I have created a nuget package, which is very convenient if you don't want to dive deep into the source code.

Install Nuget Package in your project.

Visual Studio -> Tools -> Nuget Package Manager -> Package Manager Console.

Install-Package DotNetOpen.PrerenderModule

If you want to take a look more detail about this package, you can go https://www.nuget.org/packages/DotNetOpen.PrerenderModule/

There are several versions

Version 1.01-1.0.2, they are for .NET Framework 4.6.2
From Version 1.0.2, I have changed the .NET Framework version from 4.6.2 to 4.0, so that more people can use it. Which means the latest Nuget Package is using .NET Framework 4.0

I have fully documented how to do this in my github wiki page, you can go there take a look.

Done, try it out.

Github Project

I also have created a github project to host all source code includes sample code for testing: https://github.com/dingyuliang/prerender-dotnet, in this project, it includes ASP.NET HttpModule, ASP.NET Core Middleware, IIS Configuration 3 different solution.

For ASP.NET HttpModule, you can go to https://github.com/dingyuliang/prerender-dotnet/tree/master/src/DotNetPrerender

Prerender Related

------------------------------------------------------------------------------------------------