I recently decided to build ngx_pagespeed for Ubuntu’s stock nginx, since nginx supports dynamic modules (as Apache did long ago) since 1.9.11. Being “dynamic” means that third-party modules don’t have to be built into the main nginx binary, instead can be separately built and dynamically load at runtime, as long as the module and main nginx are compatible, meaning their configuration parameters are similar enough.
I have not seen anyone else build any third-party modules separately. nginx.org official repository provides several modules as packages, but I believe they are built along with the binary they shipped. A probable reason is that even though modules built separately can be loaded, they are not guaranteed to work smoothly. I saw that nginx 1.11.5 is trying to improve compatibility of dynamic modules, but whether that works remains to be tested.
I used the latest Debian (Ubuntu) packaging tools and formats to build the package, and Launchpad PPA to host the apt repository.
debian folder: (TODO – Provide a GitHub repository)
You can use the apt commands shown on the page to enable the PPA and install the packages. The following is an example:
sudo add-apt-repository ppa:du9l/ngx-pagespeed
To use this module, first install the right flavored package, then add the line: load_module modules/ngx_pagespeed.so;
… to your /etc/nginx/nginx.conf file before reloading nginx.
Please remember that this is an experimental feature, and you should NOT use it in production. More information can be found in the PPA description below.
This repository contains “ngx_pagespeed” dynamic module for Ubuntu’s stock nginx packages, including all flavors available in the official repository.
* WARNING: Building dynamic modules alone is EXPERIMENTAL. It is NOT guaranteed to work by the nginx authors. Even though the module can be loaded and has been tested on my own server, I still don’t recommend using it in PRODUCTION environments. USE AT YOUR OWN RISK. *
The package names are “ngx-pagespeed-nginx-FLAVOR” where FLAVOR is core / light / full / extras, which should match your nginx flavor. Check with:
$ dpkg -l | grep nginx
The versions follow ngx_pagespeed’s latest stable versions and Ubuntu’s REL-updates (e.g. xenial-updates) nginx versions. For example, the first version available in this PPA is built with ngx_pagespeed 220.127.116.11 and xenial-updates’ nginx 1.10.0-0ubuntu0.16.04.4 versions.
To use this module, first install the right flavored package, and then add the line:
… to your /etc/nginx/nginx.conf file before reloading nginx.
* NOTE: This package is linked against the pre-built “psol” binaries provided by Google, so only i386 and amd64 systems are supported for now. In the future I will update the package to build psol itself. *
In this article I will show some different CDN implementations along with cases where each of them fails to bring the best performance. I am not a researcher in this area, so some of the points are based on my personal experiences.
1. Why people use CDN
Usually when people visit a website, their browsers first query the IP address from their ISP’s recursive DNS server, which in turn query the domain’s authoritative DNS. Then they will be connected to whatever IP address returned by the DNS which is usually distant (geographically far and has high latency) from them.
That is why people use CDN to solve this problem. By putting edge caching servers in different areas of the world (or in the target country) close to end-users, they can speed up the load time of their websites and improve user experience.
2. Traditional Geo-DNS setup
The most common and simple solution is to use a “Geo-DNS” service to point the same domain to different edge servers with different IP addresses (this is important). In this case, when people query the IP address of a domain, the authoritative Geo-DNS can point them to the nearest edge server, based on the users’ IP, their ISPs’ recursive DNS servers’ IP, or users’ IP provided by recursive DNS via EDNS Client Subnet (EDNS0) if supported.
This works fine in an ideal network setup, but it can easily fall apart. Here are some cases that I have come across:
(a) Unicast DNS
Sometimes Geo-DNS providers don’t use Anycast, instead provide Unicast IP addresses for different regions. The recursive DNS has no way of telling which one is closer to them, so it queries a random one, which can result in slow DNS resolution at first visit.
Example: DNSPod and CloudXNS (Popular Unicasted Geo-DNS providers in China)
(b) Bad GeoIP database
Some Geo-DNS providers don’t update their GeoIP database frequent enough, or just don’t have enough data.
(1) Amazon AWS CloudFront and Akamai don’t have servers in China for obvious reasons, but Chinese visitors are not consistently directed to nearest (Hong Kong, South Korea, Japan) locations. Sometimes a query from China can get a response of European locations, which results in ~500 ms latency.
(2) Some Geo-DNS providers in China, most notably Aliyun DNS. When both “Domestic” and “Global” records are set, they may direct Chinese users to “Global” servers.
(c) DNS different from network exit
Sometimes people may use recursive DNS servers in the network different from their actual network exit.
(1) In my university, we have mixed network exits, one in CERNET (AS4538) and one in TieTong (AS9394). Our recursive DNS has a CERNET address, so most Geo-DNS providers gives CERNET or (if the website doesn’t have CERNET servers) other networks’ addresses, for instance ChinaNet (AS4134). But our network exit is configured to use TieTong by default, so for most websites we are visiting ChinaNet servers with a TieTong network, even if they also have TieTong servers.
A more extreme case is that, in some networks they have different routing policies for TCP and UDP (which is a violation of OSI model), so when you do DNS query in UDP you have network A’s address, and when you actually connect to TCP port 80 you have network B. Magical? But true.
(2) Sometimes recursive DNS providers and/or Geo-DNS providers don’t support EDNS0. As long as either end doesn’t support it, it will not work. For instance, if user of open recursive DNS service “114DNS” (Anycasted across several Chinese networks) has a network that is not present in 114DNS’ Anycast network, and the authoritative Geo-DNS doesn’t support EDNS0, it will return the IP in the same network of 114DNS’ node, but different from the network of the user.
3. TCP Anycast setup
Some modern CDN providers use TCP Anycast technique, which means they provide a single IP address for their edge servers in multiple locations, and visitors are directed to the nearest location, decided by how they broadcast their routing tables to other networks.
Such providers include CloudFlare and MaxCDN, which use a single Anycasted IP for their edge servers across the planet. Verizon EdgeCast use a slightly different method where they provide several Anycasted IPs, each represent a geographical zone (Asia-Pacific, North America, South America, Europe).
A unified Anycasted IP solves many of the problems mentioned above, and it’s becoming harder and harder to defeat them. But here they comes:
(1) Magical routing policy (again)
Current Chinese IPv6 implementation has only one international exit (AS23911), which has two exit points: a default one in Los Angeles by HE.net (AS6939) and a premium one in Hong Kong (HKIX). When I resolve EdgeCast’s IPv6 address, I get one in 2606:2800:147::/48 network, which is Anycasted in Asia. But when I trace route to this address, the packet goes from China to Los Angeles and back to Asia, resulting in ~400 ms latency. Even if people use an Anycasted recursive DNS (like Google’s), since it has servers in Hong Kong, the result is the same. By querying the domain at OpenDNS (which doesn’t have Asian server) I get the IP in 2606:2800:11f::/48 network, which is Anycasted in North America, and the latency is only ~200 ms (same as the network exit’s).
This only happens with EdgeCast’s “Continent-based Anycast” network. CloudFlare is not affected. But it has another kind of problem.
(2) Artificial routing deterioration
CloudFlare has edge servers everywhere, including Hong Kong, Taipei, Japan, South Korea, etc. which are all very close to Chinese users. But the major Chinese ISPs’ international exit routing policy directs CloudFlare traffic to Los Angeles (ChinaNet) and San Jose (China Unicom), where they are directed to the nearest edge servers in <3 hops. They did the same thing for Softlayer’s Hong Kong locations, for some magical reasons: maybe price, maybe [censored] ;). The latency from both ISP to CloudFlare’s US west locations are 200~300 ms, where with TieTong (which use Hong Kong as international exit) the value is <100 ms.
This is obviously not CloudFlare’s fault, because they cannot control the routing policy from another AS to themselves (unless they pay the other system to do so). If your ISP is doing this, switch to another ISP; If the whole country is doing this, maybe switch to another country?
To sum it up, when you and your customers’ networks don’t have any of the quirks above, a simple Anycasted Geo-DNS solution works fine – you don’t even need a commercial CDN service. But the real networks are hard, and so far a global TCP Anycast solution is the best we can do.