In a CDN'd world, OpenDNS is the enemy!

Mon, May 17, 2010 Tweet Vote on HN While many people are happy with using DNS service providers such as OpenDNS, Google, etc... I will show you here why they may not produce optimal results. The way most CDNs work is by using DNS routing. When a user attempts to resolve a hostname, the CDN's DNS server responds with an IP which is closest based on the IP address of the requester. A more detailed insight into the workings of a CDN can be found on an earlier post "Make your own cheap charlie CDN" For my test here, I tested from the following locations listed below :-

1. True - Thailand : My personal internet connection provided by the ISP called True Internet.
2. Softlayer - United States : A server hosted at Softlayer's Washington DC Datacenter.
3. EC2 - United States : An EC2 instance at Amazons us-east-1c availability zone.
4. EC2-EU - Ireland : An EC2 instance at Amazons eu-west-1 availability zone. - Thanks Luke
5. EC2-APAC - Singapore : An EC2 instance at Amazons ap-southeast-1a availability zone.
6. Com Hem - Sweden : An ISP in sweden. - Thanks Adam
7. Tata - India : An ISP in India. - Thanks Angsuman

The following DNS servers were used to resolve the domains :-

1. OpenDNS (208.67.222.222 , 208.67.220.220 )- Has different caches in multiple locations(Anycasted) - Chicago, Illinois, USA; Dallas, Texas, USA; Los Angeles, California, USA; Miami, Florida, USA; New York, New York, USA; Palo Alto, California, USA; Seattle, Washington, USA; Washington, DC, USA; Amsterdam, The Netherlands and London, England, UK
2. Google Public DNS (8.8.8.8 , 8.8.4.4 ) - "Google Public DNS servers are available worldwide" . I think Google has their DNS servers in all countries where they have hosting infrastructure.
3. Local DNS - The ISP provided DNS in the different locations.

The test was done to the following CDN providers :-

1. Internap ( cdn.thaindian.com ) - Uses DNS routing. POPs (Point Of Presence) in the following locations : Atlanta; Boston; Chicago; Dallas; Denver; El Segundo; Houston; Miami; New York; Philadelphia; Phoenix; San Jose; Seattle; Washington, DC; Sydney; Tokyo; Singapore; Hong Kong; Amsterdam; London
2. Akamai ( profile.ak.fbcdn.net ) - AFAIK they have a POP in almost all countries including Thailand. Note: Akamai does not entertain sales queries from Thai companies.

Results:-

1) Internap ( using cdn.thaindian.com )

Location	Opendns		Google		Local
	IP Returned	Ping to IP (ms)	IP Returned	Ping to IP (ms)	IP Returned	Ping to IP (ms)
True (Thailand)	64.94.126.65	256	74.201.0.130	365	203.190.126.131	152
Softlayer (US-East Coast)	69.88.152.250	1.253	74.201.0.130	25.69	69.88.152.250	1.388
EC2 (US-East Coast)	69.88.152.250	2.144	74.201.0.130	20.229	69.88.152.250	2.094
EC2 (Europe)	77.242.194.130	13.331	64.7.222.130	159.422	77.242.194.130	12.504
EC2 (Singapore)	64.94.126.65	202	74.201.0.130	228	202.58.12.98	37.260
Com Hem (Sweden)	77.242.194.130	40.035	64.7.222.130	189.647	69.88.148.130	36.310
Tata (India)	64.7.222.130	313.2	64.74.124.65	304.1	203.190.126.131	150

2) Akamai ( using profile.ak.fbcdn.net )

Location	Opendns		Google		Local
	IP Returned	Ping to IP (ms)	IP Returned	Ping to IP (ms)	IP Returned	Ping to IP (ms)
True (Thailand)	208.50.77.112	239.4	60.254.185.83	138.9	58.97.45.59	18.88
Softlayer (US-East Coast)	72.246.31.57	1.312	72.246.31.42	1.262	24.143.196.88	0.877
EC2 (US-East Coast)	72.246.31.73	2.581	72.246.31.25	1.792	72.247.242.51	1.941
EC2 (Europe)	195.59.150.139	13.449	92.122.207.177	29.022	195.59.150.138	13.516
EC2 (Singapore)	208.50.77.94	202	60.254.185.73	71.7	124.155.222.10	7.052
Com Hem (Sweden)	217.243.192.8	51.73	92.123.69.82	35.972	92.123.155.139	13.212
Tata (India)	209.18.46.113	300	203.106.85.33	196	125.252.226.58	100.5

The ping timings represent the lag to the destination server from the location in question. I will try to update the results from more locations if I can get shell access to a server or PC in other countries. If you are willing to run the tests for me please contact me(or post in comments)

Conclusion

Using OpenDNS or Google Public DNS may be fast in resolving the DNS, but they do not give the ideal results. In the case of Global DNS providers, the IP of the original requester is not passed along to the CDN's DNS servers so they are unable to route the user to the nearest POP. As you can see in the result tables above, when using OpenDNS from Thailand, trying to access static assets of Facebook, I am directed to a server in the USA whereas when using Google's DNS i am directed to a server in Japan and when using my ISP's DNS server I access content locally, hosted within my own ISPs network! While the effect on large websites using CDN is significant, smaller non-CDN'd websites are also effected. Most websites embed widgets, advertising and other assets which are likely to be CDN'd. The solution would be to use your ISPs DNS server rather than these Global providers. If they really suck so bad, its fairly simple to set up BIND as a caching recursive resolver to resolve hostnames directly bypassing the ISPs crappy service. Bill Fumerola, ex-director of network engineering at OpenDNS confirms this problem on OpenDNS forums. You can run the tests from your own computer using this simple script: dnstest.py Here is the named.conf for a recursive server. Set your computer to use 127.0.0.1 as the DNS. - config may differ for you, RTFM and adapt accordingly.

options {
        directory "/var/named";
        listen-on {
		127.0.0.1;
        };
        auth-nxdomain yes;
        allow-recursion {
                127.0.0.1;
        };
        dump-file       "/var/named/data/cache_dump.db";
        statistics-file "/var/named/data/named_stats.txt";
        memstatistics-file "/var/named/data/named_mem_stats.txt";

};

//
// a caching only nameserver config
//
zone "." {
        type hint;
        file "named.ca";
};

include "/etc/named.rfc1912.zones";

include "/etc/named.dnssec.keys";
include "/etc/pki/dnssec-keys/dlv/dlv.isc.org.conf";

EDIT 1: Inverted the axis added test data from Europe EDIT 2: Added test data from Singapore EDIT 3: Added test data from Sweden EDIT 4: Added test data from India EDIT 5: Added link to Bill Fumerola’s explanation of the problem.

Shameless plug

Turbobytes - Faster content delivery using multiple CDNs