Preparing and delivering content on a Content Delivery Network (CDN).
- January 22nd, 2010
- Posted in Articles
- Write comment
Most CDNs employ one of 2 methods for delivering content. The first is origin pull or off-site origin. This evolves the CDN pulling the content from some outside source. This origin could be your webserver, a cloud computing service, something like Amazon S3 or any other internet connected HTTP server. The key is, that the CDN needs to access your content via HTTP GET requests.
The second method is CDN storage. This is storage that the CDN supplies to you on their network. This is usually a preferred method as the CDN does not have to go far to get your content to cache it on edge servers. You can expect to pay for this storage on top of your other CDN charges. Typically, you will FTP the content to the CDN storage or in some cases, there will be an HTTP upload option or even RSYNC may be an option. If your content is large in size, larger than 5-10MB its recommended you store the content on the CDN.
Using CNAMEs to access content
Most CDN’s will use CNAMEs to allow you to access your content. A canonical name or CNAME is simply an alias. For example ‘static.domain.com’ could point to the CDN URL. You can use this instead of the URL that CDN supplies to you. This way you can better brand your site and won’t have the CDN URL floating around on your site somewhere. Talk to your CDN on how to implement a CNAME, they may have special requirements or might not even allow them.
When you do off-site storage, the CDN usually needs to know where you store that content. So be prepared to supply the CDN the CNAME and the Source/Origin URL. Origin, being the URL where the CDN can go to, to pickup your content.
When you write your HTML instead of using a relative path to a file like “./images/logo.jpg” you will use an absolute URL instead, such as “http://static.domain.com/images/logo.jpg”. This way you are essentially embedding content from the CDN on your website.
If you are using a content management system, check to see if there is a way to address all your static elements like images, CSS, java script, PDF, MP3, FLV, MP4, etc at once. You may be able to specify a “pre-pend” URL for specific file types. This would make switching to a CDN easy and quick. You could “CDN enable” your whole site in one click.
WordPress users, see the CDN Rewrites Plugin –
How do you know if your content is cacheable?
If you are uploading the content to the CDN then it will be cacheable. If the CDN is going to pull the content from you then you need to consider a few things. Most CDNs will honor any cache control headers you put on your content. For example, if you put a Max-Age=86400 on your content, then the CDN will consider that piece of content fresh for 24 hours. Don’t think for a second you can tell the CDN how long to hold a piece of content in cache for. They will decide when that piece of content needs to be purged from an edge server. However, setting this TTL will tell the CDN that after 24 hours they need to look to see if there is a new version of the file.
Also consider this, if your content has a Private, or No-Cache header on it, then the CDN will probably not cache it, you’re wasting bandwidth. You are trying to deliver non-cacheable content through the CDN, they will go back to your origin for every request.
Some CDNs can address this issue by implementing some custom work around, so talk to your CDN of choice for advice. Also, check with your CDN to see if they require specific cache control headers to be present, you may need to alter your headers in order to make your content cacheable even if you don’t have a No-Cache type header.
Conclusion
This was a basic overview on how CDNs handle basic caching of content from different origins as well as how to deliver your content through the CDN. Consider the issues of cache control headers, these can be very powerful and allow for flexibility on how your content is cached and for how long. You should always work with your CDN of choice directly as they will have specifics for implementing their solution. No 2 CDNs are exactly alike.
Can you please let me know How I can identify whether a site is CDN enabled or not. ?
Is there any tool which can tell me the all the cdn objects with in the site. ?
I would use Firebug in Firefox. Look at the CNAMEs that are loading then do a NSLookup on the CNAME and see if it’s redirecting to a CDN
@Mike
Mike could you pls show me a screenshot (from Firebug) of the CNAMEs and the NS Lookup’s for any site of your choice that represents the CDN redirection in it.
I would apprecite your help if you could do this.
Thanks,
Siva.
What is the URL you trying to look at? You would look for anything that says like video.domain.com or static.domain.com something like that. Then go out to a command prompt and type nslookup videos.domain.com.
For example. Go to CNN.COM right click on an image, choose properties and you see the URL like this:
http://i2.cdn.turner.com/cnn/2010/WORLD/asiapcf/05/14/thailand.anti-government.protests/t1main.thaiprotest2.gi.jpg
nslookup i2.cdn.turner.com and that resolves to:
Non-authoritative answer:
Name: cdn.cnn.com.c.footprint.net
Addresses: 204.160.119.126, 8.12.218.126, 199.93.46.126
Aliases: i2.cdn.turner.com
Look up footprint.net and it belongs to Level3. Level3 is the CDN for that Image.
Finding the CDN for videos might be more difficult as many times the URL is embedded in a SWF or JS. You might need a pack sniffer software to see where the video is loading from and then resolve that URL.
@Mike
I am testing it for MSN.com.
I got it some extent here. But can you please clear me what does Level3 mean?
(If in the Addresses tag if we could found 3 IP’s does it mean that it is level 3 there? and If it doesnt show 3 IP’s/(level 3) then all those objects or not in CDN.
Is that right? Iam not sure ??)
Kindly give a clarification.
Thank you,
Siva,
Level3 is a CDN Provider. Why are you trying to analyze MSN? Seems like a a big task. Microsoft uses a combination of their own network, components from Limelight, and caching from Akamai. It also depends which Microsoft domain you are talking about too.
If you look at the page, there are a number of HOSTS, most of which are coming from 3rd parties. Things like advertisements, and other widgets. Those providers may or may not be using CDN to deliver their content.
In this case, http://www.msn.com is an A-RECORD not a CNAME. It resolves to a single IPAddress, probably a Load Balancer of something similar.
nslookup http://www.msn.com
Non-authoritative answer:
Name: us.co1.cb3.glbdns.microsoft.com
Address: 65.55.17.26
Aliases: http://www.msn.com
I have a tool which will analyze a web site. Below are all the HOSTS on the MSN.COM home page, where they resolve to and who owns the IPAddresses.
What is your goal here? Just personal knowledge or are you trying to discover some piece of information?
Hostnames for msn.com
Hostname: http://www.w3.org
Record Type: A
IP: 128.30.52.168
Owner: Massachusetts Institute of Technology
Hostname: col.stc.s-msn.com
Record Type: CNAME
CNAME Target: colstc.co1.cb3.glbdns.microsoft.com
IP: 198.78.206.126
Owner: Level 3 Communications, Inc.
Hostname: http://www.icra.org
Record Type: A
IP: 213.249.189.194
Owner: % This is the RIPE Database query service.
Hostname: http://www.msn.com
Record Type: CNAME
CNAME Target: us.co1.cb3.glbdns.microsoft.com
IP: 65.55.17.27
Owner: Microsoft Corp
Hostname: http://www.rsac.org
Record Type: CNAME
CNAME Target: http://www.icra.org
IP: 213.249.189.194
Owner: % This is the RIPE Database query service.
Hostname: col.stj.s-msn.com
Record Type: CNAME
CNAME Target: colstj.co1.cb3.glbdns.microsoft.com
IP: 205.128.65.126
Owner: Level 3 Communications, Inc.
Hostname: ads1.msn.com
Record Type: CNAME
CNAME Target: global.msads.net.c.footprint.net
IP: 198.78.206.126
Owner: Level 3 Communications, Inc.
Hostname: msnportal.112.2o7.net
Record Type: A
IP: 66.235.139.121
Owner: Omniture, Inc
Hostname: amer.rel.msn.com
Record Type: CNAME
CNAME Target: amer.hops.glbdns.microsoft.com
IP: 64.4.18.73
Owner: MS Hotmail
Hostname: exp.www.msn.com
Record Type: CNAME
CNAME Target: ro-msn.exp.glbdns.microsoft.com
IP: 65.55.18.18
Owner: Microsoft Corp
Hostname: g.msn.com
Record Type: CNAME
CNAME Target: g.msn.com.nsatc.net
IP: 207.46.216.54
Owner: Microsoft Corp
Hostname: analytics.live.com
Record Type: CNAME
CNAME Target: analytics.live.com.nsatc.net
IP: 65.55.149.119
Owner: Microsoft Corp
Hostname: http://www.bing.com
Record Type: CNAME
CNAME Target: search.ms.com.edgesuite.net
IP: 63.110.246.16
Owner: WS/Akamai Technologies/Akamai Technologies UU-63-110-246-D6 (NET-63-110-246-0-1)
Hostname: login.live.com
Record Type: CNAME
CNAME Target: login.live.com.nsatc.net
IP: 65.54.186.47
Owner: Microsoft Corp
Hostname: download.live.com
Record Type: CNAME
CNAME Target: download.live.com.nsatc.net
IP: 65.55.61.125
Owner: Microsoft Corp
Hostname: col.stb.s-msn.com
Record Type: CNAME
CNAME Target: colstb.co1.cb3.glbdns.microsoft.com
IP: 65.54.75.140
Owner: Microsoft Corp
Hostname: http://www.msnbc.msn.com
Record Type: CNAME
CNAME Target: http://www.msnbc.msn.com.edgesuite.net
IP: 63.110.246.17
Owner: WS/Akamai Technologies/Akamai Technologies UU-63-110-246-D6 (NET-63-110-246-0-1)
Hostname: local.msn.com
Record Type: CNAME
CNAME Target: local.blu.cb3.glbdns.microsoft.com
IP: 65.55.206.199
Owner: Microsoft Corp
Hostname: today.msnbc.msn.com
Record Type: CNAME
CNAME Target: msnbc.msn.com
IP: 65.55.53.233
Owner: Microsoft Corp
Hostname: entertainment.msn.com
Record Type: A
IP: 65.55.14.126
Owner: Microsoft Corp
Hostname: wonderwall.msn.com
Record Type: A
IP: 67.201.39.229
Owner: Mzima Networks, Inc. NETBLK-MZIMA-04 (NET-67-201-0-0-1)
Hostname: zone.msn.com
Record Type: A
IP: 207.46.166.10
Owner: Microsoft Corp
Hostname: movies.msn.com
Record Type: CNAME
CNAME Target: entertainment.msn.com
IP: 65.55.14.126
Owner: Microsoft Corp
Hostname: music.msn.com
Record Type: CNAME
CNAME Target: entertainment.msn.com
IP: 65.55.14.126
Owner: Microsoft Corp
Hostname: tv.msn.com
Record Type: CNAME
CNAME Target: entertainment.msn.com
IP: 65.55.14.126
Owner: Microsoft Corp
Hostname: msn.foxsports.com
Record Type: CNAME
CNAME Target: http://www.foxsports.com.edgesuite.net
IP: 63.110.246.11
Owner: WS/Akamai Technologies/Akamai Technologies UU-63-110-246-D6 (NET-63-110-246-0-1)
Hostname: moneycentral.msn.com
Record Type: CNAME
CNAME Target: moneycentral.com
IP: 65.54.150.19
Owner: Microsoft Corp
Hostname: autos.msn.com
Record Type: CNAME
CNAME Target: web.autos.glbdns.microsoft.com
IP: 65.55.182.112
Owner: Microsoft Corp
Hostname: msn.careerbuilder.com
Record Type: CNAME
CNAME Target: http://www.careerbuilder.com
IP: 208.88.80.22
Owner: CareerBuilder, LLC
Hostname: realestate.msn.com
Record Type: CNAME
CNAME Target: lifestyle.blu.cb3.glbdns.microsoft.com
IP: 65.55.206.197
Owner: Microsoft Corp
Hostname: articles.moneycentral.msn.com
Record Type: CNAME
CNAME Target: articles.moneycentral.blu.cb3.glbdns.microsoft.com
IP: 65.55.206.205
Owner: Microsoft Corp
Hostname: lifestyle.msn.com
Record Type: CNAME
CNAME Target: lifestyle.blu.cb3.glbdns.microsoft.com
IP: 65.55.206.197
Owner: Microsoft Corp
Hostname: http://www.delish.com
Record Type: CNAME
CNAME Target: http://www.delish.com.edgesuite.net
IP: 64.211.21.103
Owner: Global Crossing
Hostname: glo.msn.com
Record Type: A
IP: 68.71.242.140
Owner: Zero Lag Communications
Hostname: health.msn.com
Record Type: CNAME
CNAME Target: health.blu.cb3.glbdns.microsoft.com
IP: 65.55.206.198
Owner: Microsoft Corp
Hostname: astrocenter.astrology.msn.com
Record Type: CNAME
CNAME Target: http://www.astrocenter.com
IP: 149.7.36.148
Owner: PSINet, Inc. ASTROCENTER (UK ) Ltd
Hostname: dating.msn.com
Record Type: CNAME
CNAME Target: msn.match.com
IP: 208.83.241.16
Owner: Match.com, L.L.C.
Hostname: specials.msn.com
Record Type: CNAME
CNAME Target: specials.blu.cb3.glbdns.microsoft.com
IP: 65.55.228.106
Owner: Microsoft Corp
Hostname: my.msn.com
Record Type: CNAME
CNAME Target: my.co1.cb3.glbdns.microsoft.com
IP: 65.55.17.32
Owner: Microsoft Corp
Hostname: careers.msn.com
Record Type: CNAME
CNAME Target: us.chan.blu.cb3.glbdns.microsoft.com
IP: 65.55.228.106
Owner: Microsoft Corp
Hostname: msn.whitepages.com
Record Type: CNAME
CNAME Target: http://www.whitepages.com
IP: 64.124.61.10
Owner: Abovenet Communications, Inc ABOVENET (NET-64-124-0-0-1)
Hostname: yellowpages.msn.com
Record Type: A
IP: 65.55.241.167
Owner: Microsoft Corp
Hostname: secure.opinionlab.com
Record Type: A
IP: 66.77.28.111
Owner: Qwest Communications Company, LLC QWEST-INET-12 (NET-66-77-0-0-1)
Hostname: help.live.com
Record Type: A
IP: 65.55.162.252
Owner: Microsoft Corp
Hostname: latino.msn.com
Record Type: CNAME
CNAME Target: latino.co1.cb3.glbdns.microsoft.com
IP: 65.55.17.37
Owner: Microsoft Corp
Hostname: blst.msn.com
Record Type: CNAME
CNAME Target: blst.blu.cb3.glbdns.microsoft.com
IP: 205.128.93.126
Owner: Level 3 Communications, Inc.
Hostname: view.atdmt.com
Record Type: A
IP: 65.55.116.181
Owner: Microsoft Corp
Hostname: http://www.vacationstogo.com
Record Type: CNAME
CNAME Target: vacationstogo.com
IP: 206.222.179.99
Owner: Insync Internet Services
Hostname: http://www.forcefactor.com
Record Type: CNAME
CNAME Target: forcefactor.com
IP: 174.143.32.234
Owner: Rackspace Hosting
Hostname: ad.doubleclick.net
Record Type: CNAME
CNAME Target: dart-ad.l.doubleclick.net
IP: 74.125.19.148
Owner: Google Inc.
Hostname: ccc01.opinionlab.com
Record Type: A
IP: 66.77.28.73
Owner: Qwest Communications Company, LLC QWEST-INET-12 (NET-66-77-0-0-1)
Hostname: http://www.ie8optimized.com
Record Type: A
IP: 174.143.45.130
Owner: Rackspace Hosting RSCP-NET-4 (NET-174-143-0-0-1)
Hostname: go.microsoft.com
Record Type: CNAME
CNAME Target: http://www.go.microsoft.akadns.net
IP: 207.46.16.233
Owner: Microsoft Corp
Hostname: advertising.microsoft.com
Record Type: CNAME
CNAME Target: advertising.microsoft.com.nsatc.net
IP: 65.55.249.222
Owner: Microsoft Corp
Hostname: rss.msn.com
Record Type: CNAME
CNAME Target: rss.blu.cb3.glbdns.microsoft.com
IP: 204.160.102.126
Owner: Level 3 Communications, Inc.
Hostname: careers.microsoft.com
Record Type: CNAME
CNAME Target: careers.microsoft.akadns.net
IP: 65.55.186.234
Owner: Microsoft Corp
@Mike
Mike,
Can you please let me know the tool you are using? Is it a freeware or commericial? Actually Iam trynig to know the cdn data from MSN. What are all the data that is coming from a CDN and what are from the webserver directly.
Thanks,
The tool I’m using is something my company wrote. I can not share it with you, sorry. I think you are wasting your time trying to figure this out for Microsoft. They have built out their own CDN, they don’t a commercial one always. Figuring out which piece of content is coming from the CDN or not will be nearly impossible. Plus MSN is behind a ton of load balancers and hundreds or thousands of servers. CDN is webservers, they just cache content and deliver it closer to the end user. Mainly, it’s just Apache or IIS.
I’m just curious, what is the point of this? It seems if you want to learn more about this, you could pick a smaller target. You might as well be trying to figure out how Google Works