Related Readings: For Further Information

This is a pretty broad topic. Yet, most of the information available about it comes from IT vendors. I’ve tried to organize a few useful articles that are available. Please see the class website page for this day, as I will write an overview that provides background for the links below, and for many others that are available. Please read a few of them (whichever strike your fancy – you don’t have to read them all), and I’ll fill you in on the things I want you to know during the class lecture. 

Testing Internet Infrastructure Capacity
MD254WEBSITE: Examine this site provided by Keynote Systems that monitors the system performance at junctures along the Internet (http://www1.internetpulse.net/).

Economic Impact of Capacity Failure
MD254WEBSITE: “The Economic Impacts of Unacceptable Web-Site Download Speeds,” Zona Research, 1999 (http://www.keynote.com/downloads/whitepapers/economic_impact_of_downloadspeed.pdf)
MD254WEBSITE: “Scalability Planning: Preparing for Growth and Change,” IBM, (http://www-3.ibm.com/e-business/doc/content/resource/pdf/48160.pdf)

Overview of Capacity Management
MD254WEBSITE: “An Overview of Capacity Management Techniques and Tools for E-Services,” Witt, M, and G.R. Heim

Companies Involved in the Capacity Monitoring Industry
MD254WEBSITE: “Market Overview Update: Web Monitoring Services,” J.-P. Garbani, Giga Information Group, September 23, 2002 (http://www.keynote.com/downloads/articles/evolutionofwebmonitoring.pdf)
MD254WEBSITE: “Testing Services: Visionary Strategies and Solution for Successful e-Business Deployment,” IBM, (http://www-3.ibm.com/e-business/doc/content/resource/pdf/26546.pdf)

Web Site Load Testing
MD254WEBSITE: “The Science of Web Site Load Testing,” A. Savoia, Keynote Systems, 2000 (http://www.keynote.com/downloads/whitepapers/science_of_loadtesting.pdf)
MD254WEBSITE: “Trade Secrets from a Web Testing Expert,” A. Savoia, STQE Magazine, (http://www.keynote.com/downloads/articles/tradesecrets.pdf)
MD254WEBSITE: “Web Load Test Planning,” A. Savoia, STQE Magazine, March/April 2001 (http://www.keynote.com/downloads/articles/webloadtestplanning.pdf)
MD254WEBSITE: “Enterprise Load Testing for Web Applications,” Newport Group (http://www.keynote.com/downloads/whitepapers/newportgroup.pdf)

Visions of Future Capacity Sourcing
MD254WEBSITE: “e-Sourcing: IT on Demand,” Chief Executive Magazine, February 2002, (http://www-3.ibm.com/e-business/doc/content/resource/pdf/52115.pdf)

Academic Work on E-Business Capacity
MD254WEBSITE: “On the Quantification of E-Business Capacity,” M. Goldszmidt, et al., EC’01, Association for Computing Machinery, October 14-17, 2001. [How to Obtain: Via BC Library Link to ACM Journals]

Note: The following reading list looks like a lot, but most of the articles are actually in the 1-4 page range, with the remainder in the 8-10 page range, so in total it isn't that much. 

Prior to (and after) a website or other e-Service is deployed, managers should analyze the service process to ensure that the service will not fail under the expected demand scenarios. Failure to use such tools can lead to failed web applications, as discussed in the following article:

"Making E-Business Work," Newport Group, Inc., 2001. (Paper Index on Mercury Interactive) or (Download from Mercury Interactive) or (Download from Class Website)

Several capacity management strategies are available for making sure that expected service demands will be able to be met by the service process. The tools available for capacity management of e-Services differ substantially based on a couple of factors. First, the amount one is willing to pay for a tool will determine the type of tool available. Second, a desire for a higher (or lower) level of reliability from an e-Service will determine which tools to use.

For smaller businesses, most capacity management tools revolve around ISP-based tools that report on the historical demands experienced by a web site. We review several different types of tools available for small to large businesses in the following paper on capacity management tools: 

Witt, M., and G. R. Heim, "An Overview of Capacity Management Techniques and Tools for E-Services" 

As e-Service customer demands grow and as the need for better assurance about the reliability of an e-Service grows, managers tend to want to know that the way in which an e-Service system's capacity is tested is similar to the nature of the service demands it will see in the real world. For example:

"RadView Tests MSNBC's Olympic Mettle"

"How MSNBC Beefed Up Its Web Site for the Olympics"

One possible approach for testing the capacity of a web site (or any other e-Service) before it goes live is to have actual humans within your company test the site out. Essentially, humans could generate demand loads for the site until the site fails in the test environment. This approach has numerous shortcomings, as you can imagine after reading the above articles. For example, using this approach, each human at each computer would create a certain set of demands for digital content delivered through their browser (or other delivery device). As a result, if you would want to ensure that your e-Service could successfully service 3000 people simultaneously, you would simultaneously have 3000 of your employees on their 3000 desktop computers make requests of the service process. Obviously, it would be difficult to control these 3000 individuals to test certain features of your e-Service system. As a result, most companies would not use such a testing process to determine the realizable capacity of their service system.

Instead, large companies will use automated tools for capacity management.  Basically, a manager would set up the real website’s service-process configuration, test the site with “real” (actually simulated) demand loads generated by a demand-load simulator, and after adjusting the site’s performance, make the site go into operation. The Mercury Interactive articles linked to below lay out various issues that are important in (i) load testing before going live, and (ii) monitoring performance after going live. 

In general, tools for testing large websites can be classified as either "active" or "passive" tools:

"New Frontiers in Web Performance Monitoring," META Group, 2001. (Paper Index on Mercury Interactive) or (Download from Mercury Interactive) or (Download from Class Website)

The major applications for analyzing website capacity fall into either (i) software testing applications, or (ii) testing/monitoring services. The following white papers describe the suite of tools offered by Mercury Interactive (which are similar to those of other companies).

This paper provides an overview of the full offering of software applications and services:

"Managing Web Application Performance Beyond 2000," Newport Group, Inc., 2000. (Paper Index on Mercury Interactive) or (Download from Mercury Interactive) or (Download from Class Website)

The following article describes the typical software web testing applications and what they do:

"The Complete Testing Solution for E-Business" (Paper Index on Mercury Interactive) or (Download from Mercury Interactive) or (Download from Class Website)

The following articles provide information about the types of application performance management services available for web testing, and the performance insights that can be identified:

"ActiveWatch: Challenging Industry Perceptions of Web Performance", Mercury Interactive. (Paper Index on Mercury Interactive) or (Download from Mercury Interactive) or (Download from Class Website)

"Common Web Site Problems Identified by ActiveTest," Mercury Interactive.
(Paper Index on Mercury Interactive) or (Download from Mercury Interactive) or (Download from Class Website)

"Mercury Interactive's Top 10 Performance Management Tips," Mercury Interactive. (Paper Index on Mercury Interactive) or (Download from Mercury Interactive) or (Download from Class Website)

If you are interested in additional examples of such technology by other companies, the links found at the bottom of this page present similar tools. 

If you are interested in information about the total market size for web testing tools and the major vendors, the following paper can provide some information:

(Optional) "Annual Load Test Market Summary and Analysis," Newport Group, Inc., 2001. (Paper Index on Mercury Interactive) or (Download from Mercury Interactive)

Another approach – not considered here – is to formally model the e-Service system using OR-based mathematical tools for queueing networks. While academically interesting, this approach might not do you much good in a turbulent e-Service competitive environment, and might require you to employ someone with a PhD in Operations Research or Computer Science who knows queueing. If you are interested in this approach, I suggest you buy a book by Daniel A. Menasce and Virgilio A. F. Almeida, either Capacity Planning for Web Performance (1998) or Scaling for E-Business (2000). I'll explain these approaches in class.

Major Vendors

Compuware
http://www.compuware.com/

Empirix
http://www.empirix.com/
http://www.empirix.com/Empirix/web+test+monitoring/products/e-test+suite.html

Gomez Networks
http://www.gomeznetworks.com/

Loadtesting.com
http://www.loadtesting.com/compare.asp

Mercury Interactive
http://www.mercuryinteractive.com/

Radview
http://www.radview.com/

Radware
http://www.radware.co.il/

Rational Software
http://www.rational.com/

Segue
http://www.segue.com/

Softlight Technologies 
http://www.softlight.com/

Industry Sites for Web Application Testing and Capacity Management

“Capacity Planning,” Microsoft TechNet
http://www.microsoft.com/technet/ecommerce/capplan.asp?a=printable

Software Testing and Quality Engineering Magazine
http://www.stqemagazine.com/

StickyMinds.com
http://www.stickyminds.com/

Performance Benchmarks

ECperf
http://java.sun.com/j2ee/ecperf/

ECperf Results
http://ecperf.theserverside.com/ecperf/

Articles on Capacity Management, Load Testing, etc.
“How do the large web sites handle the load of millions of visitors per day?” 
http://www.howstuffworks.com/question342.htm

Butler, J., “E-Performance Management Gains Attention at CMG,” Software Magazine, May 8, 2000.
http://www.softwaremag.com/archive/2000feb/CMG.html

Tools for Load Testing and Web Performance Monitoring
GomezNetworks Methodology
http://www.gomeznetworks.com/mktg/method/method1.asp

“Rational Software Introduces Rational SiteLoad; Industry’s First Web-Based Load Testing Product,” Press Release, August 28, 2000.
http://www.rational.com/news/press/pr_view.jsp?ID=7978
Data Sheet: SiteLoad - http://www.rational.com/media/news/presskit/siteload.pdf
http://www.rational.com/products/siteload/readme.jsp

“What is Web Performance Monitoring?”
http://www-svca.mercuryinteractive.com/products/monitoring/
Product: ActiveWatch - http://www-svca.mercuryinteractive.com/products/activewatch/
Product: Topaz - http://www-svca.mercuryinteractive.com/products/topaz/

Rapoza, J., “SilkPerformer Tests Web-Site Fabrics,” PC Week, July 29, 1999.
http://www.zdnet.com/products/stories/reviews/0,4161,410219,00.html
Schindler, E., “Smooth Site Testing,” Sm@rtPartner, September 11, 2000.
http://www.segue.com/html/s_news/recent_articles/SmartPartner 9.00.pdf
Product: SilkPerformer - http://www.segue.com/html/s_solutions/s_performer/s_performer.htm
Product: EConfidenceScale - http://www.segue.com/html/s_solutions/s_hosted/scale_home.htm

Hendrick, K., and S. D. Hendrick, “RadView Software’s WebLoad: Integrated Testing for the Web,” IDC Bulletin #21616, February 2000.
http://www.radview.com/products/papers/IDC/paper.asp
Product: WebLoad - http://www.radview.com/products/webload.htm
Product: WebLoad Resource Manager - http://www.radview.com/products/webloadmgr.htm

Capacity Analysis Tools for Period-by-Period Demand and Demand Load Summary

Web Testing and Monitoring Newsletter – Comparison of Tools
http://members.tripod.co.uk/wtmn/

PerfMan for WebServers
http://www.perfman.com/sellSheets/WebServer/PerfManforWebServers.html

Tutorials on almost any type of capacity you might be interested in managing
http://www.iec.org/online/tutorials/

User Tracking
http://www.arsdigita.com/books/panda/user-tracking

So you want to run your own server 
http://www.arsdigita.com/books/panda/server

Related Readings: Traditional (Person-to-Person) Services

Fitzsimmons & Fitzsimmons, Service Management: Operations, Strategy, and Information Technology, Irwin/McGraw-Hill, 2001, Chapter 12 ("Queueing Models and Capacity Planning"), Chapter 13 ("Managing Capacity and Demand")

Haksever, et al., Service Management and Operations, Prentice Hall, 2000, Chapter 11 ("Managing Demand and Supply in Services"), Chapter 15 ("Forecasting Demand for Services")