Internet-scale computing

Internet use and content has grown dramatically, fueled by global reach, mobile device access, and user-generated Web content, including large audio and video files. More of the world population is looking to the mobile Web to fulfill basic economic needs. To meet this challenge, Web developers need to adopt new methods to address significant applications such as search, social networking, collaborative innovation, virtual worlds and mobile commerce.

This growth, along with fundamental changes in computer architecture and increases in network capacity, present new challenges and opportunities to the study of computer science. Computational tasks often need to be broken into hundreds or even thousands of small pieces to run across many servers simultaneously. Parallel programing techniques are also used for complex scientific analysis such as gene sequencing and climate modeling.

This initiative aims to improve students' knowledge of highly parallel computing practices and prepare them to better address the emerging paradigm of large-scale distributed computing. IBM and Google are teaming up to provide hardware, software and services to augment university curricula and expand research horizons. With their combined resources, the companies hope to lower the financial and logistical barriers for the academic community to explore this emerging model of computing.

The University of Washington is the first to join the initiative. A small number of universities will also pilot the program, including Carnegie-Mellon University, Massachusetts Institute of Technology, Stanford University, the University of California at Berkeley, and the University of Maryland. In the future, the program will be expanded to include additional researchers, educators and scientists.

The heart of the project is a large cluster of several hundred computers (a combination of Google machines and IBM® BladeCenter® and System x servers) that is planned to grow to more than 1,600 processors. Students will access the cluster through the Internet to test their parallel programming projects. The cluster is powered with open source software, including:

• Linux (Fedora)
• XEN systems virtualization
• Hadoop workload scheduling

In addition, the project includes these resources:

• Google Code for Educators. This site contains university curriculum for distributed systems developed by Google and the University of Washington. It focuses on massively parallel computing techniques, and is available through Creative Commons licensing.
  
  
• MapReduce Tools for Eclipse, which is open source software designed by IBM to help students develop programs for clusters running Hadoop. The software works with Eclipse, an open source development platform, and is available as part of Hadoop 0.14.1.
  
  
• A Web site to encourage collaboration among universities in the program, built with Web 2.0 technologies from the IBM Innovation Factory.

Computer science students at the University of Washington developed one program that scans voluminous Wikipedia edits to identify spam and one that organizes global news articles by geographic locations. These projects demonstrate the potential of the Internet-scale computing initiative.

IBM, Google and cluster computing