SciNet Supercomputing Infrastructure and Specifications
SciNet represents a massive leap in computational power, designed to tackle complex scientific challenges through high-performance computing. By leveraging a collaboration between the University of Toronto, IBM Corp, and Compute Canada, SciNet has established one of the most powerful and energy-efficient research environments in the world. The facility is specifically engineered to support diverse workloads, from tightly coupled simulations to massive general-purpose data processing.
Key Facts
- Global Scale: The General Purpose Cluster is the largest Intel processor-based IBM installation globally.
- Total Capacity: Combined theoretical peak performance reaches 366 TFlops.
- Energy Efficiency: Non-computing power consumption is kept under 20%, significantly lower than the 33% industry average.
- Sustainable Cooling: The facility utilizes a "free-air" cooling system to leverage the cold Canadian climate.
Compute Cluster Architectures
SciNet utilizes two distinct compute clusters, each optimized for specific types of computational tasks to ensure maximum efficiency and performance.
Tighly-coupled Capability System (TCS)
The TCS is designed for applications that require intense communication between nodes. It consists of 104 POWER6 nodes, with each node featuring 32 cores running at 4.7 GHz and 128 GiB of RAM. This configuration provides a theoretical peak performance of 60 TFlops and a total of 14 TiB of RAM.
General Purpose Cluster (GPC)
The GPC is a massive installation comprising 3,780 IBM System x iDataPlex dx360 M3 nodes distributed across 45 racks. Each node is equipped with two quad-core Intel Nehalem (Xeon 5540) processors running at 2.53 GHz, resulting in a total of 30,240 cores. The cluster offers a theoretical peak of 306 TFlops and 60 TiB of RAM.
Connectivity within the GPC is handled via Gigabit Ethernet, while 864 specific nodes utilize DDR InfiniBand—a high-speed, low-latency communication link—to support message passing applications.
[ไม่มีภาพประกอบ]The Green Datacenter and Infrastructure
Given that a primary research focus for SciNet is the study of climate change and global warming, the physical infrastructure was designed to be one of the greenest datacenters in existence. The computer room occupies 3,000 square feet (280 m²) on a raised floor.
Cooling is managed by a 735-ton chiller and cooling towers. To reduce energy consumption, the system employs "free-air" cooling; when external temperatures drop below a specific threshold, the chiller switches to use the naturally cold Canadian air. This innovation allows SciNet to allocate less than 20% of its energy to cooling and non-computing overhead, compared to the 33% typically seen in traditional datacenters.
The total energy requirement for the system is equivalent to the power needed for four thousand homes.
[ไม่มีภาพประกอบ]Technical Specifications Summary
| Feature | Tighly-coupled Capability System (TCS) | General Purpose Cluster (GPC) |
|---|---|---|
| Processor Type | POWER6 | Intel Nehalem (Xeon 5540) |
| Total Cores | 3,328 | 30,240 |
| Clock Speed | 4.7 GHz | 2.53 GHz |
| Theoretical Peak Performance | 60 TFlops | 306 TFlops |
| Total RAM | 14 TiB | 60 TiB |
| Primary Interconnect | N/A | Gigabit Ethernet / DDR InfiniBand |
Frequently Asked Questions
What is the difference between the TCS and GPC clusters?
The Tighly-coupled Capability System (TCS) is optimized for high-intensity, interconnected tasks using POWER6 nodes, while the General Purpose Cluster (GPC) is a larger-scale Intel-based system designed for broader computational workloads.
How does SciNet reduce its environmental impact?
SciNet uses a "free-air" cooling system that utilizes the cold Canadian climate to reduce reliance on mechanical chillers, keeping non-computing energy overhead below 20%.
What is the significance of DDR InfiniBand in the GPC?
DDR InfiniBand is used in 864 nodes to provide the high-speed and low-latency communication necessary for message passing applications, which are critical for complex scientific simulations.
Who were the partners involved in the creation of SciNet?
The supercomputer venture was a collaborative effort between the University of Toronto (SciNet), IBM Corp, and Compute Canada.
How much power does the SciNet system consume?
The system consumes an amount of energy equivalent to what would be required to power four thousand homes.