Seminar 7: On-chip interconnection networks
As we build increasingly complex parallel systems (e.g. CMPs and MPSoCs) one of the greatest challenges is in providing the interconnection networks that permit the system components to communicate. These must be must be high-performance (low-latency and high-bandwidth), flexible, scalable, simple to design with and power efficient. Beyond simple buses, the idea of an on-chip network illuminates a vast design space for building scalable interconnects.
In this seminar we will discuss the requirements that chip-multiprocessors place on the on-chip network and the trade-offs involved in designing on-chip routers and networks. As we have seen with many other aspects of chip-multiprocessor design, the trade-offs when designing on-chip are very different to those seen in larger parallel machines. Although, much of the existing underlying theory is still relevant.
Lecture Notes
LECTURE NOTES PDF (4up for printing)
Reading group material
"Route Packets, Not Wires: On-Chip Interconnection Networks", William J. Dally and Brian Towles, in Proc. ACM IEEE Design Automation Conference (DAC), pp 684-689, 2001. PDF
"Research challenges for on-chip interconnection networks", John D Owens, William J Dally, Ron Ho, D N Jayasimha, Stephen W Keckler, and Li-Shiuan Peh. IEEE Micro, 27(5):96-108, 2007. PDF
Additional reading
Deadlock issues in interconnection networks
On-Chip Networks and Fault Tolerance
Optical On-Chip Networks & Wireless On-Chip Networks
(Carbon nanotubes and Wireless on-chip Networks)
Interconnects for chip-multiprocessors
"Design Tradeoffs for Tiled CMP On-Chip Networks", James Balfour and William Dally. Proc. of the Intl. Conf. on Supercomputing (ICS) 2006 PDF
"Express Cube Topologies for On-Chip Interconnects", B. Grot, J. Hestness, S.W. Keckler, and O. Mutlu, Proceedings of the 15th International Symposium on High-Performance Computer Architecture (HPCA), February, 2009. PDF
Multidrop express channels (MECS) vs. Flattened Butterfly topology
"Design and Evaluation of Hierarchical On-Chip Network Topologies for next generation CMPs", Reetuparna Das, Soumya Eachempati, Asit K. Mishra, Vijaykrishnan, Chita R. Das, in Proc. Intl. Symp. on High Performance Computer Architecture (HPCA), Raleigh, North Carolina, 2009. PDF
"Interconnections in Multi-Core Architectures: Understanding Mechanisms, Overheads and Scaling", Rakesh Kumar, Victor Zyuban and Dean M. Tullsen, In Proc. Intl. Symp. on Computer Architecture (ISCA), 2005. PDF
- A separate address and snoop bus are discussed here, two buses are needed as both are unidirectional and pipelined.
Rings and the coherence protocol
"Coherence Ordering for Ring-based Chip Multiprocessors", Marty and Hill, MICRO 2006 PDF
Crossbar-based interconnects
"A study of the On-Chip Interconnection Network for the IBM Cyclops64 Multi-Core Architecture" PDF
The MIT RAW Processor:
"Scalar Operand Networks: Design, Implementation, Analysis", Michael Bedford Taylor, Walter Lee, Saman Amarasinghe, and Anant Agarwal. MIT/LCS Technical Memo LCS-TM-645, June 2004. In IEEE TPDS, 2005. PDF
Earlier Work:
"Virtual-Channel Flow Control", William J. Dally, In Proc. of the Intl. Symp. on Computer Architecture (ISCA), May 1990.
"The Sensitivity of Communication Mechanisms to Bandwidth and Latency", F. Chong, R. Barua, F. Dahlgren, J. Kubiatowicz, A. Agarwal, In Proc. Intl. Symp. on High-Performance Computer Architecture (HPCA), 1998. PDF
Books
"Principles and Practices of Interconnection Networks", William J. Dally and Brian Towles, Morgan Kaufmann, 2004. Google Books
"Interconnection Networks: An Engineering Approach", J. Duato, S. Yalamanchili and L. Ni, Morgan Kaufmann, 2003. Google Books
"Networks on Chips", G. De Micheli and L. Benini, Morgan Kaufmann, 2006. Google Books
Other Resources
Netmaker: A synthesizable NoC Libary
International Symposium on Networks-on-Chip
Workshop on on- and off-chip interconnection networks for multicore systems (includes videos of talks and slides)
Recent Processors
- 8-core Intel Xeon processor (ring interconnect):
"A 1.2 TB/s on-chip ring interconnect for 45nm 8-core enterprise Xeon® processor", Park et al, ISSCC'10. PDF