Dr. Ningfang Mi gave a talk in Room #109 in Mingli Builiding of the school on Apr. 25, 2018. The title of his talk is “Efficiency Improvement for Big Data Processing Platforms”. The talk was hosted by Professor Genlin Ji. Dr. Mi is an Associate Professor in Department of Electrical and Computer Engineering (ECE) at Northeastern University. She was the chair professor of the school of Computer Science and Technology.
Efficiently managing resources to improve the throughput in a large-scale cluster has become a crucial problem with the explosion of data processing applications in recent years. Hadoop YARN and Mesos, as two universal resource management platforms, have been widely adopted in the commodity cluster for co-deploying multiple data processing frameworks, such as Hadoop MapReduce and Apache Spark. In this talk, they present our new YARN scheduler, named HaSTE, which can effectively reduce the makespan of MapReduce jobs in YARN by leveraging the information of requested resources, resource capacities, and dependency between tasks. They also present an opportunistic and efficient resource allocation approach, named OPERA, which breaks the barriers among the encapsulated resource containers by leveraging the knowledge of actual runtime resource utilizations to re-assign opportunistic available resources to the pending tasks.
After the talk, Dr. Mi discussed with the audiences and answered their questions kindly. The audiences got much inspiration from the talk and the discussion.
Short Bio: Ningfang Mi is an Associate Professor in Department of Electrical and Computer Engineering (ECE) at Northeastern University since 2009. Dr. Mi graduated with a B.S. in Computer Science from Nanjing University, China in 2000 and a M.S. in Computer Science from the University of Texas at Dallas in 2004. She received her Ph.D in Computer Science from the College of William and Mary in 2009. Her research interests include cloud computing, big data processing, resource management, capacity planning, MapReduce/Hadoop scheduling, performance evaluation, simulation and virtualization. Dr. Mi was a recipient of the 2015 National Science Foundation (NSF) CAREER Award, the 2014 Air Force’s Young Investigator Research (YIP) Award and the 2010 IBM Faculty Award. She is the director of the Northeastern University Computer Systems Research Laboratory (NUCSRL) at Northeastern University.