Big Data Meets NVM: Accelerating Big Data Processing with Non-Volatile Memory (NVM)

Big Data Meets NVM: Accelerating Big Data Processing with Non-Volatile Memory (NVM)

Thursday, May 23
4:00 PM - 4:40 PM
Scarlet Oak

Advanced Big Data Processing frameworks have been proposed to harness the fast data transmission capability of Remote Direct Memory Access (RDMA) over high-speed networks such as InfiniBand, RoCEv1, RoCEv2, iWARP, and OmniPath. However, with the introduction of the Non-Volatile Memory (NVM) and NVM express (NVMe) based SSD, these designs along with the default Big Data processing models need to be re-assessed to discover the possibilities of further enhanced performance. In this talk, we will present, NRCIO, a high-performance communication runtime for non-volatile memory over modern network interconnects that can be leveraged by existing Big Data processing middleware. We will show the performance of non-volatile memory-aware RDMA communication protocols using our proposed runtime and demonstrate its benefits by incorporating it into a high-performance in-memory key-value store, Apache Hadoop, Tez, Spark, and TensorFlow. Evaluation results illustrate that NRCIO can achieve up to 3.65x performance improvement for representative Big Data processing workloads on modern data centers.

Presentation Video


Dipti Shankar
PhD Student
The Ohio State University, Columbus, OH, USA
Dipti Shankar is a Ph.D. Candidate at the Department of Computer Science and Engineering at The Ohio State University. She is currently a Graduate Research Associate at the Network-Based Computing Lab (NOWLAB) working under Dr. Dhabaleswar K. (DK) Panda and Dr. Xiaoyi Lu. Her research interests include high-performance networking and storage media for Big Data middleware, including, Remote Direct Memory Access (RDMA) aware, non-volatile memory technologies, and memory-centric storage systems. At NOWLAB, she has been working on assisting with the research and development of RDMA-based accelerations for Apache Spark, Apache Hadoop, and Memcached, which are publicly available at ( More details about Dipti are available at