Collaborative Research: CCF Core: Small: User-transparent Data Management for Persistence and Crash-consistency in Non-volatile Memories

Non-volatile memory is a type of computer memory that can retain stored data upon a power loss or system crash. Due to its large capacity and low energy footprint compared to traditional volatile memory, non-volatile memory has long been envisioned as an ideal solution for building large-scale, cost-effective, energy-efficient, and recoverable applications in many critical domains, including high-performance computing, machine learning, and embedded systems. Although non-volatile memory is available as commercial memory chips and offers numerous promises, it has not yet been widely adopted in production systems. The major obstacle is the difficulty to ensure that data is timely and correctly written to non-volatile memory, allowing it to be restored to a consistent state after a crash. Currently, application developers carry the burden of porting legacy applications to non-volatile memory, which is tedious and error-prone. This project seeks to establish a generic framework for user-transparent persistence and crash consistency that allows unmodified legacy applications to run efficiently and correctly with non-volatile memory. The success of this project will help unleash the full potential of non-volatile memory and make it easier to adopt. The research will also provide valuable insights into data management in future hybrid, disaggregated memory systems. In addition, this project involves mentoring Ph.D. students, engaging minority students, course development, and K12 outreach activities. This project integrates non-volatile memory into the page/buffer cache in memory management – i.e., an abstraction that bridges the view of byte-addressable memory and a backing memory device -- to provide persistence and crash consistency to user-space programs with no or little user involvement. The challenges lie in 1) how to intercept program updates and redirect them to non-volatile memory for persistence; 2) how to properly order the updates and ensure update atomicity to guarantee crash consistency; 3) how to efficiently integrate non-volatile memory into page/buffer cache management without incurring noticeable overhead or performance degradation. This project addresses these challenges by focusing on persisting three types of program data – file-backed data, dynamically allocated application memory, and program metadata for virtual memory management, such as page tables, and exploring various software and hardware techniques, such as copy-on-write, undo logging, shadow paging, and extended page tables, for each data type to achieve efficient crash consistency. This project advances the understanding of hybrid memory management for volatile and non-volatile memories while simultaneously achieving high usability, good backward compatibility, and high efficiency.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

非挥发存储器是一种计算机内存，可以在功率损耗或系统崩溃时保留存储的数据。由于与传统的挥发性记忆相比，由于其容量较大和能量占地面积低，因此长期以来一直将非易失性记忆作为在许多关键领域（包括高性能计算，机器学习和嵌入式系统）中构建大规模，具有成本效益，能效和可回收应用的理想解决方案。尽管非易失性记忆可作为商业记忆芯片提供，并提供了许多承诺，但在生产系统中尚未广泛采用。主要障碍是很难确保数据及时，正确地写入非易失性内存，从而使其在崩溃后恢复到一致的状态。当前，应用程序开发人员将移植传统应用程序的燃烧燃烧到非易失性存储器上，这是乏味且容易出错的。该项目旨在为用户透明的持久性和崩溃一致性建立一个通用框架，该框架允许未经修改的旧应用程序使用非挥发性内存有效，正确地运行。除非有非易失性记忆的全部潜力并使采用更容易采用，否则该项目的成功将有所帮助。该研究还将在未来的混合，分开的内存系统中对数据管理提供宝贵的见解。此外，该项目涉及心理化博士学位。学生，吸引少数族裔学生，课程发展和K12外展活动。该项目将非易失性存储器集成到内存管理中的页面/缓冲区缓存中 - 即，一个桥接字节 - 可调件内存和背景内存设备的视图的抽象，以提供无需或少量用户参与的用户空间程序的持久性和崩溃的一致性。挑战在于1）如何拦截程序更新并将其重定向到非易失性记忆以保持持久性； 2）如何正确订购更新并确保更新原子性保证崩溃一致性； 3）如何有效地将非易失性内存整合到页面/缓冲缓存管理中，而不会产生明显的开销或性能退化。该项目通过着重于持续存在三种类型的程序数据来解决这些挑战：文件支持的数据，动态分配的应用程序内存以及用于虚拟内存管理（例如页面表）的程序元数据，并探索各种软件和硬件技术，例如复制，编写纸条，删除，撤消记录，阴影要求，阴影页面和扩展的页面类型，以实现每种数据类型，以实现每种数据类型的效果一致性。该项目推动了对混合记忆管理的理解，同时又可以实现高可用性，良好的向后兼容性和高效率。该奖项反映了NSF的法定使命，并通过使用基金会的知识分子优点和更广泛的审查标准评估来诚实地获得支持。