OAC Core: Towards Zero-Carbon Data Movement at the HPC and Cloud Data Centers with GreenDataFlow

As commercial and scientific applications generate data at increasingly high rates, the carbon footprint associated with data movement is becoming a critical concern, particularly for High-Performance Computing (HPC) and Cloud data centers. While there is substantial research focusing on power management techniques at the hardware level and lower networking stack layers during data transfers, little attention has been paid to energy-saving measures at the application layer for computing systems such as servers, HPC centers, and Cloud data centers during network data transmission. The existing strategies in this realm are either prohibitively expensive, impractical in the short term, or sacrifice performance in pursuit of increased energy efficiency. This project develops an innovative application-layer solution, which is cost-effective, practical for immediate deployment, and importantly, does not compromise performance while boosting energy efficiency. It possesses the ability to adaptively fine-tune several application-layer and kernel-layer transfer parameters, thereby guaranteeing efficient utilization of computing and networking resources. This, in turn, minimizes data transfer energy consumption without undermining end-to-end performance. This revolutionary approach to energy-efficient data transfers underscores the innovation and transformative potential of this project. The models, algorithms, and tools developed within this project are poised to augment performance and reduce power consumption during end-to-end data transfers, potentially saving gigawatt hours of energy and contributing millions of dollars in savings to the US economy. Furthermore, this project seeks to permeate research insights across all tiers of education. The well-structured research activities promise to benefit for K-12, undergraduate, and graduate students alike, fostering their academic growth and nurturing future scientists in this critical field.This project develops novel application-layer models, algorithms, and tools for (1) prediction and tuning of the best cross-layer transfer parameter combination for energy-efficient and high-performance data transfers at the HPC and Cloud data centers; (2) a deep reinforcement learning-based approach that can adapt to the dynamically changing conditions in a wide range of network and end system configurations; (3) accurate estimation of the accompanying network device power consumption due to changing data transfer rate on the active intra- and inter-data center network links and dynamic readjustment of the transfer rate to balance the energy vs. performance ratio; and (4) a suite of service level agreement based energy-efficient transfer algorithms to the HPC administrators and Cloud service providers for dynamically adjustable performance and energy efficiency goals. The evaluation and validation of the proposed models and algorithms are performed in realistic scenarios in collaboration with the HPC Center at Texas Tech University and the Distributed Cloud Management group at IBM. The research outcomes of this project will fill a significant gap in the data transfer energy efficiency in HPC and Cloud data centers. This project's eventual goal is to translate the research activities into robust, production-quality software libraries that will reduce the carbon footprint of data movement for a range of user communities dealing with large amounts of data. The project will enable wider broader impacts through the development of graduate and undergraduate curricula, K-12 outreach programs, summer boot camps, the recruitment of minority groups, and broadening participation in computing.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

随着商业和科学应用以越来越高的速率生成数据，与数据流动相关的碳足迹正在成为一个关键问题，尤其是对于高性能计算（HPC）和云数据中心。尽管有大量研究重点是在硬件级别上的电源管理技术，并且在数据传输过程中降低了网络堆栈层，但在应用程序系统（例如服务器，HPC中心和云数据中心）网络数据传输过程中，在应用程序层的省力措施方面几乎没有关注。该领域中的现有策略要么是昂贵的，在短期内不切实际，要么是为了提高能源效率而牺牲绩效。该项目开发了一种创新的应用程序层解决方案，该解决方案具有成本效益，可以立即部署，并且重要的是，在提高能源效率的同时不会损害性能。它具有自适应微调几个应用程序层和内核层传输参数的能力，从而确保有效利用计算和网络资源。反过来，这将数据传输能源消耗降至最低，而不会破坏端到端的性能。这种革命性的节能数据传输的方法强调了该项目的创新和变革潜力。该项目中开发的模型，算法和工具有望提高性能并降低端到端数据传输期间的功耗，从而节省了Gigawatt小时的能源，并为美国经济节省了数百万美元的储蓄。此外，该项目旨在渗透到所有教育层次的研究洞察力。结构良好的研究活动有望为K-12，本科和研究生而受益，促进他们的学术成长并在这个关键领域中培养未来的科学家。该项目开发了新颖的应用层模型，算法和工具（1）最佳跨层传输参数的预测和调整，以供跨层次传递参数和高级效率数据和高级数据集中的数据集中，该数据集中的数据均为杂货。 （2）一种基于强化的学习方法，可以适应各种网络和最终系统配置的动态变化条件； （3）由于在主动内部和数据中心中心网络链路上的数据传输速率变化而导致随附的网络设备功耗的准确估算以及转移速率的动态调整以平衡能量与性能比率； （4）基于服务水平协议的一套基于节能的转移算法向HPC管理员和云服务提供商提供动态可调节的性能和能源效率目标。与德克萨斯理工大学的HPC中心和IBM的分布式云管理小组合作，在现实的情况下对提出的模型和算法进行评估和验证。该项目的研究结果将填补HPC和云数据中心数据传输能效的显着空白。该项目的最终目标是将研究活动转化为强大的生产质量软件库，这些库将减少一系列处理大量数据的用户社区的数据移动的碳足迹。该项目将通过开发研究生和本科课程，K-12外展计划，夏季新兵训练营，少数群体的招聘以及计算的参与来实现更广泛的影响。该奖项反映了NSF的法定任务，并通过该基金会的知识分子优点和广泛的影响来评估NSF的法定任务。