One-Stop IT Service Provider - Announcing Azure Managed Lustre for your HPC and AI workloads

Posted on July 10, 2023August 4, 2023 by admin

Announcing Azure Managed Lustre for your HPC and AI workloads

Today, we are excited to announce the general availability of Azure Managed Lustre, a managed high performance parallel file system for high performance computing (HPC) and AI workloads. Azure Managed Lustre delivers the time-tested Lustre file system as a first party managed service on Azure. Long time users of Lustre on-premises can now leverage the benefits of a complete HPC solution, including compute and high performance storage, delivered on Azure. Users benefit from the high throughput and performance capabilities of Lustre as a seamless managed experience and can now focus on their business imperatives.

Need for Azure Managed Lustre

Lustre, one of the most popular distributed parallel filesystems in the HPC world has long been deployed on-premises serving scalable and high throughput storage needs of HPC workloads. As an open-source solution, it has enjoyed a thriving ecosystem of users and developers. With the recent explosive growth seen in generative AI and the need for a high throughput storage that keeps the expensive GPU cores from waiting on data, Lustre has found a renewed growth in its adoption. Customers in industries leveraging high performance computing are looking for a seamless end-to-end infrastructure encompassing compute, storage, and networking that is both fast to deploy and easy to use.

Azure Managed Lustre enables:

A managed PaaS experience for worry-free deployment and consumption of Lustre file system.
Reduced TCO with pay-as-you-go pricing.
Throughput options at different pricing levels for different workloads.
Easy deployment of independent clusters with predictable performance, eliminating the noisy neighbor problem commonly experienced in on-premises Lustre deployments.
Quick deployment in a few minutes, alleviating the need to plan months or years in advance and meeting compliance and data residency requirements.
Region of choice for deployments across many Azure regions, supporting compliance and data residency requirements.
Scalability to hundreds of petabytes (PB), with the associated durability, availability and cost benefits of tiering to Azure Blob storage.
Seamless integration with existing Azure compute services including Azure HPC services, Azure Machine Learning and Azure Kubernetes service.
Investment protection for customers’ existing automation and platform development when migrating from on-premises Lustre deployments.

Today, two persistent durable instances are available for customers to use. These are based on solid state drives (SSDs) which are differentiated by their performance option delivered for provisioned Tebibyte of capacity.

AMLFS Standard – 125 MB/s
AMLFS Premium – 250 MB/s

Users of Azure Managed Lustre can download the Lustre client packages from packages.microsoft.com for their desired Linux distribution and kernel version. Additionally, we will have support for HPC images prebuilt with Lustre client packages for Ubuntu – 18.04, 20.04, 22.04, and Alma 8.7.

Azure Managed Lustre is available today from the Azure portal in several regions. To keep up to date on what regions you can use, please refer to our products available by region page. To learn more about pricing, please visit our pricing page.

Customers are already seeing results with Azure Managed Lustre

Azure Managed Lustre serves diverse businesses and organizations who have the need for high throughput storage access within energy, life sciences, meteorology, automotive, healthcare, and public sector industries.

Azure Managed Lustre has you covered whether you are running tools to simulate 3D product models to accelerate time-to-market, serving thousands of users querying financial information on data analytics platforms, or building the next AI supercomputer for your generative AI solution.

When Petronas, a world leader in energy, needed to accelerate their upstream energy research with highly complex quantitative interpretation of seismic processing workloads that need large data sets to be fed to the compute VMs with very high throughput, they chose Azure Managed Lustre.

“Azure Managed Lustre is an outstanding file system that has been specifically crafted to meet the unique requirements of PETRONAS’ high performance computing (HPC) and AI workloads. It delivers exceptional performance, providing high storage bandwidth and seamlessly integrating with essential Azure services.”—Faizal B Zainol Alam, PETRONAS Digital General Manager

Meteomatics, a leader and disruptor in democratizing weather intelligence needed a high performance file system for their weather modeling and simulations, they picked Azure Managed Lustre for their needs. Azure Managed Lustre’s integration with Azure CycleCloud enables Meteomatics to deploy their HPC clusters quickly and programmatically allowing them to deliver value to their end customers looking for fast and consistent access to weather data.

“At Meteomatics, we rely on high performance computing (HPC) to run complex numerical weather prediction models, delivering accurate forecasts to our customers. By leveraging Lustre, a trusted file system in the HPC community, and the seamless deployment of LustreFS, we gain the high number of IOPS and throughput crucial for handling massive data volumes. Azure Managed Lustre accelerates our weather forecast simulations with a high spatial and temporal resolution, significantly reducing time-to-completion and enabling faster delivery of forecast data to our valued customers.“—Dr. Martin Fengler, CEO Meteomatics

“Duke Clinical Research Institute was looking for a scalable solution for their high I/O needs to support their analytics tools serving clinical research. Azure Managed Lustre addressed that need with a simple deployment model allowing them to focus on providing value to their customers quickly. I see the effortless simplicity and power of AMLFS as the ultimate choice for high-speed shared storage, empowering your analytics tools with unrivaled simplicity and versatility for both immediate and long-term needs.“—Emilio Power, Research Analytics Lead, SAS System Architect, Duke Clinical Research Institute

Using Azure Managed Lustre

Azure Managed Lustre is delivered in a hosted-on-behalf-of subscription, through a simple interface within the customer virtual network. Customers don’t have to be concerned about the internals of deploying, managing, and operating the Lustre file system including metadata servers/targets (MDS/MDT), management servers/targets (MGS/MGT) and object storage servers/targets (OSS/OSTs).

Diagram showing the Azure Managed Lustre File System service topology.

The Azure portal offers a simple interface to configure and deploy Azure Managed Lustre, where customers provide some inputs like resource group id, subnet ID, cluster size, and desired performance options. Users have the ability to monitor and observe their cluster performance with throughput, latency, and IOPS metrics.

Screen shot of Azure Managed Lustre preview registration page.

Screen shot of Azure Amanged Lustre in a demo state, showing performance mentrics.

Hierarchical storage management functionality in Azure Managed Lustre enables users to seamlessly transfer data between Azure Managed Lustre and Azure Blob. This empowers users to establish data archival, retention, and protection strategies. When dealing with large datasets, users can import only the data that is currently active and crucial for their data processing tasks into Azure Managed Lustre clusters, while archiving and retaining the remaining data in Azure Blob. This approach effectively minimizes runtime costs. Furthermore, by utilizing data tiering to Azure Blob, users can take advantage of its global presence and availability to create Azure Managed Lustre clusters in multiple regions as needed. The integration with Azure Blob also facilitates the accessibility of data through multiple protocols, such as network file systems (NFS), hadoop distributed file system (HDFS), and REST.

We are also very excited to announce the general availability of the CSI driver for Azure Managed Lustre. Now, containerized applications running on Azure Kubernetes Service can easily consume Azure Managed Lustre.