Use Amazon SageMaker Studio with a customized file system in Amazon EFS

Amazon SageMaker Studio is the newest web-based expertise for operating end-to-end machine studying (ML) workflows. SageMaker Studio presents a set of built-in improvement environments (IDEs), which incorporates JupyterLab, Code Editor, in addition to RStudio. Information scientists and ML engineers can spin up SageMaker Studio personal and shared areas, that are used to handle the storage and useful resource wants of the JupyterLab and Code Editor purposes, allow stopping the purposes when not in use to save lots of on compute prices, and resume the work from the place they stopped.

The storage sources for SageMaker Studio areas are Amazon Elastic Block Retailer (Amazon EBS) volumes, which supply low-latency entry to person knowledge like notebooks, pattern knowledge, or Python/Conda digital environments. Nonetheless, there are a number of situations the place utilizing a distributed file system shared throughout personal JupyterLab and Code Editor areas is handy, which is enabled by configuring an Amazon Elastic File System (Amazon EFS) file system in SageMaker Studio. Amazon EFS offers a scalable totally managed elastic NFS file system for AWS compute situations.

Amazon SageMaker helps routinely mounting a folder in an EFS quantity for every person in a website. Utilizing this folder, customers can share knowledge between their very own personal areas. Nonetheless, customers can’t share knowledge with different customers within the area; they solely have entry to their very own folder user-default-efs within the $HOME listing of the SageMaker Studio software.

On this put up, we discover three distinct situations that display the flexibility of integrating customized Amazon EFS with SageMaker Studio.

For additional data on configuring Amazon EFS in SageMaker Studio, discuss with Attaching a customized file system to a website or person profile.

Resolution overview

Within the first state of affairs, an AWS infrastructure admin needs to arrange an EFS file system that may be shared throughout the personal areas of a given person profile in SageMaker Studio. Because of this every person throughout the area can have their very own personal area on the EFS file system, permitting them to retailer and entry their very own knowledge and information. The automation described on this put up will allow new workforce members becoming a member of the info science workforce can rapidly arrange their personal area on the EFS file system and entry the mandatory sources to begin contributing to the continuing undertaking.

The next diagram illustrates this structure.

This state of affairs presents the next advantages:

Particular person knowledge storage and evaluation – Customers can retailer their private datasets, fashions, and different information of their personal areas, permitting them to work on their very own initiatives independently. Segregation is made by their person profile.
Centralized knowledge administration – The administrator can handle the EFS file system centrally, sustaining knowledge safety, backup, and direct entry for all customers. By organising an EFS file system with a non-public area, customers can effortlessly monitor and keep their work.
Cross-instance file sharing – Customers can entry their information from a number of SageMaker Studio areas, as a result of the EFS file system offers a persistent storage resolution.

The second state of affairs is expounded to the creation of a single EFS listing that’s shared throughout all of the areas of a given SageMaker Studio area. Because of this all customers throughout the area can entry and use the identical shared listing on the EFS file system, permitting for higher collaboration and centralized knowledge administration (for instance, to share widespread artifacts). This can be a extra generic use case, as a result of there isn’t any particular segregated folder for every person profile.

The next diagram illustrates this structure.

This state of affairs presents the next advantages:

Shared undertaking directories – Suppose the info science workforce is engaged on a large-scale undertaking that requires collaboration amongst a number of workforce members. By organising a shared EFS listing at undertaking degree, the workforce can collaborate on the identical initiatives by accessing and dealing on information within the shared listing. The info science workforce can, for instance, use the shared EFS listing to retailer their Jupyter notebooks, evaluation scripts, and different project-related information.
Simplified file administration – Customers don’t have to handle their very own personal file storage, as a result of they’ll depend on the shared listing for his or her file-related wants.
Improved knowledge governance and safety – The shared EFS listing, being centrally managed by the AWS infrastructure admin, can present improved knowledge governance and safety. The admin can implement entry controls and different knowledge administration insurance policies to take care of the integrity and safety of the shared sources.

The third state of affairs explores the configuration of an EFS file system that may be shared throughout a number of SageMaker Studio domains throughout the identical VPC. This enables customers from completely different domains to entry and work with the identical set of information and knowledge, enabling cross-domain collaboration and centralized knowledge administration.

The next diagram illustrates this structure.

This state of affairs presents the next advantages:

Enterprise-level knowledge science collaboration – Think about a big group with a number of knowledge science groups engaged on varied initiatives throughout completely different departments or enterprise items. By organising a shared EFS file system accessible throughout the group’s SageMaker Studio domains, these groups can collaborate on cross-functional initiatives, share artifacts, and use a centralized knowledge repository for his or her work.
Shared infrastructure and sources – The EFS file system can be utilized as a shared useful resource throughout a number of SageMaker Studio domains, selling effectivity and cost-effectiveness.
Scalable knowledge storage – Because the variety of customers or domains will increase, the EFS file system routinely scales to accommodate the rising storage and entry necessities.
Information governance – The shared EFS file system, being managed centrally, could be topic to stricter knowledge governance insurance policies, entry controls, and compliance necessities. This might help the group meet regulatory and safety requirements whereas nonetheless enabling cross-domain collaboration and knowledge sharing.

Conditions

This put up offers an AWS CloudFormation template to deploy the primary sources for the answer. Along with this, the answer expects that the AWS account by which the template is deployed already has the next configuration and sources:

Check with Attaching a customized file system to a website or person profile for added conditions.

Configure an EFS listing shared throughout personal areas of a given person profile

On this state of affairs, an administrator needs to provision an EFS file system for all customers of a SageMaker Studio area, creating a non-public file system listing for every person. We will distinguish two use instances:

Create new SageMaker Studio person profiles – A brand new workforce member joins a preexisting SageMaker Studio area and needs to connect a customized EFS file system to the JupyterLab or Code Editor areas
Use preexisting SageMaker Studio person profiles – A workforce member is already engaged on a particular SageMaker Studio area and needs to connect a customized EFS file system to the JupyterLab or Code Editor areas

The answer supplied on this put up focuses on the primary use case. We focus on tips on how to adapt the answer for preexisting SageMaker Studio area person profiles later on this put up.

The next diagram illustrates the high-level structure of the answer.

On this resolution, we use CloudTrail, Amazon EventBridge, and Lambda to routinely create a non-public EFS listing when a brand new SageMaker Studio person profile is created. The high-level steps to arrange this structure are as follows:

Create an EventBridge rule that invokes the Lambda operate when a brand new SageMaker person profile is created and logged in CloudTrail.
Create an EFS file system with an entry level for the Lambda operate and with a mount goal in each Availability Zone that the SageMaker Studio area is positioned.
Use a Lambda operate to create a non-public EFS listing with the required POSIX permissions for the profile. The operate can even replace the profile with the brand new file system configuration.

Deploy the answer utilizing AWS CloudFormation

To make use of the answer, you possibly can deploy the infrastructure utilizing the next CloudFormation template. This template deploys three foremost sources in your account: Amazon EFS sources (file system, entry factors, mount targets), an EventBridge rule, and a Lambda operate.

Check with Create a stack from the CloudFormation console for added data. The enter parameters for this template are:

SageMakerDomainId – The SageMaker Studio area ID that might be related to the EFS file system.
SageMakerStudioVpc – The VPC related to the SageMaker Studio area.
SageMakerStudioSubnetId – One or a number of subnets related to the SageMaker Studio area. The template deploys its sources in these subnets.
SageMakerStudioSecurityGroupId – The safety group related to the SageMaker Studio area. The template configures the Lambda operate with this safety group.

Amazon EFS sources

After you deploy the template, navigate to the Amazon EFS console and make sure that the EFS file system has been created. The file system has a mount goal in each Availability Zone that your SageMaker area connects to.

Word that every mount goal makes use of the EC2 safety group that SageMaker created in your AWS account once you first created the area, which permits NFS site visitors at port 2049. The supplied template routinely retrieves this safety group when it’s first deployed, utilizing a Lambda backed customized useful resource.

It’s also possible to observe that the file system has an EFS entry level. This entry level grants root entry on the file system for the Lambda operate that may create the directories for the SageMaker Studio person profiles.

EventBridge rule

The second foremost useful resource is an EventBridge rule invoked when a brand new SageMaker Studio person profile is created. Its goal is the Lambda operate that creates the folder within the EFS file system and updates the profile that has been simply created. The enter of the Lambda operate is the occasion matched, the place you will get the SageMaker Studio area ID and the SageMaker person profile title.

Lambda operate

Lastly, the template creates a Lambda operate that creates a listing within the EFS file system with the required POSIX permissions for the person profile and updates the person profile with the brand new file system configuration.

At a POSIX permissions degree, you possibly can management which customers can entry the file system and which information or knowledge they’ll entry. The POSIX person and group ID for SageMaker apps are:

UID – The POSIX person ID. The default is 200001. A legitimate vary is a minimal worth of 10000 and most worth of 4000000.
GID – The POSIX group ID. The default is 1001. A legitimate vary is a minimal worth of 1001 and most worth of 4000000.

The Lambda operate is in the identical VPC because the EFS file system and it has connected the file system and entry level beforehand created.

Adapt the answer for preexisting SageMaker Studio area person profiles

We will reuse the earlier resolution for situations by which the area already has person profiles created. For that, you possibly can create a further Lambda operate in Python that lists all of the person profiles for the given SageMaker Studio area and creates a devoted EFS listing for every person profile.

The Lambda operate must be in the identical VPC because the EFS file system and it has connected the file system and entry level beforehand created. It is advisable add the efs_id and domain_id values as setting variables for the operate.

You’ll be able to embody the next code as a part of this new Lambda operate and run it manually:

import json
import subprocess
import boto3
import os

sm_client = boto3.shopper('sagemaker')

def lambda_handler(occasion, context):
    
    # Get EFS and Area ID
    file_system=os.environ['efs_id']
    domain_id=os.environ['domain_id']    
    
    
    # Get Area person profiles
    list_user_profiles_response = sm_client.list_user_profiles(
        DomainIdEquals=domain_id
    )
    domain_users = list_user_profiles_response["UserProfiles"]
    
    # Create directories for every person
    for person in domain_users:

        user_profile_name = person["UserProfileName"]

        # Permissions
        repository=f'/mnt/efs/{user_profile_name}'
        subprocess.name(['mkdir', repository])
        subprocess.name(['chown', '200001:1001', repository])
        
        # Replace SageMaker person
        response = sm_client.update_user_profile(
            DomainId=domain_id,
            UserProfileName=user_profile_name,
            UserSettings={
                'CustomFileSystemConfigs': [
                    {
                        'EFSFileSystemConfig': {
                            'FileSystemId': file_system,
                            'FileSystemPath': f'/{user_profile_name}'
                        }
                    }
                ]
            }
        )

Configure an EFS listing shared throughout all areas of a given area

On this state of affairs, an administrator needs to provision an EFS file system for all customers of a SageMaker Studio area, utilizing the identical file system listing for all of the customers.

To attain this, along with the conditions described earlier on this put up, you’ll want to full the next steps.

Create the EFS file system

The file system must be in the identical VPC because the SageMaker Studio area. Check with Creating EFS file programs for added data.

Add mount targets to the EFS file system

Earlier than SageMaker Studio can entry the brand new EFS file system, the file system should have a mount goal in every of the subnets related to the area. For extra details about assigning mount targets to subnets, see Managing mount targets. You may get the subnets related to the area on the SageMaker Studio console underneath Community. It is advisable create a mount goal for every subnet.

Moreover, for every mount goal, you should add the safety group that SageMaker created in your AWS account once you created the SageMaker Studio area. The safety group title has the format security-group-for-inbound-nfs-domain-id.

The next screenshot exhibits an instance of an EFS file system with two mount targets for a SageMaker Studio area related to 2 subnets. Word the safety group related to each mount targets.

Create an EFS entry level

The Lambda operate accesses the EFS file system as root utilizing this entry level. See Creating entry factors for added data.

Create a brand new Lambda operate

Outline a brand new Lambda operate with the title LambdaManageEFSUsers. This operate updates the default area settings of the SageMaker Studio area, configuring the file system settings to make use of a particular EFS file system shared repository path. This configuration is routinely utilized to all areas throughout the area.

The Lambda operate is in the identical VPC because the EFS file system and it has connected the file system and entry level beforehand created. Moreover, you’ll want to add efs_id and domain_id as setting variables for the operate.

UID – The POSIX person ID. The default is 200001.
GID – The POSIX group ID. The default is 1001.

The operate updates the default area settings of the SageMaker Studio area, configuring the EFS file system for use by all customers. See the next code:

import json
import subprocess
import boto3
import os
import logging

logger = logging.getLogger()
logger.setLevel(logging.INFO)
sm_client = boto3.shopper('sagemaker')

def lambda_handler(occasion, context):
    
    # Surroundings variables
    file_system=os.environ['efs_id']
    domain_id=os.environ['domain_id']
    
    # EFS listing title
    repository_name="shared_repository"
    repository=f'/mnt/efs/{repository_name}'
            
    # Add permissions to the brand new listing
    attempt:
        subprocess.name(['mkdir -p', repository])
        subprocess.name(['chown', '200001:1001', repository])
    besides:
        print("Repository already created")
    
    # Replace Sagemaker area to allow entry to the brand new listing
    response = sm_client.update_domain(
        DomainId=domain_id,
        DefaultUserSettings={
            'CustomFileSystemConfigs': [
                {
                    'EFSFileSystemConfig': {
                        'FileSystemId': file_system,
                        'FileSystemPath': f'/{repository_name}'
                    }
                }
            ]
        }
    )
    logger.data(f"Up to date Studio Area {domain_id} and EFS {file_system}")
    return {
        'statusCode': 200,
        'physique': json.dumps(f"Created dir and modified permissions for Studio Area {domain_id}")
    }

The execution position of the Lambda operate must have permissions to replace the SageMaker Studio area:

{ 
"Model": "2012-10-17",
    "Assertion": [ 
        { 
        "Effect": "Allow", 
        "Action": [
            "sagemaker:UpdateDomain"
        ],
        "Useful resource": "*" 
        } 
    ]
}

Configure an EFS listing shared throughout a number of domains underneath the identical VPC

On this state of affairs, an administrator needs to provision an EFS file system for all customers of a number of SageMaker Studio domains, utilizing the identical file system listing for all of the customers. The thought on this case is to assign the identical EFS file system to all customers of all domains which can be throughout the identical VPC. To check the answer, the account ought to ideally have two SageMaker Studio domains contained in the VPC and subnet.

Create the EFS file system, add mount targets, and create an entry level

Full the steps within the earlier part to arrange your file system, mount targets, and entry level.

Create a brand new Lambda operate

Outline a Lambda operate referred to as LambdaManageEFSUsers. This operate is answerable for automating the configuration of SageMaker Studio domains to make use of a shared EFS file system inside a particular VPC. This may be helpful for organizations that wish to present a centralized storage resolution for his or her ML initiatives throughout a number of SageMaker Studio domains. See the next code:

import json
import subprocess
import boto3
import os
import sys

import logging
logger = logging.getLogger()
logger.setLevel(logging.INFO)

sm_client = boto3.shopper('sagemaker')

def lambda_handler(occasion, context):
    
    #Surroundings variables
    event_domain_id =occasion["domain_id"]
    file_system=os.environ['efs_id']
    env_vpc_id =os.environ['vpc_id']
    
    #Occasion parameters 
    repository_name="shared_repository"
    repository=f'/mnt/efs/{repository_name}'
    domains =[]    

    # Listing all SageMaker domains within the specified VPC
    response = sm_client.list_domains()
    all_domains = response['Domains']
    for area in all_domains:
        domain_id =area["DomainId"]
        knowledge =sm_client.describe_domain(DomainId=domain_id)
        domain_vpc_id = knowledge['VpcId']
        if domain_vpc_id ==env_vpc_id:
            domains.append(domain_id)
    
    # Create listing and add the permission
    attempt:
        subprocess.name(['mkdir -p', repository])
        subprocess.name(['chown', '200001:1001', repository])
    besides:
        print("Repository already created")
    
    #Replace Sagemaker area
    if len(domains)>0:
        for domain_id in domains: 
            response = sm_client.update_domain(
                DomainId=event_domain_id,
                DefaultUserSettings={
                    'CustomFileSystemConfigs': [
                        {
                            'EFSFileSystemConfig': {
                                'FileSystemId': file_system,
                                'FileSystemPath': f'/{repository_name}'
                            }
                        }
                    ]
                }
            )
   
        logger.data(f"Up to date Studio for Domains {domains} and EFS {file_system}")
        return {
                'statusCode': 200,
                'physique': json.dumps(f"Created dir and modified permissions for Domains {domains}")
            }
    
    else:
        return {
            'statusCode': 400,
            'physique': json.dumps(f"VPC id of all of the domains {domain_vpc} is completely different than the vpc id configured {env_vpc_id}")
        }

The execution position of the Lambda operate must have permissions to explain and replace the SageMaker Studio area:

{ 
"Model": "2012-10-17",
    "Assertion": [ 
        { 
        "Effect": "Allow", 
        "Action": [
            "sagemaker:DescribeDomain",
            "sagemaker:UpdateDomain"
        ],
        "Useful resource": "*" 
        } 
    ]
}

Clear up

To wash up the answer you carried out and keep away from additional prices, delete the CloudFormation template you deployed in your AWS account. Once you delete the template, you additionally delete the EFS file system and its storage. For added data, discuss with Delete a stack from the CloudFormation console.

Conclusion

On this put up, we’ve explored three situations demonstrating the flexibility of integrating Amazon EFS with SageMaker Studio. These situations spotlight how Amazon EFS can present a scalable, safe, and collaborative knowledge storage resolution for knowledge science groups.

The primary state of affairs centered on configuring an EFS listing with personal areas for particular person person profiles, permitting customers to retailer and entry their very own knowledge whereas the administrator manages the EFS file system centrally.

The second state of affairs showcased a shared EFS listing throughout all areas inside a SageMaker Studio area, enabling higher collaboration and centralized knowledge administration.

The third state of affairs explored an EFS file system shared throughout a number of SageMaker Studio domains, empowering enterprise-level knowledge science collaboration and selling environment friendly use of shared sources.

By implementing these Amazon EFS integration situations, organizations can unlock the complete potential of their knowledge science groups, enhance knowledge governance, and improve the general effectivity of their data-driven initiatives. The mixing of Amazon EFS with SageMaker Studio offers a flexible platform for knowledge science groups to thrive within the evolving panorama of ML and AI.

In regards to the Authors

Irene Arroyo Delgado is an AI/ML and GenAI Specialist Options Architect at AWS. She focuses on bringing out the potential of generative AI for every use case and productionizing ML workloads, to attain prospects’ desired enterprise outcomes by automating end-to-end ML lifecycles. In her free time, Irene enjoys touring and mountaineering.

Itziar Molina Fernandez is an AI/ML Advisor within the AWS Skilled Providers workforce. In her position, she works with prospects constructing large-scale machine studying platforms and generative AI use instances on AWS. In her free time, she enjoys exploring new locations.

Matteo Amadei is a Information Scientist Advisor within the AWS Skilled Providers workforce. He makes use of his experience in synthetic intelligence and superior analytics to extract worthwhile insights and drive significant enterprise outcomes for patrons. He has labored on a variety of initiatives spanning NLP, pc imaginative and prescient, and generative AI. He additionally has expertise with constructing end-to-end MLOps pipelines to productionize analytical fashions. In his free time, Matteo enjoys touring and studying.

Giuseppe Angelo Porcelli is a Principal Machine Studying Specialist Options Architect for Amazon Net Providers. With a number of years of software program engineering and an ML background, he works with prospects of any dimension to grasp their enterprise and technical wants and design AI and ML options that make the perfect use of the AWS Cloud and the Amazon Machine Studying stack. He has labored on initiatives in several domains, together with MLOps, pc imaginative and prescient, and NLP, involving a broad set of AWS providers. In his free time, Giuseppe enjoys taking part in soccer.