AWS SageMaker Processing job

0
votes

I was able to run a simple python code in Notebook instance to read and write csv files from/to S3 bucket. Now I want to create the SageMaker processing job to run the same code without any input/output data configuration. I have downloaded the same code and pushed the image to ECR repository. How to run this code in processing job and it should be able to install 's3fs' module?I just want to run python code in processing jobs without giving any input/output algorithms/configuration. Used boto3 to read/write from s3 bucket. With the current code it's stuck in "In Progress" downloaded code in vs code

downloaded code in vs code

!pip install s3fs
import boto3
import pandas as pd
from io import StringIO
client = boto3.client('s3')
path = 's3://weatheranalysis/weatherset.csv'
df = pd.read_csv(path)
df.head()
filename = 'newdata.csv'
bucketName = 'weatheranalysis'

csv_buffer = StringIO()

df.to_csv(csv_buffer)

client = boto3.client('s3')

response = client.put_object(
ACL='private',
Body = csv_buffer.getvalue(),
Bucket =bucketName,
Key = filename

)

1
amazon-web-servicesamazon-sagemaker

1 Answers

0
votes

You can do so by defining a 'sagemaker-processing-container' and Set up the ScriptProcessor from the SageMaker Python SDK to run your existing python script in preprocessor.py.

A simple example can be found here

from sagemaker.processing import ScriptProcessor, ProcessingInput, ProcessingOutput
    
script_processor = ScriptProcessor(command=['python3'],
                    image_uri='image_uri',
                    role='role_arn',
                    instance_count=1,
                    instance_type='ml.m5.xlarge')

script_processor.run(code='preprocessing.py',
                     inputs=[ProcessingInput(
                        source='s3://path/to/my/input-data.csv',
                        destination='/opt/ml/processing/input')],
                     outputs=[ProcessingOutput(source='/opt/ml/processing/output/train'),
                               ProcessingOutput(source='/opt/ml/processing/output/validation'),
                               ProcessingOutput(source='/opt/ml/processing/output/test')])