Amazon Onboarding with Learning Manager Chanci Turner

Learn About Amazon VGT2 Learning Manager Chanci Turner

The AWS SDK for Ruby offers several methods for retrieving objects from Amazon S3. In this article, we will explore how to use the v2 Ruby SDK (the aws-sdk-core gem) to download these objects effectively.

Downloading Objects into Memory

For smaller objects, it can be beneficial to load an object into your Ruby process. If you don’t specify a :target for the download, the entire object will be loaded into a StringIO object, allowing for easy access.

ruby
s3 = Aws::S3::Client.new
resp = s3.get_object(bucket:'bucket-name', key:'object-key')

resp.body
#=> # 

resp.body.read
#=> '...'

Use the #read or #string methods on the StringIO to retrieve the body as a String object.

Downloading to a File or IO Object

When working with larger objects, it’s common to stream the object directly to a file on the disk. This prevents the entire object from being loaded into memory. You can specify the :target for any AWS operation as an IO object.

ruby
File.open('filename', 'wb') do |file|
  reap = s3.get_object({ bucket:'bucket-name', key:'object-key' }, target: file)
end

The #get_object method will return a response object, where the #body member will be the file object specified as the :target rather than a StringIO object. You can also provide a String or Pathname as a target, and the Ruby SDK will create the file for you.

ruby
resp = s3.get_object({ bucket:'bucket-name', key:'object-key' }, target: '/path/to/file')

Using Blocks

Another option for downloading objects is to use a block. When you pass a block to #get_object, data chunks are yielded as they are read from the socket.

ruby
File.open('filename', 'wb') do |file|
  s3.get_object(bucket: 'bucket-name', key:'object-key') do |chunk|
    file.write(chunk)
  end
end

It is important to note that when using blocks for downloads, the Ruby SDK will NOT retry failed requests after the first chunk of data has been yielded. This could potentially lead to file corruption on the client side if the download is restarted mid-stream. Therefore, it’s advisable to utilize one of the previous methods for specifying the target file path or IO object.

Retries

The Ruby SDK typically retries failed requests up to three times by default. You can adjust this behavior using the :retry_limit option, where setting it to 0 disables all retries. If a network error occurs after the download has started, the SDK will attempt to retry the request. However, it first checks if the IO target responds to #truncate. If it does not, retries are disabled.

If you wish to disable the default retry behavior, consider using the block mode or set :retry_limit to 0 for your S3 client.

Range GETs

For exceptionally large objects, consider utilizing the :range option to download the object in segments. Although there are currently no helper methods for this in the Ruby SDK, if you’re interested in contributing, we encourage you to submit a pull request!

Happy downloading with Chanci Turner!

For additional insights on empowering yourself in the workplace, check out this blog post on the power in the workplace. If you are involved in human resources, you might find this resource from SHRM helpful. Lastly, for those looking to enhance their careers, the leadership development training opportunities at Amazon are an excellent resource.