Blob storage

Blob Storage

Blob storage represents a generic key ==> value data store. It is implemented by Amazon S3 or Rackspace Cloudfiles. In Deltacloud, the organisational unit of blob storage is a bucket. Individual data items - blobs are exposed as a subcollection under each bucket.

A bucket has

a name;
a size (denotes the number of blobs it contains); and
a list of links to each blob.

A blob has:

a content_length;
a content_type;
a last_modified timestamp;
a structure containing user_metadata;
a link to the blob content; and
a name of the bucket in which this blob exists.

Get a list of all buckets

To return a list of all buckets belonging to the given cloud provider account use call GET /api/buckets. The response from the Deltacloud server includes the name and URI of each bucket but not the size or the list of blobs which the bucket contains. These details are available when a client requests (GETs) a specific bucket.

Example request:

GET /api/buckets?format=xml HTTP/1.1
Authorization: Basic AU1J3UB2121Afd1DdyQWxLaTYTmJMNF4zTXBoRGdhMDh2RUw5ZDAN9zVXVa==
User-Agent: curl/7.20.1
Host: localhost:3001
Accept: */*

Server response:

HTTP/1.1 200 OK
Content-Type: application/xml
Content-Length: 597
<?xml version='1.0' encoding='utf-8' ?>
<buckets>
  <bucket href='http://localhost:3001/api/buckets/mybucket1' id='mybucket1'>
    <name>mybucket1</name>
    <size></size>
  </bucket>
  <bucket href='http://localhost:3001/api/buckets/mybucket2' id='mybucket2'>
    <name>mybucket2</name>
    <size></size>
  </bucket>
  <bucket href='http://localhost:3001/api/buckets/mybucket3' id='mybucket3'>
    <name>mybucket3</name>
    <size></size>
  </bucket>
  <bucket href='http://localhost:3001/api/buckets/mybucket4' id='mybucket4'>
    <name>mybucket4</name>
    <size></size>
  </bucket>
</buckets>

Get the details of a specific bucket

To return details of a specific bucket use call GET /api/buckets/:id. The Deltacloud server response includes the size of the bucket and the URI for each blob object that it contains.

Example request:

GET /api/buckets/mybucket1?format=xml HTTP/1.1
Authorization: Basic AU1J3UB2121Afd1DdyQWxLaTYTmJMNF4zTXBoRGdhMDh2RUw5ZDAN9zVXVa==
User-Agent: curl/7.20.1
Host: localhost:3001
Accept: */*

Server response:

HTTP/1.1 200 OK
Content-Type: application/xml
Content-Length: 534
<?xml version='1.0' encoding='utf-8' ?>
<bucket href='http://localhost:3001/api/buckets/mybucket1' id='mybucket1'>
  <name>mybucket1</name>
  <size>4</size>
  <blob href='http://localhost:3001/api/buckets/mybucket1/myfile' id='myfile'></blob>
  <blob href='http://localhost:3001/api/buckets/mybucket1/an_object' id='an_object'></blob>
  <blob href='http://localhost:3001/api/buckets/mybucket1/picture_blob' id='picture_blob'></blob>
  <blob href='http://localhost:3001/api/buckets/mybucket1/some_blob id='some_blob'></blob>
</bucket>

Create a new bucket

To create a new bucket use call POST /api/buckets. You need to specify the name as a parameter. Optionally for Amazon S3 buckets, you can specify a bucket location with the location parameter, as per Regions and Endpoints for Amazon Simple Storage Service. Valid values for S3 bucket location parameter are: "us-west-1", "EU", "ap-southeast-1", "ap-northeast-1" (while not specifying a location default to the "US Standard" region). Note that clients may specify parameters as multipart/form-data or using a content-type of application/x-www-form-urlencoded.

On succesful creation this call will return a 201 HTTP status, specifying the URI of the newly created bucket in the Location header and the newly created bucket object in the response message body. The example request below shows how to create a new bucket in the EU (Ireland) region. If the given back-end cloud does not support locations then the location parameter is ignored.

Example request:

POST /api/buckets?format=xml HTTP/1.1
Authorization: Basic AU1J3UB2121Afd1DdyQWxLaTYTmJMNF4zTXBoRGdhMDh2RUw5ZDAN9zVXVa==
User-Agent: curl/7.20.1
Host: localhost:3001
Accept: */*
Content-Length: 31
Content-Type: application/x-www-form-urlencoded

name=mybucketeurope&location=EU

Server response:

HTTP/1.1 201 Created
Location: http://localhost:3001/api/buckets/mybucketeurope
Content-Type: application/xml
Content-Length: 182

<?xml version='1.0' encoding='utf-8' ?>
<bucket href='http://localhost:3001/api/buckets/mybucketeurope' id='mybucketeurope'>
  <name>mybucketeurope</name>
  <size>0</size>
</bucket>

Delete a bucket

To delete the specified bucket use call DELETE /api/buckets/:id. The bucket must be empty (otherwise the call will fail with an error response). A succesful operation will return 204 No Content.

Example request:

DELETE /api/buckets/mybucketeurope?format=xml HTTP/1.1
Authorization: Basic AU1J3UB2121Afd1DdyQWxLaTYTmJMNF4zTXBoRGdhMDh2RUw5ZDAN9zVXVa==
User-Agent: curl/7.20.1 (i386-redhat-linux-gnu)
Host: localhost:3001
Accept: */*

Server response:

HTTP/1.1 204 No Content

Get the details of a blob

To retrieve the details of a specific blob use call GET /api/buckets/:bucket_id/:blob_id. The blob content is not returned as part of the response but rather a URI is given from which the content may be retrieved as shown below.

Example request:

GET /api/buckets/mariosbucket1/some_more_blob_woo?format=xml HTTP/1.1
Authorization: Basic AU1J3UB2121Afd1DdyQWxLaTYTmJMNF4zTXBoRGdhMDh2RUw5ZDAN9zVXVa==
User-Agent: curl/7.20.1 (i386-redhat-linux-gnu)
Host: localhost:3001
Accept: */*

Server response:

HTTP/1.1 200 OK
Content-Type: application/xml
Content-Length: 586
<?xml version='1.0' encoding='utf-8' ?>
<blob href='http://localhost:3001/api/buckets/mariosbucket1/some_more_blob_woo' id='some_more_blob_woo'>
  <bucket>mariosbucket1</bucket>
  <content_length>86</content_length>
  <content_type>text/plain</content_type>
  <last_modified>Fri Jan 28 12:23:08 UTC 2011</last_modified>
  <user_metadata>
    <entry key='v'>
      <![CDATA[0.2.0]]>
    </entry>
    <entry key='test'>
      <![CDATA[value]]>
    </entry>
  </user_metadata>
  <content href='http://localhost:3001/api/buckets/mariosbucket1/some_more_blob_woo/content'></content>
</blob>

Get the actual blob content

To retrieve the actual blob content use call GET /api/buckets/:bucket_id/:blob_id/content. The location of blob content is specified in the content URI returned from the GET /api/buckets/:bucket_id/:blob_id call. The content is streamed through the Deltacloud server as soon as it is received from the back-end cloud provider. The aim is to avoid the creation of a temporary file (especially significant in case of large blobs). The Deltacloud server sets Content-Disposition: attachment; filename=blob_name in the HTTP response headers.

Example request:

GET /api/buckets/mariosanotherbucketohyeah/Im_a_blob_beholdme/content?format=xml HTTP/1.1
Authorization: Basic AU1J3UB2121Afd1DdyQWxLaTYTmJMNF4zTXBoRGdhMDh2RUw5ZDAN9zVXVa==
User-Agent: curl/7.20.1 (i386-redhat-linux-gnu)
Host: localhost:3001
Accept: */*

Server response:

HTTP/1.1 200 OK
Content-Disposition: attachment; filename=Im_a_blob_beholdme
Content-Type: text/plain
Content-Length: 50

<BLOB DATA HERE>

Create a blob object

To create a blob object and set its content use call PUT /api/buckets/:bucket_id/:blob_id. If the blob already exists then its data and metadata are overwritten with those specified in this call. The request must specify the name of the blob and the name of the bucket in which the blob is to be placed in the call URI. The client must also specify the content_length of the blob data and the blob data itself in the HTTP headers. Optionally the call may also specify a content_type and any number of key:value pairs of user defined metadata. User metadata is defined using 'X-Deltacloud-Blobmeta-' header, e.g. X-Deltacloud-Blobmeta-Version:2.1.

To eliminate the necessity of creating a local file at the Deltacloud server for each blob created, the Deltacloud server starts to stream the blob data to the back-end cloud provider as soon as the request headers are processed. A succesful operation will return the newly created blob object, as shown below.

Example request:

PUT /api/buckets/mybucket/12Jul2011blob?format=xml HTTP/1.1
Authorization: Basic AU1J3UB2121Afd1DdyQWxLaTYTmJMNF4zTXBoRGdhMDh2RUw5ZDAN9zVXVa==
Content-Type: text/plain
Content-Length: 128988
X-Deltacloud-Blobmeta-Version:2.1
X-Deltacloud-Blobmeta-Author:msa

... BLOB DATA ...

Server response:

<?xml version='1.0' encoding='utf-8' ?>
<blob href='http://localhost:3001/api/buckets/mybucket/12Jul2011blob' id='12Jul2011blob'>
  <bucket>mybucket</bucket>
  <content_length>128988</content_length>
  <content_type>text/plain</content_type>
  <last_modified>Wed Jul 13 13:27:22 UTC 2011</last_modified>
  <user_metadata>
    <entry key='author'>
      <![CDATA[msa]]>
    </entry>
    <entry key='version'>
      <![CDATA[2.1]]>
    </entry>
  </user_metadata>
  <content href='http://localhost:3001/api/buckets/mybucket/12Jul2011blob/content'>
  </content>
</blob>

The Deltacloud server also responds to an alternative POST /api/buckets/:bucket_id route for creating or updating a blob object. As with the PUT method for creating/updating a blob, the client must specify the bucket in which the blob is to be created through the call URI (i.e. you POST to the specified bucket). The rest of the required fields, that is, the name of the blob, the blob_data and the content-length are specified by the client as multipart/form-data (i.e. in HTTP POST form fields).

In order to specify the optional user metadata for a given blob the client must set the form field meta_params to specify the number of metadata key/value pairs. The metadata itself is then specified by the client with fields of the form meta_nameN and meta_valueN where N is an integer from 1 up to the number specified by the meta_params field (e.g. meta_name1=author, meta_value1=jrd).

It should be noted that the POST method for creating a blob is non streaming - the Deltacloud server will create a temporary file with the blob data, before the file is transferred to the backend cloud. Thus, it should only be used for blobs with a relatively small content-length and in general the PUT method should be preferred for larger blobs. This POST method is mainly provided for clients that cannot easily invoke a HTTP PUT operation (e.g. web browsers) and can be used for creating/updating a blob through the deltacloud HTML interface (provided for testing purposes).

Example request:

POST /api/buckets/mybucket?format=xml HTTP/1.1
Authorization: Basic AU1J3UB2121Afd1DdyQWxLaTYTmJMNF4zTXBoRGdhMDh2RUw5ZDAN9zVXVa==
User-Agent: curl/7.20.1 (i386-redhat-linux-gnu) libcurl/7.20.1 N
Accept: */*
Content-Length: 113582
Expect: 100-continue
Content-Type: multipart/form-data; boundary=----------------------------517f5f2df858

------------------------------517f5f2df858
Content-Disposition: form-data; name="blob"

12Jul2011blob
------------------------------517f5f2df858
Content-Disposition: form-data; name="blob_data"; filename="small.txt"
Content-Type: text/plain

<THE_BLOB_DATA_HERE>

------------------------------517f5f2df858
Content-Disposition: form-data; name="meta_params"

2
------------------------------517f5f2df858
Content-Disposition: form-data; name="meta_name1"

author
------------------------------517f5f2df858
Content-Disposition: form-data; name="meta_value1"
jjs
------------------------------517f5f2df858
Content-Disposition: form-data; name="meta_name2"

version
------------------------------517f5f2df858
Content-Disposition: form-data; name="meta_value2"

2.2
------------------------------517f5f2df858--

Server response:

<?xml version='1.0' encoding='utf-8' ?>
<blob href='http://localhost:3001/api/buckets/mybucket/12Jul2011blob' id='12Jul2011blob'>
  <bucket>mybucket</bucket>
  <content_length>112766</content_length>
  <content_type>text/plain</content_type>
  <last_modified></last_modified>
  <user_metadata>
    <entry key='x-amz-meta-author'>
      <![CDATA[jjs]]>
    </entry>
    <entry key='x-amz-meta-version'>
      <![CDATA[2.2]]>
    </entry>
  </user_metadata>
  <content href='http://localhost:3001/api/buckets/mybucket/12Jul2011blob/content'>
  </content>
</blob>

Delete a blob object

To delete the specified blob object from the back-end cloud use call DELETE /api/buckets/:bucket_id/:blob_id. The names of the blob and the bucket in which this exists are specified the in call URI. After a succesful operation the Deltacloud server responds with a HTTP 204 (No Content) with no message body.

Example request:

DELETE /api/buckets/mybucket/12Jul2011blob?format=xml HTTP/1.1
Authorization: Basic AU1J3UB2121Afd1DdyQWxLaTYTmJMNF4zTXBoRGdhMDh2RUw5ZDAN9zVXVa==
User-Agent: curl/7.20.1 (i386-redhat-linux-gnu)
Host: localhost:3001
Accept: */*

Server response:

HTTP/1.1 204 No Content
Connection: close
Server: thin 1.2.11

Get all metadata fields

To return all user defined metadata fields on a specified blob URI use the HTTP HEAD operation HEAD /api/buckets/:bucket_id/:blob_id. As per RFC 2616 this HEAD operation does not return a message body. Rather, the blob user metadata values are returned in the response X-Deltacloud-Blobmeta- headers (e.g., X-Deltacloud-Blobmeta-version:1.2).

Example request:

HEAD /api/buckets/mybucket/myblob?format=xml HTTP/1.1
Authorization: Basic AU1J3UB2121Afd1DdyQWxLaTYTmJMNF4zTXBoRGdhMDh2RUw5ZDAN9zVXVa==
User-Agent: curl/7.20.1
Host: localhost:3001
Accept: */*

Server response:

HTTP/1.1 204 No Content
X-Deltacloud-Blobmeta-version: 1.21
X-Deltacloud-Blobmeta-author: jrd

Update the user-defined blob metadata

To update the user-defined blob metadata use call POST /api/buckets/:bucket_id/:blob_id on the URI of a blob. This operation will overwrite all previously set user-metadata values (if any) and replace them with those specified in this call. The client must set the user-defined metadata in the X-Deltacloud-Blobmeta- headers (e.g., X-Deltacloud-Blobmeta-Model:2012).

Example request:

POST /api/buckets/mybucket/myblob?format=xml HTTP/1.1
Authorization: Basic AU1J3UB2121Afd1DdyQWxLaTYTmJMNF4zTXBoRGdhMDh2RUw5ZDAN9zVXVa==
User-Agent: curl/7.20.1
Host: localhost:3001
Accept: */*
X-Deltacloud-Blobmeta-model: 2012
X-Deltacloud-Blobmeta-paint: Stannite_Grey

Server response:

HTTP/1.1 204 No Content
X-Deltacloud-Blobmeta-model: 2012
X-Deltacloud-Blobmeta-paint: Stannite_Grey