Setting up IBM Cloud Object Storage

For a little website that I’m writing, I thought that I’d use IBM Cloud Object Storage (equivalent to AWS S3) as I’m generating the pages using OpenWhisk on IBM Cloud Functions. The documentation is quite good if you want to use the website, but is a bit spread all over the place if you’re using the command line, which is how I do things.

As Cloud Object Storage is a bit of a mouthful, I’ll no doubt abbreviate to COS at times.

Also, make sure that you have the ibmcloud command line tool installed and are logged in to it.

Install Cloud Object Storage

While COS is availabel on the CF marketplace, it’s a global service rather than a per-instance one, so it needs to be added to your account via the website. At least, I couldn’t find a way to use the ibmcloud CLI to add a global service. Hopefully this will be provided at some point.

To install, log into your IBM Cloud account, find Cloud Object Storage in the Catalog(sic) and run through the wizard to create an instance on the Lite plan. I called my instance objectstorage.

Add the COS CLI plugin

18th June 2019. Update: The COS plugin is back!

8 April 2019. Update: The COS plugin has been removed with no explanation, so you’ll need to use curl nowadays.

There’s a handy COS plugin for the CLI tool, so we’ll install it.

$ ibmcloud plugin install cloud-object-storage

To use the COS plugin, we need to configure it with our region and objectstore‘s unique number, known as a CRN. The CRN is available using the command:

$ ibmcloud resource service-instance objectstore

The CRN is labelled ID and starts with “crn:v1:“. We now configure the COS plugin using:

$ ibmcloud cos config --crn --region eu-geo

And enter your CRN at the prompt. The region defaults to us-geo which may be fine, but I wanted Europe.

Test that it’s working using:

$ ibmcloud cos list-buckets

Create a bucket

Creating a bucket via the CLI is easy:

$ ibmcloud cos create-bucket --bucket mybucket

Where mybucket is the name of the bucket. Note that this name needs to be globally unique, so prefix it with your initials or something similar.

You can also do it via the API. For this you need your bearer token which can be found in ibmcloud iam oauth-tokens. It’s the really long string after "IAM token: Bearer", so the easiest thing is to put it into an environment variable:

$ export COS_ADMIN_TOKEN=`ibmcloud iam oauth-tokens \
    | grep IAM | awk '{printf("%s", $5)}'`

You also need the service instance’s GUID. This can be found using ibmcloud resource service-instance objectstore, but again, it’s easiest to put into an environment variable using:

$ export COS_GUID=`ibmcloud resource service-instance objectstore \
    | grep GUID | awk '{printf("%s", $2)}'`

To create the bucket via curl:

curl -X "PUT" "https://s3.eu.cloud-object-storage.appdomain.cloud/mybucket" \
 -H "Authorization: Bearer $COS_ADMIN_TOKEN" \
 -H "ibm-service-instance-id: $COS_GUID"

Uploading and downloading files

To add files to the bucket, PUT to the filename:

curl -X "PUT" "https://s3.eu.cloud-object-storage.appdomain.cloud/mybucket/test.txt" \
       -H "x-amz-acl: public-read" \
       -H "Authorization: Bearer $COS_ADMIN_TOKEN" \
       -H "Content-Type: text/plain; charset=utf-8" \
       -D "The plain text contents for this text file."

You must set the correct Content-Type header and use the -T option for files. For example, to upload an image:

curl -X "PUT" "https://s3.eu.cloud-object-storage.appdomain.cloud/mybucket/some_image.jpg" \
       -H "x-amz-acl: public-read" \
       -H "Authorization: Bearer $COS_ADMIN_TOKEN" \
       -H "Content-Type: image/jpeg" \
       -T some_image.jpg

All the possible operations are documented on the Using curl page. For example, you can delete the file by using the DELETE method

We can retrieve the file from the URL we PUT to. As we set the x-amz-acl header to public-read, no authentication is required. Remove this header if the file is to be kept private.

Creating application-specific credentials

While it is easy to use the bearer token attached to our account, we don’t want to use this for our applications where we also want to limit what any given application can do. To create application-specfic credentials, we create a service ID:

$ ibmcloud iam service-id-create myappservice -d "Service ID for My Application"

We then create a permissions policy for our bucket:

ibmcloud iam service-policy-create myappservice \
      --roles Writer \
      --service-name cloud-object-storage \
      --service-instance $COS_GUID \
      --resource-type bucket \
      --resource mybucket

This command creates a new policy for our “myappservice” service ID which assigns the Writer role to the mybucket bucket within our objectstore service. The available roles can be viewed using ibmcloud iam roles. Note that we need the objectstore GUID for the argument to –service-instance, not its ID/CRN.

You can view all policies that a service ID has using ibmcloud iam service-policies myappservice which can be handy to ensure that you’ve set it up correctly.

To attach these to our actual bucket, we create a service-key to join our service ID to the service:

ibmcloud resource service-key-create myappservice-mybucket Reader \
    --instance-name cloud-object-storage \
    --service-id myappservice

This will return a set of credentials. Make a note of the apikey as we’ll need it to create the token in the app.

Creating a bearer token from an api key

To access the COS bucket, we need to use a bearer token. The system is OAuth, so we can get the token we need with a POST request to a /token endpoint:

curl -X "POST" "https://iam.bluemix.net/oidc/token" \
     -H 'Accept: application/json' \
     -H 'Content-Type: application/x-www-form-urlencoded' \
     --data-urlencode "apikey={api key here}" \
     --data-urlencode "response_type=cloud_iam" \
     --data-urlencode "grant_type=urn:ibm:params:oauth:grant-type:apikey"

{
  "access_token": "eyKra ... [lots of characters] ... ljpKTmpgJK4EHTopFw",
  "refresh_token": "J2Bq ... [lots of characters] ... 7SWA",
  "token_type": "Bearer",
  "expires_in": 3600,
  "expiration": 1546785203,
  "scope": "ibm openid"
}

The response holds the bearer token that we need in the access_token property. We can then use that with the COS API to manage files within our bucket.

Adding a CDN

To add a CDN, we have to use the website from what I can tell. I used the instructions from the middle of this tutorial.