Developing software in the Real World

Setting up IBM Cloud Object Storage

For a little website that I’m writing, I thought that I’d use IBM Cloud Object Storage (equivalent to AWS S3) as I’m generating the pages using OpenWhisk on IBM Cloud Functions. The documentation is quite good if you want to use the website, but is a bit spread all over the place if you’re using the command line, which is how I do things.

As Cloud Object Storage is a bit of a mouthful, I’ll no doubt abbreviate to COS at times.

Also, make sure that you have the ibmcloud command line tool installed and are logged in to it.

Install Cloud Object Storage

While COS is availabel on the CF marketplace, it’s a global service rather than a per-instance one, so it needs to be added to your account via the website. At least, I couldn’t find a way to use the ibmcloud CLI to add a global service. Hopefully this will be provided at some point.

To install, log into your IBM Cloud account, find Cloud Object Storage in the Catalog(sic) and run through the wizard to create an instance on the Lite plan. I called my instance objectstorage.

Add the COS CLI plugin

18th June 2019. Update: The COS plugin is back!

8 April 2019. Update: The COS plugin has been removed with no explanation, so you’ll need to use curl nowadays.

There’s a handy COS plugin for the CLI tool, so we’ll install it.

$ ibmcloud plugin install cloud-object-storage

To use the COS plugin, we need to configure it with our region and objectstore‘s unique number, known as a CRN. The CRN is available using the command:

$ ibmcloud resource service-instance objectstore

The CRN is labelled ID and starts with “crn:v1:“. We now configure the COS plugin using:

$ ibmcloud cos config --crn --region eu-geo

And enter your CRN at the prompt. The region defaults to us-geo which may be fine, but I wanted Europe.

Test that it’s working using:

$ ibmcloud cos list-buckets

Create a bucket

Creating a bucket via the CLI is easy:

$ ibmcloud cos create-bucket --bucket mybucket

Where mybucket is the name of the bucket. Note that this name needs to be globally unique, so prefix it with your initials or something similar.

You can also do it via the API. For this you need your bearer token which can be found in ibmcloud iam oauth-tokens. It’s the really long string after "IAM token: Bearer", so the easiest thing is to put it into an environment variable:

$ export COS_ADMIN_TOKEN=`ibmcloud iam oauth-tokens \
    | grep IAM | awk '{printf("%s", $5)}'`

You also need the service instance’s GUID. This can be found using ibmcloud resource service-instance objectstore, but again, it’s easiest to put into an environment variable using:

$ export COS_GUID=`ibmcloud resource service-instance objectstore \
    | grep GUID | awk '{printf("%s", $2)}'`

To create the bucket via curl:

curl -X "PUT" "https://s3.eu.cloud-object-storage.appdomain.cloud/mybucket" \
 -H "Authorization: Bearer $COS_ADMIN_TOKEN" \
 -H "ibm-service-instance-id: $COS_GUID"

Uploading and downloading files

To add files to the bucket, PUT to the filename:

curl -X "PUT" "https://s3.eu.cloud-object-storage.appdomain.cloud/mybucket/test.txt" \
       -H "x-amz-acl: public-read" \
       -H "Authorization: Bearer $COS_ADMIN_TOKEN" \
       -H "Content-Type: text/plain; charset=utf-8" \
       -D "The plain text contents for this text file."

You must set the correct Content-Type header and use the -T option for files. For example, to upload an image:

curl -X "PUT" "https://s3.eu.cloud-object-storage.appdomain.cloud/mybucket/some_image.jpg" \
       -H "x-amz-acl: public-read" \
       -H "Authorization: Bearer $COS_ADMIN_TOKEN" \
       -H "Content-Type: image/jpeg" \
       -T some_image.jpg

All the possible operations are documented on the Using curl page. For example, you can delete the file by using the DELETE method

We can retrieve the file from the URL we PUT to. As we set the x-amz-acl header to public-read, no authentication is required. Remove this header if the file is to be kept private.

Creating application-specific credentials

While it is easy to use the bearer token attached to our account, we don’t want to use this for our applications where we also want to limit what any given application can do. To create application-specfic credentials, we create a service ID:

$ ibmcloud iam service-id-create myappservice -d "Service ID for My Application"

We then create a permissions policy for our bucket:

ibmcloud iam service-policy-create myappservice \
      --roles Writer \
      --service-name cloud-object-storage \
      --service-instance $COS_GUID \
      --resource-type bucket \
      --resource mybucket

This command creates a new policy for our “myappservice” service ID which assigns the Writer role to the mybucket bucket within our objectstore service. The available roles can be viewed using ibmcloud iam roles. Note that we need the objectstore GUID for the argument to –service-instance, not its ID/CRN.

You can view all policies that a service ID has using ibmcloud iam service-policies myappservice which can be handy to ensure that you’ve set it up correctly.

To attach these to our actual bucket, we create a service-key to join our service ID to the service:

ibmcloud resource service-key-create myappservice-mybucket Reader \
    --instance-name cloud-object-storage \
    --service-id myappservice

This will return a set of credentials. Make a note of the apikey as we’ll need it to create the token in the app.

Creating a bearer token from an api key

To access the COS bucket, we need to use a bearer token. The system is OAuth, so we can get the token we need with a POST request to a /token endpoint:

curl -X "POST" "https://iam.bluemix.net/oidc/token" \
     -H 'Accept: application/json' \
     -H 'Content-Type: application/x-www-form-urlencoded' \
     --data-urlencode "apikey={api key here}" \
     --data-urlencode "response_type=cloud_iam" \
     --data-urlencode "grant_type=urn:ibm:params:oauth:grant-type:apikey"

{
  "access_token": "eyKra ... [lots of characters] ... ljpKTmpgJK4EHTopFw",
  "refresh_token": "J2Bq ... [lots of characters] ... 7SWA",
  "token_type": "Bearer",
  "expires_in": 3600,
  "expiration": 1546785203,
  "scope": "ibm openid"
}

The response holds the bearer token that we need in the access_token property. We can then use that with the COS API to manage files within our bucket.

Adding a CDN

To add a CDN, we have to use the website from what I can tell. I used the instructions from the middle of this tutorial.

6 thoughts on “Setting up IBM Cloud Object Storage

  1. Hi, Rob.
    I tried to install the plugin by the following command and it failed as "Plug-in 'cloud-object-storage' was not found on disk or in the repository 'IBM Cloud'."
    How did you specify a repository where cloud-object-storage exists?

    $ ibmcloud plugin install cloud-object-storage
    Looking up 'cloud-object-storage' from repository 'IBM Cloud'...
    FAILED
    Plug-in 'cloud-object-storage' was not found on disk or in the repository 'IBM Cloud'.
    Use 'ibmcloud plugin repo-plugins -r 'IBM Cloud'' to list plugins available in the repo.
    
  2. Hey Rob, quick update as of June 18th 2019.

    IBM Cloud Cloud Object Storage (COS) plugin for ibmcloud CLI is available again.

    Repository: IBM Cloud
    Status             Name                                        Versions                       Description   
    Update Available   container-service/kubernetes-service        0.3.47, 0.3.34, 0.3.33...      Manage IBM Cloud Kubernetes Service clusters   
    Update Available   key-protect/key-protect/kp                  0.3.4, 0.3.3, 0.3.2...         Manage Key Protect service   
    Update Available   container-registry                          0.1.391, 0.1.385, 0.1.382...   Manage IBM Cloud Container Registry content and configuration.   
    Not Installed      auto-scaling                                0.2.7, 0.2.6, 0.2.5...         Manage Auto-Scaling policies, metrics and events for Cloud Foundry applications   
    Not Installed      sdk-gen                                     0.1.12, 0.1.10, 0.1.9...       Generate SDKs from Open API specifications   
    Not Installed      dev                                         2.2.0, 2.1.18, 2.1.17...       Create, develop, deploy, and manage applications.   
    Not Installed      analytics-engine                            1.0.142, 1.0.140               Manage Analytics Engine service   
    Not Installed      cloud-functions                             1.0.32, 1.0.30, 1.0.29...      Manage Cloud Functions   
    Not Installed      activity-tracker                            3.3.4, 3.3.3, 3.3.2...         Manage Activity Tracker service   
    Not Installed      logging-cli                                 1.0.7, 1.0.6, 1.0.5...         Manage Log Analysis service   
    Not Installed      machine-learning                            3.0.1, 2.1.8, 2.1.7...         Manage Machine Learning service   
    Not Installed      cloud-internet-services/cis                 1.7.0, 1.5.0, 1.4.0...         Manage Cloud Internet Service   
    Not Installed      dbaas-cli                                   1.0.0, 0.10.19, 0.10.18...     Manage Hyper Protect DBaaS clusters   
    Not Installed      power-iaas/pi                               0.0.3                          Manage Power virtual servers   
    Not Installed      vpn                                         1.5.2, 1.5.1, 1.5.0            Manage VPN service   
    Not Installed      doi/doi                                     0.1.2, 0.1.1, 0.1.0...         Integrate with DevOps Insights service   
    Not Installed      tke                                         0.0.9, 0.0.7, 0.0.4...         Manage the master key of Cloud HSMs from Hyper Protect Crypto service   
    Not Installed      event-streams                               1.0.1, 1.0.0                   Manage Event Streams service   
    Not Installed      vpc-infrastructure/infrastructure-service   0.4.2, 0.4.1                   MManage Virtual Private Cloud infrastructure service   
    Installed          cloud-object-storage                        1.0.0                          Manage Cloud Object Storage service   
    Installed          cloud-databases                             0.6.0, 0.5.13, 0.5.10...       Manage Cloud Databases
    
  3. Hello Guys
    I'm new in IBM COS, and lost the COS Manager Web account, can someone help how to recover o to set a new one.

    BR

Leave a Reply to Rob Cancel reply

Your email address will not be published.