7.4. RAIN

FutureGrid image management defines the full lifecycle of the images in FutureGrid. It involves the process of creating, customizing, storing, sharing, and deploying images for different FutureGrid environments. For information and instructions on managing your images in FutureGrid, and the Rain environment, see the Quickstart Guide.

7.4.1. Generate and Register an OS Image on FutureGrid using the FG Shell

Below, we summarize the different steps needed to create a new image and register it in a FutureGrid infrastructure. In this case, we will register the image in OpenStack, but it can be registered in any other FutureGrid infrastructure, such as Eucalyptus, Nimbus, or HPC. Detailed information about these tools can be found in http://futuregrid.github.com/rain/. Request access

Submit a ticket to request access help

Please use the subject: “Request to access FutureGrid Rain”. Include in the body “I would like to obtain access to FutureGrid Rain”.

Note: In future we may just create a checkbox for this request in the help form.

Once you have access, you can continue the tutorial. Typically, it will take a business day for you to get added (a business day is 9am-5pm EST Mon-Fri). Log into India

$ ssh <username>@india.futuregrid.org
$ module load futuregrid Start the Shell

Execute the FG Shell. Please remember to replace <username> with your portal user name. Then, you will be asked for a password (which is your portal password):

$ fg-shell -u <username>

This command will change your shell and the prompt will be fg-rain>. If your prompt is different, you need to execute “use rain”. Generate the Image

We are going to generate an Ubuntu 12.04 image. The -s parameter allows you to specify the software you want to install in your image. Currently, only the software available in the official repositories can be installed:

fg-rain> generate -o ubuntu -v 12.04 -a x86_64 -s wget, openmpi-bin

After a while, your image will be stored in the repository, and your image ID will be printed. Image Repository

We can consult the information of the image by executing the following (please replace <imageid> with the ID that you got from the previous command):

fg-rain> list * where imgId=<imageId> Register Image

Next, we need to register that image in the infrastructure we want to use. In this case, we will register the image in OpenStack (if you followed the FG Openstack tutorial, your novarc will probably be in ~/openstack/novarc). This command will return an <ami-ID>, which is the ID of the image in OpenStack:

fg-rain> register -r <imageId> -s india -v ~/novarc

Once you have registered the image, you need to wait until it becames available. To check the status of the image, you can execute the following command:

fg-rain> cloudlist -s india -v ~/novarc | grep <imageId>

The status is the second field. You may experience that this command takes time to respond when the image is being uploaded (it is an OpenStack issue). Start Image

Once the image is in “available” status, we can proceed to start a VM using the ami-ID that we got from the register command. In this case, we are going in Interative mode, which means that we are going to get logged into the VM once it is running:

fg-rain> launch -i <ami-ID> -s india -v ~/novarc -I

7.4.2. FutureGrid Standalone Image Repository

The FutureGrid image repository is a standalone service that is not tied to FutureGrid nor to any IaaS infrastructure. This image repository offers a common interface that can distinguish image types for different IaaS frameworks like Nimbus, Eucalyptus, and also bare-metal images. This allows us in FG to include a diverse image set not only contributed by the FG development team, but also by the user community that generates such images and wishes to share them. The images can be described with information about the software stack that is installed on them including versions, libraries, and available services. This information is maintained in the catalog and can be searched by users and/or other FutureGrid services. Users looking for a specific image can discover available images fitting their needs, and find their location in the repository by using the catalog interface.

The repository supports different storage systems that allow you to choose the most appropriate one for you. It includes MySQL, where the image files are stored directly in the POSIX file system; MongoDB, where both data and files are stored in the NoSQL database; the OpenStack Object Store (Swift); and Cumulus from the Nimbus project. For the last two cases, the data can be stored in either MySQL or in MongoDB. These storage plugins not only increase the interoperability of the image repository, but can also be used by the community as templates to create their own plugins to support other storage systems. Requirement

  • Python 2.6 or 2.7
  • LDAP server for user authentication Software

The software is open-source under an Apache 2 license. The image repository can be found in github as part of our Image Management and Rain software https://portal.futuregrid.org/rain. Documentation

The documentation can be found in http://futuregrid.github.com/rain/index.html. Since this link provides information about all our tools, in this section we collect the links that refer to the image repository. User Manual

Information about the command-line interface is found in http://futuregrid.github.com/rain/man-repo.html. If you are interested on using the shell, the manual is in http://futuregrid.github.com/rain/man-shell.html. Installation

You need to install the complete software, even if you only want to use the image repository. This information is found in http://futuregrid.github.com/rain/install.html.

7.4.3. Manual Image Customization

Sometimes users need to further customize their images by installing their own software-software which may not be able to be installed with the image generation tool. In this tutorial, we explain how users can perform this customization in the images created with the FG image generation tool. Detailed information about FG image management tools can be found in http://futuregrid.github.com/rain/. Logging into India

$ ssh <username>@india.futuregrid.org
$ module load futuregrid Requesting access

Submit a ticket to request access help Obtaining the image

You can get your image from our repository, or generate a new one. Getting an image from the repository

$ fg-repo -u <username> -g <imageId> Generating a new image

We are going to generate an Ubuntu image. Since we want to modify the image, we have specified the -g parameter to retrieve the image after it is generated. By default, it is uploaded to the image repository:

$ fg-generate -u jdiaz -o ubuntu -v 12.04 -a x86_64 -s wget, openmpi-bin -g

In both cases, we will obtain a tgz file that contains the image file (.img) and a manifest (.manifest.xml). For example, our image could be in /N/u/<username>/123123123.tgz . Customizing the image

To continue with the rest of the tutorial, we need a UNIX machine where we have root privileges.

Assuming that we are in a UNIX machine with root privileges:

  1. Retrieve the image from India:

    $ scp <username>@india.futuregrid.org:/N/u/<username/123123123.tgz .
  2. Decompress the image:

    $ tar vxfz 123123123.tgz

3. Mount the image. This will mount the image file into a directory. In this way, we will have access to the OS files that are inside the image:

$ mkdir image
$ sudo mount -o loop jdiaz859434.img image

Copy your software into the image directory (if needed). In this case, I am going to copy a software directory to the tmp directory of the image:

$ cp -r /home/javi/mysoftware image/tmp

Chroot into the image. This changes the root of the OS to the one of the image. In this way, evey operation we execute will have effect only inside the image. After executing this command, you will be the root users inside the image:

$ sudo chroot image

Now you can install whatever software you need. You can also use yum/apt to install packages from the software repository. Remember that they will be installed inside the image. As example, here I install tomcat with apt and compile my software with make:

# apt-get install tomcat6
# cd /tmp/mysoftware
# make && make install

You can also configure the OS of image to start services during the boot time. The easiest way to do that is by modifying the/etc/rc.localfile.

Once you have finished customizing your image, you need to exit from the chroot command, unmount the image, and compress it again:

# exit
$ sudo umount image
$ tar vxfz mynewimage.tgz jdiaz859434.img jdiaz859434.manifest.xml Transfer the image back to India

$ scp mynewimage.tgz <username>@india.futuregrid.org:/N/u/<username>/ Log into India

$ ssh <username>@india.futuregrid.org
$ module load futuregrid Upload the image to the repository

We upload the image to the repository so we can reuse it and share it with other users. When uploading the image, you can specify some metadata to describe the properties of the image:

$ fg-repo -p mynewimage.tgz "os=Ubuntu12 & arch=x86_64 & description=My new customized image & tag=tomcat, openmpi"

This command will provide you the ID your image has in the repository. This ID is needed for the next step. Register your image in different infrastructures

You now have your image ready to be registered in the different FutureGrid infrastructures. To use OpenStack (tutorial <openstackhavana.html>) and Eucalyptus (tutorial <eucalyptus.html>), you need to indicate the location of you novarc and eucarc file. Register the image in Openstack

$ fg-register -u <username> -r <imageID> -s india -v ~/novarc Register the image in Eucalyptus

$ fg-register -u <username> -r <imageID> -e india -v ~/eucarc Register the image in HPC

$ fg-register -u <username> -r <imageID> -x india

Each command will provide you with the ami-ID that your image has in the specified infrastructure. Using your Registered Image


fix the links in this section

OpenStack (more info in openstack <openstackhavana.html>):

source novarc
euca-run-instance -k <keyname> <ami-ID>

Eucalyptus (more info in eucalyptus <eucalyptus.html>):

source eucarc
euca-run-instance -k <keyname> <ami-ID>

HPC (more info in hpc):

Provision a machine with our image and go into Interactive mode (you are logged into the machine):

qsub -l os=<ami-ID> -I

Provision two machines with our image and execute an script:

qsub -l os=<ami-ID> -l nodes=2:ppn=8 myscript.sh

7.4.4. RAIN Manual Pages

FutureGrid provides a number of manual pages, listed below.

(For Rain, our up-to-date documentation of rain can be found at doc/rain/index.html andhttp://futuregrid.github.com/rain/index.html . fg-portal-manage

The tool is for admin purpose only. An admin person or an system script can call this to get user/project information from the portal:

 -h, --help            show this help message and exit
 -n NAME, --name=NAME  querying by name
 -m MAIL, --mail=MAIL  querying by email
 -i UID, --uid=UID     querying by portal uid
 -u USERNAME, --username=USERNAME
                       querying by portal username
 -k, --attrib          specifying attrib(s) to be displayed, inlcuding:
                       firstname, lastname, email, phone, organization,
                       institution_name, citizenship, sshkey, projectall,
 -l, --ldif            output ldif format of user info(no dn). If provided
                       all attributes will be included in the output
                       disregarding the setting from -k
 -v, --vetted          Show only those vetted users. If omitted all users
                       that meet the creteria will be shown
 -w, --waiting         Show the waiting list in which a user is eligible to
                       get an LDAP account
 -y, --withsshkey      A sub-option for -w, which shows the waiting list, but
                       only for those who have submitted a sshkey
 -p, --project         Get list of project in table view
 -s STATUS, --status=STATUS
                       A sub-option for -p, which specifies project status
                       that would like to be retrieved. Try pending,
                       approved, completed, denied
 -1, --pidonly         A sub-option for -p which prints out only the project

FutureGrid account admin tool queries user profile info from the portal and also checks status against the ldap.

To run the script, python 2.7 is needed (if trying on lower version, try to install the argparse module; it may also work), as well as the mysqldb module. Depending on the os distribution, the installation could be different. E.g., in ubuntu, it’s simply: apt-get install python-mysqldb .

The futuregrid.cfg file needs to be put into the same directory as the script, and the dummy config values need to be replaced with those real ones. (Whoever has access to the portal server should have knowledge on this. We will not distribute this to anymore other than the portal admin and sys admin.)


fix this command

Run this to get the help info:

./fg-portal-manage.py -h

This will give a table viewed info of user with uid 3:

./fg-portal-manage.py -i 3

However, the following:

./fg-portal-manage.py -i 3 -l

will print the long quasi-ldif formatted info.

Other command:

-u for username; -n for realname(first or last); as well as -m for email does similar thing
./fg-portal-manage.py -l

-will simply print out all users in the portal in the ldif-like format.

./fg-portal-manage.py -l -v

-will print out a similar list but only for (all) those vetted users, i.e., approved users with an active project.

./fg-portal-manage.py -w

-prints out all users who have approved portal account and are members of an active project(s) (i.e., vetted users), but don’t have an LDAP account yet.

./fg-portal-manage.py -l -w

-the same as -w but in long quasi-ldif format.

./fg-portal-manage.py -w -y

-the same as -w, but prints out only those who have submitted a sshkey.

./fg-portal-manage.py -p

-lists all FG projects registered in the portal, in the format of:

pid: title|project lead|project manager|status|[members list]

-project lead/manager and members are represented by the portaluid.

./fg-portal-manage.py -p -s approved

-prints a list of projects but only those in ‘approved’ status. Other valid statuses are: pending, completed, denied.

./fg-portal-manage.py -p -s completed -1(number '1')

-prints a list of completed projects but only display the projectids.