7.4. RAIN¶

7.4.1.1. Request access¶

Submit a ticket to request access help

Please use the subject: “Request to access FutureGrid Rain”. Include in the body “I would like to obtain access to FutureGrid Rain”.

Note: In future we may just create a checkbox for this request in the help form.

Once you have access, you can continue the tutorial. Typically, it will take a business day for you to get added (a business day is 9am-5pm EST Mon-Fri).

7.4.1.2. Log into India¶

$ ssh <username>@india.futuregrid.org
$ module load futuregrid

7.4.1.3. Start the Shell¶

Execute the FG Shell. Please remember to replace <username> with your portal user name. Then, you will be asked for a password (which is your portal password):

$ fg-shell -u <username>

This command will change your shell and the prompt will be fg-rain>. If your prompt is different, you need to execute “use rain”.

7.4.1.4. Generate the Image¶

We are going to generate an Ubuntu 12.04 image. The -s parameter allows you to specify the software you want to install in your image. Currently, only the software available in the official repositories can be installed:

fg-rain> generate -o ubuntu -v 12.04 -a x86_64 -s wget, openmpi-bin

After a while, your image will be stored in the repository, and your image ID will be printed.

7.4.1.5. Image Repository¶

We can consult the information of the image by executing the following (please replace <imageid> with the ID that you got from the previous command):

fg-rain> list * where imgId=<imageId>

7.4.1.6. Register Image¶

Next, we need to register that image in the infrastructure we want to use. In this case, we will register the image in OpenStack (if you followed the FG Openstack tutorial, your novarc will probably be in ~/openstack/novarc). This command will return an <ami-ID>, which is the ID of the image in OpenStack:

fg-rain> register -r <imageId> -s india -v ~/novarc

Once you have registered the image, you need to wait until it becames available. To check the status of the image, you can execute the following command:

fg-rain> cloudlist -s india -v ~/novarc | grep <imageId>

The status is the second field. You may experience that this command takes time to respond when the image is being uploaded (it is an OpenStack issue).

7.4.1.7. Start Image¶

Once the image is in “available” status, we can proceed to start a VM using the ami-ID that we got from the register command. In this case, we are going in Interative mode, which means that we are going to get logged into the VM once it is running:

fg-rain> launch -i <ami-ID> -s india -v ~/novarc -I

7.4.2.1. Requirement¶

• Python 2.6 or 2.7
• LDAP server for user authentication

7.4.2.2. Software¶

The software is open-source under an Apache 2 license. The image repository can be found in github as part of our Image Management and Rain software https://portal.futuregrid.org/rain.

7.4.2.3. Documentation¶

The documentation can be found in http://futuregrid.github.com/rain/index.html. Since this link provides information about all our tools, in this section we collect the links that refer to the image repository.

7.4.3.1. Logging into India¶

$ ssh <username>@india.futuregrid.org
$ module load futuregrid

7.4.3.2. Requesting access¶

Submit a ticket to request access help

7.4.3.3. Obtaining the image¶

You can get your image from our repository, or generate a new one.

$ fg-repo -u <username> -g <imageId>

$ fg-generate -u jdiaz -o ubuntu -v 12.04 -a x86_64 -s wget, openmpi-bin -g

7.4.3.4. Customizing the image¶

To continue with the rest of the tutorial, we need a UNIX machine where we have root privileges.

Assuming that we are in a UNIX machine with root privileges:

1. Retrieve the image from India:

   $ scp <username>@india.futuregrid.org:/N/u/<username/123123123.tgz .
   

2. Decompress the image:

   $ tar vxfz 123123123.tgz
   
   jdiaz859434.img
   jdiaz859434.manifest.xml
   

3. Mount the image. This will mount the image file into a directory. In this way, we will have access to the OS files that are inside the image:

$ mkdir image
$ sudo mount -o loop jdiaz859434.img image

Copy your software into the image directory (if needed). In this case, I am going to copy a software directory to the tmp directory of the image:

$ cp -r /home/javi/mysoftware image/tmp

Chroot into the image. This changes the root of the OS to the one of the image. In this way, evey operation we execute will have effect only inside the image. After executing this command, you will be the root users inside the image:

$ sudo chroot image

Now you can install whatever software you need. You can also use yum/apt to install packages from the software repository. Remember that they will be installed inside the image. As example, here I install tomcat with apt and compile my software with make:

# apt-get install tomcat6
# cd /tmp/mysoftware
# make && make install

You can also configure the OS of image to start services during the boot time. The easiest way to do that is by modifying the/etc/rc.localfile.

Once you have finished customizing your image, you need to exit from the chroot command, unmount the image, and compress it again:

# exit
$ sudo umount image
$ tar vxfz mynewimage.tgz jdiaz859434.img jdiaz859434.manifest.xml

7.4.3.5. Transfer the image back to India¶

$ scp mynewimage.tgz <username>@india.futuregrid.org:/N/u/<username>/

7.4.3.6. Log into India¶

$ ssh <username>@india.futuregrid.org
$ module load futuregrid

7.4.3.7. Upload the image to the repository¶

We upload the image to the repository so we can reuse it and share it with other users. When uploading the image, you can specify some metadata to describe the properties of the image:

$ fg-repo -p mynewimage.tgz "os=Ubuntu12 & arch=x86_64 & description=My new customized image & tag=tomcat, openmpi"

This command will provide you the ID your image has in the repository. This ID is needed for the next step.

7.4.3.8. Register your image in different infrastructures¶

You now have your image ready to be registered in the different FutureGrid infrastructures. To use OpenStack (tutorial <openstackhavana.html>) and Eucalyptus (tutorial <eucalyptus.html>), you need to indicate the location of you novarc and eucarc file.

$ fg-register -u <username> -r <imageID> -s india -v ~/novarc

$ fg-register -u <username> -r <imageID> -e india -v ~/eucarc

$ fg-register -u <username> -r <imageID> -x india

7.4.3.9. Using your Registered Image¶

OpenStack (more info in openstack <openstackhavana.html>):

source novarc
euca-run-instance -k <keyname> <ami-ID>

Eucalyptus (more info in eucalyptus <eucalyptus.html>):

source eucarc
euca-run-instance -k <keyname> <ami-ID>

HPC (more info in hpc):

Provision a machine with our image and go into Interactive mode (you are logged into the machine):

qsub -l os=<ami-ID> -I

Provision two machines with our image and execute an script:

qsub -l os=<ami-ID> -l nodes=2:ppn=8 myscript.sh

7.4.4.1. fg-portal-manage¶

The tool is for admin purpose only. An admin person or an system script can call this to get user/project information from the portal:

fg-portal-manage
 -h, --help            show this help message and exit
 -n NAME, --name=NAME  querying by name
 -m MAIL, --mail=MAIL  querying by email
 -i UID, --uid=UID     querying by portal uid
 -u USERNAME, --username=USERNAME
                       querying by portal username
 -k, --attrib          specifying attrib(s) to be displayed, inlcuding:
                       firstname, lastname, email, phone, organization,
                       institution_name, citizenship, sshkey, projectall,
                       project
 -l, --ldif            output ldif format of user info(no dn). If provided
                       all attributes will be included in the output
                       disregarding the setting from -k
 -v, --vetted          Show only those vetted users. If omitted all users
                       that meet the creteria will be shown
 -w, --waiting         Show the waiting list in which a user is eligible to
                       get an LDAP account
 -y, --withsshkey      A sub-option for -w, which shows the waiting list, but
                       only for those who have submitted a sshkey
 -p, --project         Get list of project in table view
 -s STATUS, --status=STATUS
                       A sub-option for -p, which specifies project status
                       that would like to be retrieved. Try pending,
                       approved, completed, denied
 -1, --pidonly         A sub-option for -p which prints out only the project

FutureGrid account admin tool queries user profile info from the portal and also checks status against the ldap.

To run the script, python 2.7 is needed (if trying on lower version, try to install the argparse module; it may also work), as well as the mysqldb module. Depending on the os distribution, the installation could be different. E.g., in ubuntu, it’s simply: apt-get install python-mysqldb .

The futuregrid.cfg file needs to be put into the same directory as the script, and the dummy config values need to be replaced with those real ones. (Whoever has access to the portal server should have knowledge on this. We will not distribute this to anymore other than the portal admin and sys admin.)

Run this to get the help info:

./fg-portal-manage.py -h

This will give a table viewed info of user with uid 3:

./fg-portal-manage.py -i 3

However, the following:

./fg-portal-manage.py -i 3 -l

will print the long quasi-ldif formatted info.

Other command:

-u for username; -n for realname(first or last); as well as -m for email does similar thing

./fg-portal-manage.py -l

-will simply print out all users in the portal in the ldif-like format.

./fg-portal-manage.py -l -v

-will print out a similar list but only for (all) those vetted users, i.e., approved users with an active project.

::

./fg-portal-manage.py -w

-prints out all users who have approved portal account and are members of an active project(s) (i.e., vetted users), but don’t have an LDAP account yet.

./fg-portal-manage.py -l -w

-the same as -w but in long quasi-ldif format.

./fg-portal-manage.py -w -y

-the same as -w, but prints out only those who have submitted a sshkey.

./fg-portal-manage.py -p

-lists all FG projects registered in the portal, in the format of:

pid: title|project lead|project manager|status|[members list]

-project lead/manager and members are represented by the portaluid.

./fg-portal-manage.py -p -s approved

-prints a list of projects but only those in ‘approved’ status. Other valid statuses are: pending, completed, denied.

./fg-portal-manage.py -p -s completed -1(number '1')

-prints a list of completed projects but only display the projectids.