The FutureGrid shell simplifies the access to the different FutureGrid software components. This shell is a common entry point for every components that offer a customized environment where only the FutureGrid commands are available. It has features similar to the regular GNU/Linux shell like command auto-completion.
usage: fg-shell [-h] -u user [-q] [-i] [-f script_file] [--nopasswd]
Options between brackets are not required. Parenthesis means that you need to specify one of the options.
Option | Description |
-h/--help | Shows help information and exit. |
-u/--user <userName> | FutureGrid HPC user name, that is, the one used to login into the FG resources. |
-q/--quiet | Prevent to load banner and welcome messages |
-i/--interactive | After the commands are interpreted the shell is put into interactive mode |
-f/--file <script_file> | Execute commands from a file. |
--nopasswd | If this option is used, the password is not requested. This is intended for systems daemons like Inca. |
The shell is executed by typing:
fg-shell -u <username>
After executing the previous command and typing your password, the prompt should change to fg>.
Note
Using shell from outside FutureGrid: If the shell is installed outside of FutureGrid, users will not be able to enter in the shell using their passwords. The security reasons, the LDAP server cannot be contacted from outside of FutureGrid. Therefore, users will have to use the option --nopasswd and set their password inside the shell by executing the command setpasswd. In this way, users can authenticate against the different FutureGrid components without typing the password everytime.
This is useful when a user install the shell in his local machine because he will not be able to enter in the shell typing his password.
Note
Context concept: It is essential to understand how to use the shell the concept CONTEXT. A context is an environment specialized for a particular tool or service. This allows us to use only the components we are interested on and organize the commands by component. For example, if we want to use the image repository, we initialize its context by typing use repo and only the image repository commands will be available. See contexts and use commands.
Next, we explain the available commands that you can find inside the FutureGrid Shell.
Commands listed in this section are available in any context.
Help Related Commands
Command
Description
help [command]
Show help message for a command or List of all available commands organized by context if we are in the generic context fg> (see use and context commands). However, if we are in a specific context (e.g. fg-repo>) it only show the generic commands and the specific commands of this particular context.
manual
List help message for all commands in the shell.
shortcuts
List all available shortcuts.
CONTEXT Related Commands
Command
Description
contexts
List of available contexts.
use [context]
Change the Shell CONTEXT to use a specified FG component. If no argument is provided, it returns to the default context.
History Related Commands
Command
Description
history, hist and hi
Show historic of executed commands.
historysession, hists and his
Show historic of the commands executed in the current session.
l, li
List last executed command.
save [N]
Save session history to a file. N => number of command (from historysession), or *. Most recent command if omitted.
Execution Related Commands
Command
Description
load <filename> and exec <filename>
Load commands from an script and stay in the shell
pause [text]
Displays the specified text then waits for the user to press RETURN.
py [command]
py <command>: Executes a Python command. py: Enters interactive Python mode. End with Ctrl-D (Unix) / Ctrl-Z (Windows), quit(), exit().
run, r
Re-run the last executed command
script [filename]
When Script is active, all commands executed are stored in a file. Activate it by executing: script [file]. If no argument is provided, the file will be called script and will be located in your current directory. To finish and store the commands execute: script end
User-Settable Parameters Related Commands
Command
Description
setpassword
Set the password for the current session without leaving the shell. The password is stored encrypted
set [parameter] [value]
Sets a cmd2 parameter. Call without arguments for a list of settable parameters with their values.
show
List of settable parameters with their values.
These commands are available when Image Repository (repo) or Image Management (image) contexts are active. To activate the image repository context execute use repo. If we execute help, we will see which commands are generic and which ones are specific of this context.
Image Related Commands
Command
Description
list [queryString]
Get list of images that meet the criteria.
get <imgId>
Get an image by specifying its unique identifier.
put <imgFile> [attributeString]
Store image into the repository and its metadata defined in attributeString. Default metadata is provided if the argument is missing.
modify <imgId> <attributeString>
Modify the metadata associated with the image.
remove <imgId>
Delete images from the Repository.
setpermission <imgId> <permissionString>
Change the permission of a particular image. Valid values are public, private.
histimg [imgId]
Get usage information an image. If no argument provided, it shows the usage information of all images.
User Related Commands
The following options are available only for users with admin role.
Command
Description
user <options>
Manage image management user’s database.
options
-a/--add <userId> Add a new user to the image management database.
-d/--del <userId> Delete an user from the image management database.
-l, --list List of users.
-m/--modify <userId> <quota/role/status> <value> Modify quota, role or status of an user.
histuser [userId]
Get usage info of an User. If no argument provided, it shows the usage information of all users. This option can be used by normal users to show their own information
These commands are available when the Image Management (image) or the Rain (rain) contexts are active. To activate the image management context execute use image. If we execute help, we will see which commands are generic and which ones are specific of this context.
Command | Description |
generate <options> | Generates images with the requested OS and software stacks specification. options -o/--os <osName> Specify the desired Operating System for the new image. Currently, CentOS and Ubuntu are supported -v/--version <osVersion> Operating System version. In the case of Centos, it can be 5 or 6. In the case of Ubuntu, it can be karmic(9.10), lucid(10.04), maverick(10.10), natty(11.04) -a/--arch <arch> Destination hardware architecture (x86_64 or i386) --baseimage Generate a Base Image that will be used to generate other images. In this way, the image generation process will be faster. -s/--software <software> List of software packages, separated by commas, that will be installed in the image. --scratch Generate the image from scratch without using any Base Image from the repository. -n/--name <givenname> Desired recognizable name of the image. -e/--description <description> Short description of the image and its purpose. -g/--getimg Retrieve the image instead of uploading to the image repository. -z/--size <SIZE> Specify the size of the Image in GigaBytes. The size must be large enough to install all the software required. The default and minimum size is 1.5GB, which is enough for most cases. |
These commands are available when the Image Management (image) or the Rain (rain) contexts are active. To activate the image management context execute use image. If we execute help, we will see which commands are generic and which ones are specific of this context.
Command | Description |
register <options> | Registers images in the selected infrastructures. After this process, images become available for instantiation in such infrastructures. Options -k/--kernel <version> Specify the desired kernel. -i/--image <imgFile> Select the image to register by specifying its location. The image is a tgz file that contains the manifest and image files. -r/--imgid <imgId> Select the image to register by specifying its Id in the repository. -x/--xcat <SiteName> Select the HPC infrastructure named SiteName (minicluster, india ...). -e/--euca <SiteName> Select the Eucalyptus Infrastructure located in SiteName (india, sierra...). -s/--openstack <SiteName> Select the OpenStack Infrastructure located in SiteName (india, sierra...). -n/--nimbus <SiteName> Select the Nimbus Infrastructure located in SiteName (india, sierra...). -o/--opennebula <SiteName> Select the OpenNebula Infrastructure located in SiteName (india, sierra...). -v/--varfile <VARFILE> Path of the environment variable files. Currently this is used by Eucalyptus, OpenStack and Nimbus. -g/--getimg Customize the image for a particular cloud framework but does not register it. So the user gets the image file. -p/--noldap If this option is active, FutureGrid LDAP will not be configured in the image. This option only works for Cloud registrations. LDAP configuration is needed to run jobs using fg-rain -w/--wait Wait until the image is available in the targeted infrastructure. Currently this is used by Eucalyptus and OpenStack. |
cloudlist <options> | List images registered in the Cloud infrastructures. Options -e/--euca <SiteName> Select the Eucalyptus Infrastructure located in SiteName (india, sierra...). -n / --nimbus <SiteName> Select the Nimbus Infrastructure located in SiteName (india, sierra...). -o / --opennebula <SiteName> Select the OpenNebula Infrastructure located in SiteName (india, sierra...). -s / --openstack <SiteName> Select the OpenStack Infrastructure located in SiteName (india, sierra...). -v/--varfile <VARFILE> Path of the environment variable files. Currently this is used by Eucalyptus, OpenStack and Nimbus. |
cloudlistkernels <options> | List kernels available for the Cloud infrastructures. Options -e/--euca <SiteName> Select the Eucalyptus Infrastructure located in SiteName (india, sierra...). -n / --nimbus <SiteName> Select the Nimbus Infrastructure located in SiteName (india, sierra...). -o / --opennebula <SiteName> Select the OpenNebula Infrastructure located in SiteName (india, sierra...). -s / --openstack <SiteName> Select the OpenStack Infrastructure located in SiteName (india, sierra...). |
hpclist <SiteName> | List images registered in the HPC infrastructure named SiteName (minicluster, india ...). |
hpclistkernels <SiteName> | List kernels available for HPC infrastructure named SiteName (minicluster, india ...). |
listsites | List supported sites with their respective HPC and Cloud services. |
deregister <options> | Deregister an image from the specified infrastructure. Options --deregister <imageId> -x/--xcat <SiteName> Select the HPC infrastructure named SiteName (minicluster, india ...). -e/--euca <SiteName> Select the Eucalyptus Infrastructure located in SiteName (india, sierra...). -n / --nimbus <SiteName> Select the Nimbus Infrastructure located in SiteName (india, sierra...). -o / --opennebula <SiteName> Select the OpenNebula Infrastructure located in SiteName (india, sierra...). -s / --openstack <SiteName> Select the OpenStack Infrastructure located in SiteName (india, sierra...). -v/--varfile <VARFILE> Path of the environment variable files. Currently this is used by Eucalyptus, OpenStack and Nimbus. |
These commands are available when the Rain (rain) contexts is active. To activate the rain context execute use rain. If we execute help, we will see which commands are generic and which ones are specific of this context.
Command | Description | |
launch <options> | Provision machines or VMs with the requested OS and execute a job or enter in interactive mode Options -k/--kernel <version> Specify the desired kernel. -i/--image <imgFile> Select the image to register by specifying its location. The image is a tgz file that contains the manifest and image files. -r/--imgid <imgId> Select the image to register by specifying its Id in the repository. -x/--xcat <SiteName> Select the HPC infrastructure named SiteName (minicluster, india ...). -e/--euca <SiteName> Select the Eucalyptus Infrastructure located in SiteName (india, sierra...). -s/--openstack <SiteName> Select the OpenStack Infrastructure located in SiteName (india, sierra...). -n/--nimbus <SiteName> Select the Nimbus Infrastructure located in SiteName (india, sierra...). -o/--opennebula <SiteName> Select the OpenNebula Infrastructure located in SiteName (india, sierra...). -v/--varfile <VARFILE> Path of the environment variable files. Currently this is used by Eucalyptus, OpenStack and Nimbus. -m/--numberofmachines <#instances> Number of machines needed. --volume <size> This creates and attaches a volume of the specified size (in GiB) to each instance. The volume will be mounted in /mnt/. This is supported by Eucalyptus and OpenStack. -t/--instance-type <instancetype> VM Image type to run the instance as. Valid values: [‘m1.small’, ‘m1.large’, ‘m1.xlarge’] -w/--walltime <hours> How long to run (in hours). You may use decimals. This is used for HPC and Nimbus. -j/--jobscript <JOBSCRIPT> Script to execute on the provisioned images. In the case of Cloud environments, the user home directory is mounted in /tmp/N/u/<username>. The /N/u/<username> is only used for ssh between VM and store the ips of the parallel job in a file called /N/u/<username>/machines -I/--interactive Interactive mode. It boots VMs or provisions bare-metal machines. Then, the user is automatically logged into one of the VMs/machines. -b, --background Background mode. It boots VMs or provisions bare-metal machines. Then, it gives you the information you need to know to log in anytime. -w/--wait Wait until the image is available in the targeted infrastructure. Currently this is used by Eucalyptus and OpenStack. |
|
launchhadoop <options> | Provision machines or VMs with the requested OS, install/configure Hadoop and execute a job or enter in interactive mode. Options Same options that launch. Additional Hadoop Option --inputdir <inputdir> Location of the directory containing the job input data that has to be copied to HDFS. The HDFS directory will have the same name. Thus, if this option is used, the job script has to specify the name of the directory (not to all the path). --outputdir <outputdir> Location of the directory to store the job output data from HDFS. The HDFS directory will have the same name. Thus, if this option is used, the job script has to specify the name of the directory (not to all the path). -hdfsdir <hdfsdir> Location of the HDFS directory to use in the machines. If not provided /tmp/ will be used. |
|
cloudinstanceslist <options> | List the information of the instance/s submitted to the selected cloud. Options -i/--instance [InstanceId/s] Id of the instance to check status. This is optional, if not provided all instances will be listed. -e/--euca <SiteName> Select the Eucalyptus Infrastructure located in SiteName (india, sierra...). -s/--openstack <SiteName> Select the OpenStack Infrastructure located in SiteName (india, sierra...). -n/--nimbus <SiteName> Select the Nimbus Infrastructure located in SiteName (india, sierra...). -o/--opennebula <SiteName> Select the OpenNebula Infrastructure located in SiteName (india, sierra...). -v/--varfile <VARFILE> Path of the environment variable files. Currently this is used by Eucalyptus, OpenStack and Nimbus. |
|
hpcjobslist [job/s] | List the information of the HPC job/s. | |
cloudinstancesterminate <options> | Terminate instance/s from the selected cloud. You can specify a list of instances ids and also reservations ids. Options -i/--instance <InstanceId/s> Id/s of the instance/s or reservation/s to terminate. -e/--euca <SiteName> Select the Eucalyptus Infrastructure located in SiteName (india, sierra...). -s/--openstack <SiteName> Select the OpenStack Infrastructure located in SiteName (india, sierra...). -n/--nimbus <SiteName> Select the Nimbus Infrastructure located in SiteName (india, sierra...). -o/--opennebula <SiteName> Select the OpenNebula Infrastructure located in SiteName (india, sierra...). -v/--varfile <VARFILE> Path of the environment variable files. Currently this is used by Eucalyptus, OpenStack and Nimbus. |
|
hpcjobsterminate <job/s> | Terminate HPC job/s. |
Context Usage
Show list of available contexts
$ fg-shell -u jdiaz
fg> contexts
The output shows all available contexts
FG Contexts:
------------
repo
image
rain
hadoop
Users can select any of the previous contexts with the use command. Then, the environment of this particular context is initialized.
fg> use repo
fg-repo>
Return to the normal context
fg-repo> use
fg>
Help Usage
List available commands in the generic context
$ fg-shell -u jdiaz
fg> help
The output shows the list of generic commands and the list of commands that are available in each of the contexts. Note that the commands listed for each context are only available when that particular context has been loaded. Some contexts load other contexts as part of their requirements, as we explained before.
A complete manual can be found in https://portal.futuregrid.org/man/fg-shell
Generic Documented commands (type help <topic>):
================================================
contexts history load py save setpasswd use
exec historysession manual quit script shortcuts
help li pause run set show
Image Repository commands. Execute "use repo" to use them. (type help <topic>):
===============================================================================
get histimg histuser list modify put remove setpermission user
Image Management commands. Execute "use image" to use them. (type help <topic>):
================================================================================
cloudlist cloudlistkernels generate hpclist hpclistkernels register
FG Dynamic Provisioning commands. Execute "use rain" to use them. (type help <topic>):
======================================================================================
launch launchhadoop
Please select a CONTEXT by executing use <context_name>
Execute 'contexts' command to see the available context names
List available commands in the image context (this contexts also loads the repo contexts)
fg> use image
fg-image> help
The output is something like this.
A complete manual can be found in https://portal.futuregrid.org/man/fg-shell
General documented commands (type help <topic>):
================================================
contexts history load py save setpasswd use
exec historysession manual quit script shortcuts
help li pause run set show
Specific documented commands in the repo context (type help <topic>):
=====================================================================
get histimg histuser list modify put remove setpermission user
Specific documented commands in the image context (type help <topic>):
======================================================================
cloudlist cloudlistkernels generate hpclist hpclistkernels register
General Shell Usage
Session example where we get an image, list all the images which os is centos, add an user and activate it.
$ fg-shell
fg> use repo
fg-repo> get image123123123
fg-repo> list * where os=centos
fg-repo> user -a javi
fg-repo> user -m javi status active
Record the executed commands in an script.
$fg-shell
fg> script myscript.txt
fg> use repo
fg-repo> put /tmp/image.img vmtype=xen & imgtype=opennebula & os=linux & arch=x86_64
fg-repo> list
fg-repo> script end
use repo put /tmp/image.img vmtype=xen & imgtype=opennebula & os=linux & arch=x86_64 list
Execute shell commands stored in a file. Then exits from the shell
$ cat myscript.txt| fg-shell
Execute shell commands stored in a file from the shell. This stay in the shell.
$ fg-shell -u jdiaz
fg> load myscript.txt