SUBMIT(1) General Commands Manual SUBMIT(1) NAME qsub - submit a batch job to Univa Grid Engine. qsh - submit an interactive X-windows session to Univa Grid Engine. qlogin - submit an interactive login session to Univa Grid Engine. qrsh - submit an interactive rsh session to Univa Grid Engine. qalter - modify a pending or running batch job of Univa Grid Engine. qresub - submit a copy of an existing Univa Grid Engine job. SYNTAX qsub [ options ] [ command [ command_args ] | -- [ command_args ]] qsh [ options ] [ -- xterm_args ] qlogin [ options ] qrsh [ options ] [ command [ command_args ]] qalter [ options ] wc_job_range_list [ -- [ command_args ]] qalter [ options ] -u user_list | -uall [ -- [ command_args ]] qresub [ options ] job_id_list DESCRIPTION Qsub submits batch jobs to the Univa Grid Engine queuing system. Univa Grid Engine supports single- and multiple-node jobs. Command can be a path to a binary or a script (see -b below) which contains the commands to be run by the job using a shell (for example, sh(1) or csh(1)). Arguments to the command are given as command_args to qsub . If com- mand is handled as a script then it is possible to embed flags in the script. If the first two characters of a script line either match '#$' or are equal to the prefix string defined with the -C option described below, the line is parsed for embedded command flags. Qsh submits an interactive X-windows session to Univa Grid Engine. An xterm(1) is brought up from the executing machine with the display directed either to the X-server indicated by the DISPLAY environment variable or as specified with the -display qsh option. Interactive jobs are not spooled if no resource is available to execute them. They are either dispatched to a suitable machine for execution immediately or the user submitting the job is notified by qsh that appropriate resources to execute the job are not available. xterm_args are passed to the xterm(1) executable. Note, however, that the -e and -ls xterm options do not work with qsh . Qlogin is similar to qsh in that it submits an interactive job to the queuing system. It does not open an xterm(1) window on the X display, but uses the current terminal for user I/O. Usually, qlogin establishes a telnet(1) connection with the remote host, using standard client- and server-side commands. These commands can be configured with the qlo- gin_daemon (server-side, Univa Grid Engine telnetd if not set, other- wise something like /usr/sbin/in.telnetd) and qlogin_command (client- side, Univa Grid Engine telnet if not set, otherwise something like /usr/bin/telnet) parameters in the global and local configuration set- tings of sge_conf(5). The client side command is automatically parame- terized with the remote host name and port number to which to connect, resulting in an invocation like /usr/bin/telnet my_exec_host 2442 for example. Qlogin is invoked exactly like qsh and its jobs can only run on INTERACTIVE queues. Qlogin jobs can only be used if the sge_execd(8) is running under the root account. Qrsh is similar to qlogin in that it submits an interactive job to the queuing system. It uses the current terminal for user I/O. Usually, qrsh establishes a rsh(1) connection with the remote host. If no com- mand is given to qrsh, an rlogin(1) session is established. It inherits all SGE_ environment variables plus SHELL, HOME, TERM, LOGNAME, TZ, HZ, PATH and LANG. The server-side commands used can be configured with the rsh_daemon and rlogin_daemon parameters in the global and local configuration settings of sge_conf(5). An Univa Grid Engine rshd or rlogind is used if the parameters are not set. If the parameters are set, they should be set to something like /usr/sbin/in.rshd or /usr/sbin/in.rlogind. On the client-side, the rsh_command and rlogin_command parameters can be set in the global and local configura- tion settings of sge_conf(5). If they are not set, special Univa Grid Engine rsh(1) and rlogin(1) binaries delivered with Univa Grid Engine are used. Use the cluster configuration parameters to integrate mecha- nisms like ssh or the rsh(1) and rlogin(1) facilities supplied with the operating system. Qrsh jobs can only run in INTERACTIVE queues unless the option -now no is used (see below). They can also only be run, if the sge_execd(8) is running under the root account. Qrsh provides an additional useful feature for integrating with inter- active tools providing a specific command shell. If the environment variable QRSH_WRAPPER is set when qrsh is invoked, the command inter- preter pointed to by QRSH_WRAPPER will be executed to run qrsh commands instead of the users login shell or any shell specified in the qrsh command-line. The options -cwd, -v, -V, and -display only apply to batch jobs. Qalter can be used to change the attributes of pending jobs. For array jobs with a mix of running and pending tasks (see the -t option below), modification with qalter only affects the pending tasks. Qalter can change most of the characteristics of a job (see the corresponding statements in the OPTIONS section below), including those which were defined as embedded flags in the script file (see above). Some submit options, such as the job script, cannot be changed with qalter. Qresub allows the user to create jobs as copies of existing pending or running jobs. The copied jobs will have exactly the same attributes as the ones from which they were copied, except with a new job ID and with a cleared hold state. The only modification to the copied jobs sup- ported by qresub is assignment of a new hold state with the -h option. This option can be used to first copy a job and then change its attributes via qalter. Only a manager can use qresub on jobs submitted by another user. Regu- lar users can only use qresub on their own jobs. For qsub, qsh, qrsh, and qlogin the administrator and the user may define default request files (see sge_request(5)) which can contain any of the options described below. If an option in a default request file is understood by qsub and qlogin but not by qsh the option is silently ignored if qsh is invoked. Thus you can maintain shared default request files for both qsub and qsh. A cluster wide default request file may be placed under $SGE_ROOT/$SGE_CELL/common/sge_request. User private default request files are processed under the locations $HOME/.sge_request and $cwd/.sge_request. The working directory local default request file has the highest precedence, then the home directory located file and then the cluster global file. The option arguments, the embedded script flags and the options in the default request files are processed in the following order: left to right in the script line, left to right in the default request files, from top to bottom of the script file (qsub only), from top to bottom of default request files, from left to right of the command line. In other words, the command line can be used to override the embedded flags and the default request settings. The embedded flags, however, will override the default settings. Note, that the -clear option can be used to discard any previous set- tings at any time in a default request file, in the embedded script flags, or in a command-line option. It is, however, not available with qalter. The options described below can be requested either hard or soft. By default, all requests are considered hard until the -soft option (see below) is encountered. The hard/soft status remains in effect until its counterpart is encountered again. If all the hard requests for a job cannot be met, the job will not be scheduled. Jobs which cannot be run at the present time remain spooled. OPTIONS -@ optionfile Forces qsub, qrsh, qsh, or qlogin to use the options contained in optionfile. The indicated file may contain all valid options. Comment lines must start with a "#" sign. -a date_time Available for qsub and qalter only. Defines or redefines the time and date at which a job is eligi- ble for execution. Date_time conforms to [[CC]]YY]MMDDhhmm[.SS], for the details, please see date_time in sge_types(1). If this option is used with qsub or if a corresponding value is specified in qmon then a parameter named a with the format CCYYMMDDhhmm.SS will be passed to the defined JSV instances (see -jsv option below or find more information concerning JSV in jsv(1)). -ac variable[=value],... Available for qsub, qsh, qrsh, qlogin and qalter only. Adds the given name/value pair(s) to the job's context. Value may be omitted. Univa Grid Engine appends the given argument to the list of context variables for the job. Multiple -ac, -dc, and -sc options may be given. The order is important here. The variable name must not start with the letters "+", "-" or "=". The outcome of the evaluation of all -ac, -dc, and -sc options is passed to defined JSV instances as parameter with the name ac. (see -jsv option below or find more information concerning JSV in jsv(1)). QALTER allows changing this option even while the job executes. -adds parameter key value Available for qsub, qrsh and qalter of Univa Grid Engine only. Gives the user the possibility to add additional entries to list based job parameters like resource requests, job context, envi- ronment variables and more. The -mods and -clears switches can be used to modify or remove a single entry of a job parameter list. The parameter argument specifies the job parameter that should be enhanced. The names used here are names of command line switches that are also used in job classes or JSV to address job parameters. Currently the -adds switch supports following param- eters: ac, CMDARG, cwd, e, hold_jid, i, l_hard, l_soft, M, mas- terl, masterq, o, q_hard, q_hard, rou, S and v. The same set of parameters is also supported by the -mods and -clears switches. The -clearp switch allows to reset all list based parameters mentioned above and also non-list based parameters. Find corre- sponding non-list based parameter names in the -clearp section below. Please note that the cwd parameter is a list-based parameter that can be addressed with the -adds, -mods and -clears switches although this list can only have one entry. The key argument depends on the used parameter argument. For the ac and v parameter it has to specify the name of a variable that should either be added to the job context or environment vari- able list. For the parameters o, i, e or S it is a hostname. An empty key parameter might be used to define a default value that is not host specific. The key of l_hard or l_soft has to refer to a resource name (name of a complex entry) whereas q_hard, q_soft and masterq expect a queue name. CMDARG expects a string that should be passed as command line argument, hold_jid a name or job ID of a job and M a mail address. All parameter/key combinations expect a value argument. For CMDARG, q_hard, q_soft, hold_jid, M and rou parameter this value has to be an empty argument. ac, v, l_hard and l_soft allow also empty values. Independent of the position within the command line the switches -adds, -mods and -clears will be evaluated after modifications of all other switches that will be passed to q submit command or qalter and the sequence in which they are applied is the same as specified on the command line. If the -adds parameter is used to change a list based job param- eter that was derived from a job class, then this operation might be rejected by the Univa Grid Engine system if within the job class access specifiers were used that do not allow to add new elements to the list. (see -jc option below or find more information concerning job classes and access specifiers in sge_job_class(5)). -ar ar_id Available for qsub, qalter, qrsh, qsh, or qlogin only. Assigns the submitted job to be a part of an existing Advance Reservation. The complete list of existing Advance Reservations can be obtained using the qrstat(1) command. Note that the -ar option adds implicitly the -w e option if not otherwise requested. Qalter allows changing this option even while the job executes. The modified parameter will only be in effect after a restart or migration of the job however. When this option is used for a job or when a corresponding value in qmon is specified then this value will be passed to defined JSV instances as parameter with the name ar. (see -jsv option below or find more information concerning JSV in jsv(1)) -A account_string Available for qsub, qsh, qrsh, qlogin and qalter only. Identifies the account to which the resource consumption of the job should be charged. The account_string should conform to the name definition in sge_types(1). In the absence of this parame- ter Univa Grid Engine will place the default account string "sge" in the accounting record of the job. Qalter allows changing this option even while the job executes. If this option or a corresponding value in qmon is specified then this value will be passed to defined JSV instances as parameter with the name A. (see -jsv option below or find more information concerning JSV in jsv(1)) -bgio bgio_params This option will bypass the problem that the controlling termi- nal will suspend the qrsh process when it is reading from STDIN or writing to STDOUT/STDERR file descriptors. Available for qrsh with builtin interactive job support mecha- nism only. Supported if e.g. "&" is used to start the qrsh in the back- ground of a terminal. The terminal must support job control and must also have a supported tty assigned. Supported bgio_params options are nr (no read), fw (forced write) and bw= (buffered write up to the specified buffer size). The combination of the options is supported by using the "," character as delimiter (no spaces allowed): bgio_params nr|bw=|fw[,nr|bw=|fw,...] nr: no read If the user terminal supports job control the qrsh will not read from STDIN when it is running in background. If a user is entering some input in the terminal the default behavior often is that the process running in the background is suspended when it reads the user input from STDIN. This is done by the user's terminal for all background jobs which try to read from STDIN. By using the "nr" option the qrsh will not read from STDIN as long it is running in the background. fw: force write If the "stty tostop" option is active for the user's terminal any job running in the background of the terminal will be sus- pended when it tries to write to STDOUT or STDERR. The "fw" option is used to tell qrsh to ignore this setting and force writing without being suspended. bw=: buffered write If the user terminal has the "stty tostop" option set (back- ground jobs will be suspended when writing to STDOUT or STDERR) it is possible to simply buffer the messages in the qrsh client to avoid being suspended by using this option. The qrsh will write to STDOUT or STDERR if one of the following items occur: - When the process is in foreground again - When the buffer is full - When the qrsh is terminating -binding [ binding_instance ] binding_strategy A job can request a specific processor core binding (processor affinity) with this parameter. This request is treated since version 8.1 as a hard resource request, i.e. the job is only dispatched to an host which is able to fulfill the request. In contrast to previous versions the request is now processed in the Univa Grid Engine scheduler component. To enforce Univa Grid Engine to select a specific hardware architecture please use the -l switch in combination with the complex attribute m_topology. binding_instance is an optional parameter. It might either be env, pe or set depending on which instance should accomplish the job to core binding. If the value for binding_instance is not specified then set will be used. env means that only the environment variable SGE_BINDING will be exported to the job environment of the job. This variable con- tains the selected operating system internal processor numbers. They might be more than selected cores in presence of SMT or CMT because each core could be represented by multiple processor identifiers. The processor numbers are space separated. This variable is also available in case of real core binding when set or pe was requested. pe means that the information about the selected cores appears in the fourth column of the pe_hostfile. Here the logical core and socket numbers are printed (they start at 0 and have no holes) in colon separated pairs (i.e. 0,0:1,0 which means core 0 on socket 0 and core 0 on socket 1). For more information about the $pe_hostfile check sge_pe(5) set (default if nothing else is specified). The binding strategy is applied by Univa Grid Engine. How this is achieved depends on the underlying operating system of the execution host where the submitted job will be started. On Solaris 10 hosts a processor set will be created where the job can exclusively run in. Because of operating system limita- tions at least one core must remain unbound. This resource could of course used by an unbound job. On Linux (lx-amd64 or lx-x86) hosts a processor affinity mask will be set to restrict the job to run exclusively on the selected cores. The operating system allows other unbound pro- cesses to use these cores. Please note that on Linux the bind- ing requires a Linux kernel version of 2.6.16 or greater. It might be even possible to use a kernel with lower version number but in that case additional kernel patches have to be applied. The loadcheck tool in the utilbin directory can be used to check if the hosts capabilities. You can also use the -sep in combi- nation with -cb of qconf(5) command to identify if Univa Grid Engine is able to recognize the hardware topology. Possible values for binding_strategy are as follows: linear:[:,] striding::[:,] explicit:[,;...], For the binding strategy linear and striding there is an optional socket and core pair attached. These denotes the manda- tory starting point for the first core to bind on. linear means that Univa Grid Engine tries to bind the job on amount successive cores. If socket and core is omitted then Univa Grid Engine first allocates successive cores on the first empty socket found. Empty means that there are no jobs bound to the socket by Univa Grid Engine. If this is not possible or is not sufficient Univa Grid Engine tries to find (further) cores on the socket with the most unbound cores and so on. If the amount of allocated cores is lower than requested cores, no binding is done for the job. If socket and core is specified then Univa Grid Engine tries to find amount of empty cores beginning with this starting point. If this is not possible then binding is not done. In order to force a job to be scheduled to at most one socket a special complex value needs to be requested together with the linear request. Please note that if there is no host, which offers at least the requested amount of cores per socket, the job is never going to be scheduled. For such jobs always the socket with the highest number of free cores per socket is used. The special complex needs to be created in the complex configu- ration and needs to have the name: sched_binding_per_socket The configuration of the complex is meaningless for this feature, i.e. it can be freely adapted. Please also note that the com- plex needs to be initialized in order to be usable, either on global, host local, or on queue level in the complex_values field. striding means that Univa Grid Engine tries to find cores with a certain offset. It will select amount of empty cores with a offset of n -1 cores in between. Start point for the search algorithm is socket 0 core 0. As soon as amount cores are found they will be used to do the job binding. If there are not enough empty cores or if correct offset cannot be achieved then there will be no binding done. explicit binds the specified sockets and cores that are men- tioned in the provided socket/core list. Each socket/core pair has to be specified only once. If a socket/core pair is already in use by a different job the whole binding request will be ignored. Qalter allows changing this option even while the job executes. The modified parameter will only be in effect after a restart or migration of the job, however. If this option or a corresponding value in qmon is specified then these values will be passed to defined JSV instances as parameters with the names binding_strategy, binding_type, bind- ing_amount, binding_step, binding_socket, binding_core, bind- ing_exp_n, binding_exp_socket, binding_exp_core. Please note that the length of the socket/core value list of the explicit binding is reported as binding_exp_n. will be replaced by the position of the socket/core pair within the explicit list (0 <= id < binding_exp_n). The first socket/core pair of the explicit binding will be reported with the parameter names binding_exp_socket0 and binding_exp_core0. Following values are possible for binding_strategy: linear_auto- matic, linear, striding, striding_automatic, and explicit. The value linear_automatic corresponds to the command line request -binding linear:. Hence binding_amount must be set to the amount of requested cores. The value linear corresponds to the command line request "-binding linear::,". Additionally to the binding_amount the start socket (bind- ing_socket) and start core (binding_core) must be set. Otherwise the request is treated as "-binding linear::0,0" which is different to "-binding linear:". The same rules apply to striding_automatic and striding. In the automatic case the scheduler seeks free cores itself while in non-automatic case the scheduler starts to fill up cores at the position given with binding_socket and binding_core if possible (otherwise it skips the host). Values that do not apply for the specified binding will not be reported to JSV. E.g. binding_step will only be reported for the striding binding and all binding_exp_* values will passed to JSV if explicit binding was specified. If the binding strategy should be changed with JSV, it is impor- tant to set all parameters that do not belong to the selected binding strategy to zero, to avoid combinations that could get rejected. E.g., if a job requesting striding via commandline should be changed to linear, the JSV has to set binding_step and possibly binding_exp_n to zero, in addition to changing bind- ing_strategy (see -jsv option below or find more information concerning JSV in jsv(1)). -b y[es]|n[o] Available for qsub, qrsh only. Univa Grid Engine also supports the modification with qalter. Gives the user the possibility to indicate explicitly whether command should be treated as binary or script. If the value of -b is 'y', then command may be a binary or script. The command might not be accessible from the submission host. Nothing except the path of the command will be transferred from the sub- mission host to the execution host. Path aliasing will be applied to the path of command before command will be executed. If the value of -b is 'n' then command needs to be a script and it will be handled as script. The script file has to be accessi- ble by the submission host. It will be transferred to the execu- tion host. qsub/qrsh will search directive prefixes within script. qsub will implicitly use -b n whereas qrsh will apply the -b y option if nothing else is specified. Qalter can only be used to change the job type from binary to script when a script is specified additionally with -CMDNAME. Please note that submission of command as script (-b n) can have a significant performance impact, especially for short running jobs and big job scripts. Script submission adds a number of operations to the submission process: The job script needs to be - parsed at client side (for special comments) - transferred from submit client to qmaster - spooled in qmaster - transferred to execd at job execution - spooled in execd - removed from spooling both in execd and qmaster once the job is done If job scripts are available on the execution nodes, e.g. via NFS, binary submission can be the better choice. The value specified with this option or the corresponding value specified in qmon will only be passed to defined JSV instances if the value is yes. The name of the parameter will be b. The value will be y also when then long form yes was specified dur- ing submission. (see -jsv option below or find more information concerning JSV in jsv(1)). -CMDNAME command Only available in Univa Grid Engine. Available for qalter only. Changes the command (script or binary) to be run by the job. In combination with the -b switch it is possible to change binary jobs to script jobs and vice versa. The value specified as command during the submission of a job will be passed to defined JSV instances as parameter with the name CMDNAME. (see -jsv option below or find more information concerning JSV in jsv(1)) -c occasion_specifier Available for qsub and qalter only. Defines or redefines whether the job should be checkpointed, and if so, under what circumstances. The specification of the check- pointing occasions with this option overwrites the definitions of the when parameter in the checkpointing environment (see checkpoint(5)) referenced by the qsub -ckpt switch. Possible values for occasion_specifier are n no checkpoint is performed. s checkpoint when batch server is shut down. m checkpoint at minimum CPU interval. x checkpoint when job gets suspended. checkpoint in the specified time interval. The minimum CPU interval is defined in the queue configuration (see queue_conf(5) for details). has to be specified in the format hh:mm:ss. The maximum of and the queue's minimum CPU interval is used if is specified. This is done to ensure that a machine is not overloaded by checkpoints being generated too frequently. The value specified with this option or the corresponding value specified in qmon will be passed to defined JSV instances. The will be available as parameter with the name c_inter- val. The character sequence specified will be available as parameter with the name c_occasion. Please note that if you change c_occasion via JSV then the last setting of c_interval will be overwritten and vice versa. (see -jsv option below or find more information concerning JSV in jsv(1)) -ckpt ckpt_name Available for qsub and qalter only. Selects the checkpointing environment (see checkpoint(5)) to be used for checkpointing the job. Also declares the job to be a checkpointing job. If this option or a corresponding value in qmon is specified then this value will be passed to defined JSV instances as parameter with the name ckpt. (see -jsv option below or find more information concerning JSV in jsv(1)) -clear Available for qsub, qsh, qrsh, and qlogin only. Causes all elements of the job to be reset to the initial default status prior to applying any modifications (if any) appearing in this specific command. -clearp parameter Available for qsub, qrsh and qalter of Univa Grid Engine only. Gives the user the possibility to clear list bases and non-list based job parameters. As result the specified job parameter will be reset to the same default value that would also be used when a job is submitted with the qsub command without an addi- tional specification of parameter (e.g -clearp N would reset the job name to the default name. For script based jobs this is the basename of the command script). If a job is derived from a job class and if the access speci- fiers that is defined before (or within a list based attribute) does not allow to delete the parameter then the use of the -clearp switch is forbidden for the corresponding entry. (see -jc option below or find more information concerning job classes and access specifiers in sge_job_class(5)). The parameter argument might either be the name of a list based job parameter as explained in the section -adds above or it might be a non-list parameter. Non-list parameters names are a, A, ar, binding, ckpt, c_occasion, c_interval, dl, j, js, m, mbind, N, now, notify, P, p, pe_name, pe_range, r and shell. -clears parameter key Available for qsub, qrsh and qalter of Univa Grid Engine only. Gives the user the possibility to remove single entries of list based job parameters like resource requests, job context, envi- ronment variables and more. The -adds and -mods switches can be used to add or modify a sin- gle entry of a job parameter list. If a job is derived from a job class and if the access specifier that is defined before or within a list based attribute does not allow the removal of a specific entry from the list then the use of the -clears switch is forbidden for the corresponding entry. (see -jc option below or find more information concerning job classes and access specifiers in sge_job_class(5)). Parameter and key arguments are explained in more detail in the -adds section above. -cwd Available for qsub, qsh, qrsh and qalter only. Execute the job from the current working directory. This switch will activate Univa Grid Engine's path aliasing facility, if the corresponding configuration files are present (see sge_aliases(5)). In the case of qalter, the previous definition of the current working directory will be overwritten if qalter is executed from a different directory than the preceding qsub or qalter. Qalter allows changing this option even while the job executes. The modified parameter will only be in effect after a restart or migration of the job, however. If this option or a corresponding value in qmon is specified then this value will be passed to defined JSV instances as parameter with the name cwd. The value of this parameter will be the absolute path to the working directory. JSV scripts can remove the path from jobs during the verification process by setting the value of this parameter to an empty string. As a result the job behaves as if -cwd was not specified during job submission. (see -jsv option below or find more information concerning JSV in jsv(1)) -C prefix_string Available for qsub and qrsh with script submission (-b n). Prefix_string defines the prefix that declares a directive in the job's command. The prefix is not a job attribute, but affects the behavior of qsub and qrsh. If prefix is a null string, the command will not be scanned for embedded directives. The directive prefix consists of two ASCII characters which, when appearing in the first two bytes of a script line, indicate that what follows is an Univa Grid Engine command. The default is "#$". The user should be aware that changing the first delimiting character can produce unforeseen side effects. If the script file contains anything other than a "#" character in the first byte position of the line, the shell processor for the job will reject the line and may exit the job prematurely. If the -C option is present in the script file, it is ignored. -dc variable,... Available for qsub, qsh, qrsh, qlogin and qalter only. Removes the given variable(s) from the job's context. Multiple -ac, -dc, and -sc options may be given. The order is important. Qalter allows changing this option even while the job executes. The outcome of the evaluation of all -ac, -dc, and -sc options is passed to defined JSV instances as parameter with the name ac. (see -jsv option below or find more information concerning JSV in jsv(1)). -display display_specifier Available for qsh and qrsh with command. Directs xterm(1) to use display_specifier in order to contact the X server. The display_specifier has to contain the hostname part of the display name (e.g. myhost:1). Local display names (e.g. :0) cannot be used in grid environments. Values set with the -display option overwrite settings from the submission envi- ronment and from -v command line options. If this option or a corresponding value in qmon is specified then this value will be passed to defined JSV instances as parameter with the name display. This value will also be avail- able in the job environment which might optionally be passed to JSV scripts. The variable name will be DISPLAY. (see -jsv option below or find more information concerning JSV in jsv(1)) -dl date_time Available for qsub, qsh, qrsh, qlogin and qalter only. Specifies the deadline initiation time in [[CC]YY]MMDDhhmm[.SS] format (see -a option above). The deadline initiation time is the time at which a deadline job has to reach top priority to be able to complete within a given deadline. Before the deadline initiation time the priority of a deadline job will be raised steadily until it reaches the maximum as configured by the Univa Grid Engine administrator. This option is applicable only for users allowed to submit dead- line jobs. If this option or a corresponding value in qmon is specified then this value will be passed to defined JSV instances as parameter with the name dl. The format for the date_time value is CCYYMMDDhhmm.SS (see -jsv option below or find more informa- tion concerning JSV in jsv(1)) -e [[hostname]:]path,... Available for qsub, qsh, qrsh, qlogin and qalter only. Defines or redefines the path used for the standard error stream of the job. For qsh, qrsh and qlogin only the standard error stream of prolog and epilog is redirected. If the path consti- tutes an absolute path name, the error-path attribute of the job is set to path, including the hostname. If the path name is rel- ative, Univa Grid Engine expands path either with the current working directory path (if the -cwd switch (see above) is also specified) or with the home directory path. If hostname is present, the standard error stream will be placed in the corre- sponding location only if the job runs on the specified host. If the path contains a ":" without a hostname, a leading ":" has to be specified. By default the file name for interactive jobs is /dev/null. For batch jobs the default file name has the form job_name.ejob_id and job_name.ejob_id.task_id for array job tasks (see -t option below). If path is a directory, the standard error stream of the job will be put in this directory under the default file name. If the pathname contains certain pseudo environment variables, their value will be expanded at runtime of the job and will be used to constitute the standard error stream path name. The fol- lowing pseudo environment variables are supported currently: $HOME home directory on execution machine $USER user ID of job owner $JOB_ID current job ID $JOB_NAME current job name (see -N option) $HOSTNAME name of the execution host $TASK_ID array job task index number Alternatively to $HOME the tilde sign "~" can be used as common in csh(1) or ksh(1). Note, that the "~" sign also works in com- bination with user names, so that "~" expands to the home directory of . Using another user ID than that of the job owner requires corresponding permissions, of course. If path or any component of it does not exist, it will be cre- ated with the permissions of the current user. A trailing "/" indicates that the last component of path is a directory. For example the command "qsub -e myjob/error.e $SGE_ROOT/exam- ples/sleeper.sh" will create the directory "myjob" in the cur- rent working directory if it does not exist, and write the stan- dard error stream of the job into the file "error.e". The com- mand "qsub -e myotherjob/ $SGE_ROOT/examples/sleeper.sh" will create the directory "myotherjob", and write the standard error stream of the job into a file with the default name (see description above). If it is not possible to create the direc- tory (e.g. insufficient permissions), the job will be put in error state. Qalter allows changing this option even while the job executes. The modified parameter will only be in effect after a restart or migration of the job, however. If this option or a corresponding value in qmon is specified then this value will be passed to defined JSV instances as parameter with the name e. (see -jsv option below or find more information concerning JSV in jsv(1)) -hard Available for qsub, qsh, qrsh, qlogin and qalter only. Signifies that all -q and -l resource requirements following in the command line will be hard requirements and must be satisfied in full before a job can be scheduled. As Univa Grid Engine scans the command line and script file for Univa Grid Engine options and parameters it builds a list of resources required by a job. All such resource requests are con- sidered as absolutely essential for the job to commence. If the -soft option (see below) is encountered during the scan then all following resources are designated as "soft requirements" for execution, or "nice-to-have, but not essential". If the -hard flag is encountered at a later stage of the scan, all resource requests following it once again become "essential". The -hard and -soft options in effect act as "toggles" during the scan. If this option or a corresponding value in qmon is specified then the corresponding -q and -l resource requirements will be passed to defined JSV instances as parameter with the names q_hard and l_hard. Find for information in the sections describ- ing -q and -l. (see -jsv option below or find more information concerning JSV in jsv(1)) -h | -h {u|s|o|n|U|O|S}... Available for qsub (only -h), qrsh, qalter and qresub (hold state is removed when not set explicitly). List of holds to place on a job, a task or some tasks of a job. `u' denotes a user hold. `s' denotes a system hold. `o' denotes a operator hold. `n' denotes no hold (requires manager privileges). As long as any hold other than `n' is assigned to the job the job is not eligible for execution. Holds can be released via qalter and qrls(1). In case of qalter this is supported by the following additional option specifiers for the -h switch: `U' removes a user hold. `S' removes a system hold. `O' removes a operator hold. Univa Grid Engine managers can assign and remove all hold types, Univa Grid Engine operators can assign and remove user and oper- ator holds, and users can only assign or remove user holds. In the case of qsub only user holds can be placed on a job and thus only the first form of the option with the -h switch alone is allowed. As opposed to this, qalter requires the second form described above. An alternate means to assign hold is provided by the qhold(1) facility. If the job is a array job (see the -t option below), all tasks specified via -t are affected by the -h operation simultane- ously. Qalter allows changing this option even while the job executes. The modified parameter will only be in effect after a restart or migration of the job, however. If this option is specified with qsub or during the submission of a job in qmon then the parameter h with the value u will be passed to the defined JSV instances indicating that the job will be in user hold after the submission finishes. (see -jsv option below or find more information concerning JSV in jsv(1)) -help Prints a listing of all options. -hold_jid wc_job_list Available for qsub, qrsh, and qalter only. See sge_types(1). for wc_job_list definition. Defines or redefines the job dependency list of the submitted job. A reference by job name or pattern is only accepted if the referenced job is owned by the same user as the referring job. The submitted job is not eligible for execution unless all jobs referenced in the comma-separated job id and/or job name list have completed. If any of the referenced jobs exits with exit code 100, the submitted job will remain ineligible for execu- tion. With the help of job names or regular pattern one can specify a job dependency on multiple jobs satisfying the regular pattern or on all jobs with the requested name. The name dependencies are resolved at submit time and can only be changed via qalter. New jobs or name changes of other jobs will not be taken into account. Qalter allows changing this option even while the job executes. The modified parameter will only be in effect after a restart or migration of the job, however. If this option or a corresponding value in qmon is specified then this value will be passed to defined JSV instances as parameter with the name hold_jid. (see -jsv option below or find more information concerning JSV in jsv(1)) -hold_jid_ad wc_job_list Available for qsub, qrsh, and qalter only. See sge_types(1). for wc_job_list definition. Defines or redefines the job array dependency list of the sub- mitted job. A reference by job name or pattern is only accepted if the referenced job is owned by the same user as the referring job. Each sub-task of the submitted job is not eligible for exe- cution unless the corresponding sub-tasks of all jobs referenced in the comma-separated job id and/or job name list have com- pleted. If any array task of the referenced jobs exits with exit code 100, the dependent tasks of the submitted job will remain ineligible for execution. With the help of job names or regular pattern one can specify a job dependency on multiple jobs satisfying the regular pattern or on all jobs with the requested name. The name dependencies are resolved at submit time and can only be changed via qalter. New jobs or name changes of other jobs will not be taken into account. If either the submitted job or any job in wc_job_list are not array jobs with the same range of sub-tasks (see -t option below), the request list will be rejected and the job create or modify operation will error. qalter allows changing this option even while the job executes. The modified parameter will only be in effect after a restart or migration of the job, however. If this option or a corresponding value in qmon is specified then this value will be passed to defined JSV instances as parameter with the name hold_jid_ad. (see -jsv option below or find more information concerning JSV in jsv(1)) -i [[hostname]:]file,... Available for qsub, and qalter only. Defines or redefines the file used for the standard input stream of the job. If the file constitutes an absolute filename, the input-path attribute of the job is set to path, including the hostname. If the path name is relative, Univa Grid Engine expands path either with the current working directory path (if the -cwd switch (see above) is also specified) or with the home directory path. If hostname is present, the standard input stream will be placed in the corresponding location only if the job runs on the specified host. If the path contains a ":" with- out a hostname, a leading ":" has to be specified. By default /dev/null is the input stream for the job. It is possible to use certain pseudo variables, whose values will be expanded at runtime of the job and will be used to express the standard input stream as described in the -e option for the standard error stream. Qalter allows changing this option even while the job executes. The modified parameter will only be in effect after a restart or migration of the job, however. If this option or a corresponding value in qmon is specified then this value will be passed to defined JSV instances as parameter with the name i. (see -jsv option below or find more information concerning JSV in jsv(1)) -inherit Available only for qrsh and qmake(1). qrsh allows the user to start a task in an already scheduled parallel job. The option -inherit tells qrsh to read a job id from the environment variable JOB_ID and start the specified command as a task in this job. Please note that in this case, the hostname of the host where the command will be executed must precede the command to execute; the syntax changes to qrsh -inherit [ other options ] hostname command [ command_args ] Note also, that in combination with -inherit, most other command line options will be ignored. Only the options -verbose, -v and -V will be interpreted. As a replacement to option -cwd please use -v PWD. Usually a task should have the same environment (including the current working directory) as the corresponding job, so specify- ing the option -V should be suitable for most applications. Note: If in your system the qmaster tcp port is not configured as a service, but rather via the environment variable SGE_QMAS- TER_PORT, make sure that this variable is set in the environment when calling qrsh or qmake with the -inherit option. If you call qrsh or qmake with the -inherit option from within a job script, export SGE_QMASTER_PORT with the option "-v SGE_QMASTER_PORT" either as a command argument or an embedded directive. This parameter is not available in the JSV context. (see -jsv option below or find more information concerning JSV in jsv(1)) -j y[es]|n[o] Available for qsub, qsh, qrsh, qlogin and qalter only. Specifies whether or not the standard error stream of the job is merged into the standard output stream. If both the -j y and the -e options are present, Univa Grid Engine sets but ignores the error-path attribute. Qalter allows changing this option even while the job executes. The modified parameter will only be in effect after a restart or migration of the job, however. The value specified with this option or the corresponding value specified in qmon will only be passed to defined JSV instances if the value is yes. The name of the parameter will be j. The value will be y also when then long form yes was specified dur- ing submission. (see -jsv option below or find more information concerning JSV in jsv(1)) -jc jc_name Available for qsub, qrsh, and qalter only. Specifies if the job specification of a job should be derived from a job class. jc_name might either be a name of a job class or the combination of a job class name and a variant name, both names separated by a dot (.). If this switch is used then within the sge_qmaster(8) process following 6 steps will be executed: (1) A new job will be created (2) This job structure will be initialized with default values. (3) Then all those default values will be replaced with that values that are specified as job template attributes in the job class (or job class variant). (4) If the -jc switch was combined with other command line switches that specify job characteristics then those settings will be applied to the job. This step might overwrite default values and values that where copied from the job class specifi- cation. (5) Server JSV will be triggered if configured. This server JSV script will receive the specification of the job and if the server JSV adjusts the job specification then default values, values derived from the job class specification and values spec- ified at the command line might be overwritten. (6) With the last step sge_qmaster checks if any access speci- fiers were violated during the steps (4) or (5). If this is the case then the job is rejected. Otherwise it will enter the list of pending jobs. The server JSV that might be triggered with step (5) will receive the jc_name as a parameter with the name jc. If a server JSV decides to change the jc attribute then the process described above will restart at step (1) and the new jc_name will be used for step (3). Please note that the violation of the access specifiers is checked in the last step. As result a server JSV is also not allowed to apply modifications to the job that would violate any access specifiers defined in the job class specification. Any attempt to change a job attribute of a job that was derived from a job class will be rejected. Owners of the job class can soften this restriction by using access specifiers within the specification of a job class. Details concerning access speci- fiers can be found in sge_job_class(5.) The qalter -jc NONE command can be used by managers to release the link between a submitted job class job and its parent job class. In this case all other job parameters won't be changed but it will be possible to change all settings with qalter afterwards independent on the access specifiers that were used. -js job_share Available for qsub, qsh, qrsh, qlogin and qalter only. Defines or redefines the job share of the job relative to other jobs. Job share is an unsigned integer value. The default job share value for jobs is 0. The job share influences the Share Tree Policy and the Func- tional Policy. It has no effect on the Urgency and Override Policies (see share_tree(5), sched_conf(5) and the Univa Grid Engine Installation and Administration Guide for further infor- mation on the resource management policies supported by Univa Grid Engine). In case of the Share Tree Policy, users can distribute the tick- ets to which they are currently entitled among their jobs using different shares assigned via -js. If all jobs have the same job share value, the tickets are distributed evenly. Otherwise, jobs receive tickets relative to the different job shares. Job shares are treated like an additional level in the share tree in the latter case. In connection with the Functional Policy, the job share can be used to weight jobs within the functional job category. Tickets are distributed relative to any uneven job share distribution treated as a virtual share distribution level underneath the functional job category. If both the Share Tree and the Functional Policy are active, the job shares will have an effect in both policies, and the tickets independently derived in each of them are added to the total number of tickets for each job. If this option or a corresponding value in qmon is specified then this value will be passed to defined JSV instances as parameter with the name js. (see -jsv option below or find more information concerning JSV in jsv(1)) -jsv jsv_url Available for qsub, qsh, qrsh and qlogin only. Defines a client JSV instance which will be executed to verify the job specification before the job is sent to qmaster. In contrast to other options this switch will not be overwritten if it is also used in sge_request files. Instead all specified JSV instances will be executed to verify the job to be submit- ted. The JSV instance which is directly passed with the command-line of a client is executed as first to verify the job specifica- tion. After that the JSV instance which might have been defined in various sge_request files will be triggered to check the job. Find more details in man page jsv(1) and sge_request(5). The syntax of the jsv_url is specified in sge_types(1).() -masterl resource=value,... Available for qsub, qsh, qrsh, qlogin and qalter only. Available only in combination with parallel jobs. Launch the parallel job in a Univa Grid Engine queue meeting the given resource request list for the master task of that parallel job. Other resource requests as they can be specified with the l-switch will only specify the requirements of slave tasks if the masterl-switch is used during job submission. If a master queue or a master host are reqeusted with the mas- terl-switch, depending on this request and other queue and host requests - implicit or explicit ones -, depending on how the queues are spread out over the execution hosts and depending on the allocation_rule of the parallel environment, requesting a master queue or master host may make it necessary for Univa Grid Engine to allocate one task more for this parallel job than the user requested. See sge_pe(5), "allocation_rule" for details. In case of qalter the previous definition is replaced by the specified one. complex(5) describes how a list of available resources and their associated valid value specifiers can be obtained. Qalter does allow changing the value of this option while the job is running, however the modification will only be effective after a restart or migration of the job. If this option or a corresponding value in qmon is specified then the hard resource requirements will be passed to defined JSV instances as parameter with the name masterl. If regular expressions will be used for resource requests, then these expressions will be passed as they are. Also shortcut names will not be expanded. (see -jsv option above or find more informa- tion concerning JSV in jsv(1)) -l resource=value,... Available for qsub, qsh, qrsh, qlogin and qalter only. Launch the job in a Univa Grid Engine queue meeting the given resource request list. In case of qalter the previous defini- tion is replaced by the specified one. complex(5) describes how a list of available resources and their associated valid value specifiers can be obtained. If the resource request is specified while the -soft option is active the value for consumables can also be specified as range. You can find the format description and an example in the com- plex(5) man page. There may be multiple -l switches in a single command. You may request multiple -l options to be soft or hard both in the same command line. In case of a serial job multiple -l switches refine the definition for the sought queue. Qalter allows changing the value of this option even while the job is running, but only if the initial list of resources does not contain a resource that is marked as consumable. However the modification will only be effective after a restart or migration of the job. If this option or a corresponding value in qmon is specified the these hard and soft resource requirements will be passed to defined JSV instances as parameter with the names l_hard and l_soft. If regular expressions will be used for resource requests, then these expressions will be passed as they are. Also shortcut names will not be expanded. (see -jsv option above or find more information concerning JSV in jsv(1)) -m b|e|a|s|n,... Available for qsub, qsh, qrsh, qlogin and qalter only. Defines or redefines under which circumstances mail is to be sent to the job owner or to the users defined with the -M option described below. The option arguments have the following mean- ing: `b' Mail is sent at the beginning of the job. `e' Mail is sent at the end of the job. `a' Mail is sent when the job is aborted or rescheduled. `s' Mail is sent when the job is suspended. `n' No mail is sent. Currently no mail is sent when a job is suspended. Qalter allows changing the b, e, and a option arguments even while the job executes. The modification of the b option argu- ment will only be in effect after a restart or migration of the job, however. If this option or a corresponding value in qmon is specified then this value will be passed to defined JSV instances as parameter with the name m. (see -jsv option above or find more information concerning JSV in -M user[@host],... Available for qsub, qsh, qrsh, qlogin and qalter only. Defines or redefines the list of users to which the server that executes the job has to send mail, if the server sends mail about the job. Default is the job owner at the originating host. Qalter allows changing this option even while the job executes. If this option or a corresponding value in qmon is specified then this value will be passed to defined JSV instances as parameter with the name M. (see -jsv option above or find more information concerning JSV in jsv(1)) -masterq wc_queue_list Available for qsub, qrsh, qsh, qlogin and qalter. Only meaning- ful for parallel jobs, i.e. together with the -pe option. Defines or redefines a list of cluster queues, queue domains and queue instances which may be used to become the so called master queue of this parallel job. A more detailed description of wc_queue_list can be found in sge_types(1). The master queue is defined as the queue where the parallel job is started. The other queues to which the parallel job spawns tasks are called slave queues. A parallel job only has one master queue. Depending on the requested master queue and other queue requests - implicit or explicit ones -, depending on how the queues are spread out over the execution hosts and depending on the alloca- tion_rule of the parallel environment, requesting a master queue may make it necessary for Univa Grid Engine to allocate one task more for this parallel job than the user requested. See sge_pe(5), "allocation_rule" for details. This parameter has all the properties of a resource request and will be merged with requirements derived from the -l option described above. Qalter allows changing this option even while the job executes. The modified parameter will only be in effect after a restart or migration of the job, however. If this option or a corresponding value in qmon is specified the this hard resource requirement will be passed to defined JSV instances as parameter with the name masterq. (see -jsv option above or find more information concerning JSV in jsv(1)) -mods parameter key value Available for qsub, qrsh, qalter of Univa Grid Engine only. Gives the user the possibility to modify entries of list based job parameters like resource requests, job context, environment variables and more. The -adds and -clears switches can be used to add or remove a single entry of a job parameter list. Parameter, key and value arguments are explained in more detail in the -adds section above. -mbind Available for qsub, qrsh, and qalter. Supported on lx-amd64 exe- cution hosts only (for more details try utilbin/loadcheck -cb on the execution host). Sets the memory allocation strategy for all processes and sub- processes of a job. On execution hosts with a NUMA architec- ture, the memory access latency and memory throughput depends on which NUMA node the memory is allocated and on which socket/core the job runs. In order to influence the memory allocation dif- ferent submit options are provided: -mbind cores Prefers memory on the NUMA node where the job is bound with core binding. Requires core binding set with -bind- ing. The optional "m_mem_free" request is enhanced during sched- uling time with implicit per NUMA node requests ("m_mem_free_n"), depending on which cores the job is bound to. For more details see -mbind cores:strict -mbind cores:strict The job is only allowed to allocate memory on the NUMA node where it is bound to. Requires core binding set with -binding. The optional "m_mem_free" request is extended during scheduling time by implicit per NUMA node requests ("m_mem_free_n"), depending on which cores the job is bound to. The amount of selected cores per NUMA node and the total memory per slot request determining the amount of required memory per NUMA node. Hence when using the "m_mem_free" memory request the job is only scheduled to sockets which offer the specific amount of free memory. -mbind round_robin Sets the memory allocation strategy for the job to interleaved memory access. When the memory resource request "m_mem_free" is used, the scheduler also adds implicit memory requests for all NUMA nodes on the execution host ("m_mem_free_n"). -mbind nlocal Only allowed for serial jobs or jobs using a par- allel environment with allocation rule "$pe_slots". Unspecified behavior for other PEs. Automatically binds a sequential or multi-threaded job to cores or sockets and sets an appropriate memory allocation strategy. Requires a resource request for the "m_mem_free" host complex. The behavior of the scheduler depends on the execution hosts characteristics as well whether the job is a serial job or a multi-threaded parallel job (PE job with allocation rule "$pe_slots"). It is not allowed to override the implicit core binding with the -binding switch. The scheduler algorithm for sequential jobs is as follows: - If the host can't fulfill the "m_mem_free" request then the host is skipped. - If the job requests more ram than free on each socket but less than installed on the sockets the host is skipped. - If memory request is "smaller" than amount of free memory on a socket, try to bind the job to "one core on the socket" and decrements the amount of memory on this socket ("m_mem_free_n"). The global host memory "m_mem_free" on this host is decremented as well. - If memory request is "greater" than the amount of free memory on any socket, find an unbound socket and bind it there com- pletely and allow memory overflow. Decrement from "m_mem_free" as well as from "m_mem_free_n" and the remaining memory round robin from the remaining sockets. - If both are not possible go to the next host. The scheduler algorithm for parallel jobs is as follows: - Hosts that don't offer "m_mem_free" memory are skipped (of course hosts that doesn't offer the amount of free slots requested are skipped as well). - If the amount of requested slots is greater than the amount of cores per socket, the job is dispatched to the host without any binding. - If the amount of requested slots is smaller than the amount of cores per socket do following: - If there is any socket which offers enough memory ("m_mem_free_n") and enough free cores bind the job to these cores and set memory allocation mode to "cores:strict" (so that only local memory requests can be done by the job). - If this is not possible try to find a socket which is com- pletely unbound and has more than the required amount of memory installed ("m_mem_total_n"). Bind the job to the complete socket, decrement the memory on that socket at "m_mem_free_n" (as well as host globally on "m_mem_free") and set the memory allocation strategy to "cores" (preferred usage of socket local memory). If the parameters are requesting the "m_mem_free" complex, the resulting NUMA node memory requests can be seen in the "implicit_requests" row in the qstat output. Note that resource reservation for implicit per NUMA node requests as well as topology selections for core binding are not part of resource reservation yet. The value specified with the -mbind option will be passed to defined JSV instances (as "mbind") only when set. JSV can set the parameter as "round_robin", "cores", "cores:strict", or "NONE". The same values can be used for job classes. -notify Available for qsub, qrsh (with command) and qalter only. This flag, when set causes Univa Grid Engine to send "warning" signals to a running job prior to sending the signals them- selves. If a SIGSTOP is pending, the job will receive a SIGUSR1 several seconds before the SIGSTOP. If a SIGKILL is pending, the job will receive a SIGUSR2 several seconds before the SIGKILL. This option provides the running job, before receiving the SIGSTOP or SIGKILL, a configured time interval to do e.g. cleanup operations. The amount of time delay is controlled by the notify parameter in each queue configuration (see queue_conf(5)). Note that the Linux operating system "misused" the user signals SIGUSR1 and SIGUSR2 in some early Posix thread implementations. You might not want to use the -notify option if you are running multi-threaded applications in your jobs under Linux, particu- larly on 2.0 or earlier kernels. Qalter allows changing this option even while the job executes. Only if this option is used the parameter named notify with the value y will be passed to defined JSV instances. (see -jsv option above or find more information concerning JSV in jsv(1)) -now y[es]|n[o] Available for qsub, qsh, qlogin and qrsh. -now y tries to start the job immediately or not at all. The command returns 0 on success, or 1 on failure (also if the job could not be scheduled immediately). For array jobs submitted with the -now option, if one or more tasks can be scheduled immediately the job will be accepted, otherwise it will not be started at all. Jobs submitted with -now y option, can ONLY run on INTERACTIVE queues. -now y is default for qsh, qlogin and qrsh With the -now n option, the job will be put into the pending queue if it cannot be executed immediately. -now n is default for qsub. The value specified with this option or the corresponding value specified in qmon will only be passed to defined JSV instances if the value is yes. The name of the parameter will be now. The value will be y also when then long form yes was specified dur- ing submission. (see -jsv option above or find more information concerning JSV in jsv(1)) -N name Available for qsub, qsh, qrsh, qlogin and qalter only. The name of the job. The name should follow the "name" defini- tion in sge_types(1). Invalid job names will be denied at sub- mit time. If the -N option is not present, Univa Grid Engine assigns the name of the job script to the job after any directory pathname has been removed from the script-name. If the script is read from standard input, the job name defaults to STDIN. In the case of qsh or qlogin with the -N option is absent, the string `INTERACT' is assigned to the job. In the case of qrsh if the -N option is absent, the resulting job name is determined from the qrsh command line by using the argument string up to the first occurrence of a semicolon or whitespace and removing the directory pathname. Qalter allows changing this option even while the job executes. The value specified with this option or the corresponding value specified in qmon will be passed to defined JSV instances as parameter with the name N. (see -jsv option above or find more information concerning JSV in jsv(1)) -noshell Available only for qrsh with a command line. Do not start the command line given to qrsh in a user's login shell, i.e. execute it without the wrapping shell. This option can be used to speed up execution as some overhead, like the shell startup and sourcing the shell resource files, is avoided. This option can only be used if no shell-specific command line parsing is required. If the command line contains shell syntax like environment variable substitution or (back) quoting, a shell must be started. In this case, either do not use the -noshell option or include the shell call in the command line. Example: qrsh echo '$HOSTNAME' Alternative call with the -noshell option qrsh -noshell /bin/tcsh -f -c 'echo $HOSTNAME' -nostdin Available only for qrsh. Suppress the input stream STDIN - qrsh will pass the option -n to the rsh(1) command. This is especially useful, if multiple tasks are executed in parallel using qrsh, e.g. in a make(1) process - it would be undefined, which process would get the input. -o [[hostname]:]path,... Available for qsub, qsh, qrsh, qlogin and qalter only. The path used for the standard output stream of the job. The path is handled as described in the -e option for the standard error stream. By default the file name for standard output has the form job_name.ojob_id and job_name.ojob_id.task_id for array job tasks (see -t option below). Qalter allows changing this option even while the job executes. The modified parameter will only be in effect after a restart or migration of the job, however. If this option or a corresponding value in qmon is specified then this value will be passed to defined JSV instances as parameter with the name o. (see -jsv option above or find more information concerning JSV in jsv(1)) -ot override_tickets Available for qalter only. Changes the number of override tickets for the specified job. Requires manager/operator privileges. -P project_name Available for qsub, qsh, qrsh, qlogin and qalter only. Specifies the project to which this job is assigned. The admin- istrator needs to give permission to individual users to submit jobs to a specific project. (see -aprj option to qconf(1)). If this option or a corresponding value in qmon is specified then this value will be passed to defined JSV instances as parameter with the name ot. (see -jsv option above or find more information concerning JSV in jsv(1)) -p priority Available for qsub, qsh, qrsh, qlogin and qalter only. Defines or redefines the priority of the job relative to other jobs. Priority is an integer in the range -1023 to 1024. The default priority value for jobs is 0. Users may only decrease the priority of their jobs. If the parameter ALLOW_INCREASE_POSIX_PRIORITY is set as qmaster_param in the global configuration then users are also allowed to increase the priority of their own jobs up to 0. Univa Grid Engine managers and operators may also increase the priority associated with jobs independent from ALLOW_INCREASE_POSIX_PRIORITY setting. If a pending job has higher priority, it is earlier eligible for being dispatched by the Univa Grid Engine scheduler. If this option or a corresponding value in qmon is specified and the priority is not 0 then this value will be passed to defined JSV instances as parameter with the name p. (see -jsv option above or find more information concerning JSV in jsv(1)) -pe parallel_environment n[-[m]]|[-]m,... Available for qsub, qsh, qrsh, qlogin and qalter only. Parallel programming environment (PE) to instantiate. For more detail about PEs, please see the sge_types(1). Qalter allows changing this option even while the job executes. The modified parameter will only be in effect after a restart or migration of the job, however. If this option or a corresponding value in qmon is specified then the parameters pe_name, pe_min and pe_max will be passed to configured JSV instances where pe_name will be the name of the parallel environment and the values pe_min and pe_max represent the values n and m which have been provided with the -pe option. A missing specification of m will be expanded as value 9999999 in JSV scripts and it represents the value infinity. Since it is possible to specify more than one range with the -pe option the JSV instance pe_n will contain the number of speci- fied ranges. The content of of the ranges can be addressed by adding the index to the variable name. The JSV variable pe_min_0 is representing the first minimum value of the first range. (see -jsv option above or find more information concerning JSV in jsv(1)) -pty y[es]|n[o] Available for qrsh, qlogin and qsub only. -pty yes enforces the job to be started in a pseudo terminal (pty). If no pty is available, the job start fails. -pty no enforces the job to be started without a pty. By default, qrsh without a command and qlogin start the job in a pty, qrsh with a command and qsub start the job without a pty. Information that this switch was specified during submission is not available in the JSV context of the open source version of Grid Engine. In Univa Grid Engine a variable named pty will be available and it will have the value u when the switch was omitted or the value y or n depending if y[es] or n[o] was passed as parameter with the switch. This parameter can be changed in the JSV con- text to influence the behavior of the command line client and job. (see -jsv option above or find more information concerning JSV in jsv(1)) -q wc_queue_list Available for qsub, qrsh, qsh, qlogin and qalter. Defines or redefines a list of cluster queues, queue domains or queue instances which may be used to execute this job. Please find a description of wc_queue_list in sge_types(1). This parameter has all the properties of a resource request and will be merged with requirements derived from the -l option described above. Qalter allows changing this option even while the job executes. The modified parameter will only be in effect after a restart or migration of the job, however. If this option or a corresponding value in qmon is specified the these hard and soft resource requirements will be passed to defined JSV instances as parameters with the names q_hard and q_soft. If regular expressions will be used for resource requests, then these expressions will be passed as they are. Also shortcut names will not be expanded. (see -jsv option above or find more information concerning JSV in jsv(1)) -R y[es]|n[o] Available for qsub, qrsh, qsh, qlogin and qalter. Indicates whether a reservation for this job should be done. Reservation is never done for immediate jobs, i.e. jobs submit- ted using the -now yes option. Please note that regardless of the reservation request, job reservation might be disabled using max_reservation in sched_conf(5) and might be limited only to a certain number of high priority jobs. By default jobs are submitted with the -R n option. The value specified with this option or the corresponding value specified in qmon will only be passed to defined JSV instances if the value is yes. The name of the parameter will be R. The value will be y also when then long form yes was specified dur- ing submission. (see -jsv option above or find more information concerning JSV in jsv(1)) -r y[es]|n[o] Available for qsub and qalter only. Identifies the ability of a job to be rerun or not. If the value of -r is 'yes', the job will be rerun if the job was aborted without leaving a consistent exit state. (This is typi- cally the case if the node on which the job is running crashes). If -r is 'no', the job will not be rerun under any circum- stances. Interactive jobs submitted with qsh, qrsh or qlogin are not rerunnable. Qalter allows changing this option even while the job executes. The value specified with this option or the corresponding value specified in qmon will only be passed to defined JSV instances if the value is yes. The name of the parameter will be r. The value will be y also when then long form yes was specified dur- ing submission. (see -jsv option above or find more information concerning JSV in jsv(1)) -rou variable,... Available for qsub, qsh, qrsh, qlogin and qalter only. Used to specify which job report attributes (e.g. cpu, mem, vmem, ...) shall get written to the reporting file and the reporting database. Variables are specified as comma separated list. Specifying reporting variables per job will overwrite a global setting done in the global cluster configuration, report- ing_params, see also sge_conf(5). -sc variable[=value],... Available for qsub, qsh, qrsh, qlogin and qalter only. Sets the given name/value pairs as the job's context. Value may be omitted. Univa Grid Engine replaces the job's previously defined context with the one given as the argument. Multiple -ac, -dc, and -sc options may be given. The order is important. The variable name must not start with the letters "+", "-" or "=". Contexts provide a way to dynamically attach and remove meta- information to and from a job. The context variables are not passed to the job's execution context in its environment. Qalter allows changing this option even while the job executes. The outcome of the evaluation of all -ac, -dc, and -sc options or corresponding values in qmon is passed to defined JSV instances as parameter with the name ac. (see -jsv option above or find more information concerning JSV in jsv(1)) -shell y[es]|n[o] Available only for qsub. -shell n causes qsub to execute the command line directly, as if by exec(2). No command shell will be executed for the job. This option only applies when -b y is also used. Without -b y, -shell n has no effect. This option can be used to speed up execution as some overhead, like the shell startup and sourcing the shell resource files is avoided. This option can only be used if no shell-specific command line parsing is required. If the command line contains shell syntax, like environment variable substitution or (back) quoting, a shell must be started. In this case either do not use the -shell n option or execute the shell as the command line and pass the path to the executable as a parameter. If a job executed with the -shell n option fails due to a user error, such as an invalid path to the executable, the job will enter the error state. -shell y cancels the effect of a previous -shell n. Otherwise, it has no effect. See -b and -noshell for more information. The value specified with this option or the corresponding value specified in qmon will only be passed to defined JSV instances if the value is yes. The name of the parameter will be shell. The value will be y also when then long form yes was specified during submission. (see -jsv option above or find more informa- tion concerning JSV in jsv(1)) -si session_id Available for qsub, qsh, qrsh, qlogin and qalter. Requests sent by this client to the sge_qmaster(1) daemon will be done as part of the specified session. If the switch is omit- ted or if NONE is specified as session_id then such requests will be executed outside the control of a session. Find more information concerning sessions in session_conf(5). -soft Available for qsub, qsh, qrsh, qlogin and qalter only. Signifies that all resource requirements following in the com- mand line will be soft requirements and are to be filled on an "as available" basis. It is possible to specify ranges for consumable resource requirements if they are declared as -soft requests. More information about soft ranges can be found in the description of the -l option. As Univa Grid Engine scans the command line and script file for Univa Grid Engine options and parameters, it builds a list of resources required by the job. All such resource requests are considered as absolutely essential for the job to commence. If the -soft option is encountered during the scan then all follow- ing resources are designated as "soft requirements" for execu- tion, or "nice-to-have, but not essential". If the -hard flag (see above) is encountered at a later stage of the scan, all resource requests following it once again become "essential". The -hard and -soft options in effect act as "toggles" during the scan. If this option or a corresponding value in qmon is specified then the corresponding -q and -l resource requirements will be passed to defined JSV instances as parameter with the names q_soft and l_soft. Find for information in the sections describ- ing -q and -l. (see -jsv option above or find more information concerning JSV in jsv(1)) -sync y|n|l|r Available for qsub. -sync y causes qsub to wait for the job to complete before exit- ing. If the job completes successfully, qsub's exit code will be that of the completed job. If the job fails to complete suc- cessfully, qsub will print out a error message indicating why the job failed and will have an exit code of 1. If qsub is interrupted, e.g. with CTRL-C, before the job completes, the job will be canceled. With the -sync n option, qsub will exit with an exit code of 0 as soon as the job is submitted successfully. -sync n is default for qsub. If -sync y is used in conjunction with -now y, qsub will behave as though only -now y were given until the job has been success- fully scheduled, after which time qsub will behave as though only -sync y were given. If -sync y is used in conjunction with -t n[-m[:i]], qsub will wait for all the job's tasks to complete before exiting. If all the job's tasks complete successfully, qsub's exit code will be that of the first completed job tasks with a non-zero exit code, or 0 if all job tasks exited with an exit code of 0. If any of the job's tasks fail to complete successfully, qsub will print out an error message indicating why the job task(s) failed and will have an exit code of 1. If qsub is interrupted, e.g. with CTRL-C, before the job completes, all of the job's tasks will be canceled. With the -sync l option, qsub will print an appropriate message as soon as the job changes into the l-state (license request sent to License Orchestrator). With the -sync r option, qsub will print an appropriate message as soon as the job is running. All those options can be combined. qsub will exit when the last event occurs. Information that this switch was specified during submission is not available in the JSV context of the open source version of Grid Engine. In Univa Grid Engine a variable named sync will be available and it will have the value y when the switch was used. The parameter value cannot be changed within the JSV context. (see -jsv option above or find more information concerning JSV in jsv(1)) -S [[hostname]:]pathname,... Available for qsub, qsh and qalter. Specifies the interpreting shell for the job. Only one pathname component without a host specifier is valid and only one path name for a given host is allowed. Shell paths with host assign- ments define the interpreting shell for the job if the host is the execution host. The shell path without host specification is used if the execution host matches none of the hosts in the list. Furthermore, the pathname can be constructed with pseudo envi- ronment variables as described for the -e option above. In the case of qsh the specified shell path is used to execute the corresponding command interpreter in the xterm(1) (via its -e option) started on behalf of the interactive job. Qalter allows changing this option even while the job executes. The modified parameter will only be in effect after a restart or migration of the job, however. If this option or a corresponding value in qmon is specified then this value will be passed to defined JSV instances as parameter with the name S. (see -jsv option above or find more information concerning JSV in jsv(1)) -t n[-m[:s]] Available for qsub only. qalter cannot be used to change the array job size but -t might be used in combination with a job ID to address the tasks that should be changed. Submits a so called Array Job, i.e. an array of identical tasks being differentiated only by an index number and being treated by Univa Grid Engine almost like a series of jobs. The option argument to -t specifies the number of array job tasks and the index number which will be associated with the tasks. The index numbers will be exported to the job tasks via the environment variable SGE_TASK_ID. The option arguments n, m and s will be available through the environment variables SGE_TASK_FIRST, SGE_TASK_LAST and SGE_TASK_STEPSIZE. Following restrictions apply to the values n and m: 1 <= n <= MIN(2^31-1, max_aj_tasks) 1 <= m <= MIN(2^31-1, max_aj_tasks) n <= m max_aj_tasks is defined in the cluster configuration (see sge_conf(5)) The task id range specified in the option argument may be a sin- gle number, a simple range of the form n-m or a range with a step size. Hence, the task id range specified by 2-10:2 would result in the task id indexes 2, 4, 6, 8, and 10, for a total of 5 identical tasks, each with the environment variable SGE_TASK_ID containing one of the 5 index numbers. All array job tasks inherit the same resource requests and attribute definitions as specified in the qsub or qalter command line, except for the -t option. The tasks are scheduled indepen- dently and, provided enough resources exist, concurrently, very much like separate jobs. However, an array job or a sub-array there of can be accessed as a single unit by commands like qmod(1) or qdel(1). See the corresponding manual pages for fur- ther detail. Array jobs are commonly used to execute the same type of opera- tion on varying input data sets correlated with the task index number. The number of tasks in a array job is unlimited. STDOUT and STDERR of array job tasks will be written into dif- ferent files with the default location .['e'|'o']'.' In order to change this default, the -e and -o options (see above) can be used together with the pseudo environment vari- ables $HOME, $USER, $JOB_ID, $JOB_NAME, $HOSTNAME, and $TASK_ID. Note, that you can use the output redirection to divert the out- put of all tasks into the same file, but the result of this is undefined. If this option or a corresponding value in qmon is specified then this value will be passed to defined JSV instances as parameters with the name t_min, t_max and t_step (see -jsv option above or find more information concerning JSV in jsv(1)) -tc max_running_tasks Available for qsub and qalter only. Can be used in conjunction with array jobs (see -t option) to set a self-imposed limit on the maximum number of concurrently running tasks per job. If this option or a corresponding value in qmon is specified then this value will be passed to defined JSV instances as parameter with the name tc. (see -jsv option above or find more information concerning JSV in jsv(1)) -tcon y[es]|n[o] Available for qsub only. Can be used in conjunction with array jobs (see -t option) to submit a concurrent array job. For a concurrent array job either all tasks can be started in one scheduling run or the whole job will stay pending. When combined with the -now y option the immediate concurrent array job will be rejected if not all tasks can be scheduled immediately. The -tcon y switch cannot be combined with the -tc and the -R switch. If this option is specified then its value (y or n) will be passed to defined JSV instances as parameter with the name tcon. (see -jsv option above or find more information concerning JSV in jsv(1)) Array task concurrency can be enabled and limited with the MAX_TCON_TASKS qmaster_param setting in the global cluster con- figuration, see sge_conf(1). By default array task concurrency is disabled. Submission of concurrent array jobs will be rejected when their size exceeds the settings of max_aj_tasks or max_aj_instances, see sge_conf(1). When max_aj_instances is lowered below the size of a pending concurrent array job then this job will stay pending. -terse Available for qsub only. -terse causes the qsub to display only the job-id of the job being submitted rather than the regular "Your job ..." string. In case of an error the error is reported on stderr as usual. This can be helpful for scripts which need to parse qsub output to get the job-id. Information that this switch was specified during submission is not available in the JSV context of the open source version of Grid Engine. In Univa Grid Engine a variable named terse will be available and it will have the value y when the switch was used. This parameter can be changed in the JSV context to influence the behavior of the command line client. (see -jsv option above or find more information concerning JSV in jsv(1)) -u username,... Available for qalter only. Changes are only made on those jobs which were submitted by users specified in the list of user- names. For managers it is possible to use the qalter -u '*' command to modify all jobs of all users. If you use the -u switch it is not permitted to specify an addi- tional wc_job_range_list. -v variable[=value],... Available for qsub, qrsh (with command argument) and qalter. Defines or redefines the environment variables to be exported to the execution context of the job. If the -v option is present Univa Grid Engine will add the environment variables defined as arguments to the switch and, optionally, values of specified variables, to the execution context of the job. Qalter allows changing this option even while the job executes. The modified parameter will only be in effect after a restart or migration of the job, however. All environment variables specified with -v, -V or the DISPLAY variable provided with -display will be exported to the defined JSV instances only optionally when this is requested explicitly during the job submission verification. Information that the -V switch was specified during submission is not available in the JSV context of the open source version of Grid Engine. (see -jsv option above or find more information concerning JSV in jsv(1)) -verbose Available only for qrsh and qmake(1). Unlike qsh and qlogin, qrsh does not output any informational messages while establishing the session, compliant with the standard rsh(1) and rlogin(1) system calls. If the option -ver- bose is set, qrsh behaves like the qsh and qlogin commands, printing information about the process of establishing the rsh(1) or rlogin(1) session. -verify Available for qsub, qsh, qrsh, qlogin and qalter. Instead of submitting a job, prints detailed information about the would-be job as though qstat(1) -j were used, including the effects of command-line parameters and the external environment. -V Available for qsub, qsh, qrsh with command and qalter. Specifies that all environment variables active within the qsub utility be exported to the context of the job. All environment variables specified with -v, -V or the DISPLAY variable provided with -display will be exported to the defined JSV instances only optionally when this is requested explicitly during the job submission verification. In Univa Grid Engine a variable named -V will be available and it will have the value y when the switch was used. (see -jsv option above or find more information concerning JSV in jsv(1)) -w e|w|n|p|v Available for qsub, qsh, qrsh, qlogin and qalter. Specifies a validation level applied to the job to be submitted (qsub, qlogin, and qsh) or the specified queued job (qalter). The information displayed indicates whether the job can possibly be scheduled assuming an empty system with no other jobs. Resource requests exceeding the configured maximal thresholds or requesting unavailable resource attributes are possible causes for jobs to fail this validation. The specifiers e, w, n and v define the following validation modes: `e' error - jobs with invalid requests will be rejected. `w' warning - only a warning will be displayed for invalid requests. `n' none - switches off validation; the default for qsub, qalter, qrsh, qsh and qlogin. `p' poke - does not submit the job but prints a validation report based on a cluster as is with all resource utilizations in place. `v' verify - does not submit the job but prints a validation report based on an empty cluster. Note, that the necessary checks are performance consuming and hence the checking is switched off by default. It should also be noted that load values are not taken into account with the verification since they are assumed to be too volatile. To cause -w e verification to be passed at submission time, it is possi- ble to specify non-volatile values (non-consumables) or maximum values (consumables) in complex_values. -wd working_dir Available for qsub, qsh, qrsh and qalter only. Execute the job from the directory specified in working_dir. This switch will activate Univa Grid Engine's path aliasing facility, if the corresponding configuration files are present (see sge_aliases(5)). Qalter allows changing this option even while the job executes. The modified parameter will only be in effect after a restart or migration of the job, however. The parameter value will be available in defined JSV instances as parameter with the name cwd (see -cwd switch above or find more information concerning JSV in jsv(1)) -when [now|on_reschedule] Available for qalter only. Qalter allows to change resource requests of running jobs. If -when now is set the changes will be done immediately if possi- ble (only consumables). If -when on_reschedule (default) is set the changes take effect when the job gets re-scheduled. command Available for qsub and qrsh only. The job's scriptfile or binary. If not present or if the oper- and is the single-character string '-', qsub reads the script from standard input. The command will be available in defined JSV instances as param- eter with the name CMDNAME (see -jsv option above or find more information concerning JSV in jsv(1)) -xd docker_option Available for qsub, qsh, qrsh, qlogin and qalter only when sub- mitting Docker jobs. Use the -xd switch for specifying arbitrary docker run options to be used in the creation of the container for Docker jobs. docker run means the run option of the docker command that is part of the Docker Engine. If a docker run option and/or its arguments contain spaces, quoting is required, e.g. qsub -xd "-v /tmp:/hosts_tmp". Multi- ple docker run options can be specified as a comma separated list with one -xd option, e.g. qsub -xd --net=user- net,--ip=192.168.99.10,--hostname=test. -xd --help prints a list of docker run options, if they are sup- ported by Univa Grid Engine, and a comment describing why an option is not supported, which option to use instead or how the docker run option is passed via the docker API to docker. Placeholders can be used in arguments to Docker options. These placeholders are resolved with values the Univa Grid Engine Scheduler selected for the job based on the resource map (RSMAP) requests of the job that correspond to the placeholders. These placeholders have the format: := ${ "(" ")" } complex_name is defined in sge_types(1) and is the name of the resource map which is requested for this job. index is the index of the element of the resource map to use, the first element has index 0. Using these placeholders is supported only if the RSMAP is of type "consumable=HOST", "consumable=YES" and "consumable=JOB" are not supported, because the list of resource map elements granted for the parallel tasks on a certain host then depend on the number of tasks scheduled there. Only -l requests are taken into account, no -masterl requests, because the master task of a PE job never runs in a Docker con- tainer. The substitution is equal for all PE tasks running on the same host, so likely it makes sense only to use the placeholder sub- stitution in conjunction with PE allocation rule=1 E.g.: If a resource map defines these values on a host: gpu_map=4(0 1 2 3) this qsub command line is used: # qsub -l docker,docker_images="*some_image*",gpu_map=2 -xd "--device=/dev/gpu${gpu_map(0)}:/dev/gpu0, --device=/dev/gpu${gpu_map(1)}:/dev/gpu1" ... and the scheduler selects the elements "1" and "3" from the resource map, the command line is resolved to # qsub -l docker,docker_images"*some_image*",gpu_map=2 -xd "--device=/dev/gpu1:/dev/gpu0, --device=/dev/gpu3:/dev/gpu1"... which means the physical GPUs "gpu1" and "gpu3" are mapped to the virtual GPUs "gpu0" and "gpu1" inside the container and at the same time are exclusively reserved for the current job among all Univa Grid Engine jobs. -xdv docker_volume Available for qsub, qsh, qrsh, qlogin and qalter only. When a job is running within a docker container the -xdv switch can be used to specify docker volumes to be mounted into the docker container. docker_volume is specified following the syn- tax of the docker run command line option -v, see docker run(1) man page. Multiple volumes can be mounted by passing a comma separated list of volumes to the -xdv switch or by repeating the -xdv switch. The -xdv switch is deprecated and will be removed in future ver- sions of Univa Grid Engine, use -xd --volume instead. command_args Available for qsub, qrsh and qalter only. Arguments to the job. Not valid if the script is entered from standard input. Qalter allows changing this option even while the job executes. The modified parameter will only be in effect after a restart or migration of the job, however. The number of command arguments is provided to configured JSV instances as parameter with the name CMDARGS. Also the argument values can by accessed. Argument names have the format CMDARG where is a integer between 0 and CMDARGS - 1. (see -jsv option above or find more information concerning JSV in jsv(1)) xterm_args Available for qsh only. Arguments to the xterm(1) executable, as defined in the configu- ration. For details, refer to sge_conf(5)). Information concerning xterm_args will be available in JSV con- text as parameters with the name CMDARGS and CMDARG. Find more information above in section command_args. (see -jsv option above or find more information concerning JSV in jsv(1)) ENVIRONMENTAL VARIABLES SGE_ROOT Specifies the location of the Univa Grid Engine standard configuration files. SGE_CELL If set, specifies the default Univa Grid Engine cell. To address a Univa Grid Engine cell qsub, qsh, qlogin or qalter use (in the order of precedence): The name of the cell specified in the environment variable SGE_CELL, if it is set. The name of the default cell, i.e. default. SGE_DEBUG_LEVEL If set, specifies that debug information should be writ- ten to stderr. In addition the level of detail in which debug information is generated is defined. SGE_QMASTER_PORT If set, specifies the tcp port on which sge_qmaster(8) is expected to listen for communication requests. Most installations will use a services map entry for the ser- vice "sge_qmaster" instead to define that port. DISPLAY For qsh jobs the DISPLAY has to be specified at job sub- mission. If the DISPLAY is not set by using the -dis- play or the -v switch, the contents of the DISPLAY envi- ronment variable are used as default. In addition to those environment variables specified to be exported to the job via the -v or the -V option (see above) qsub, qsh, and qlogin add the following variables with the indicated values to the variable list: SGE_O_HOME the home directory of the submitting client. SGE_O_HOST the name of the host on which the submitting client is running. SGE_O_LOGNAME the LOGNAME of the submitting client. SGE_O_MAIL the MAIL of the submitting client. This is the mail directory of the submitting client. SGE_O_PATH the executable search path of the submitting client. SGE_O_SHELL the SHELL of the submitting client. SGE_O_TZ the time zone of the submitting client. SGE_O_WORKDIR the absolute path of the current working directory of the submitting client. Furthermore, Univa Grid Engine sets additional variables into the job's environment, as listed below. ARC SGE_ARCH The Univa Grid Engine architecture name of the node on which the job is running. The name is compiled-in into the sge_execd(8) binary. SGE_BINDING This variable contains the selected operating system internal processor numbers. They might be more than selected cores in presence of SMT or CMT because each core could be represented by multiple processor identifiers. The processor numbers are space sepa- rated. SGE_CKPT_ENV Specifies the checkpointing environment (as selected with the -ckpt option) under which a checkpointing job executes. Only set for checkpointing jobs. SGE_CKPT_DIR Only set for checkpointing jobs. Contains path ckpt_dir (see checkpoint(5) ) of the checkpoint interface. SGE_CWD_PATH Specifies the current working directory where the job was started. SGE_STDERR_PATH the pathname of the file to which the standard error stream of the job is diverted. Commonly used for enhanc- ing the output with error messages from prolog, epilog, parallel environment start/stop or checkpointing scripts. SGE_STDOUT_PATH the pathname of the file to which the standard output stream of the job is diverted. Commonly used for enhanc- ing the output with messages from prolog, epilog, paral- lel environment start/stop or checkpointing scripts. SGE_STDIN_PATH the pathname of the file from which the standard input stream of the job is taken. This variable might be used in combination with SGE_O_HOST in prolog/epilog scripts to transfer the input file from the submit to the execu- tion host. SGE_JOB_SPOOL_DIR The directory used by sge_shepherd(8) to store job related data during job execution. This directory is owned by root or by a Univa Grid Engine administrative account and commonly is not open for read or write access to regular users. SGE_TASK_ID The index number of the current array job task (see -t option above). This is an unique number in each array job and can be used to reference different input data records, for example. This environment variable is set to "undefined" for non-array jobs. It is possible to change the predefined value of this variable with -v or -V (see options above). SGE_TASK_FIRST The index number of the first array job task (see -t option above). It is possible to change the predefined value of this variable with -v or -V (see options above). SGE_TASK_LAST The index number of the last array job task (see -t option above). It is possible to change the predefined value of this variable with -v or -V (see options above). SGE_TASK_STEPSIZE The step size of the array job specification (see -t option above). It is possible to change the predefined value of this variable with -v or -V (see options above). ENVIRONMENT The ENVIRONMENT variable is set to BATCH to identify that the job is being executed under Univa Grid Engine control. HOME The user's home directory path from the passwd(5) file. HOSTNAME The hostname of the node on which the job is running. JOB_ID A unique identifier assigned by the sge_qmaster(8) when the job was submitted. The job ID is a decimal integer in the range 1 to 99999. JOB_NAME The job name. For batch jobs or jobs submitted by qrsh with a command, the job name is built as basename of the qsub script filename resp. the qrsh command. For inter- active jobs it is set to `INTERACTIVE' for qsh jobs, `QLOGIN' for qlogin jobs and `QRLOGIN' for qrsh jobs without a command. This default may be overwritten by the -N. option. JOB_SCRIPT The path to the job script which is executed. The value can not be overwritten by the -v or -V option. LOGNAME The user's login name from the passwd(5) file. NHOSTS The number of hosts in use by a parallel job. NQUEUES The number of queues allocated for the job (always 1 for serial jobs). NSLOTS The number of queue slots in use by a parallel job. PATH A default shell search path of: /usr/local/bin:/usr/ucb:/bin:/usr/bin SGE_BINARY_PATH The path where the Univa Grid Engine binaries are installed. The value is the concatenation of the cluster configuration value binary_path and the architecture name $SGE_ARCH environment variable. PE The parallel environment under which the job executes (for parallel jobs only). PE_HOSTFILE The path of a file containing the definition of the vir- tual parallel machine assigned to a parallel job by Univa Grid Engine. See the description of the $pe_host- file parameter in sge_pe(5) for details on the format of this file. The environment variable is only available for parallel jobs. QUEUE The name of the cluster queue in which the job is run- ning. REQUEST Available for batch jobs only. The request name of a job as specified with the -N switch (see above) or taken as the name of the job script file. RESTARTED This variable is set to 1 if a job was restarted either after a system crash or after a migration in case of a checkpointing job. The variable has the value 0 other- wise. SHELL The user's login shell from the passwd(5) file. Note: This is not necessarily the shell in use for the job. TMPDIR The absolute path to the job's temporary working direc- tory. TMP The same as TMPDIR; provided for compatibility with NQS. TZ The time zone variable imported from sge_execd(8) if set. USER The user's login name from the passwd(5) file. SGE_JSV_TIMEOUT If the response time of the client JSV is greater than this timeout value, then the JSV will attempt to be re- started. The default value is 10 seconds, and this value must be greater than 0. If the timeout has been reached, the JSV will only try to re-start once, if the timeout is reached again an error will occur. SGE_JOB_EXIT_STATUS This value contains the exit status of the job script itself. This is the same value that can later be found in the exit_status field in the qacct -j out- put. This variable is available in the pe_stop and epi- log environment only. # setting the SGE_JOB_FAILED environment variable doesn't work yet #.IP "SGE_JOB_FAILED" 1.5i #This value contains the failed status of the job. This is the same value that can later be #found in the failed field in the qacct -j output. #This variable is available in the pe_stop and epilog environment only. SGE_RERUN_REQUESTED This value denotes if a job rerun on error was explicitely requested. This value is 0 if the -r option was not specified on the job submit command line, by the job class or set by a JSV script, 1 if -r y was requested and 2 if -r n was requested. For interactive jobs submitted by qrsh or qlogin, always implicitely -r n is requested and therefore the value of this environ- ment variable always is 2. This variable is available in the prolog, pe_start, job, pe_stop and epilog envi- ronment. SGE_RERUN_JOB This value denotes if the job is going to be rescheduled on error. This value is 0 if the job will not be rerun and 1 if it will be rerun on error. To determine this value, the explicitely - or for interactive jobs: implicitely - requested -r option is used. If this is option is not specified, the queue configuration value rerun is used as the default value. This variable is available in the prolog, pe_start, job, pe_stop and epi- log environment. RESTRICTIONS There is no controlling terminal for batch jobs under Univa Grid Engine, and any tests or actions on a controlling terminal will fail. If these operations are in your .login or .cshrc file, they may cause your job to abort. Insert the following test before any commands that are not pertinent to batch jobs in your .login: if ( $?JOB_NAME) then echo "Univa Grid Engine spooled job" exit 0 endif Don't forget to set your shell's search path in your shell start-up before this code. EXIT STATUS The following exit values are returned: 0 Operation was executed successfully. 25 It was not possible to register a new job according to the config- ured max_u_jobs or max_jobs limit. Additional information may be found in sge_conf(5) >0 Error occurred. EXAMPLES The following is the simplest form of a Univa Grid Engine script file. ===================================================== #!/bin/csh a.out ===================================================== The next example is a more complex Univa Grid Engine script. ===================================================== #!/bin/csh # Which account to be charged cpu time #$ -A santa_claus # date-time to run, format [[CC]yy]MMDDhhmm[.SS] #$ -a 12241200 # to run I want 6 or more parallel processes # under the PE pvm. the processes require # 128M of memory #$ -pe pvm 6- -l mem=128 # If I run on dec_x put stderr in /tmp/foo, if I # run on sun_y, put stderr in /usr/me/foo #$ -e dec_x:/tmp/foo,sun_y:/usr/me/foo # Send mail to these users #$ -M santa@nothpole,claus@northpole # Mail at beginning/end/on suspension #$ -m bes # Export these environmental variables #$ -v PVM_ROOT,FOOBAR=BAR # The job is located in the current # working directory. #$ -cwd a.out ========================================================== FILES $REQUEST.oJID[.TASKID] STDOUT of job #JID $REQUEST.eJID[.TASKID] STDERR of job $REQUEST.poJID[.TASKID] STDOUT of par. env. of job $REQUEST.peJID[.TASKID] STDERR of par. env. of job $cwd/.sge_aliases cwd path aliases $cwd/.sge_request cwd default request $HOME/.sge_aliases user path aliases $HOME/.sge_request user default request //common/sge_aliases cluster path aliases //common/sge_request cluster default request //common/act_qmaster Univa Grid Engine master host file SEE ALSO sge_intro(1), qconf(1), qdel(1), qhold(1), qmod(1), qrls(1), qstat(1), accounting(5), session_conf(5), sge_aliases(5), sge_conf(5), sge_job_class(5), sge_request(5), sge_types(1), sge_pe(5), complex(5). COPYRIGHT If configured correspondingly, qrsh and qlogin contain portions of the rsh, rshd, telnet and telnetd code copyrighted by The Regents of the University of California. Therefore, the following note applies with respect to qrsh and qlogin: This product includes software developed by the University of California, Berkeley and its contributors. See sge_intro(1) as well as the information provided in /3rd_party/qrsh and /3rd_party/qlogin for a state- ment of further rights and permissions. Univa Grid Engine User Commands UGE 8.5.4 SUBMIT(1)