Installation, Deployment and Run

Overview:
Sun Grid Engine (SGE) can be used as a globus jobmanager. The jobmanager type for SGE is `grd'. The globus toolkit includes job script files for SGE (globus-script-grd-{submit,poll,rm,queue*}).

Installation, deployment and run:
In order to enable SGE jobmanager, following steps are required:

1. When installing the globus toolkit (when running globus-build command), you need to set your executable search path and other environmental conditions properly. This can be done by executing the following command.

      % source ${SGE_ROOT}/default/common/settings.csh
        or
      # . ${SGE_ROOT}/default/common/settings.sh
   

2. For the globus build, no additional options are required for enabling SGE in the globus environment.
3. Before deploying the globus, you should edit the ${GLOBUS_INSTALL_PATH}/etc/globus-services.conf. See the Globus Toolkit System Administration Guide, page 30. For example, for SGE jobmanager on the host xxx.aist.go.jp, you would add the line:

   xxx.aist.go.jp jobmanager-grd stderr_log,local_cred - ${libexecdir}/globus-jobmanager \
   globus-jobmanager -conf ${sysconfdir}/globus-jobmanager.conf -type grd -rdn jobmanager-grd \
   -machine-type cluster
   

4. Deploy the globus by running globus-local-deploy command and start the gatekeeper. These are regular procedures for running globus gatekeeper.
5. In order to use SGE jobmanager, you have to specify the jobmanager as a gatekeeper contact. Following is a sample command line sequence for contacting to SGE jobmanager:

   % globus-job-submit "xxx.aist.go.jp/jobmanager-grd" a.out
   

Enhancement of the job script for enabling jobarray submission:
For Globus 1:
The original version of the job script (globus-script-grd-submit.in) does not support the jobarray submission. In order to enable jobarray submission, we've slightly modified the job script for SGE jobmanager in the globus package at:

${GLOBUS_SRC_DIR}/ResourceManagement/gram/libraries/job_manager/globus-script-grd-submit.in

Modified version of the job script is here. In order to enable jobarray submission, you should replace this script and re-install the globus toolkit.
For Globus 2:
In Globus 2, these jobscripts are included in Essential Grid Services I bundle.

globus_services_1_bundle.tar.gz
  -> globus_gram_job_manager-2.1.tar.gz
      -> scripts/globus-script-*-{submit,poll,rm}.

Here is a job script for globus2 package. In order to enable jobarray submission, replace this with the original version and re-install the globus toolkit.
Detailed overview of this enhancement

----------------------------------------------------------------------
Globus supports the following four jobtypes.

jobtype 0 = mpi
jobtype 1 = single
jobtype 2 = multiple
jobtype 3 = condor

These jobtypes are assinged to grami_job_type variable by GRAM.
In order to enable jobarray submissions, I took the following
approach:

If jobtype ($grami_job_type) is single and count ($grami_count) is 1,
grd_jobtype is set to "single".
If jobtype ($grami_job_type) is single and count ($grami_count) is not
1, grd_jobtype is set to "jobarray".

Therefore, there could be the following five jobtypes:

# 5 jobtypes exist               GRD result
# -----------------              ------------------
# jobtype 0 = mpi         -----> run mpirun
# jobtype 1 = single      -----> submit one copy of grd script (if grami_count = 1)
# jobtype 1 = jobarray    -----> submit jobarray (if grami_count != 1)
# jobtype 2 = multiple    -----> submit count copy(s) of grd script
# jobtype 3 = condor      -----> ERROR

In order to submit jobarray through SGE gram, user have to specify
the resource like this:

% globusrun -r "host.xxx.yyy:/jobmanager-grd" \
'&(jobtype=single)(count=10)(executable=a.out)'

When the SGE gram is invoked on host.xxx.yyy, the script file for SGE
qsub command includes the following line:

#$ -t 10

When the user wish to specify the base index and stride for jobarray
submission, ex.

qsub -t 1001:1019:2

the user can specify them as environment variables like this:

% globusrun -r "host.xxx.yyy:/jobmanager-grd" \
'&(jobtype=single)(count=10)\
  (environments=(GRD_JOBARRAY_BASE 1001)(GRD_JOBARRAY_STRIDE 2))\
  (executable=a.out)'

In this case, the script file includes the following line:

#$ -t 1001:1019:2

1019 is automatically calculated by jobarray_base, count, and
jobarray_stride.

GRD_JOBARRAY_STRIDE is an optional and it's default value is 1.
----------------------------------------------------------------------

---

Yoshio Tanaka

Last modified: Mon Feb 5 10:00:00 JST 2002