| This document describes the Grid Engine source code modules, and is
written for use by software developers. Our goal is to provide you with useful
information if you are considering participating in the Grid Engine project
or will be using Grid Engine source code. In addition to this source documentation,
the user product documentation is necessary for a full understanding of the
functional operation of the software. Consequently the man pages are refered to herein where appropriate.
If you have problems with finding functions or browsing through the
souce code we suggest the use of grep or preferably ctags. The module documentation for version 5.3 can be found here. 1. Job life cycleThe job life cycle is a good way to understand to overall system.
2. Components2.1 Daemon componentsIn the "source/daemons" subdirectory there is one directory for each daemon.QmasterThe Qmaster (Queue Master) controls the overall behaviour in a cluster. For the purpose of scheduling jobs Qmaster cooperates with Schedd. The Qmaster is the responsible for answering GDI requests from clients and for delivering dispatched jobs to the assigned Execd's.ScheddThe Schedd (Scheduling Daemon) gets notified via events about all scheduling relevant information. The resulting scheduling decisions are sent as orders to Qmaster.ExecdThe Execd (Execution Daemon) provides Qmaster with information about utilization and availability of resources. Each job sent to Execd by Qmaster is started by writing all relevant information into a couple files describing the job and forking a Sheperd. After Shepherds termination Execd reports details about the job run back to Qmaster.ShepherdThe Shepherd starts all kinds of jobs according to what he finds in the per-job configuration files written by Execd. If necessary administrator supplied procedures are started before the job is started (prolog, PE start-up), during job run (checkpoint/migrate, terminate/suspend/resume) and after job termination (PE shut-down, epilog).CommdNearly all network communication in a cluster is handled via the Commd (Communication Daemon). Daemons and clients use the commlib (communication library) calls to send and receive messages (sync/async), but also a hostname resolving service is provided by Commd.ShadowdThe Shadowd (Shadow Daemon) detects failures of the Qmaster and starts a new Qmaster if necessary.2.2 Client componentsIn the "source/clients" subdirectory there is one directory for each client.QsubQsub combines all information provided with the command line and in per-cluster or per-user default settings files in a job object and sends the job via GDI to Qmaster. As response Qsub receives a job ID and prints it.QrshQrsh combines all job relevant information in a job object like Qsub does and sends it as an interactive job via GDI to Qmaster. For stdin/stdout/stderr redirection rsh command is forked by qrsh. The rsh command cooperates with the rshd forked by shepherd for jobs of that type. Instead of rsh/rshd also other commands can be used.QmonQmon is a Motif GUI allowing to administer and monitor the system but also to submit and monitor jobs. Qmon uses GDI to retrieve from Qmaster the whole configuration and to change it.QdelQdel uses GDI to initiate job termination.QacctQacct produces accounting reports by scanning the accounting data file.QloginQlogin invokes an interactive login session in the current terminal using GDI.QresubQresub creates a copy from an existing job using GDI.QselectQselect prints a list of queues matching the specified criteria from the command line. The necessary information about the clusters configuration is fetched from Qmaster using GDI.QalterQalter modifies an already submitted job using GDI.QholdQhold puts a hold on a job using Qalter.QrlsQhold releases a hold from a job using Qalter.QshQsh submits an interactive X-windows session job using GDI.QconfQconf allows to add, delete, modify and view the configuration of all kind of configuration objects. Qconf uses GDI to do this.QhostQhost shows the current status of all hosts in the cluster based on information retrieved using GDI.QstatQstat shows the current status of all queues in the cluster based on information retrieved using GDI.QmodQmod allows to modify the state of queues and jobs by using GDI.2.3 LibrariesIn the "source/libs" subdirectory there is one directory for each library.librmonThis low level library provides functionality to run daemons in a monitoring mode.libutiA low level library with several utility functions. This library bases on functionality of librmon.libcullThe generic list library which is used in nearly all components of the system. The cull library bases on functionality of libuti and librmon.libcommThe communication libraryis the interface providing communication via Commd. This library bases on librmon.libgdiThe Gridengine Database Interface library provides methods to retrieve, add, delete and modify all internal objects hosted by the Qmaster. The GDI library bases on libcomm and libcull.libschedThis library contains many important high-level functions mainly used by Schedd. Some of them however have turned out to be very helpful also in other clients like Qstat, Qhost or qmon and also in Qmaster.
libevcThe event client interface allows a client to register at qmaster and receive events about changes in the system more user friendly than with polling.libevmThis library contains code providing the master side of the event client interface.It is currently used in qmaster to serve event clients but could be used to build any event server, e.g. an event caching / proxy component. libmirThe mirror interface provides a mechanism to mirror the data held in qmaster, e.g. the job list or the queue list.It is based on the event client interface. libjapiThe job API interface (JAPI) provides a to-the-point interface providing those operations typical Grid Engine end users depend upon. That is submission of jobs, enquire and control of job status, synchronizing with jobs end as well as examining how a job finished. Operations required to administer Grid Engine are not covered by JAPI. This library bases on functionality of libgdi and libevc.libdrmaaThe DRMAA library is an implementation of the DRMAA C binding standard based on Grid Engine. The purpose of DRMAA is to provide an interface that facilitates integration of applications with DRM systems such as Grid Engine in a DRM system independent fashion. The DRMAA library is kept in the libs/japi subdirectory due to the close relationship with JAPI.2.4 Third-party componentsThe "source/3rdparty" directory contains modules with different licenses than SISSL.QmakeThis client is a GNU make enhanced with transparent remote execution. The implementation of transparent remote execution bases on a forked 'qrsh -inherit' command.QtcshThis client program is an extension to the popular csh(1) derivative tcsh. The implementation of transparent remote execution bases on libgdi.Remote toolsThis directory is the home of rsh, rshd, rlogin.2.5 UtilitiesThe "source/utilbin" directory contains binaries that are needed during installation of Grid Engine, for diagnosis purposes or for the job samples.2.6 DistThe "source/dist" directory contains files and directories which are part of a Grid Engine distribution.3. SecurityThe "source/security" directory contains modules handling security.4. ExperimentalEach subdirectory of the "source/experimental" directory contains an experimental module. Those modules cannot be expected to work - although some may, at least with a bit of work. The experimental modules are provided here for information purposes mainly and may be helpful if you plan to enhance Grid Engine in a direction similar to one of the modules. Some also can be a valid starting point for such an effort. |
