Grid Engine Portal

Installation Guide

Dan.Fraser@Sun.COM

Ron.Selva@Sun.COM

Table of Contents

 

Introduction & Technical Acknowledgments

Illustration 1
 

The illustration details the component layers that need to be integrated in the solution.

We would like to especially thank those who made significant technical contributions to the GEP:

Frédéric Parienté for his vision and foundational implementation of the GEP servlet source code.

David Bunker for the practical implementation of SunONE Portal Server with the GEP code and significant documentation contributions.

Eric Sharakan for adding & documenting (single-sign-on) X-window support to GEP.

Solution Components Detail

Data Set

The Data Set layer represents the information that is processed by the compute engine. It includes the raw data that is accessed by applications for compare and process operations, the input arguments and parameters and the output from the processing commands.

Compute Engine

The Compute Engine layer is essentially Grid Engine. Grid Engine is a distributed application processing solution from Sun Microsystems. Grid Engine controls the submission of jobs to the Grid Cluster and Grid Cells (Sub Clusters), the distribution of jobs to execution hosts within the cluster and the administration, management and auditing of these jobs to and from processing queues. Grid Engine can be integrated with other applications used in the processing of compute intensive problems.

Java Servlet

Portal Environment

The Portal Environment is provided using the SunONE Portal Server. Portal Server provides the capabilities of personalization, customization, security and aggregation to a Java enabled web browser. Portal Server can be deployed in a multi-domain scenario, utilizing varying authentication methods and role functions within each domain.

Multi- Domain Environment

The Multi-Domain Environment enables organizations to provision specific content, manage multiple authentication and security structures and create a personalized and customizable user experience with consistent look and feel within the domains. Within domains user roles are implemented allowing varying security levels and responsibilities to be assigned to users within the portal environment. In this solution these roles are used to differentiate between administrators, support personnel and end users. The administrative controls allowed within a domain provide a level of delegated administration to users of the solution. The Portal Server, it's domains and the roles and users within the domain can be customized at all levels, allowing organizations to provision content and services at a certain level and have those functions "inherited" by lower level categories. For example an application could be provisioned at the portal server level and this application could then be made available to all domains, roles and users within the portal environment.

Illustration 2 represents graphically the portal domain structure.
 
 

Illustration 2
 
 
 

Illustration 2 shows the flexibility of the portal environment to provision domains each with a different look and feel and authentication method; roles to perform specific tasks within the portal framework with different authentication methods; sub-roles; and users within the role structure with differing authentication methods.
 

Installation Procedure

Installation Procedure Summary

We will assume that the computer hardware, base operating system (including latest patches) and network configuration has been completed.

Step 1: Install Grid Engine (and Sun ClusterTools w/ loose integration for MPI support).

Step 2: Install SunONE Portal Server.

Step 3: Install the Grid Engine Portal Integration Package.

Step 4: Adjust the configuration of SunONE Portal Server.

Step 5: Test the Grid Engine Portal

Step 6: Register new applications with Grid Engine Portal
 

Step 1: Grid Engine

Download the latest version of Grid Engine from http://www.sun.com/gridware.

Follow the installation instructions. The default queues and the default cell will work fine.

If MPI jobs will be run through the portal:
 

Download and install the latest version of the HPC ClusterTools software at http://supportforum.sun.com/clustertools.

Create a loose integration for HPC ClusterTools and Grid Engine. The Loose Integration package is a part of the Grid Engine download (version 5.3 or later) and may be found under the directory HPC_GE_Integration. Follow the directions in the README file.

Step 2: SunONE Portal Server

Generally you should install in the /opt (default) directory.

When asked if this will be an open Portal install this is SunONE Portal Server lingo that means without a gateway. Some have found it simpler to install without a gateway and add a gateway later if desired (specifically for using the X-windows graphics capability.) The default is to use a gateway, but make sure to install the gateway after the server has been installed.

If asked about using SSL connectivity to the server, this refers only to the communications between the Server and the Gateway. This may be required in ultra-sensitive environments, but is not usually necessary. All external communication through the gateway automatically uses SSL (i.e. https). When in doubt, take the default.

When asked about the hostname, sub-domain, and domain for the server (or gateway). The following examples apply: mercury.sun.com would have mercury as the hostname, no subdomain (use . ) and sun.com as the domain. If the system were mercury.iforce.sun.com, then iforce would be the sub-domain and sun.com would be the domain name. These names must be specified for both the server and gateway components. In other words there must be a line in the /etc/hosts file that links both names such as the following:

                          129.146.60.220 scserv220 loghost scserv220.iforce.sun.com

(This must be done prior to beginning the SunONE Portal Server installation).

When specifying the root of the profile role tree the default may be something like sun.com which is unacceptable, since the GEP requires this name to be a valid username. Here instead of the default, use suntcp. Whatever name is used however must be the same when later installing the gateway.

Next, the user for the profile role tree must be root. Take the default.

It is good to select the default (y) to start the server after it has been installed.

Next select 2 to begin installing the gateway (unless this is not required). Be sure the root of tree is the same name specified above, and that all corresponding domain names match exactly.

Step 3: Grid Engine Portal Integration (Glue) Package.

Become root and create the SGP_ROOT installation directory # mkdir -p /export/GridPortal       (Note: /export/GridPortal must be an NFS shared file system across all Grid Engine nodes)   # chmod 755 /export/GridPortal # SGE_ROOT=<where Grid Engine has been installed> # SGE_CELL=<Grid Engine sge_cell if not default cell> # COMMD_PORT=<Grid Engine commd port> # export SGE_ROOT SGE_CELL COMMD_PORT # cd /export/GridPortal # gzip -dc /tmp/gridportal.tar.gz | tar xvf -      (Prior to the official open-source release of Grid Engine Portal, the gridportal.tar.gz file includes a        directory named infotext that contains the infotext.tar file, which is needed for environments        that have installed early releases of Sun Grid Engine 5.3 or prior releases of Sun Grid Engine.        The required infotext binaries were not supplied in these prior releases of Sun Grid Engine, but        are needed for the purposes of this Grid Engine Portal installation.  Check the SGE_ROOT/utilbin       directory for the presence of the infotext binaries in both the solaris and solaris64 subdirectories.       If they are not there, and the gridportal.tar.gz file contained them, please install them using the        following two commands.  Otherwise, skip to the cd /tmp command below.) # cd $SGE_ROOT/utilbin # tar xvf /export/GridPortal/infotext/infotext.tar  # cd /tmp # unzip cos-27May2002.zip lib/cos.jar # cd /export/GridPortal # gzip -dc /tmp/vnc-3.3.3r2_sun4_sosV_5.5.tgz | tar xvf - # cd vnc_sun4_sosV_5.5 # gzip -dc /tmp/vnc-3.3.3r2_javasrc.tgz | tar xvf - # cd .. # vnc_patches/applyVncPatches    Note: You will be asked by the install script below to supply  the following parameters:              O'Reilly servlet path - use /tmp/lib/cos.jar assuming you followed the exact steps above              Vnc root directory - use /export/GridPortal/vnc_sun4_sosV_5.5 assuming you followed                                               the exact steps above              Grid Portal domain - for consistency with the nomenclature in this guide, use suntcp # ./install -gp Installing Grid Portal servlets ------------------------------- We will ask for - the O'Reilly servlet location - the SunONE Portal Server (iPS) root directory - the Vnc root directory (optional) - the Grid Portal domain The Grid Portal is using the O'Reilly servlet cos.jar The version we used for testing is available as http://www.servlets.com/cos/cos-27May2002.zip Please read the license conditions carefully and download this file. Then proceed by unzipping it, e.g. like this % cd /tmp % unzip cos-27May2002.zip lib/cos.jar This path must be entered here, e.g. /tmp/lib/cos.jar Please enter the O'Reilly servlet path >> /tmp/lib/cos.jar Please enter the iPS root directory ( default: /opt ) >> /opt Do you want to use VNC >> y Please enter the VNC root directory >> /export/GridPortal/vnc_sun4_sosV_5.5 Please enter the Grid Portal domain >> suntcp You entered the following installation information O'Reilly servlet: /tmp/lib/cos.jar Grid Portal root: /export/GridPortal iPS install directory: /opt VNC root directory: /export/GridPortal/vnc_sun4_sosV_5.5 Grid Portal domain: suntcp Do you want to use these parameters (y/n) [y]>> The Grid Portal servlets will now be installed. iPS is automatically restarted. Do you want to proceed (y/n) [y]>>  stopping auth helpers ... done. stopping web server ... done. stopping directory server ... done. stopping gateway ... done. No change to /opt/netscape/server4/https-bilbo.Germany.Sun.COM/config/rules.properties No change to /opt/netscape/server4/https-bilbo.Germany.Sun.COM/config/jvm12.conf Do you want to install example applications (y/n) [y]>> stopping auth helpers ... done. stopping web server ... done. stopping directory server ... done. starting auth helpers ... done. removing /opt/netscape/directory4/slapd-bilbo/locks...done starting directory server ... done. starting web server ... done. starting gateway ... done. Do you want to install channels (y/n) [n]>>  The installation has completed.

Every user that utilizes VNC for X-Windows applications must change his/her environment settings. VNC is called from the servlet with a
 

Next the channels can be added.  If this is a clean install of SunONE Portal Server SP4 (i.e., no additional channels have been added to the portal server) then you can say yes when asked for installing channels during installation. Otherwise you MUST manually add the channels by following the steps in Appendix A.

If you have previously installed earlier versions of Grid Engine Portal (then known as the Technical Computing Portal), you will need to delete the suntcp.jar file from /opt/SUNWips/lib.
 

Step 4: Configure SunONE Portal Server

Open a browser window and go to the URL http://mercury.sun.com:8080/console
Login as root with the system root passwd
Select Manage Domains
Click suntcp (or whatever domain name was used in the installation)
Select Authentication
Unselect all but UNIX authentication
Select Submit

Select Manage Domains
Click suntcp
Expand the Applications tab by clicking on the key next to Applications
Select Netlet
Decide how many simultaneous X server sessions you want to support (one X server session corresponds to one logged in GEP user). Add netlet rules as shown , following the pattern for as many simultaneous sessions as desired (incrementing the port numbers by one for each subsequent rule) based on the portal server hostname (in this example mercury -- you can also use mercury.sun.com). Add a minimum of three rules for debugging purposes.

            Xvnc:1|http://localhost:5801/portal.vnc|5801:mercury:5801|5901|mercury|5901
            Xvnc:2|http://localhost:5802/portal.vnc|5802:mercury:5802|5902|mercury|5902
            Xvnc:3|http://localhost:5803/portal.vnc|5803:mercury:5803|5903|mercury|5903

               ...

Uncheck Warning Popup For Connections
Select Submit then Continue
Select Logout (must be done to set the changes)

Step 5: Using Grid Engine Portal

Access SunONE Portal Server via the link https://mercury.sun.com/. When you login, a small Netlet window will pop up. The first time it runs, it will pop up a series of dialogs asking for additional permissions. You must Grant all these permissions, or the netlet will not work. This window hosts the netlet applet, which is required for tunneling the Xvnc protocol through the portal. Heed the warning in the window about not closing it. You will also find a netlet channel on your desktop, with a note saying No netlet targets configured (unless you have configured other, non Xvnc netlet targets). This comment is not entirely accurate; targets are configured for Xvnc, but they do not show up in the Netlet channel. Do not remove this channel from your desktop. Also, the Netlet window does not dismiss itself when you logout. You must manually dismiss it, or netlet functions will not work the next time you login.

In the portal server, the Project List, Job List, and Application List channels should be visible. If logged in as root or as an administrative user there will be Admin Application List and Job Control channels. (If these channels display errors, check for incorrect paths and make sure Grid Engine is accessible from the portal. Also it is sometimes helpful to view the source behind the channels, which can be accomplished by right clicking on the mouse and selecting the View Source option.)

Users can select which channels they wish to have visible by checking on the Content switch. The order in which channels are displayed can be set by the Layout switch.

The Job List channel should contain the message You have no running jobs. This message shows that contact with Grid Engine has been established (qstat has executed successfully). (A Java error in this window could indicate a problem communicating with Grid Engine. Go back and verify that Grid Engine and SunONE Portal Server were installed correctly. Try logging in directly as the user, executing the Grid Engine settings script (settings.sh), and running qstat.

If you get Java exceptions, edit the /opt/netscape/server4/https-mercury.sun.com/config/servlets.properties file by hand to reflect the correct paths. This should not happen after a successful installation.

The Project List contains a list of projects for the current user. Projects are stored in the suntcp subdirectory of the /export/GridPortal/workspace/username directory. A list of projects is kept in the file .suntcp-list in the user's home directory. An example file might look like:

Fasta_project Fasta Genome
Blast_project Blast Genome
Qmon_project qmon
P1002129493410 New Project name

where the first entry on each line corresponds to a specific directory that was created by GEP in the user's workspace (e.g. /export/GridPortal/workspace/username/suntcp/Fasta_project) and the second string (separated by a tab) corresponds to the name of the project that is used in the Project List channel. If there are no Projects, then you can add a project by clicking on the Create new project link. (Note:  Use of any other character than a Tab to delineate the entries will result in a variety of strange errors.  If you edit these files by hand, be sure that your editor is not inserting multiple spaces in place of tabs.)

Finally the Application List channel should contain a list of applications that was installed in the Application directory (/export/GridPortal/apps) during the GEP installation process.  If there are no applications, make sure the application files were installed correctly in the applications directory.

Now you are ready to run a job. Click the Submit new job link in the Job List channel. Select the Fasta Genome project. Use the default values for Algorithm, and Query file. Select the first database in the list. Click submit. (If this project is not available, create a project for the Sleeper Demo application. Then submit this job. The input argument is simply the number of seconds you want the processor to sleep). When the screen refreshes, the job should appear. Click on the job for detailed information from Grid Engine. Make sure the job is executing. If it is waiting for a queue, for example, Grid Engine may not be properly configured.

If an error is detected, you can attempt to debug it by executing the job by hand. First, you will need to go to the /export/GridPortal/workspace/username/suntcp directory of the user that logged into the portal. This directory should contain a list of projects. Look at the .suntcp-list file to determine which project directory you just used, then go to the appropriate project directory.

Two files that should be in this directory are .suntcp-su and .suntcp-qsub. The Grid Engine Portal executes the .suntcp-su script, which in turn executes the .suntcp-qsub script. Try executing these by hand. Any errors detected should indicate possible errors in the Grid Engine, SunONE Portal Server or Grid Engine Portal setup. Common errors include incorrect permission settings on the application or queue problems with Grid Engine.

Next try submitting a job that uses an X-windows interface. Go to the Qmon subdirectory of the applications directory. Create a qmon project if there is not already one. Then submit this project as a new job.

[Note: The SunONE Portal Server netlet which tunnels the X11 SGP connection between client and server is sensitive to your browser's proxy settings. In addition, it cannot understand so-called PAC (proxy auto-config) files (yet). This means the netlet will use whatever settings are under your Netscape Manual ProxyConfiguration settings, even if you're configured to use the Automatic Configuration. So either make sure the settings under the Manual configuration are reasonable, or select Direct connection to the internet, whichever is appropriate for your configuration.]

When submitting a job that utilizes X11, the first time you do so after logging in, a new X server is launched for you, and an applet window is displayed in your browser. This applet is essentially your X display. Your job should appear in this window shortly. You can leave this window open for the duration of your portal session, and any new applications utilizing X will appear in this window.

Alternatively, you can explicitly start the X session by clicking the Launch Xvnc server bookmark, which you'll find in the Bookmarks channel of your desktop. Once the X session has begun, other jobs launched which utilize X will use the existing session.

There is absolutely no X state kept on the applet side. If the applet crashes, or you inadvertently close the window, you can restart the applet one of two ways: via the Launch Xvnc server bookmark in your Bookmarks channel, or by attempting to launch another X-based application. Either way, GEP notices you already have a session active, and provides you a link to just restart the viewer applet. Click that link, and you should find the X display exactly as you left it.

The X session automatically terminates when you logout of the portal. In addition, a Kill Xvnc server bookmark is provided for you if you want to do so manually. While killing the Xvnc server seems to kill any jobs running under it, it is not recommended to rely on this behavior. It's better to make sure your X-based jobs are no longer running before killing the X session explicitly or logging out.

Step 6: Register New Applications with Grid Engine Portal

Edit the .suntcp-list file in the applications directory specified above. This file contains lines, each with two entries of the form directory (tab) Application Description. Add a new line with the directory that the application will execute from, then a tab character, then a description that will show up in the Portal. Create the directory and place the executable there. The executable can be a script, a link, or an actual binary. Make sure any relevant data files are accessible from this directory. (Note: Use of any other character than a Tab to delineate the entries will result in a variety of strange errors.  If you edit thse files by hand, make sure that your editor is not inserting multiple spaces in place of Tabs.)

Create a .suntcp-app file in the directory just created. For examples, look at the equivalent file in other application directories. This file contains the following lines:

Line 1- An application description (version etc.)
Line 2- The name of the executable
Line 3- yes/mpi/no Whether or not the application runs SMP/MPI/ or Sequential
Line 4- yes/no Whether the application uses an X-windows interface
Line 5- yes/no Whether the application uses a form
Line 6- List of users who have access to the application (optional) (Note: If no user access list is being
            defined, Line 6 must be a blank line.)
Line 7- Additional lines can be inserted directly into the script that is submitted to Grid Engine. Useful examples can be
            Grid Engine commands (e.g. #$ -q qname) or environment variables (e.g. export Option=2). Place each command
            on a separate line.

Create an HTML/JavaScript form if Line 5 contains yes. This file must be named .suntcp-form. Example forms for use as templates can be found in the existing application directories. Forms are responsible for creating the execution line arguments. (Technical note: the form.elements[0] element is reserved to make sure the form returns to the proper location. Do not modify the lines of code that use this element. Begin modifying the [1] element.)

A Tip for Developers

Appendix A: Adding Channels Manually

Open a browser window and go to the URL http://mercury.sun.com:8080/console
Login as root (or an administrative user) with the system root passwd
Select Manage Domains
Click on suntcp
Select Desktop Channel Wizard
    Channel name: suntcpProjectList
    Channel description: Directory spaces (projects) for the applications to run under
    Select URL scraper
    Select Next
    Title: Project List
    Content URL: /SunTCP?action=projectList
    Select Finish
    Select'Continue
Select Desktop Channel Wizard
    Channel name: suntcpApplicationList
    Channel description: List of applications available through the GEP
    Select URL scraper
    Select Next
    Title: Application List
    Content URL: /SunTCP?action=applicationList
    Select Finish
    Select Continue
Select Desktop Channel Wizard
    Channel name: suntcpJobList
    Channel description: List of running jobs in the GEP
    Select URL scraper
    Select Next
    Title: Job List
    Content URL: /SunTCP?action=jobList
    Select Finish
    Select Continue

Now provide the channels for the user

Select Manage Domains
Select suntcp
    Select the key next to Applications
    Select Desktop
    Select suntcpJobList in the Available Channels box
    Select Edit Channel
    Select Show Advanced Options
    Change width from thick to thin (Puts the joblist on the left of the window)
    Select Submit, then Continue, then Back to Overview
    Move the three new channels that we just added to the Selected Channels box
    Select Submit
   Select the iwtBookmarkProvider in the Available Channels box and click Edit Channel
    Add the following bookmarks exactly as shown:
        Launch Xvnc server|/SunTCP?action=launchserver
        Kill XVNC server|/SunTCP?action=launchvncserver&kill=yes
    Select Submit and then Continue

Select Manage Domains
Select suntcp
Select Admin Role
Select Desktop Channel Wizard
    Channel name: suntcpAdminApplicationList
    Channel description: List of applications available through the GEP
    Select URL scraper
    Select Next
    Title: Admin Application List
    Content URL: /AdminTCP?action=applicationList
    Select Finish
    Select Continue
Select Desktop Channel Wizard
    Channel name: suntcpAdminJobControl
    Channel description: Monitoring console for jobs submitted through the GEP
    Select URL scraper
    Select Next
    Title: Job Control
    Content URL: /AdminTCP?action=jobList
    Select Finish
    Select Continue

Select Manage Domains
Select suntcp
    Select the key next to Applications
    Select Desktop
    Select the suntcpJobControl channel in the Available Channels box
    Select Edit Channel
   Select Show Advanced Options
    Change width from thick to thin
    Click Submit and then Continue
    Select Back to Overview
Move the two new Admin channels and the three previous User channels to the Selected Channels box as was
    done previously
Select Submit and then Continue
 
Now forward the cookies of all the channels
 
Select Manage Domains
Select suntcp
Select Policy
    In the iwt"channel-name"-cookiesToForward section add a * or the name of your SunONE Portal Server
    session cookie (which is SunONE Portal Server by default) to each of the channels that you just added.
Click Submit
Logout (This is important. Changes are not updated until Logout.)

Appendix B - Screen Shots

The image below is the authentication window for a domain created during a Proof of Concept project undertaken by Sun Microsystems and the University of Queensland in Australia. This domain is for the Institute for Molecular Biology and uses UNIX as an authentication method into their NIS domain.
 
 
 

 
 
 
 
 
 
 
 

The image below shows the Admin user window for the IMB domain.  (Note that the channel names are slightly different from what you will see when you install the GEP software)

Note that all five applications as channels have been provisioned into this role user. You can see that in the Job List window there is a Submit option, whereas in the Job Monitor window this option has been replaced by Perform accounting. You can also see that the AdminApplication List and Application List content is differentiated by the option Add a new application. This option does not appear in the Application List window (seen as the middle window above Project List).

 
 
 
 
 
 
 
 
 
 
 

The image below shows the end user's desktop. End users are provisioned into a role referred to as researcher.

This user has the channels to be able to create projects, see and select applications and submit jobs against the Grid Engine cluster.