YAIM Overview (Tutorial Session, International Symposium On Grid Computing 2008, Taipei, Taiwan)

NOTES on the EGEE Tutorial, 7 April 2008, Academia Sinica, Taipei, Taiwan.


http://quattor.grid.sinica.edu.tw/UI_practical.html

Topic: YAIM Introduction

• How to Begin


• Install a node


• Configure a node

YAIM = Yet Another Installation Manage
Developed by EGEE SA3 team

This tutorial will guide you through the process of installing a User Interface (UI). In particular we will install a TAR UI which is installed into your own home directory and does not require root permission. The standard UI installation will require root permissions, but will allow all users to access UI commands. For this tutorial we will use the Yaim Tutorial Server. Please refer to the tutorial handout sheet for server and account details.


1. Installing the UI TAR
Install the UI_TAR packages
Configure the UI

2. Preparing your UI environment
Configure the Grid Environment
Install your user certificate

3. Test the Grid jobs submission by your UI
Query the Grid Computing Environment
Simple job submission
Submitting jobs by the Resource Broker
Query the job status
Get your job results

4. Test the Grid data stor by your UI
Query the Grid Storage Environment
Set the DPNS HOST
Create directory on diskpool
Put your file into diskpool
Down file from diskpool

Installing the UI TAR

1. Please make sure the following file is already into your home directory
[taipei01@t-yaim~]$ ls -1F certificates/gLite/glite-UI-3.1.8-0-external.tar.gzglite-UI-3.1.8-0.tar.gzjobs/site-info.def.isgc2008
You can also find the last packages by the following link:The relocatable UI
Notice: The PPS is the pre-production packages, that sometime have several new features, but might also have some bugs, If you are not certainty to use the new features, we recommend you to use the production packages.
To install the UI_TAR, you have only to untar the UI tar ball in your own directory.
2. Untar the tar balls
[taipei01@t-yaim~]$ tar -xzf glite-UI-3.1.8-0-external.tar.gz -C gLite [taipei01@t-yaim~]$ tar -xzf glite-UI-3.1.8-0.tar.gz -C gLite
3. Change the work directory and check the installed files[taipei01@t-yaim gLite]$ cd gLite[taipei01@t-yaim gLite]$ ls -Fclassads/ edg/ external/ globus/ lcg/d-cache/ etc/ glite/ gpt/ tmp.31683/[taipei01@t-yaim gLite]$
4. Run the yaim script to configure the UI_TAR. Here we use the -c "configure command" to configure the UI. The configuration has already been completed for you in site-info.def. Please take a look inside the file to get an idea of how to fill in the yaim parameters. This particular site-info.def allows you to configure a UI that uses the gilda grid infrastructure.[taipei01@t-yaim gLite]$./glite/yaim/bin/yaim -c -s ~/site-info.def.isgc2008 -n UI_TAR
You shouldn't get any error messages.
Notice:Please pay attention to the X509_CERT_DIR of site-info.def variable, here we used "/etc/grid-security/certificates/", if you have not root permission and your computer never installed any certificates, you should modify the X509_CERT_DIR to fit your own directory, the yaim script will go to install all the certificates into this directory.For example:If you have specifyX509_CERT_DIR=$HOME/gLite/external/etc/certificatesin your site-info.def,The yaim script will install all the certificates into $HOME/gLite/external/etc/certificatesHere we were already installed gilda certificates into /etc/grid-security/certificates/ for this tutorial, so please don't modify the X509_CERT_DIR in this practical anyway.
Preparing your UI environment
5. Setup the environment. UI tar package contains two environment scripts that must be sourced in order to use the UI client tools. Here we add these scripts to your startup script, that can help to auto configure the UI environment every time you login the shell.[taipei01@t-yaim gLite]$ echo 'source $HOME/gLite/external/etc/profile.d/grid-env.sh' >>~/.bashrc
Now, you can re-login to the shell or just running following command:[taipei01@t-yaim gLite]$ source ~/.bashrc
6. Put your user certificate and private key into ~/.globus. After you obtain your certificate from your CA, you will need to install them on an UI in the .globus directory.The commands below show you how this is done. Pay close attention to the permission of these certificates.[taipei01@t-yaim gLite]$cd ~[taipei01@t-yaim ~]$ mkdir .globus[taipei01@t-yaim ~]$ chmod 700 .globus[taipei01@t-yaim ~]$ cp certificates/TAIPEIxx-cert.pem .globus/usercert.pem[taipei01@t-yaim ~]$ cp certificates/TAIPEIxx-key.pem .globus/userkey.pem[taipei01@t-yaim ~]$ ls .globus/usercert.pem userkey.pem[taipei01@t-yaim ~]$ chmod 644 .globus/usercert.pem[taipei01@t-yaim ~]$ chmod 400 .globus/userkey.pem
The following step is only for this tutorial that will add gilda vo certificate into your own UI environment [taipei01@t-yaim ~]$ cp /opt/gilda-voms/gilda-voms.ct.infn.it ~/gLite/external/etc/grid-security/vomsdir/
7. Create a proxy certificate.
[taipei01@t-yaim ~]$ voms-proxy-init -voms gildaEnter GRID pass phrase: Your identity: /C=IT/O=GILDA/OU=Personal Certificate/L=TAIPEI/CN=TAIPEI01/Email=jennsu@ccweb.sinica.edu.twCreating temporary proxy ...................................... DoneContacting voms.ct.infn.it:15001 [/C=IT/O=INFN/OU=Host/L=Catania/CN=voms.ct.infn.it] "gilda" DoneCreating proxy ........................................................ Done Your proxy is valid until Fri Mar 26 21:56:48 2008
Congratulations, you have now configured and setup a Grid User Interface environment that is required to access the Grid resources. We will now test your environment by performing simple job submission and data management task.
Test the grid jobs submission by your UI
8. Check which CE are available on for gilda Virtual Organization with lcg-infosites:[taipei01@t-yaim ~]$ lcg-infosites --vo gilda ce
valor del bdii: grid004.ct.infn.it:2170
#CPU Free Total Jobs Running Waiting ComputingElement
----------------------------------------------------------
11 11 0 0 0 grid-ce.bio.dist.unige.it:2119/jobmanager-lcglsf-gilda
10 10 0 0 0 grid011f.cnaf.infn.it:2119/jobmanager-lcgpbs-gilda
34 34 0 0 0 grid010.ct.infn.it:2119/jobmanager-lcgpbs-long
34 34 0 0 0 grid010.ct.infn.it:2119/jobmanager-lcgpbs-short
34 34 0 0 0 grid010.ct.infn.it:2119/jobmanager-lcgpbs-infinite
56 46 2 2 0 iceage-ce-01.ct.infn.it:2119/jobmanager-lcgpbs-short
56 46 1 1 0 iceage-ce-01.ct.infn.it:2119/jobmanager-lcgpbs-long
56 46 0 0 0 iceage-ce-01.ct.infn.it:2119/jobmanager-lcgpbs-infinite
10 10 0 0 0 grid004.iucc.ac.il:2119/jobmanager-lcgpbs-short
10 10 0 0 0 grid004.iucc.ac.il:2119/jobmanager-lcgpbs-long
10 10 0 0 0 grid004.iucc.ac.il:2119/jobmanager-lcgpbs-infinite
16 16 0 0 0 gn0.hpcc.sztaki.hu:2119/jobmanager-lcgpbs-gilda
9.Test the CE by submitting a simple job that runs directly on the CE itself:[taipei01@t-yaim ~]$ globus-job-run grid011f.cnaf.infn.it /bin/hostname
grid011f.cnaf.infn.it
Make sure you have a correct output which is the hostname of the CE.
10. Test job submission through a resource broker:
Edit a jdl file first[taipei01@t-yaim ~]$ cd jobs/
[taipei01@t-yaim jobs]$ ls
edglog.log testJob.jdl testJob.sh
[taipei01@t-yaim jobs]$ vim testJob.jdl
* Please add following contents in to the testJob.jdl:
Executable = "testJob.sh";
StdOutput = "testJob.out";
StdError = "testJob.err";
InputSandbox = {"./testJob.sh"};
OutputSandbox = {"testJob.out","testJob.err"};
VirtualOrganisation = "gilda";
Feel free to modify the JDL and specify your job requirements.
Edit testJob.sh[taipei01@t-yaim jobs]$ vim testJob.sh
#!/bin/bash
date
hostname
echo "****************************************"
echo "env sort"
echo "****************************************"
env sort
echo "****************************************"
echo "mount"
echo "****************************************"
mount
date
11. Check which CEs match your JDL requirements.
usage: edg-job-list-match
[taipei01@t-yaim jobs]$ edg-job-list-match testJob.jdl
Selected Virtual Organisation name (from JDL): gilda
Connecting to host glite-rb.ct.infn.it, port 7772
***************************************************************************
COMPUTING ELEMENT IDs LIST
The following CE(s) matching your job requirements have been found:
*CEId*
ce-nano-37.to.infn.it:2119/jobmanager-lcgpbs-infinite
ce-nano-37.to.infn.it:2119/jobmanager-lcgpbs-long
ce-nano-37.to.infn.it:2119/jobmanager-lcgpbs-short
dgt01.ui.savba.sk:2119/jobmanager-lcgpbs-infinite
dgt01.ui.savba.sk:2119/jobmanager-lcgpbs-long
dgt01.ui.savba.sk:2119/jobmanager-lcgpbs-short
gilda-ce-01.pd.infn.it:2119/jobmanager-lcgpbs-infinite
gilda-ce-01.pd.infn.it:2119/jobmanager-lcgpbs-long
gilda-ce-01.pd.infn.it:2119/jobmanager-lcgpbs-short
gilda01.ihep.ac.cn:2119/jobmanager-lcgpbs-gilda
gn0.hpcc.sztaki.hu:2119/jobmanager-lcgpbs-gilda
grid010.ct.infn.it:2119/jobmanager-lcgpbs-infinite
grid010.ct.infn.it:2119/jobmanager-lcgpbs-long
grid010.ct.infn.it:2119/jobmanager-lcgpbs-short
iceage-ce-01.ct.infn.it:2119/jobmanager-lcgpbs-short
iceage-ce-01.ct.infn.it:2119/jobmanager-lcgpbs-infinite
iceage-ce-01.ct.infn.it:2119/jobmanager-lcgpbs-long
***************************************************************************
12. Submit the test job:
usage: edg-job-submit [taipei01@t-yaim jobs]$ edg-job-submit testJob.jdl
Selected Virtual Organisation name (from JDL): gilda
Connecting to host glite-rb.ct.infn.it, port 7772
Logging to host glite-rb.ct.infn.it, port 9002
*********************************************************************************************
JOB SUBMIT OUTCOME
The job has been successfully submitted to the Network Server.
Use edg-job-status command to check job current status. Your job identifier (edg_jobId) is:
- https://glite-rb.ct.infn.it:9000/js8FEQjvTiVDaIPwQhWqMA
*********************************************************************************************
* Notice, please remember your job id, in this case the job id is:
"https://glite-rb.ct.infn.it:9000/js8FEQjvTiVDaIPwQhWqMA"
Get the job status:
usage: edg-job-status [taipei01@t-yaim jobs]$ edg-job-status https://glite-rb.ct.infn.it:9000/js8FEQjvTiVDaIPwQhWqMA
*************************************************************
BOOKKEEPING INFORMATION:
Status info for the Job : https://glite-rb.ct.infn.it:9000/js8FEQjvTiVDaIPwQhWqMA
Current Status: Scheduled
Status Reason: Job successfully submitted to Globus
Destination: gilda-ce-01.pd.infn.it:2119/jobmanager-lcgpbs-infinite
reached on: Fri Oct 26 13:22:35 2007
*************************************************************
Check the status until the “Current Status” changes to "Done"[taipei01@t-yaim jobs]$ edg-job-status https://glite-rb.ct.infn.it:9000/js8FEQjvTiVDaIPwQhWqMA
*************************************************************
BOOKKEEPING INFORMATION:
Status info for the Job : https://glite-rb.ct.infn.it:9000/js8FEQjvTiVDaIPwQhWqMA
Current Status: Done (Success)
Exit code: 0
Status Reason: Job terminated successfully
Destination: gilda-ce-01.pd.infn.it:2119/jobmanager-lcgpbs-infinite
reached on: Fri Oct 26 13:28:08 2007
*************************************************************
12. Download the job results from the Resource Broker to your User Interface[taipei01@t-yaim jobs]$ edg-job-get-output https://glite-rb.ct.infn.it:9000/js8FEQjvTiVDaIPwQhWqMA
Retrieving files from host: glite-rb.ct.infn.it ( for https://glite-rb.ct.infn.it:9000/js8FEQjvTiVDaIPwQhWqMA )
*********************************************************************************
JOB GET OUTPUT OUTCOME
Output sandbox files for the job:
- https://glite-rb.ct.infn.it:9000/js8FEQjvTiVDaIPwQhWqMA
have been successfully retrieved and stored in the directory:
/home/taipei01/jobOutput/taipe01_js8FEQjvTiVDaIPwQhWqMA
*********************************************************************************
The result output path was define in site-info.def the OUTPUT_STORAGE variable, for this practical, the OUTPUT_STORAGE is $HOME/jobOutput/.
so the result will put into $HOME/jobOutput/taipei01_js8FEQjvTiVDaIPwQhWqMA
Check the result:[taipei01@t-yaim jobs]$ less ~/jobOutput/camp_js8FEQjvTiVDaIPwQhWqMA/testJob.out
Test the Grid data store by your UI
13. Get SE information:[taipei01@t-yaim gLite]$ lcg-infosites --vo gilda se
Avail Space(Kb) Used Space(Kb) Type SEs
----------------------------------------------------------
32350000 210590 n.a grid-se.bio.dist.unige.it
74980000 5153576 n.a egee016.cnaf.infn.it
321010000 50293604 n.a aliserv6.ct.infn.it
50660000 7340000 n.a iceage-se-01.ct.infn.it
884662792 27744624 n.a grid005.iucc.ac.il
100120000 30 n.a fn2.hpcc.sztaki.hu
14. Set your DPNS_HOST:[taipei01@t-yaim jobs]$ export DPNS_HOST=grid-se.bio.dist.unige.it
Here we use the "grid-se.bio.dist.unige.it" for dpns host, you can feel free to select any listed one, but please remember to modify the domain name in following exercise.For example, if you use "egee016.cnaf.infn.it" for DPNS_HOST, your dpm home should be "/dpm/cnaf.infn.it/home/gilda/"
15. Create your own directory:[taipei01@t-yaim jobs]$ dpns-mkdir /dpm/bio.dist.unige.it/home/gilda/taipeixx
[taipei01@t-yaim jobs]$ dpns-ls /dpm/bio.dist.unige.it/home/gilda/
taipeixx
generated
sh_copy
16. Generate a file:[taipei01@t-yaim jobs]$ echo 'hello!! just a test'> data
17. Put file to SE:[taipei01@t-yaim jobs]$ globus-url-copy file://$HOME/jobs/data gsiftp://grid-se.bio.dist.unige.it/dpm/bio.dist.unige.it/home/gilda/taipeixx/
18. Check file:[taipei01@t-yaim jobs]$ dpns-ls /dpm/bio.dist.unige.it/home/gilda/taipeixx/
data
19. Delete local file:[taipei01@t-yaim jobs]$ rm data
20. Download the file from the Storage Element back to the User Interface:[taipei01@t-yaim jobs]$ globus-url-copy gsiftp://grid-se.bio.dist.unige.it/dpm/bio.dist.unige.it/home/gilda/camp/data file://$HOME/jobs/
[taipei01@t-yaim jobs]$ cat data
hello!! just a test
21. Finish