[Contents] [Prev] [Next] [End]
This chapter describes the system resources LSF keeps track of and how you use LSF resource specifications. Topics covered in this chapter are:
A computer may be thought of as a collection of resources used to execute programs. Different applications often require different resources. For example, a number crunching program may take a lot of CPU power, whereas a large spreadsheet may need a lot of memory to run well. If the machines are of different types, certain applications may run on some of them, but not on others.
To run applications quickly and correctly, the LSF system needs to know their resource requirements. Resource requirements are strings that contain resource names and operators. There are several types of resource. Load indices measure dynamic resource availability such as a host's CPU load or available swap space. Static resources represent unchanging information such as the number of CPUs a host has, the host type, and the maximum available swap space.
Resources can have numeric, string, or boolean values. Memory, swap space, and CPU load are examples of resources with numeric values. Host type and model are string valued resources. Boolean valued resources are static resources assigned by the LSF administrator to represent features available on particular hosts such as the availability of host-locked software licences or special hardware.
Resource names may be any string of up to 29 characters, excluding the characters reserved as operators. Resource names are case sensitive. The lsinfo command lists the resources available in your cluster.
Load indices measure the availability of resources on hosts in the LSF cluster. Load indices built into the LIM are updated at fixed time intervals. External load indices are updated when new values are received from the external load collection program, ELIM, configured by the LSF administrator. Table 1 summarizes the load indices collected by the LIM.
Index |
Measures |
Units |
Direction |
Averaged over |
Update Interval |
| status | host status | string | 15 seconds | ||
| r15s | run queue length | processes | increasing | 15 seconds | 15 seconds |
| r1m | run queue length | processes | increasing | 1 minute | 15 seconds |
| r15m | run queue length | processes | increasing | 15 minutes | 15 seconds |
| ut | CPU utilisation | (per cent) | increasing | 1 minute | 15 seconds |
| pg | paging activity | pages in + pages out per second | increasing | 1 minute | 15 seconds |
| ls | logins | users | increasing | N/A | 30 seconds |
| it | idle time | minutes | decreasing | N/A | 30 seconds |
| swp | available swap space | megabytes | decreasing | N/A | 15 seconds |
| mem | available memory | megabytes | decreasing | N/A | 15 seconds |
| tmp | available space in /tmp filesystem | megabytes | decreasing | N/A | 120 seconds |
| io | disk I/O (shown by lsload -l) | kilobytes per second | increasing | 1 minute | 15 seconds |
| name | external load index configured by LSF administrator | site defined | |||
The status index is a string indicating the current status of the host. This status applies to the LIM and RES. The possible values for status are:
If the LIM is available but the RES server is not responding, status begins with a '-'.
Here is an example of the output from lsload:
% lsload HOST_NAME status r15s r1m r15m ut pg ls it tmp swp mem hostN ok 0.0 0.0 0.1 1% 0.0 1 224 43M 67M 3M hostK -ok 0.0 0.0 0.0 3% 0.0 3 0 38M 40M 7M hostG busy *6.2 6.9 9.5 85% 1.1 30 0 5M 400M 385M hostF busy 0.1 0.1 0.3 7% *17 6 0 9M 23M 28M hostV unavail
The r15s, r1m, and r15m load indices are the 15-second, 1-minute and 15-minute average CPU run queue lengths. This is the average number of processes ready to use the CPU during the given interval.
Note
Run queue length indices are not necessarily the same as the load averages printed by the uptime command; uptime load averages on some platforms also include processes that are in short term wait states (such as paging or disk I/O).
On multiprocessor systems more than one process can execute at a time. LSF scales the run queue value on multiprocessor systems to make the CPU load of uniprocessors and multiprocessors comparable. The scaled value is called the effective run queue length. The -E option shows the effective run queue length.
LSF also adjusts the CPU run queue based on the relative speeds of the processors (the CPU factor). The normalized run queue length is adjusted for both number of processors and CPU speed. The host with the lowest normalized run queue length will run a CPU intensive job the fastest. The -N option shows the normalized CPU run queue lengths.
The ut index measures CPU utilization, which is the percentage of time spent running system and user code. A host with no process running has a ut value of 0 percent; a host on which the CPU is completely busy has a ut of 100 percent.
The pg index gives the virtual memory paging rate in pages per second. This index is closely tied to the amount of available RAM memory and the total size of the processes running on a host; if there is not enough RAM to satisfy all processes, the paging rate will be high. Paging rate is a good measure of how a machine will respond to interactive use; a machine that is paging heavily feels very slow.
The paging rate is reported in units of pages rather than kilobytes, because the relationship between interactive response and paging rate is largely independent of the page size.
The ls index gives the number of users logged in. Each user is counted once, no matter how many times they have logged into the host. The ls index is not supported on NT.
The it index is the interactive idle time of the host, in minutes. Idle time is measured from the last input or output on a directly attached terminal or a network pseudo-terminal supporting a login session. The it index is not supported on NT.
Note
This does not include activity directly through the X server such as CAD applications or emacs windows, except on SunOS 4, Solaris, and HP-UX systems.
The tmp index is the space available on the file system that contains the /tmp directory in megabytes.
The swp index gives the currently available swap space in megabytes. This represents the largest process that can be started on the host.
The mem index is an estimate of the real memory currently available to user processes. This represents the approximate size of the largest process that could be started on a host without causing the host to start paging. This is an approximation because the virtual memory behaviour of UNIX systems varies from system to system and is hard to predict.
The io index is only displayed with the -l option to lsload. This index measures I/O throughput to disks attached directly to this host, in kilobytes per second. It does not include I/O to disks that are NFS mounted from other hosts.
External load indices are defined by the LSF administrator. The lsinfo command lists the external load indices and the lsload -l command displays the values of all load indices. If you need more information about the external load indices defined at your site, contact your LSF administrator.
Static resources represent host information that does not change over time such as the maximum RAM available to user processes and the number of processors in a machine. Most static resources are determined by the LIM at start-up time. Table 2 lists the static resources reported by the LIM.
Index |
Measures |
Units |
Determined by |
| type | host type | string | configuration |
| model | host model | string | configuration |
| hname | host name | string | configuration |
| cpuf | CPU factor | relative | configuration |
| server | host can run remote jobs | boolean | configuration |
| rexpri | execution priority | nice(2) argument | configuration |
| ncpus | number of processors | processors | LIM |
| ndisks | number of local disks | disks | LIM |
| maxmem | maximum RAM memory available to users | megabytes | LIM |
| maxswp | maximum available swap space | megabytes | LIM |
| maxtmp | maximum available space in temporary file system | megabytes | LIM |
The type and model static resources are strings specifying the host type and model.
The CPU factor is the speed of the host's CPU relative to other hosts in the cluster. If one processor is twice the speed of another, its CPU factor should be twice as large. The CPU factors are defined by the LSF administrator. For multiprocessor hosts the CPU factor is the speed of a single processor; LSF automatically scales the host CPU load to account for additional processors.
The server static resource is boolean; its value is 1 if the host is configured to execute tasks from other hosts, and 0 if the host is a client.
Static resources can be used to select appropriate hosts for particular jobs based on binary architecture, relative CPU speed, and system configuration.
Boolean resource names are used to describe features that may be available only on some machines in a cluster. For example:
Any characteristics or attributes of certain hosts that can be useful in selecting hosts for remote jobs may be configured as boolean resources. Specifying a boolean resource in the resource requirements of a job limits the set of computers that can execute the job. Table 3 lists some examples of boolean resources.
Resource Name |
Describes |
Meaning of Example Name |
| cs | role in cluster | compute server |
| fs | role in cluster | file server |
| solaris | operating system | Solaris operating system |
| frame | available software | FrameMaker license |
Dedicated resources are a special form of boolean resource used to identify hosts that are dedicated to running a particular application or class of applications. For example, a software group might have a compute server with very fast local disk and a special compiler license. This compute server is intended to run compilation jobs only.
If a host is dedicated to a resource, the LIM only selects that host if the application requires the dedicated resource. For example, the LSF administrator can add an f77 resource to the snacks cluster, and dedicate host apple to that resource. The LIM will only send jobs to apple if they require the f77 resource.
The lsinfo command lists all the resources configured in the LSF cluster. See 'Displaying Available Resources' for an example of the lsinfo command. The lsinfo -l option gives more detail about each index:
% lsinfo -l r1m RESOURCE_NAME: r1m DESCRIPTION: 1-minute CPU run queue length (alias: cpu) TYPE ORDER INTERVAL BUILTIN DYNAMIC Numeric Inc 15 Yes Yes
A resource requirement string describes the resources a job needs. LSF uses resource requirements to select hosts for remote execution and job execution.
A resource requirement string is divided into four sections:
The selection section specifies the criteria for selecting hosts from the system. The ordering section indicates how the hosts that meet the selection criteria should be sorted. The resource usage section specifies the expected resource consumption of the task. The job spanning section indicates if a (parallel) batch job should span across multiple hosts.
The syntax of a resource requirement expression is:
select[selectstring] order[orderstring] rusage[usagestring] span [spanstring]
The section names are select, order, rusage, and span. The syntax for each of selectstring, orderstring, usagestring, and spanstring is defined below.
Note
The square brackets are an essential part of the resource requirement expression.
Depending on the command, one or more of these sections may apply. The lshosts command only selects hosts, but does not order them. The lsload command selects and orders hosts, while lsplace uses the information in select, order, and rusage sections to select an appropriate host for a task. The lsloadadj command uses the rusage section to determine how the load information should be adjusted on a host, while bsub uses all four sections. Sections that do not apply for a command are ignored.
If no section name is given, then the entire string is treated as a selection string. The select keyword may be omitted if the selection string is the first string in the resource requirement.
The selection string specifies the characteristics a host must have to match the resource requirement. It is a logical expression built from a set of resource names. The lsinfo command lists all the resource names and their descriptions. The resource names swap, idle, logins, and cpu are accepted as aliases for swp, it, ls, and r1m respectively.
The selection string can combine resource names with logical and arithmetic operators. Non-zero arithmetic values are treated as logical TRUE, and zero as logical FALSE. Boolean resources (for example, server to denote LSF server hosts) have a value of one if they are defined for a host, and zero otherwise.
Table 4 shows the operators that can be used in selection strings. The operators are listed in order of decreasing precedence.
Syntax |
Meaning |
| -a !a |
Negative of a Logical not: 1 if a==0, 0 otherwise |
| a * b a / b |
Multiply a and b Divide a by b |
| a + b a - b |
Add a and b Subtract b from a |
| a > b a < b a >= b a <= b |
1 if a is greater than
b, 0 otherwise 1 if a is less than b, 0 otherwise 1 if a is greater than or equal to b, 0 otherwise 1 if a is less than or equal to b, 0 otherwise |
| a == b a != b |
1 if a is equal to b,
0 otherwise 1 if a is not equal to b, 0 otherwise |
| a && b | Logical AND: 1 if both a and b are non-zero, 0 otherwise |
| a || b | Logical OR: 1 if either a or b is non-zero, 0 otherwise |
The selection string is evaluated for each host; if the result is non-zero, then that host is selected. For example:
select[(swp > 50 && mem >= 10 && type == MIPS) || (swp > 35 && type == ALPHA)]
select[((2*r15s + 3*r1m + r15m) / 6 < 1.0) && !fs && (cpuf > 4.0)]
For the string resources type and model, the special value any selects any value and local selects the same value as that of the local host. For example, type==local selects hosts of the same type as the host submitting the job. If a job can run on any type of host, include type==any in the resource requirements. If no type is specified, the default depends on the command. For lshosts, lsload, lsmon and lslogin the default is type==any. For lsplace, lsrun, lsgrun, lsmake and bsub the default is type==local unless a model or boolean resource is specified, in which case it is type==any.
Versions prior to LSF 2.0 used a different syntax for resource requirement strings. LSF accepts resource requirements that use the old syntax. The old syntax combines resource names using ':' (logical AND) and '-' (logical NOT). The single equal sign '=' means that the load level must be at least as good as requested; for example, the requirement mem=20 means that the host must have 20 megabytes or more of memory available. The following resource requirement string uses the old syntax:
r15m=1.5:mem=20:swp=12:-aix
This selection string is equivalent to this selection string in the new syntax:
r15m<=1.5 && mem>=20 && swp>=12 && !aix
A selection string in the old syntax is of the form
[-]res[=value][:[-]res[=value]]...
Each res is a resource name. A value may only be specified for numeric or string resources. The semantics of '=' depend on the type of resource. For string resources, '=' is equivalent to '=='. For numeric resources, '=' is equivalent to '<=' for increasing load indices (the value of the index is higher when the load is higher) or '>=' for decreasing load indices (the value is lower when the load is higher).
A '-' may only be used in front of a boolean resource name or in isolation. In isolation '-' is equivalent to type==any.
Other examples of a selection string in the old syntax are:
-:swp=12
type=MIPS:maxmem=20
The order string allows the selected hosts to be sorted according to the values of resources. The syntax of the order string is
[-]res[:[-]res]...
Each res must be a dynamic load index; that is, one of the indices r15s, r1m, r15m, ut, pg, io, ls, it, tmp, swp, mem, or an external load index defined by the LSF administrator. For example, swp:r1m:tmp:r15s is a valid order string.
Note
The values of r15s, r1m, and r15m used for sorting are the normalized load indices returned by lsload -N (see 'Load Indices').
The order string is used for host sorting and selection. The ordering begins with the rightmost index in the order string and proceeds from right to left. The hosts are sorted into order based on each load index, and if more hosts are available than were requested, the LIM drops the least desirable hosts according to that index. The remaining hosts are then sorted by the next index.
After the hosts are sorted by the leftmost index in the order string, the final phase of sorting orders the hosts according to their status, with hosts that are currently not available for load sharing (that is, not in the ok state) listed at the end.
Because the hosts are resorted for each load index, only the host status and the leftmost index in the order string actually affect the order in which hosts are listed. The other indices are only used to drop undesirable hosts from the list.
When sorting is done on each index, the direction in which the hosts are sorted (increasing vs decreasing values) is determined by the default order returned by lsinfo for that index. This direction is chosen such that after sorting, the hosts are ordered from best to worst on that index.
When an index name is preceded by a minus sign '-', the sorting order is reversed so that hosts are ordered from worst to best on that index.
The default sorting order is r1m:pg (except for lslogin(1): ls:r1m).
This string defines the expected resource usage of the task. It is used to specify resource reservations for LSF Batch jobs, or for mapping tasks onto hosts and adjusting the load when running interactive jobs.
For LSF Batch jobs, the resource usage section is used along with the queue configuration parameter RES_REQ (see 'Resource Requirement Parameters'). External indices are also considered in the resource usage string.
The syntax of the resource usage string is
res=value[:res=value]...[:res=value][:duration=value][:decay=value]
The res parameter can be any load index. The value parameter is the initial reserved amount. If res or value is not given, the default is not to reserve that resource.
The duration parameter is the time period within which the specified resources should be reserved. It is specified in minutes by default. If the value is followed by the letter 'h', it is specified in hours. For example, 'duration=30' and 'duration=2h' specify a duration of 30 minutes and two hours respectively. If duration is not specified, the default is to reserve the total amount for the lifetime of the job.
The decay parameter indicates how the reserved amount should decrease over the duration. A value of 1, 'decay=1', indicates that system should linearly decrease the amount reserved over the duration. The default decay value is 0, which causes the total amount to be reserved for the entire duration. Values other than 0 or 1 are unsupported. If duration is not specified decay is ignored.
rusage[mem=50:duration=100:decay=1
The above example indicates that 50MB memory should be reserved for the job. As the job runs, the amount reserved will decrease at approximately 0.5 megabytes per minute until the 100 minutes is up.
When the job is interactive (see lsplace(1) and lsloadadj(1)), the external indices are not considered in the resource usage string. The LIM uses resource usage to determine the placement of interactive jobs. Interactive placement is limited in comparison to LSF Batch in that the LIM does not track when an application finishes. Resource usage requests are used to temporarily increase the load so that a host is not overloaded.
The syntax of the resource usage string is
res[=value]:res[=value]: ... :res[=value]
The res is one of the resources whose value is returned by the lsload command.
rusage[r1m=0.5:mem=20:swp=40]
The above example indicates that the task is expected to increase the 1-minute run queue length by 0.5, consume 20 Mbytes of memory and 40 Mbytes of swap space.
If no value is specified, the task is assumed to be intensive in using that resource. In this case no more than one task will be assigned to a host regardless of how many CPUs it has.
The default resource usage for a task is r15s=1.0:r1m=1.0:r15m=1.0. This indicates a CPU intensive task which consumes few other resources.
This string specifies the locality of a parallel job (see 'Specifying Locality'). Currently only the following two cases are supported:
span[hosts=1]
This indicates that all the processors allocated to this job must be on the same host.
span[ptile=1]
This indicates that only one processor on each host should be allocated to the job regardless of how many processors the host possesses.
If span is omitted, LSF Batch will allocate the required processors for the job from the available set of processors.
Some applications require resources other than the default. LSF can store resource requirements for specific applications so that the application automatically runs with the correct resources. For frequently used commands and software packages, the LSF administrator can set up cluster--wide resource requirements available to all users in the cluster. See the LSF Administrator's Guide for more information.
You may have applications that you need to control yourself. Perhaps your administrator did not set them up for load sharing for all users, or you need a non-standard setup. You can use LSF commands to find out resource names available in your system, and tell LSF about the needs of your applications. LSF stores the resource requirements for you from then on.
The resource requirements of applications are stored in task list files. A task is a UNIX or NT command or a user-created executable program; the terms application or job are also used to refer to tasks.
A task list file contains two lists of task names. The remote task list contains tasks that are eligible for remote execution and the resource requirements of each task. The local task list contains tasks that should be executed on the local machine, usually because the tasks depend on local information (such as ps and hostname).
There are three sets of task list files: the system-wide default file lsf.task, the cluster default file lsf.task.cluster, and the user file $HOME/.lsftask. The system and cluster default files apply to all users. The user file specifies the tasks to be added to or removed from the system lists for your jobs. Resource requirements specified in your user file override those in the system lists.
For an interactive job, its resource requirements are sent to the LIM. The LIM selects the best machine based on the resource requirements. For a batch job, its resource requirements are sent to mbatchd.
The lsrtasks and lsltasks commands inspect and modify the remote and local task lists, respectively. Invoking these commands with no arguments displays the current lists:
The lsrtasks command displays the resource requirements of tasks in the remote list, separated from the task name by '/'.
% lsrtasks CC fgrep sort ar g++/cpu spell as gcc/cpu split c89/cpu gprof tbl cc/cpu grap/- tpc compress/-:cpu:mem grep/- troff/cpu compressdir/cpu:mem ispell ttc deroff latex/cpu uncompress/cpu:mem diff lint uuencode/cpu ditroff make/cpu wc dvi2ps nroff/cpu what egrep od zcat/cpu:mem eqn pic zmore/cpu f77/cpu psroff
You can specify resource requirements when tasks are added to the user's remote task list. If the task to be added is already in the list, its resource requirements are replaced.
% lsrtasks + "f77/order[cpu]" "myprog/mem>=8 && cpu" % lsrtasks - troff
These commands add myprog to the remote tasks list with its resource requirements, change the resource requirements for f77, and remove troff from the list:
% lsrtasks CC fgrep psroff ar g++/cpu sort as gcc/cpu spell c89/cpu gprof split cc/cpu grap/- tbl compress/-:cpu:mem grep/- tpc compressdir/cpu:mem ispell ttc deroff latex/cpu uncompress/cpu:mem diff lint uuencode/cpu ditroff make/cpu wc dvi2ps myprog/mem>=8 && cpu what egrep nroff/cpu zcat/cpu:mem eqn od zmore/cpu f77/order[cpu] pic
lsltasks command can be used in a similar way.
Most LSF commands accept a -R resreq argument to specify resource requirements. The exact behaviour depends on the command; for example, specifying a resource requirement for the lsload command displays the load levels for all hosts that have the requested resources.
Specifying resource requirements for the lsrun command causes LSF to select the best host out of the set of hosts that have the requested resources. The -R resreq option overrides any resource requirements specified in the task lists. For an example of the lsrun command with the -R resreq option see 'Running Remote Jobs with lsrun'.
[Contents] [Prev] [Next] [End]
doc@platform.comCopyright © 1994-1997 Platform Computing Corporation.
All rights reserved.