Microway Working Note 22

Condor is a software system that creates a HighThroughput Computing (HTC) environment. It effectively utilizes the computing power of workstations that communicate over a network. Condor can manage a dedicated cluster of workstations. Its power comes from the ability to effectively harness nondedicated, preexisting resources under distributed ownership. A user submits the job to Condor. Condor finds an available machine on the network and begins running the job on that machine. Condor has the capability to detect that a machine running a Condor job is no longer available (perhaps because the owner of the machine came back from lunch and started typing on the keyboard). It can checkpoint the job and move (migrate) the jobs to a different machine which would otherwise be idle. Condor continues job on the new machine from precisely where it left off. In those cases where Condor can checkpoint and migrate a job, Condor makes it easy to maximize the number of machines which can run a job. In this case, there is no requirement for machines to share file systems (for example, with NFS or AFS), so that machines across an entire enterprise can run a job, including machines in different administrative domains. Condor can be a real time saver when a job must be run many (hundreds of) different times, perhaps with hundreds of different data sets. With one command, all of the hundreds of jobs are submitted to Condor. Depending upon the number of machines in the Condor pool, dozens or even hundreds of otherwise idle machines can be running the job at any given moment. Condor does not require an account (login) on machines where it runs a job. Condor can do this because of its remote system call technology, which traps library calls for such operations as reading or writing from disk files. The calls are transmitted over the network to be performed on the machine where the job was submitted. Condor provides powerful resource management by matchmaking resource owners with resource consumers. This is the cornerstone of a successful HTC environment. Other compute cluster resource management systems attach properties to the job queues themselves, resulting in user confusion over which queue to use as well as administrative hassle in con stantly adding and editing queue properties to satisfy user demands. Condor implements ClassAds, a clean design that simplifies the user's submission of jobs. ClassAds work in a fashion similar to the newspaper classified advertising wantads. All machines in the Condor pool advertise their resource properties, both static and dynamic, such as available RAM memory, CPU type, CPU speed, virtual memory size, physical location, and current load average, in a resource offer ad. A user specifies a resource request ad when submitting a job. The request defines both the required and a desired set of properties of the resource to run the job. Condor acts as a broker by matching and ranking resource offer ads with resource request ads, making certain that all requirements in both ads are satisfied. During this matchmaking process, Condor also considers several layers of priority values: the priority the user assigned to the resource request ad, the priority of the user which submitted the ad, and desire of machines in the pool to accept certain types of ads over others.

Exceptional Features:

Checkpoint and Migration: Where programs can be linked with Condor libaries, users of Condor may be assured that their jobs will eventually complete, even in the ever changing environment that Condor utilizes. As a machine running a job submitted to Condor becomes unavailable, the job can be checkpointed. The job may continue after migrating to another machine. Condor's periodic checkpoint feature periodically checkpoints a job even in lieu of migration in order to safeguard the accumulated computation time on a job from being lost in the event of a system failure such as the machine being shutdown or a crash.
Remote System Calls: Despite running jobs on remote machines, the Condor standard universe execution mode preserves the local execution environment via remote system calls. Users do not have to worry about making data files available to remote worksta tions or even obtaining a login account on remote workstations before Condor executes their programs there. The program behaves under Condor as if it were running as the user that submitted the job on the workstation where it was originally submitted, no matter on which machine it really ends up executing on.
No Changes Necessary to User's Source Code: No special programming is required to use Condor. Condor is able to run noninteractive programs. The checkpoint and migration of programs by Condor is transparent and automatic, as is the use of remote system calls. If these facilities are desired, the user only relinks the program. The code is neither recompiled nor changed. Pools of Machines can be Hooked Together. Flocking is a feature of Condor that al lows jobs submitted within a first pool of Condor machines to execute on a second pool. The mechanism is flexible, following requests from the job submission, while al lowing the second pool, or a subset of machines within the second pool to set policies over the conditions under which jobs are executed.
Jobs can be Ordered: The ordering of job execution required by dependencies among jobs in a set is easily handled. The set of jobs is specified using a directed acyclic graph, where each job is a node in the graph. Jobs are submitted to Condor following the dependencies given by the graph.
Condor Enables Grid Computing: As grid computing becomes a reality, Condor is already there. The technique of glidein allows jobs submitted to Condor to be executed on grid machines in various locations worldwide. As the details of grid computing evolve, so does Condor's ability, starting with Globuscontrolled resources.
Sensitive to the Desires of Machine Owners: The owner of a machine has complete priority over the use of the machine. An owner is generally happy to let others compute on the machine while it is idle, but wants it back promptly upon returning. The owner does not want to take special action to regain control. Condor handles this automatically.
ClassAds: The ClassAd mechanism in Condor provides an extremely flexible, expressive framework for matchmaking resource requests with resource o#ers. Users can easily request both job requirements and job desires. For example, a user can require that a job run on a machine with 64 Mbytes of RAM, but state a preference for 128 Mbytes, if available. A workstation owner can state a preference that the workstation runs jobs from a specified set of users. The owner can also require that there be no interactive workstation activity detectable at certain hours before Condor could start a job. Job requirements/preferences and resource availability constraints can be described in terms of powerful expressions, resulting in Condor's adaptation to nearly any desired policy.

Current Limitations:

Limitations on Jobs which can be Checkpointed: Although Condor can schedule and run any type of process, Condor does have some limitations on jobs that it can transparently checkpoint and migrate:

Multiprocess jobs are not allowed. This includes system calls such as fork(), exec(), and system().
Interprocess communication is not allowed This includes pipes, semaphores, and shared memory. (Refer to Microway Application Note 19 for more information on Interprocess Communications. This is why Condor does not offer an MPI universe though it still supports an intelligent PVM one. This is discussed in more detail later. Note however that MPI jobs can be run if a dedicated job is submitted. In this case Condor needs to be configured for a specific MPI environment. This is discussed in more detail later.)
Network communication must be brief. A job may make network connections using system calls such as socket(), but a network connection left open for long periods will delay checkpointing and migration.
Sending or receiving the SIGUSR2 or SIGTSTP signals is not allowed. Condor reserves these signals for its own use. Sending or receiving all other signals is allowed.
Alarms, timers, and sleeping are not allowed. This includes system calls such as alarm(), getitimer(), and sleep().
Multiple kernellevel threads are not allowed. However, multiple userlevel threads are allowed.
Memory mapped files are not allowed. This includes system calls such as mmap() and munmap().
File locks are allowed, but not retained between checkpoints.
All files must be opened readonly or writeonly. A file opened for both reading and writing will cause trouble if a job must be rolled back to an old checkpoint image. For compatibility reasons, a file opened for both reading and writing will result in a warning but not an error.
A fair amount of disk space must be available on the submitting machine for storing a job's checkpoint images. A checkpoint image is approximately equal to the virtual memory consumed by a job while it runs. If disk space is short, a special checkpoint server can be designated for storing all the checkpoint images for a pool.
On Digital Unix (OSF/1), HPUX, and Linux, your job must be statically linked. Dynamic linking is allowed on all other platforms.

Note: these limitations only apply to jobs which Condor has been asked to transparently checkpoint. If job checkpointing is not desired, the limitations above do not apply.

Security Implications: Condor does a significant amount of work to prevent security hazards, but loopholes are known to exist. Condor can be instructed to run user programs only as the UNIX user nobody, a user login which traditionally has very restricted access. But even with access solely as user nobody, a su#ciently malicious individual could do such things as fill up /tmp (which is world writable) and/or gain read access to world readable files. Furthermore, where the security of machines in the pool is a high concern, only machines where the UNIX user root on that machine can be trusted should be admitted into the pool. Condor provides the administrator with IPbased security mechanisms to enforce this.

Jobs Need to be Relinked to get Checkpointing and Remote System Calls: Although typically no source code changes are required, Condor requires that the jobs be relinked with the Condor libraries to take advantage of checkpointing and remote system calls. This often precludes commercial software binaries from taking advantage of these services because commercial packages rarely make their object code available. Condor's other services are still available for these commercial packages.

Condor Universe

A universe in Condor defines an execution environment. Condor Version 6.3.1 supports four different universes for user jobs:

Standard
Vanilla
PVM
Globus

Standard Universe: In the standard universe, Condor provides checkpointing and remote system calls. These features make a job more reliable and allow it uniform access to resources from anywhere in the pool. To prepare a program as a standard universe job, it must be relinked with condor compile. Most programs can be prepared as a standard universe job, but there are a few restrictions. Condor checkpoints a job at regular intervals. A checkpoint image is essentially a snap shot of the current state of a job. If a job must be migrated from one machine to another, Condor makes a checkpoint image, copies the image to the new machine, and restarts the job continuing the job from where it left off. If a machine should crash or fail while it is running a job, Condor can restart the job on a new machine using the most recent check point image. In this way, jobs can run for months or years even in the face of occasional computer failures. Remote system calls make a job perceive that it is executing on its home machine, even though the job may execute on many different machines over its lifetime. When a job runs on a remote machine, a second process, called a condor shadow runs on the machine where the job was submitted. When the job attempts a system call, the condor shadow performs the system call instead and sends the results to the remote machine. For example, if a job attempts to open a file that is stored on the submitting machine, the condor shadow will find the file, and send the data to the machine where the job is running. To convert your program into a standard universe job, you must use condor compile to relink it with the Condor libraries. Put condor compile in front of your usual link command. (Configuring Condor to use condor_compile will be discussed shortly.) You do not need to modify the program's source code, but you do need access to the unlinked object files. A commercial program that is packaged as a single executable file cannot be converted into a standard universe job.

Vanilla Universe: The vanilla universe in Condor is intended for programs which cannot be successfully relinked. Shell scripts are another case where the vanilla universe is useful. Unfortunately, jobs run under the vanilla universe cannot checkpoint or use remote system calls. This has unfortunate consequences for a job that is partially completed when the remote machine running a job must be returned to its owner. Condor has only two choices. It can suspend the job, hoping to complete it at a later time, or it can give up and restart the job from the beginning on another machine in the pool. Under Unix, jobs submitted as vanilla universe jobs rely on an external mechanism for accessing data files, such as NFS or AFS. The job must be able to access the data files from any machine on which it could potentially run. Condor deals with this restriction imposed by the vanilla universe by using the FileSystemDomain and UidDomain machine ClassAd attributes. These attributes reflect the reality of the pool's disk mounting structure. For a large pool spanning multiple UidDomain and/or FileSystemDomains, the job must specify its requirements to use the correct UidDomain and/or FileSystemDomains. This mechanism is not required under Windows NT/200. The vanilla universe does not require a shared file system due to the Condor File Transfer mechanism. More details about Condor NT/2000 will be considered shortly.

PVM: The PVM universe allows programs written for the Parallel Virtual Machine interface to be used within the opportunistic Condor environment. Applications that use PVM (Parallel Virtual Machine) may use Condor. PVM offers a set of message passing primitives for use in C and C++ language programs. The primitives, to gether with the PVM environment allow parallelism at the program level. Multiple processes may run on multiple machines, while communicating with each other. CondorPVM provides a framework to run PVM applications in Condor's opportunistic environment. Where PVM needs dedicated machines to run PVM applications, Condor does not. Condor can be used to dynamically construct PVM virtual machines from a Condor pool of machines. In CondorPVM, Condor acts as the resource manager for the PVM daemon. Whenever a PVM program asks for nodes (machines), the request is forwarded to Condor. Condor finds a machine in the Condor pool using usual mechanisms, and adds it to the virtual machine. If a machine needs to leave the pool, the PVM program is notified by normal PVM mechanisms. The PVM module is not installed by default. We do not consider the installation of the PVM module here. There are several different parallel programming paradigms. One of the more common is the masterworker (or pool of tasks) arrangement. In a masterworker program model, one node acts as the controlling master for the parallel application and sends pieces of work out to worker nodes. The worker node does some computation, and it sends the result back to the master node. The master has a pool of work that needs to be done, so it assigns the next piece of work out to the next worker that becomes available. CondorPVM is designed to run PVM applications which follow the masterworker paradigm. Condor runs the master application on the machine where the job was sub mitted and will not preempt it. Workers are pulled in from the Condor pool as they become available. Not all parallel programming paradigms lend themselves to Condor's opportunistic en vironment. In such an environment, any of the nodes could be preempted and disappear at any moment. The masterworker model does work well in this environment. The master keeps track of which piece of work it sends to each worker. The master node is informed of the addition and disappearance of nodes. If the master node is informed that a worker node has disappeared, the master places the unfinished work it had assigned to the disappearing node back into the pool of tasks. This work is sent again to the next available worker node. If the master notices that the number of workers has dropped below an acceptable level, it requests more workers (using pvm addhosts()). Alternatively, the master requests a replacement node every time it is notified that a worker has gone away. The benefit of this paradigm is that the number of workers is not important and changes in the size of the virtual machine are easily handled. A tool called MW has been developed to assist in the development of masterworker style applications for distributed, opportunistic environments like Condor. MW provides a C++ API which hides the complexities of managing a masterworker CondorPVM application. It is better to consider modifying your PVM application to use MW instead of developing your own dynamic PVM master from scratch. CondorPVM does not define a new API (application program interface); programs use the existing resource management PVM calls such as pvm addhosts() and pvm notify(). Because of this, some masterworker PVM applications are ready to run under CondorPVM with no changes at all. Regardless of using CondorPVM or not, it is good masterworker design to handle the case of a disappearing worker node, and therefore all master programs must be constructed with all the necessary fault tolerant logic. The same binary which runs under regular PVM will run under Condor, and viceversa. There is no need to relink for CondorPVM. This permits easy application development (develop your PVM application interactively with the regular PVM console, XPVM, etc.) as well as binary sharing between Condor and some dedicated MPP systems. This release of CondorPVM is based on PVM 3.4.2. The following list is a summary of the changes and new features of PVM running within the Condor environment:

Condor introduces the concept of machine class. A pool of machines is likely to con tain machines of more than one platform. Under CondorPVM, machines of di#erent architectures belong to different machine classes. With the concept machine class, Condor can be told what type of machine to allocate. Machine classes are assigned integer values, starting with 0. A machine class is specified in a submit description file when the job is submitted to Condor.
pvm addhosts(). When an application adds a host machine, it calls pvm addhosts(). The first argument to pvm addhosts() is a string that specifies the machine class. For example, to specify class 0, a pointer to the string ``0'' is the first argument. Condor finds a machine that satisfies the requirements of class 0 and adds it to the PVM virtual machine. The function pvm addhosts() does not block. It returns immediately, before hosts are added to the virtual machine. In a nondedicated environment the amount of time it takes until a machine becomes available is not bound. A program should call pvm notify() before calling pvm addhosts(). When a host is added later, the program will be notified in the usual PVM fashion (with a PvmHostAdd notification message). After receiving a PvmHostAdd notification, the PVM master can unpack the following information about the added host: an integer specifying the TID of the new host, a string specifying the name of the new host, followed by a string specifying the machine class of the new host. The PVM master can then call pvm spawn() to start a worker process on the new host, specifying this machine class as the architecture and using the appropriate executable path for this machine class. Note that the name of the host is given by the startd and may be of the form ``vmN@hostname'' on SMP machines.
pvm notify(). Under Condor, there are two additional possible notification types to the function pvm notify(). They are PvmHostSuspend and PvmHostResume. The program calls pvm notify() with a host tid and PvmHostSuspend (or PvmHostResume) as arguments, and the program will receive a notification for the event of a host being suspended. Note that a notification occurs only once for each request. As an example, a PvmHostSuspend notification request for tid 4 results in a single PvmHostSuspend message for tid 4. There will not be another PvmHostSuspend message for that tid without another notification request. The easiest way to handle this is the following: When a worker node starts up, set up a notification for PvmHostSuspend on its tid. When that node gets suspended, set up a PvmHostResume notification. When it resumes, set up a PvmHostSuspend notification. If your application uses the PvmHostSuspend and PvmHostResume notification types, you will need to modify your PVM distribution to support them as follows. First, go to your $(PVM ROOT). In include/pvm3.h, add #define PvmHostSuspend 6 /* condor suspension */ #define PvmHostResume 7 /* condor resumption */ to the list of ''pvm notify kinds''. In src/lpvmgen.c, in pvm notify(), change } else { switch (what) { case PvmHostDelete: .... to } else { switch (what) { case PvmHostSuspend: /* for condor */ case PvmHostResume: /* for condor */ case PvmHostDelete: .... And that's it. Recompile, and you're done.
pvm spawn(). If the flag in pvm spawn() is PvmTaskArch, then a machine class string should be used. If there is only one machine class in a virtual machine, ``0'' is the string for the desired architecture. Under Condor, only one PVM task spawned per node is currently allowed, due to Condor's machine load checks. Most Condor sites will suspend or vacate a job if the load on its machine is higher than a specified threshold. Having more than one PVM task per node pushes the load higher than the threshold. Also, Condor only supports starting one copy of the executable with each call to pvm spawn() (i.e., the fifth argument must always be equal to one). To spawn multiple copies of an executable in Condor, you must call pvm spawn() once for each copy. A good fault tolerant program will be able to deal with pvm spawn() failing. It happens more often in opportunistic environments like Condor than in dedicated ones.
pvm exit(). If a PVM task calls pvm catchout() during its run to catch the output of child tasks, pvm exit() will attempt to gather the output of all child tasks before returning. Due to the dynamic nature of the virtual machine in Condor, this cleanup procedure (in the PVM library and daemon) is errorprone and should be avoided. So, any PVM tasks which call pvm catchout() should be sure to call it again with a NULL argument to disable output collection before calling pvm exit().

Globus Universe: The Globus universe in Condor is intended to provide the standard Condor interface to users who wish to start Globus system jobs from Condor. Each job queued in the job submission file is translated into a Globus RSL string and used as the arguments to the globusrun program. .

NOTE: Here we consider CONDOR 6.3.1 only and the standard universe.

Condor Machine and Activity States

A Condor pool is comprised of a single machine which serves as the central manager, and an arbitrary number of other machines that have joined the pool. Conceptually, the pool is a collection of resources (machines) and resource requests (jobs). The role of Condor is to match waiting requests with available resources. Every part of Condor sends periodic updates to the central manager, the centralized repository of information about the state of the pool. Periodically, the central manager assesses the current state of the pool and tries to match pending requests with the appropriate resources. Each resource has an owner, the user who works at the machine. This person has absolute power over their own resource and Condor goes out of its way to minimize the impact on this owner caused by Condor. It is up to the resource owner to define a policy for when Condor requests will serviced and when they will be denied. Each resource request has an owner as well: the user who submitted the job. These people want Condor to provide as many CPU cycles as possible for their work. Often the interests of the resource owners are in conflict with the interests of the resource requesters. The job of the Condor administrator is to configure the Condor pool to find the happy medium that keeps both resource owners and users of resources satisfied.

The Different Roles a Machine Can Play:

Every machine in a Condor pool can serve a variety of roles. Most machines serve more than one role simultaneously. Certain roles can only be performed by single machines in your pool. The following list describes what these roles are and what resources are required on the machine that is providing that service:

Central Manager: There can be only one central manager for your pool. The machine is the collector of information, and the negotiator between resources and resource re quests. These two halves of the central manager's responsibility are performed by separate daemons, so it would be possible to have different machines providing those two services. However, normally they both live on the same machine. This machine plays a very important part in the Condor pool and should be reliable. If this machine crashes, no further matchmaking can be performed within the Condor system (although all current matches remain in e#ect until they are broken by either party involved in the match). Therefore, choose for central manager a machine that is likely to be online all the time, or at least one that will be rebooted quickly if something goes wrong. The central manager will ideally have a good network connection to all the machines in your pool, since they all send updates over the network to the central manager. All queries go to the central manager.
Execute: Any machine in your pool (including your Central Manager) can be configured for whether or not it should execute Condor jobs. Obviously, some of your machines will have to serve this function or your pool won't be very useful. Being an execute machine doesn't require many resources at all. About the only resource that might matter is disk space, since if the remote job dumps core, that file is first dumped to the local disk of the execute machine before being sent back to the submit machine for the owner of the job. However, if there isn't much disk space, Condor will simply limit the size of the core file that a remote job will drop. In general the more resources a machine has (swap space, real memory, CPU speed, etc.) the larger the resource requests it can serve. However, if there are requests that don't require many resources, any machine in your pool could serve them.
Submit: Any machine in your pool (including your Central Manager) can be configured for whether or not it should allow Condor jobs to be submitted. The resource requirements for a submit machine are actually much greater than the resource requirements for an execute machine. First of all, every job that you submit that is currently running on a remote machine generates another process on your submit machine. So, if you have lots of jobs running, you will need a fair amount of swap space and/or real memory. In addition all the checkpoint files from your jobs are stored on the local disk of the machine you submit from. Therefore, if your jobs have a large memory image and you submit a lot of them, you will need a lot of disk space to hold these files. This disk space requirement can be somewhat alleviated with a checkpoint server (described below), however the binaries of the jobs you submit are still stored on the submit machine.
Checkpoint Server: One machine in your pool can be configured as a checkpoint server. This is optional, and is not part of the standard Condor binary distribution. The checkpoint server is a centralized machine that stores all the checkpoint files for the jobs submitted in your pool. This machine should have lots of disk space and a good network connection to the rest of your pool, as the traffic can be quite heavy. We do not consider the configuration of a checkpoint server here.

The Condor Daemons:

The following list describes all the daemons and programs that could be started under Condor and what they do:

condor_master: This daemon is responsible for keeping all the rest of the Condor daemons running on each machine in your pool. It spawns the other daemons, and periodically checks to see if there are new binaries installed for any of them. If there are, the master will restart the affected daemons. In addition, if any daemon crashes, the master will send email to the Condor Administrator of your pool and restart the daemon. The condor master also supports various administrative commands that let you start, stop or reconfigure daemons remotely. The condor_master will run on every machine in your Condor pool, regardless of what functions each machine are performing.
condor_startd: This daemon represents a given resource (namely, a machine capable of running jobs) to the Condor pool. It advertises certain attributes about that resource that are used to match it with pending resource requests. The startd will run on any machine in your pool that you wish to be able to execute jobs. It is responsible for enforcing the policy that resource owners configure which determines under what conditions remote jobs will be started, suspended, resumed, vacated, or killed. When the startd is ready to execute a Condor job, it spawns the condor starter, described below.
condor_starter: This program is the entity that actually spawns the remote Condor job on a given machine. It sets up the execution environment and monitors the job once it is running. When a job completes, the starter notices this, sends back any status information to the submitting machine, and exits.
condor_schedd: This daemon represents resources requests to the Condor pool. Any machine that you wish to allow users to submit jobs from needs to have a condor schedd running. When users submit jobs, they go to the schedd, where they are stored in the job queue, which the schedd manages. Various tools to view and manipulate the job queue (such as condor_submit, condor_q, or condor_rm) all must connect to the schedd to do their work. If the schedd is down on a given machine, none of these commands will work. The schedd advertises the number of waiting jobs in its job queue and is responsible for claiming available resources to serve those requests. Once a schedd has been matched with a given resource, the schedd spawns a condor shadow (described below) to serve that particular request.
condor_shadow: This program runs on the machine where a given request was submitted and acts as the resource manager for the request. Jobs that are linked for Condor's standard universe, which perform remote system calls, do so via the condor_shadow. Any system call performed on the remote execute machine is sent over the network, back to the condor_shadow which actually performs the system call (such as file I/O) on the submit machine, and the result is sent back over the network to the remote job. In addition, the shadow is responsible for making decisions about the request (such as where checkpoint files should be stored, how certain files should be accessed, etc).
condor_collector: This daemon is responsible for collecting all the information about the status of a Condor pool. All other daemons (except the negotiator) periodically send ClassAd updates to the collector. These ClassAds contain all the information about the state of the daemons, the resources they represent or resource requests in the pool (such as jobs that have been submitted to a given schedd). The condor_status command can be used to query the collector for specific information about various parts of Condor. In addition, the Condor daemons themselves query the collector for important information, such as what address to use for sending commands to a remote machine.
condor_negotiator: This daemon is responsible for all the matchmaking within the Condor system. Periodically, the negotiator begins a negotiation cycle, where it queries the collector for the current state of all the resources in the pool. It contacts each schedd that has waiting resource requests in priority order, and tries to match avail able resources with those requests. The negotiator is responsible for enforcing user priorities in the system, where the more resources a given user has claimed, the less priority they have to acquire more resources. If a user with a better priority has jobs that are waiting to run, and resources are claimed by a user with a worse priority, the negotiator can preempt that resource and match it with the user with better priority. NOTE: A higher numerical value of the user priority in Condor translate into worse priority for that user. The best priority you can have is 0.5, the lowest numerical value, and your priority gets worse as this number grows.
condor_kbdd: This daemon is only needed on Digital Unix and IRIX. On these platforms, the condor startd cannot determine console (keyboard or mouse) activity directly from the system. The condor_kbdd connects to the X Server and periodically checks to see if there has been any activity. If there has, the kbdd sends a command to the startd. That way, the startd knows the machine owner is using the machine again and can perform whatever actions are necessary, given the policy it has been configured to enforce.
condor_ckpt server: This is the checkpoint server. It services requests to store and retrieve checkpoint files. If your pool is configured to use a checkpoint server but that machine (or the server itself is down) Condor will revert to sending the checkpoint files for a given job back to the submit machine. See Figure 1 for a graphical representation of the pool architecture.

Figure 1: Condor Pool Architecture in a graphical form.

Machine States:

A machine is assigned a state by Condor. The state depends on whether or not the machine is available to run Condor jobs, and if so, what point in the negotiations has been reached. The possible states are

Owner: The machine is being used by the machine owner, and/or is not available to run Condor jobs. When the machine first starts up, it begins in this state.
Unclaimed: The machine is available to run Condor jobs, but it is not currently doing so.
Matched: The machine is available to run jobs, and it has been matched by the negotiator with a specific schedd. That schedd just has not yet claimed this machine. In this state, the machine is unavailable for further matches.
Claimed: The machine has been claimed by a schedd.
Preempting: The machine was claimed by a schedd, but is now preempting that claim for one of the following reasons. 1. the owner of the machine came back 2. another user with higher priority has jobs waiting to run 3. another request that this resource would rather serve was found Figure 2 shows the states and the possible transitions between the states.

Figure 2: Condor Machine States.

Machine Activities:

Within some machine states, activities of the machine are defined. The state has meaning regardless of activity. Differences between activities are significant. Therefore, a ``state/activity'' pair describes a machine. The following list describes all the possible state/activity pairs.

Owner

Idle: This is the only activity for Owner state. As far as Condor is concerned the machine is Idle, since it is not doing anything for Condor. .

Unclaimed

Idle: This is the normal activity of Unclaimed machines. The machine is still Idle in that the machine owner is willing to let Condor jobs run, but Condor is not using the machine for anything.
Benchmarking: The machine is running benchmarks to determine the speed on this machine. This activity only occurs in the Unclaimed state. How often the activity occurs is determined by the RunBenchmarks expression.

Matched

Idle: When Matched, the machine is still Idle to Condor.

Claimed

Idle: In this activity, the machine has been claimed, but the schedd that claimed it has yet to activate the claim by requesting a condor starter to be spawned to service a job.
Busy: Once a condor starter has been started and the claim is active, the machine moves to the Busy activity to signify that it is doing something as far as Condor is concerned.
Suspended: If the job is suspended by Condor, the machine goes into the Suspended activity. The match between the schedd and machine has not been broken (the claim is still valid), but the job is not making any progress and Condor is no longer generating a load on the machine.

Preempting

The preempting state is used for evicting a Condor job from a given machine. When the machine enters the Preempting state, it checks the WANT VACATE expression to determine its activity.

Vacating: In the Vacating activity, the job that was running is in the process of checkpointing. As soon as the checkpoint process completes, the machine moves into either the Owner state or the Claimed state, depending on the reason for its preemption.
Killing: Killing means that the machine has requested the running job to exit the machine immediately, without checkpointing.

Figure 3 gives the overall view of all machine states and activities and shows the possible transitions from one to another within the Condor system. Each transition is labeled with a number on the diagram, and transition numbers referred to will be bold. Various expressions are used to determine when and if many of these state and activity transitions occur. Other transitions are initiated by parts of the Condor protocol (such as when the condor negotiator matches a machine with a schedd). The following describes the conditions that lead to the various state and activity transitions.

Figure 3: Condor Machine States and Activities Diagram.

State and Activity Transitions:

This part traces through all possible state and activity transitions within a machine and describes the conditions under which each one occurs. Whenever a transition occurs, Condor records when the machine entered its new activity and/or new state. These times are often used to write expressions that determine when further transitions occurred. For example, enter the Killing activity if a machine has been in the Vacating activity longer than a specified amount of time.

Owner State: When the startd is first spawned, the machine it represents enters the Owner state. The machine will remain in this state as long as the START expression locally evaluates to FALSE. If the START locally evaluates to TRUE or cannot be locally evaluated (it evaluates to UNDEFINED, transition 1 occurs and the machine enters the Unclaimed state. As long as the START expression evaluates locally to FALSE, there is no possible request in the Condor system that could match it. The machine is unavailable to Condor and stays in the Owner state. For example, if the START expression is

START = KeyboardIdle > 15 * $(MINUTE) && Owner = = "coltrane"

and if KeyboardIdle is 34 seconds, then the machine would remain in the Owner state. Owner is undefined, and anything && FALSE is FALSE. If, however, the START expression is

START = KeyboardIdle > 15 * $(MINUTE) || Owner = = "coltrane"

and KeyboardIdle is 34 seconds, then the machine leaves the Owner state and becomes Unclaimed. This is because FALSE || UNDEFINED is UNDEFINED. So, while this machine is not available to just anybody, if user coltrane has jobs submitted, the machine is willing to run them. Any other user's jobs have to wait until KeyboardIdle exceeds 15 minutes. However, since coltrane might claim this resource, but has not yet, the machine goes to the Unclaimed state. While in the Owner state, the startd polls the status of the machine every UPDATE INTERVAL to see if anything has changed that would lead it to a different state. This minimizes the impact on the Owner while the Owner is using the machine. Frequently waking up, computing load averages, checking the access times on files, computing free swap space take time, and there is nothing time critical that the startd needs to be sure to notice as soon as it happens. If the START expression evaluates to TRUE and five minutes pass before the startd notices, that's a drop in the bucket of highthroughput computing. The machine can only transition to the Unclaimed state from the Owner state. It only does so when the START expression no longer locally evaluates to FALSE. In general, if the START expression locally evaluates to FALSE at any time, the machine will either transition directly to the Owner state or to the Preempting state on its way to the Owner state, if there is a job running that needs preempting.

Unclaimed State: While in the Unclaimed state, if the START expression locally evaluates to FALSE, the machine returns to the Owner state by transition 2. When in the Unclaimed state, the RunBenchmarks expression is relevant. If RunBenchmarks evaluates to TRUE while the machine is in the Unclaimed state, then the machine will transition from the Idle activity to the Benchmarking activity (transition 3) and perform benchmarks to determine MIPS and KFLOPS. When the benchmarks complete, the machine returns to the Idle activity (transition 4). The startd automatically inserts an attribute, LastBenchmark, whenever it runs bench marks, so commonly RunBenchmarks is defined in terms of this attribute, for example:

BenchmarkTimer = (CurrentTime - LastBenchmark)

RunBenchmarks = $(BenchmarkTimer) >= (4 * $(HOUR))

Here, a macro, BenchmarkTimer is defined to help write the expression. This macro holds the time since the last benchmark, so when this time exceeds 4 hours, we run the benchmarks again. The startd keeps a weighted average of these benchmarking results to try to get the most accurate numbers possible. This is why it is desirable for the startd to run them more than once in its lifetime. NOTE: LastBenchmark is initialized to 0 before benchmarks have ever been run. So, if you want the startd to run benchmarks as soon as the machine is Unclaimed (if it hasn't done so already), include a term for LastBenchmark as in the example above. NOTE: If RunBenchmarks is defined and set to something other than FALSE, the startd will automatically run one set of benchmarks when it first starts up. To disable benchmarks, both at startup and at any time thereafter, set RunBenchmarks to FALSE or comment it out of the configuration file. From the Unclaimed state, the machine can go to two other possible states: Matched or Claimed/Idle. Once the condor_negotiator matches an Unclaimed machine with a requester at a given schedd, the negotiator sends a command to both parties, notifying them of the match. If the schedd receives that notification and initiates the claiming procedure with the machine before the negotiator's message gets to the machine, the Match state is skipped, and the machine goes directly to the Claimed/Idle state (transition 5). However, normally the machine will enter the Matched state (transition 6), even if it is only for a brief period of time.

Matched State: The Matched state is not very interesting to Condor. Noteworthy in this state is that the machine lies about its START expression while in this state and says that Requirements are false to prevent being matched again before it has been claimed. Also interesting is that the startd starts a timer to make sure it does not stay in the Matched state too long. The timer is set with the MATCH TIMEOUT configuration file macro. It is specified in seconds and defaults to 300 (5 minutes). If the schedd that was matched with this machine does not claim it within this period of time, the machine gives up, and goes back into the Owner state via transition 7. It will probably leave the Owner state right away for the Unclaimed state again and wait for another match. At any time while the machine is in the Matched state, if the START expression locally evaluates to FALSE, the machine enters the Owner state directly (transition 7). If the schedd that was matched with the machine claims it before the MATCH TIMEOUT expires, the machine goes into the Claimed/Idle state (transition 8).

Claimed State: The Claimed state is certainly the most complex state. It has the most possible activities and the most expressions that determine its next activities. In addition, the condor_checkpoint and condor_vacate commands affect the machine when it is in the Claimed state. In general, there are two sets of expressions that might take effect. They depend on the universe of the request: standard or vanilla. The standard universe expressions are the normal expressions. For example:

WANT_SUSPEND = True

WANT_VACATE = $(ActivationTimer) > 10 * $(MINUTE)

SUSPEND = $(KeyboardBusy) || $(CPUBusy)

...

The vanilla expressions have the string`` VANILLA'' appended to their names. For example:

WANT_SUSPEND_VANILLA = True

WANT_VACATE_VANILLA = True

SUSPEND_VANILLA = $(KeyboardBusy) || $(CPUBusy)

...

Without specific vanilla versions, the normal versions will be used for all jobs, including vanilla jobs. Here, the normal expressions are referenced. The difference exists for the the resource owner that might want the machine to behave differently for vanilla jobs, since they cannot checkpoint. For example, owners may want vanilla jobs to remain suspended for longer than standard jobs. While Claimed, the POLLING_INTERVAL takes effect, and the startd polls the machine much more frequently to evaluate its state. If the machine owner starts typing on the console again, it is best to notice this as soon as possible to be able to start doing whatever the machine owner wants at that point. For SMP machines, if any virtual machine is in the Claimed state, the startd polls the machine frequently. If already polling one virtual machine, it does not cost much to evaluate the state of all the virtual machines at the same time. In general, when the startd is going to take a job off a machine (usually because of activity on the machine that signifies that the owner is using the machine again), the startd will go through successive levels of getting the job out of the way. The first and least costly to the job is suspending it. This works for both standard and vanilla jobs. If suspending the job for a short while does not satisfy the machine owner (the owner is still using the machine after a specific period of time), the startd moves on to vacating the job. Vacating a job involves performing a checkpoint so that the work already completed is not lost. If even that does not satisfy the machine owner (usually because it is taking too long and the owner wants their machine back now ), the final, most drastic stage is reached: killing. Killing is a quick death to the job, without a checkpoint. For vanilla jobs, vacating and killing are equivalent, although a vanilla job can request to have a specific softkill signal sent to it at vacate time so that the job itself can perform applicationspecific checkpointing. The WANT_SUSPEND expression determines if the machine will evaluate the SUSPEND ex pression to consider entering the Suspended activity. The WANT_VACATE expression deter mines what happens when the machine enters the Preempting state. It will go to the Vacating activity or directly to Killing. If one or both of these expressions evaluates to FALSE, the machine will skip that stage of getting rid of the job and proceed directly to the more drastic stages. When the machine first enters the Claimed state, it goes to the Idle activity. From there, it has two options. It can enter the Preempting state via transition 9 (if a condor vacate arrives, or if the START expression locally evaluates to FALSE), or it can enter the Busy activity (transition 10) if the schedd that has claimed the machine decides to activate the claim and start a job. From Claimed/Busy, the machine can transition to three other state/activity pairs. The startd evaluates the WANT_SUSPEND expression to decide which other expressions to evaluate. If WANT_SUSPEND is TRUE, then the startd evaluates the SUSPEND expression. If SUSPEND is FALSE, then the startd will evaluate the PREEMPT expression and skip the Suspended activity entirely. By transition, the possible state/activity destinations from Claimed/Busy:

Claimed/Idle: If the starter that is serving a given job exits (for example because the jobs completes), the machine will go to Claimed/Idle (transition 11).
Preempting: If WANT_SUSPEND is FALSE and the PREEMPT expression is TRUE, the machine enters the Preempting state (transition 12). The other reason the machine would go from Claimed/Busy to Preempting is if the condor_negotiator matched the machine with a ``better'' match. This better match could either be from the machine's perspec tive using the RANK Expression above, or it could be from the negotiator's perspective due to a job with a higher user priority. In this case, WANT_VACATE is assumed to be TRUE, and the machine transitions to Preempting/Vacating.
Claimed/Suspended: If both the WANT_SUSPEND and SUSPEND expressions evaluate to TRUE, the machine suspends the job (transition 13). If a condor_checkpoint command arrives, or the PeriodicCheckpoint expression evaluates to TRUE, there is no state change. The startd has no way of knowing when this process completes, so periodic checkpointing can not be another state. Periodic checkpointing remains in the Claimed/Busy state and appears as a running job. From the Claimed/Suspended state, the following transitions may occur:
Claimed/Busy: If the CONTINUE expression evaluates to TRUE, the machine resumes the job and enters the Claimed/Busy state (transition 14).
Preempting: If the PREEMPT expression is TRUE, the machine will enter the Preempting state (transition 15).

Preempting State: The Preempting state is less complex than the Claimed state. There are two activities. Depending on the value of WANT_VACATE, a machine will be in the Vacating activity (if TRUE) or the Killing activity (if FALSE). While in the Preempting state (regardless of activity) the machine advertises its Requirements expression as FALSE to signify that it is not available for further matches, either because it is about to transition to the Owner state, or because it has already been matched with one preempting match, and further preempting matches are disallowed until the machine has been claimed by the new match. The main function of the Preempting state is to get rid of the starter associated with the resource. If the condor_starter associated with a given claim exits while the machine is still in the Vacating activity, then the job successfully completed its checkpoint. If the machine is in the Vacating activity, it keeps evaluating the KILL expression. As soon as this expression evaluates to TRUE, the machine enters the Killing activity (transition 16). When the starter exits, or if there was no starter running when the machine enters the Preempting state (transition 9), the other purpose of the Preempting state is completed: notifying the schedd that had claimed this machine that the claim is broken. At this point, the machine enters either the Owner state by transition 17 (if the job was preempted because the machine owner came back) or the Claimed/Idle state by transition 18 (if the job was preempted because a better match was found). The machine enters the Killing activity, and it starts a timer, the length of which is defined by the KILLING TIMEOUT macro. This macro is defined in seconds and defaults to 30. If this timer expires and the machine is still in the Killing activity, something has gone seriously wrong with the condor starter and the startd tries to vacate the job immediately by sending SIGKILL to all of the condor starter 's children, and then to the condor starter itself. Once the starter is gone and the schedd that had claimed the machine is notified that the claim is broken, the machine will either enter the Owner state by transition 19 (if the job was preempted because the machine owner came back) or the Claimed/Idle state by transition 20 (if the job was preempted because a better match was found).

Default Policy Settings Used in this installation of Condor:

These settings are the default. The vanilla expressions are identical to the regular ones. (They are not listed here. If not defined, the standard expressions are used for vanilla jobs as well).

The following are macros to help write the expressions clearly.

StateTimer Amount of time in the current state.

ActivityTimer Amount of time in the current activity.

ActivationTimer Amount of time the job has been running on this machine.

LastCkpt Amount of time since the last periodic checkpoint.

NonCondorLoadAvg The difference between the system load and the Condor load (the load generated by everything but Condor).

BackgroundLoad Amount of background load permitted on the machine and still start a Condor job.

HighLoad If the $(NonCondorLoadAvg) goes over this, the CPU is considered too busy, and eviction of the Condor job should start.

StartIdleTime Amount of time the keyboard must to be idle before Condor will start a job.

ContinueIdleTime Amount of time the keyboard must to be idle before resumption of a suspended job.

MaxSuspendTime Amount of time a job may be suspended before more drastic measures are taken.

MaxVacateTime Amount of time a job may be checkpointing before we give up kill it outright.

KeyboardBusy A boolean string that evaluates to TRUE when the keyboard is being used.

CPU_Idle A boolean string that evaluates to TRUE when the CPU is idle.

CPU_Busy A boolean string that evaluates to TRUE when the CPU is busy.

MachineBusy The CPU or the Keyboard is busy.

## These macros are here to help write legible expressions:

MINUTE = 60 HOUR = (60 * $(MINUTE))

StateTimer = (CurrentTime - EnteredCurrentState)

ActivityTimer = (CurrentTime - EnteredCurrentActivity)

ActivationTimer = (CurrentTime - JobStart)

NonCondorLoadAvg = (LoadAvg - CondorLoadAvg)

BackgroundLoad = 0.3

HighLoad = 0.5

StartIdleTime = 15 * $(MINUTE)

ContinueIdleTime = 5 * $(MINUTE)

MaxSuspendTime = 10 * $(MINUTE)

MaxVacateTime = 5 * $(MINUTE)

KeyboardBusy = KeyboardIdle < $(MINUTE)

CPU_Idle = $(NonCondorLoadAvg) <= $(BackgroundLoad)

CPU_Busy = $(NonCondorLoadAvg) >= $(HighLoad)

MachineBusy = ($(CPU_Busy) || $(KeyboardBusy))

Macros are defined to always want to suspend jobs. If that is not enough, always try to gracefully vacate them, unless they have only been running for less than 10 minutes anyway, in which case just kill them, instead of trying to checkpoint those 10 minutes of work.