bcftbx.JobRunner

Legacy module providing job runner classes for starting, stopping and managing jobs.

The core functionality has been moved to the jobs.runners module, which implements the base JobRunner class and subclasses for various job management systems.

This module provides legacy job runner classes based on the new classes (which mostly have different names and different interfaces) for backwards compatibility. The legacy runner classes implemented here are:

  • SimpleJobRunner: run jobs (e.g. scripts) on a local file system.

  • GEJobRunner : run jobs using Grid Engine (GE) i.e. qsub, qdel etc

  • SlurmRunner : run jobs using Slurm i.e. sbatch, scancel etc

These classes use the legacy names for interface methods (e.g. isRunning, logFile etc) but otherwise provide the same functionality as the corresponding classes in the jobs.runners module.

Simple usage example:

>>> # Create a JobRunner instance
>>> runner = SimpleJobRunner()
>>> # Start a job using the runner and collect its id
>>> job_id = runner.run('Example',None,'myscript.sh')
>>> # Wait for job to complete
>>> import time
>>> while runner.isRunning(job_id):
>>>     time.sleep(10)
>>> # Get the names of the output files
>>> log,err = (runner.logFile(job_id),runner.errFile(job_id))

The version of fetch_runner provided here can be used to create legacy runner instances based on a definition string; it is separate from the version in jobs.runners to maintain backwards compatibility.

class bcftbx.JobRunner.BaseJobRunner(name='JobRunner')

Provided for backwards compatibility

class bcftbx.JobRunner.GEJobRunner(queue=None, log_dir=None, ge_extra_args=None, poll_interval=5, timeout=30)

Class implementing job runner for Grid Engine

GEJobRunner is a wrapper for the GridEngineRunner class from the jobs.runners module.

Parameters:

  • queue (str) – name of GE queue to use (set to ‘None’ to use default queue)

  • log_dir (str) – directory to write log files to (set to ‘None’ to use CWD)

  • ge_extra_args (list) – arbitrary additional arguments to supply to qsub

  • poll_interval (int) – time interval (in seconds) to use when polling Grid Engine e.g. to acquire qacct information (default: 5)

  • timeout (int) – maximum length of time (in seconds) to wait before giving up when polling Grid Engine (default: 30)

errFile(job_id)

Return the error file name for a job

The name should be ‘<name>.e<job_id>’

errorState(job_id)

Check if the job is in an error state

Return True if the job is deemed to be in an ‘error state’ (i.e. qstat returns the state as ‘E..’), False otherwise.

isRunning(job_id)

Check if a job is running

logFile(job_id)

Return the log file name for a job

The name should be ‘<name>.o<job_id>’

class bcftbx.JobRunner.ResourceLock

Provided for backwards compatibility

class bcftbx.JobRunner.SimpleJobRunner(log_dir=None, join_logs=False, nslots=1)

Class implementing job runner for local system

SimpleJobRunner is a wrapper for the LocalRunner class from the jobs.runners module.

Parameters:

  • log_dir – Directory to write log files to (set to ‘None’ to use CWD)

  • join_logs – Combine stderr and stdout into a single log file (by default stdout and stderr have their own log files)

  • nslots – Number of threads associated with this runner instance

errFile(job_id)

Return the error file name for a job

errorState(job_id)

Check if the job is in an error state

Return True if the job is deemed to be in an ‘error state’ (i.e. qstat returns the state as ‘E..’), False otherwise.

isRunning(job_id)

Check if a job is running

logFile(job_id)

Return the log file name for a job

class bcftbx.JobRunner.SlurmRunner(log_dir=None, nslots=None, partition=None, join_logs=None, slurm_extra_args=None, poll_interval=300, timeout=30, missing_job_timeout=600)

Class implementing job runner for Slurm

SlurmRunner is a wrapper for the SlurmRunner` class from the jobs.runners module.

Parameters:

  • log_dir (str) – path of directory to write log files to (set to ‘None’ to use cwd)

  • nslots (int) – number of threads assigned to the runner instance

  • partition (str) – name of Slurm partition to target (set to ‘None’ to use default queue)

  • join_logs (bool) – if True then combine stderr and stdout into a single log file (default is to write stdout and stderr to separate log files)

  • slurm_extra_args (list) – arbitrary additional arguments to supply to ‘sbatch’ (e.g. ‘[“-n”, 8]’)

  • poll_interval (int) – time interval to use (in seconds) when polling Slurm using ‘squeue’ (seconds) (default: 300.0)

  • timeout (int) – maximum length of time to wait (in seconds) before giving up when submitting jobs to Slurm and finalizing jobs (default: 30)

  • missing_job_timeout (int) – minimum time (in seconds) that a job needs to be flagged as “missing” before it’s removed from the runner (default: 600)

errFile(job_id)

Return the error file name for a job

The name should be ‘<name>.e<job_id>’

errorState(job_id)

Check if the job is in an error state

Return True if the job is deemed to be in an ‘error state’, False otherwise.

isRunning(job_id)

Check if a job is running

logFile(job_id)

Return the log file name for a job

The name should be ‘<name>.o<job_id>’

bcftbx.JobRunner.fetch_runner(definition)

Return job runner instance based on a definition string

Given a definition string, returns an appropriate runner instance.

Definitions are of the form:

RunnerName[(args)]

RunnerName can be ‘SimpleJobRunner’, ‘GEJobRunner’ or ‘SlurmRunner’. If ‘(args)’ are also supplied then:

  • for SimpleJobRunners, this can be a list of optional arguments separated by spaces:

    • ‘nslots=N’ (where N is an integer; sets a non-default number of slots

    • ‘join_logs=BOOLEAN’ (where BOOLEAN can be ‘True’, ‘true’,’y’,’False’,’false’,’n’; sets whether stdout and stderr should be written to the same file)

  • for GEJobRunners, this is a set of arbitrary ‘qsub’ options that will be used on job submission

  • for SlurmRunners, this can be a list of optional arguments separated by spaces:

    • ‘nslots=N’ (where N is an integer; sets a non-default number of slots

    • ‘partition=STRING’ (where STRING is the name of the target Slurm partition)

    • ‘join_logs=BOOLEAN’ (where BOOLEAN can be ‘True’, ‘true’,’y’,’False’,’false’,’n’; sets whether stdout and stderr should be written to the same file)

    • a sting with arbitrary ‘sbatch’ options that will be included on job submission (note: ‘-J’, ‘-o’, ‘-e’ and ‘–export’ cannot be specified)