batch_tool.py

Look through the output to identify which jobs are not done, and report task status, optionally in detail, with the option of fixing problems. With 1.0, also invoke Tool.py for errors.

Invocation looks something like:

>>> syqada manage batchrootdirectory [options]

Options:

‘–showmax N’

Maximum items of a given state to show

‘–details <list>’

Provide detailed listing of jobs for the categories listed.

‘–fix <list>’

Request corrections to be made to the location of jobs in the status directories based on jobs’ self-reported status.

‘–errors [options]’

See Tool.py

‘–reset’

See Tool.py

‘–purge’

See Tool.py

Both –fix and –details accept one or more of [‘begun’, ‘done’, ‘failed’, ‘output’]

Jobs are reported in the output category if they self-reported as successfully completed but did not produce the same number of outputs in the output directory as other successful jobs. Fixing the output category means treating them as errors by moving them to ERROR.

Example usage:

Show a summary, as well as details on which jobs have completed that are not properly managed

>>> syqada manage 01-bwa --details done

Show a summary, as well as fix those jobs that have insufficient output

>>> syqada manage 01-bwa --fix output

Show a summary, as well as show one each of each kind of error, as well as 20 lines of its standard error and standard output files from the LOGS directory.

>>> syqada manage 01-bwa --errors classify 20 stdout

Developer Documentation Only Below This Point:

class batch_tool.batch_tool(batchroot, fix, details, times=False, showmax=2, qa=[], stdout=None, debug=False)

Looks through management directories (qm.STATUS_PENDING, qm.STATUS_RUNNING, qm.STATUS_ERROR, DONE) to find jobs and evaluates their progress based on the creation of .begun, .failed, and .done suffixes in the LOG directory.

check_control_directories(errors=<_io.TextIOWrapper name='<stderr>' mode='w' encoding='UTF-8'>)

Build dictionaries of jobs found in the various control directories

check_log_directory(errors=None, read_times=False)

Build dictionaries of jobs found in the log directory. Count the outputs of any job marked DONE.

check_running_jobs(stdout=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'>)

Still not functional

evaluate_problems(reportterm, manageterm)

For a given pair, build the two sorted lists of conditions A - B and B - A

reportterm states the job script can report about itself. One of [begun, done, or failed]

manageterm states syqada can think the job is in, one of [qm.STATUS_PENDING, qm.STATUS_RUNNING, qm.STATUS_DONE, qm.STATUS_ERROR]

manage_problem_details(fixable, jobs, errors=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'>)

Fix a given problem

process_output_problems(failures, reason, errors=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'>)

Report jobs that have failed output qa.

process_problems(reportterm, manageterm, errors=None)

Report and/or manage the specified problem if so specified in self.details or self.fix

process_times(output=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'>)

Report times from LOGS directory.

qa_exec(method)

Confirm that all outputs pass the executed qa script.

qa_inconsistent_output(ignored_parameter=None)

Determine how many outputs are being produced by each successful job, and confirm that each produces the same. Complain about differences.

qa_match_output(method)

Confirm that all outputs have the same number of lines.

qa_non_empty_output(ignored_parameter=None)

Confirm that each output is non_empty

qa_output_generated(ignored_parameter=None)

Confirm only that outputs are produced by each successful job.

report_problem_details(detail, sets, errors=None)

Report a specified set of details

detail the kind of problem

sets two sets, managed but not reported (should be empty), and reported but not managed (indicating that the JobBatch stopped running for some reason).

summarize_status(stdout=None)

Report batch status. Return True if there are discrepancies, else False

batch_tool.parse(input=None, stream=<_io.TextIOWrapper name='<stderr>' mode='w' encoding='UTF-8'>)

Return and create all of the necessary parameters. Complain about semantic validation issues.

batch_tool.restartable_error(batchroot, job)

Conditions under which syqada would simply repend and rerun a job.

batch_tool.set_logging(object, metadata, level='INFO')

Duck-type object as an entity compatible with syqada logging and assign it some new attributes.

batch_tool.test_console_input(flag='Y', stdin=<_io.TextIOWrapper name='<stdin>' mode='r' encoding='UTF-8'>)

Take a character from the console, and return True if it matches the flag.