ShellLogger

class shell_logger.ShellLogger(name, *, log_dir=None, stream_dir=None, html_file=None, indent=0, login_shell=False, log=None, init_time=None, done_time=None, duration=None)[source]

Bases: object

Run commands in the shell, while logging various metadata.

This class will keep track of commands run in the shell, their durations, descriptions, stdout, stderr, and return_code. When the finalize() method is called, the ShellLogger object will aggregate all the data from its commands and child ShellLogger objects (see example below) into both JSON and HTML files.

Example:

> Parent ShellLogger Object Name
    Duration: 18h 20m 35s
  > cmd1  (Click arrow '>' to expand for more details)
    Duration: 0.25s
  > Child ShellLogger Object Name (i.e. Trilinos)
    Duration: 3h 10m 0s
    > Child ShellLogger Object Name (i.e. Configure)
      Duration: 1m 3s
      > cmd1
etc...

Note

Because some stdout/stderr streams can be quite long, they will be written to files in a temporary directory (log_dir/tmp/YYYY-MM-DD_hh:mm:ss/). Once the finalize() method is called, they will be aggregated in an HTML file (log_dir/html_file). The JSON file (log_dir/json_file) will contain references to the stdout/stderr files so that an HTML file can be recreated again later if needed.

name

The name of the ShellLogger object.

Type:: str

log_dir

Path to where the logs are stored for the parent ShellLogger and all its children.

Type:: Path

stream_dir

Path to directory where stdout/stderr stream logs are stored.

Type:: Path

html_file

Path to main HTML file for the parent and child ShellLogger objects.

Type:: Path

indent

The indentation level of this ShellLogger object. The parent has a level 0. Each successive child’s indent is increased by 1.

Type:: int

login_shell

Whether or not the Shell spawned should be a login shell.

Type:: bool

log_book

A list containing log entries and child ShellLogger objects in the order they were created.

Type:: list

init_time

The time this ShellLogger object was created.

Type:: datetime

done_time

The time this ShellLogger object is done with its commands/messages.

Type:: datetime

duration

The string-formatted duration of this ShellLogger, updated when the finalize() method is called.

Type:: str

shell

The Shell in which all commands will be run when logging.

Type:: Shell

Initialize a ShellLogger object.

Parameters:

name (str) – The name to give to this ShellLogger object.
log_dir (Optional[Path]) – Where to store the log files.
stream_dir (Optional[Path]) – Where the stdout/stderr stream logs are stored. This is helpful for parent ShellLogger objects to give to child ShellLogger objects in order to keep things in the same directory.
html_file (Optional[Path]) – The path to the main HTML file for the parent and children ShellLogger objects. If omitted, this is the parent ShellLogger object, and it will need to create the file.
indent (int) – The indentation level of this ShellLogger object. The parent has a level 0. Each successive child’s indent is increased by 1.
login_shell (bool) – Whether or not the Shell spawned should be a login shell.
log (Optional[list[object]]) – Optionally provide an existing log list to the ShellLogger object.
init_time (Optional[datetime]) – Optionally specify when this ShellLogger was initialized.
done_time (Optional[datetime]) – Optionally specify when this ShellLogger was finalized.
duration (Optional[str]) – A string representation of the total duration of the ShellLogger.

Note

The log, init_time, done_time, and duration parameters are mainly used when importing ShellLogger objects from a JSON file, and can generally be omitted.

add_child(child_name)[source]

Add a child logger.

Creates and returns a ‘child’ ShellLogger object. This will be one step indented in the tree of the output log (see the example in the class docstring). The total time for this child will be recorded when the finalize() method is called in the child object.

Parameters:: child_name (str) – The name of the child ShellLogger object.
Returns:: A child ShellLogger object.
Return type:: ShellLogger

static append(path)[source]

Append to a prior log.

Create a ShellLogger to append to the HTML log file generated by a prior ShellLogger.

Parameters:: path (Path) – The location of the prior ShellLogger ‘s output, either the log directory, or the HTML log file itself.
Return type:: ShellLogger
Returns:: A new ShellLogger populated with the data from the prior one.

auxiliary_information()[source]

Grab auxiliary information.

Capture all sorts of auxiliary information before running a command.

Return type:: SimpleNamespace
Returns:: The working directory, environment, umask, hostname, user, group, shell, and ulimit.

change_log_dir(new_log_dir)[source]

Change the log directory.

Change the log_dir of this ShellLogger object and all children recursively.

Parameters:: new_log_dir (Path) – Path to the new log_dir.
Raises:: RuntimeError – If this is called on a child ShellLogger.
Return type:: None

check_duration()[source]

Get the current duration.

From the beginning of the ShellLogger object’s creation until now.

Return type:: str
Returns:: A string representation of the total duration.

finalize()[source]

Finalize the ShellLogger object.

Write the HTML log file.

Return type:: None

html_print(msg, msg_title='HTML Message')[source]

Save a message to the log but don’t print it in the console.

Parameters:

msg (str) – Message to save to the log.
msg_title (str) – Title of the message to save to the log.

Return type:

None

is_parent()[source]

Check whether this is a parent logger.

Determine whether or not this is the parent ShellLogger object, as indicated by the object’s indent attribute.

Return type:: bool
Returns:: True if this is the parent; False otherwise.

log(msg, cmd, *, cwd=None, live_stdout=False, live_stderr=False, return_info=False, verbose=False, stdin_redirect=True, **kwargs)[source]

Execute a command, and log the corresponding information.

Parameters:

msg (str) – A message to be recorded with the command. This could be documentation of what your command is doing and why.
cmd (str) – The shell command to be executed.
cwd (Optional[Path]) – Where to execute the command.
live_stdout (bool) – Print stdout as it is being produced, as well as saving it to the file.
live_stderr (bool) – Print stderr as it is being produced, as well as saving it to the file.
return_info (bool) – If set to True, stdout, stderr, and return_code will be stored and returned in a dictionary. Consider leaving this set to False if you anticipate your command producing large stdout/stderr streams that could cause memory issues.
verbose (bool) – Print the command before it is executed.
stdin_redirect (bool) – Whether or not to redirect stdin to /dev/null. We do this by default to handle issues that arise when the cmd involves MPI; however, in some cases (e.g., involving bsub) the redirect causes problems, and we need the flexibility to revert back to standard behavior.
**kwargs – Any other keyword arguments to pass on to _run().

Return type:

dict

Returns:

A dictionary containing stdout, stderr, trace, and return_code keys. If return_info is set to False, the stdout and stderr values will be None. If return_info is set to True and trace is specified in kwargs, trace in the dictionary will contain the output of the specified trace; otherwise, it will be None.

Note

To conserve memory, stdout and stderr will be written to files as they are being generated.

print(msg, end='\\n')[source]

Print a message and save it to the log.

Parameters:

msg (str) – The message to print and save to the log.
end (str) – The string appended after the message:

Return type:

None

static strfdelta(delta, fmt)[source]

Convert a time delta to a string.

Format a time delta object. Use this like you would datetime.strftime().

Parameters:

delta (timedelta) – The time delta object.
fmt (str) – The delta format string.

Return type:

str

Returns:

A string representation of the time delta.

to_html()[source]

Convert the log entries to HTML.

This method iterates through each entry in this ShellLogger object’s log list and builds up a list of corresponding HTML snippets. For each entry, the stdout/stderr are copied from their respective files in the stream_dir.

Return type:: Union[Iterator[str], list[Iterator[str]]]
Returns:: A generator (or list of generators) that will lazily yield strings corresponding to the elements of the HTML file. This lazy evaluation was done to avoid loading all the data for the log file into memory at once.

update_done_time()[source]

Update the done time.

Allows the done_time to be updated before finalize() is called. This is especially useful for child ShellLogger objects who might finish their commands before the parent finalizes everything.

Return type:: None

JSON Serialization

class shell_logger.ShellLoggerDecoder[source]

Bases: JSONDecoder

Convert JSON to ShellLogger objects.

This is a helper class to make the ShellLogger class JSON serializable. It is used in the process of retrieving ShellLogger objects from JSON.

Usage:

import json
with open('path_to_json_file', 'r') as jf:
    logger = json.load(jf, cls=ShellLoggerDecoder)

Initialize the decoder.

static dict_to_object(obj)[source]

Convert a dict to a corresponding object.

This converts data dictionaries given by the JSONDecoder into objects of type ShellLogger, datetime, etc.

Parameters:: obj (dict) – The JSON-serialized representation of an object.
Return type:: object
Returns:: The object represented by the JSON serialization.

Todo

Figure out why the documentation below is pulling in the docstring for the base class as well.

class shell_logger.ShellLoggerEncoder(*, skipkeys=False, ensure_ascii=True, check_circular=True, allow_nan=True, sort_keys=False, indent=None, separators=None, default=None)[source]

Bases: JSONEncoder

Convert ShellLogger objects to JSON.

This is a helper class to make the ShellLogger class JSON serializable. It is used in the process of saving ShellLogger objects to JSON.

Usage:

import json
with open('path_to_json_file', 'w') as jf:
    json.dump(data, jf, cls=ShellLoggerEncoder)

Constructor for JSONEncoder, with sensible defaults.

If skipkeys is false, then it is a TypeError to attempt encoding of keys that are not str, int, float, bool or None. If skipkeys is True, such items are simply skipped.

If ensure_ascii is true, the output is guaranteed to be str objects with all incoming non-ASCII characters escaped. If ensure_ascii is false, the output can contain non-ASCII characters.

If check_circular is true, then lists, dicts, and custom encoded objects will be checked for circular references during encoding to prevent an infinite recursion (which would cause an RecursionError). Otherwise, no such check takes place.

If allow_nan is true, then NaN, Infinity, and -Infinity will be encoded as such. This behavior is not JSON specification compliant, but is consistent with most JavaScript based encoders and decoders. Otherwise, it will be a ValueError to encode such floats.

If sort_keys is true, then the output of dictionaries will be sorted by key; this is useful for regression tests to ensure that JSON serializations can be compared on a day-to-day basis.

If indent is a non-negative integer, then JSON array elements and object members will be pretty-printed with that indent level. An indent level of 0 will only insert newlines. None is the most compact representation.

If specified, separators should be an (item_separator, key_separator) tuple. The default is (’, ‘, ‘: ‘) if indent is None and (‘,’, ‘: ‘) otherwise. To get the most compact JSON representation, you should specify (‘,’, ‘:’) to eliminate whitespace.

If specified, default is a function that gets called for objects that can’t otherwise be serialized. It should return a JSON encodable version of the object or raise a TypeError.

default(obj)[source]

Serialize an object; that is, encode it in a string format.

Parameters:: obj (object) – Any Python object.
Return type:: object
Returns:: The JSON serialization of the given object.