MaestroPipelineList: Class for a list of MaestroPipelines A MaestroPipelineList is created when there are multiple maestro pipelines in a single script

Description

Class for a list of MaestroPipelines A MaestroPipelineList is created when there are multiple maestro pipelines in a single script

Arguments

Public fields

MaestroPipelines: list of pipelines
n_pipelines: number of pipelines in the list

Methods

Public methods

`MaestroPipelineList$new()`

Create a MaestroPipelineList object

Usage

MaestroPipelineList$new(MaestroPipelines = list(), network = NULL)

Arguments

MaestroPipelines: list of MaestroPipelines
network: initialize a network

Returns

MaestroPipelineList

`MaestroPipelineList$print()`

Print the MaestroPipelineList

Usage

MaestroPipelineList$print()

Returns

`MaestroPipelineList$add_pipelines()`

Add pipelines to the list

Usage

MaestroPipelineList$add_pipelines(MaestroPipelines = NULL)

Arguments

MaestroPipelines: list of MaestroPipelines

Returns

invisible

`MaestroPipelineList$update_pipelines()`

Update pipelines in a list

Usage

MaestroPipelineList$update_pipelines(MaestroPipelines = NULL)

Arguments

MaestroPipelines: list of MaestroPipelines

Returns

invisible

`MaestroPipelineList$get_pipe_names()`

Get names of the pipelines in the list arranged by priority

Usage

MaestroPipelineList$get_pipe_names()

Returns

character

`MaestroPipelineList$get_pipe_by_name()`

Get a MaestroPipeline by its name

Usage

MaestroPipelineList$get_pipe_by_name(pipe_name)

Arguments

pipe_name: name of the pipeline

Returns

MaestroPipeline

`MaestroPipelineList$get_pipes_by_name()`

Get a MaestroPipelineList with selected pipelines

Usage

MaestroPipelineList$get_pipes_by_name(pipe_names)

Arguments

pipe_names: names of the pipelines

Returns

MaestroPipelineList

`MaestroPipelineList$get_priorities()`

Get priorities

Usage

MaestroPipelineList$get_priorities()

Returns

numeric

`MaestroPipelineList$get_schedule()`

Get the schedule as a data.frame

Usage

MaestroPipelineList$get_schedule()

Returns

data.frame

`MaestroPipelineList$get_timely_pipelines()`

Get a new MaestroPipelineList containing only those pipelines scheduled to run

Usage

MaestroPipelineList$get_timely_pipelines(...)

Arguments

...: arguments passed to self$check_timeliness

Returns

MaestroPipelineList

`MaestroPipelineList$get_primary_pipes()`

Get pipelines that are primary (i.e., don't have an inputting pipeline)

Usage

MaestroPipelineList$get_primary_pipes()

Returns

list of MaestroPipelines

`MaestroPipelineList$check_timeliness()`

Check whether pipelines in the list are scheduled to run based on orchestrator frequency and current time

Usage

MaestroPipelineList$check_timeliness(...)

Arguments

...: arguments passed to self$check_timeliness

Returns

logical

`MaestroPipelineList$get_status()`

Get status of the pipelines as a data.frame

Usage

MaestroPipelineList$get_status()

Returns

data.frame

`MaestroPipelineList$get_errors()`

Get list of errors from the pipelines

Usage

MaestroPipelineList$get_errors()

Returns

list

`MaestroPipelineList$get_warnings()`

Get list of warnings from the pipelines

Usage

MaestroPipelineList$get_warnings()

Returns

list

`MaestroPipelineList$get_messages()`

Get list of messages from the pipelines

Usage

MaestroPipelineList$get_messages()

Returns

list

`MaestroPipelineList$get_artifacts()`

Get artifacts (return values) from the pipelines

Usage

MaestroPipelineList$get_artifacts()

Returns

list

`MaestroPipelineList$get_run_sequences()`

Get run sequences from the pipelines

Usage

MaestroPipelineList$get_run_sequences(
  n = NULL,
  min_datetime = NULL,
  max_datetime = NULL
)

Arguments

n: optional sequence limit
min_datetime: optional minimum datetime
max_datetime: optional maximum datetime

Returns

list

`MaestroPipelineList$get_flags()`

Get the flags of the pipelines as a named list

Usage

MaestroPipelineList$get_flags()

Returns

list

`MaestroPipelineList$get_labels()`

Get the labels of the pipelines as a data.frame

Usage

MaestroPipelineList$get_labels()

Returns

data.frame

`MaestroPipelineList$get_network()`

Get the network structure as a edge list

Usage

MaestroPipelineList$get_network()

Returns

data.frame

`MaestroPipelineList$validate_network()`

Validates whether all inputs and outputs exist and that the network is a valid DAG

Usage

MaestroPipelineList$validate_network()

Returns

warning or invisible

`MaestroPipelineList$run()`

Runs all the pipelines in the list

Usage

MaestroPipelineList$run(..., cores = 1L, pipes_to_run = NULL)

Arguments

...: arguments passed to MaestroPipeline$run
cores: if using multicore number of cores to run in (uses furrr)
pipes_to_run: an optional vector of pipe names to run. If NULL defaults to all primary pipelines

Returns

invisible

`MaestroPipelineList$run_pending_collects()`

Run any collect pipelines that are ready but were skipped during a parallel run. Called on the main process after update_pipelines() has synced worker state back. Uses run_pipe internally so that the collect pipe's own downstream outputs are recursed into normally.

Usage

MaestroPipelineList$run_pending_collects(...)

Arguments

...: arguments forwarded to MaestroPipeline$run (same dots as run())

Returns

list of MaestroPipeline objects that were run

`MaestroPipelineList$reset_pipelines()`

Resets the run time attributes

Usage

MaestroPipelineList$reset_pipelines()

`MaestroPipelineList$apply_cascade()`

Propagate @maestroCascadeTags metadata through the DAG. For each pipeline that declares a non-empty cascade vector, the selected tag types (label, flags, loglevel) are applied to every downstream pipeline in topological order (root to leaves) so that the nearest ancestor wins on conflict.

Usage

MaestroPipelineList$apply_cascade()

Returns

invisible

`MaestroPipelineList$clone()`

The objects of this class are cloneable with this method.

Usage

MaestroPipelineList$clone(deep = FALSE)

Arguments

deep: Whether to make a deep clone.