Cancellation of Processing

MMF provides the ability to cancel the processing of ModuleGraphs. This is implemented and demonstrated in SimpleML through the GradientDescentOptimizer class and LinearRegressionGradientDescentOptimizer module.

Overview

Cancellation is facilitated by the .NET CancellationTokenSource and CancellationToken class/struct. The MMF ModuleBase class contains a CancellationToken member called 'cancellationToken', and protected utility method ThrowIfCancellationRequested() and property IsCancellationRequested. This method and property duplicate the same-named method and property on the CancellationToken struct... i.e. IsCancellationRequested will return true if cancellation has been requested, ThrowIfCancellationRequested() will throw an OperationCanceledException if cancellation has been requested.

Implementation

Classes used by MMF Modules which contain long running operations should support cancellation via a CancellationToken. This is demonstrated in the GradientDescentOptimizer class. A CancellationToken is passed to its constructor, and the loop performing gradient descent iterations in the Optimize() method periodically checks for a cancellation request...

        Double cost = costFunctionCalculator.Calculate(trainingDataSeries, trainingDataResults, currentThetaParameters);
        totalIterationsCounter++;
        loggingUtilities.Log(this, LogLevel.Debug, "Gradient descent iteration " + totalIterationsCounter + ", cost = " + cost + ".");

        currentThetaParameters = newThetaParameters;
    }

    cancellationToken.ThrowIfCancellationRequested();
    remainingIterations = remainingIterations - performedIterations;
}

MMF Modules which compose cancellable objects should pass the ModuleBase CancellationToken to the cancellable object(s). This is demonstrated in the LinearRegressionGradientDescentOptimizer module, which passes the CancellationToken to the composed GradientDescentOptimizer object's constructor...

try
{
    MatrixUtilities matrixUtilities = new MatrixUtilities();
    // Add a column of 1's to the training data
    Matrix biasedTrainingDataSeries = matrixUtilities.AddColumns(trainingSeriesData, 1, true, 1);
    GradientDescentOptimizer gradientDescentOptimizer = new GradientDescentOptimizer(logger, metricLogger, cancellationToken);
    MultivariateLinearRegressionHypothesisCalculator hypothesisCalculator = new MultivariateLinearRegressionHypothesisCalculator();
    MultivariateLinearRegressionCostFunctionCalculator costFunctionCalculator = new MultivariateLinearRegressionCostFunctionCalculator();
    Matrix optimizedThetaParameters = gradientDescentOptimizer.Optimize(initialThetaParameters, biasedTrainingDataSeries, trainingSeriesResults, hypothesisCalculator, costFunctionCalculator, learningRate, maxIterations);
    GetOutputSlot(optimizedThetaParametersOutputSlotName).DataValue = optimizedThetaParameters;
}
catch (OperationCanceledException)
{
    throw;
}
catch (Exception e)
{
    logger.Log(this, LogLevel.Critical, "Error occurred whilst running gradient descent for linear regression.", e);
    throw;
}

Performing Cancellation

An example of cancelling the processing of a ModuleGraph is demonstrated in the Program classes' ProcessAndCancelGraph() method. A worker thread is used to cancel the processing of a long running gradient descent process after 8 seconds...

using (FileApplicationLogger logger = new FileApplicationLogger(LogLevel.Information, '|', "  ", Path.Combine(logFilePath, "SimpleMLSampleLog.txt")))
using (SizeLimitedBufferProcessor bufferProcessor = new SizeLimitedBufferProcessor(10))
using (FileMetricLogger metricLogger = new FileMetricLogger('|', Path.Combine(logFilePath, "SimpleMLSampleMetrics.txt"), bufferProcessor, true))
using (ModuleGraphProcessor processor = new ModuleGraphProcessor(logger, metricLogger))
{
    // Create a thread to call CancelProcessing() after 8 seconds
    Thread cancellationThread = new Thread
    (() =>
    {
        Thread.Sleep(8000);
        processor.CancelProcessing();
    }
    );
    cancellationThread.Start();

    // Process the module graph
    bufferProcessor.Start();
    try
    {
        processor.Process(moduleGraph, false);
    }
    finally
    {
        bufferProcessor.Stop();
    }
}

Logging of Cancellation

The MMF ModuleGraphProcessor class records a ModuleGraphProcessingCancelled metric when processing is cancelled, and also logs details of the module that was processing at the time of cancellation...

try
{
    module.Process();
}
catch (OperationCanceledException)
{
    metricsUtilities.Increment(new ModuleGraphProcessingCancelled());
    loggingUtilities.Log(this, LogLevel.Information, "Processing of module '" + module.GetType().FullName + "' cancelled.");
    metricsUtilities.CancelBegin(new ModuleProcessingTime());
    throw;
}

... resulting in the following information being written to the log and metric log...

2017-04-17 09:55:06.440 | Source = ModuleGraphProcessor | Processing of module 'SimpleML.Samples.Modules.LinearRegressionGradientDescentOptimizer' cancelled.

2017-04-17 09:55:06.440 | ModuleGraphProcessingCancelled

Additional information about the cancellation can also be logged from the module class, or classes used by the module, as is demonstrated in the GradientDescentOptimizer class...

catch(OperationCanceledException)
{
    logger.Log(this, LogLevel.Information, "Gradient descent optimization cancelled after "  + totalIterationsCounter + " iterations.");
    metricsUtilities.CancelBegin(new GradientDescentOptimizationTime());
    throw;
}