PipelineStageEvaluator

NexusLabs.Needlr.AgentFramework.Evaluation

PipelineStageEvaluator Class

Deterministic evaluator that scores per-stage success/failure and overall pipeline health from the captured NexusLabs.Needlr.AgentFramework.Diagnostics.IPipelineRunResult snapshot carried in a PipelineEvaluationContext.

public sealed class PipelineStageEvaluator : Microsoft.Extensions.AI.Evaluation.IEvaluator

Inheritance System.Object 🡒 PipelineStageEvaluator

Implements Microsoft.Extensions.AI.Evaluation.IEvaluator

Remarks

This evaluator never contacts a language model. It reads the NexusLabs.Needlr.AgentFramework.Diagnostics.IPipelineRunResult to produce: - pipeline.succeeded — whether the pipeline succeeded. - pipeline.total_stages — total number of stages. - pipeline.completed_stages — stages with non-null diagnostics. - pipeline.skipped_stages — stages with null diagnostics AND null response. - pipeline.total_duration_ms — total pipeline duration in milliseconds. - pipeline.error_message — error message if the pipeline failed (nullable).

When no PipelineEvaluationContext is present in the additionalContext collection, the evaluator returns an empty Microsoft.Extensions.AI.Evaluation.EvaluationResult — callers should treat that as "not applicable".

Fields

PipelineStageEvaluator.CompletedStagesMetricName Field

Metric name for the number of completed stages (those with diagnostics).

public const string CompletedStagesMetricName = "pipeline.completed_stages";

Field Value

System.String

PipelineStageEvaluator.ErrorMessageMetricName Field

Metric name for the error message if the pipeline failed.

public const string ErrorMessageMetricName = "pipeline.error_message";

Field Value

System.String

PipelineStageEvaluator.SkippedStagesMetricName Field

Metric name for the number of skipped stages (null diagnostics AND null response).

public const string SkippedStagesMetricName = "pipeline.skipped_stages";

Field Value

System.String

PipelineStageEvaluator.SucceededMetricName Field

Metric name for whether the pipeline succeeded.

public const string SucceededMetricName = "pipeline.succeeded";

Field Value

System.String

PipelineStageEvaluator.TotalDurationMsMetricName Field

Metric name for the total pipeline duration in milliseconds.

public const string TotalDurationMsMetricName = "pipeline.total_duration_ms";

Field Value

System.String

PipelineStageEvaluator.TotalStagesMetricName Field

Metric name for the total number of stages.

public const string TotalStagesMetricName = "pipeline.total_stages";

Field Value

System.String

Properties

PipelineStageEvaluator.EvaluationMetricNames Property

Gets the Microsoft.Extensions.AI.Evaluation.EvaluationMetric.Names of the Microsoft.Extensions.AI.Evaluation.EvaluationMetrics produced by this Microsoft.Extensions.AI.Evaluation.IEvaluator.

public System.Collections.Generic.IReadOnlyCollection<string> EvaluationMetricNames { get; }

Implements EvaluationMetricNames

Property Value

System.Collections.Generic.IReadOnlyCollection<System.String>

Methods

PipelineStageEvaluator.EvaluateAsync(IEnumerable<ChatMessage>, ChatResponse, ChatConfiguration, IEnumerable<EvaluationContext>, CancellationToken) Method

Evaluates the supplied modelResponse and returns an Microsoft.Extensions.AI.Evaluation.EvaluationResult containing one or more Microsoft.Extensions.AI.Evaluation.EvaluationMetrics.

public System.Threading.Tasks.ValueTask<Microsoft.Extensions.AI.Evaluation.EvaluationResult> EvaluateAsync(System.Collections.Generic.IEnumerable<Microsoft.Extensions.AI.ChatMessage> messages, Microsoft.Extensions.AI.ChatResponse modelResponse, Microsoft.Extensions.AI.Evaluation.ChatConfiguration? chatConfiguration=null, System.Collections.Generic.IEnumerable<Microsoft.Extensions.AI.Evaluation.EvaluationContext>? additionalContext=null, System.Threading.CancellationToken cancellationToken=default(System.Threading.CancellationToken));

Parameters

messages System.Collections.Generic.IEnumerable<Microsoft.Extensions.AI.ChatMessage>

The conversation history including the request that produced the supplied modelResponse.

modelResponse Microsoft.Extensions.AI.ChatResponse

The response that is to be evaluated.

chatConfiguration Microsoft.Extensions.AI.Evaluation.ChatConfiguration

A Microsoft.Extensions.AI.Evaluation.ChatConfiguration that specifies the Microsoft.Extensions.AI.IChatClient that should be used if one or more composed Microsoft.Extensions.AI.Evaluation.IEvaluators use an AI model to perform evaluation.

additionalContext System.Collections.Generic.IEnumerable<Microsoft.Extensions.AI.Evaluation.EvaluationContext>

Additional contextual information (beyond that which is available in messages) that the Microsoft.Extensions.AI.Evaluation.IEvaluator may need to accurately evaluate the supplied modelResponse.

cancellationToken System.Threading.CancellationToken

A System.Threading.CancellationToken that can cancel the evaluation operation.

Implements EvaluateAsync(IEnumerable<ChatMessage>, ChatResponse, ChatConfiguration, IEnumerable<EvaluationContext>, CancellationToken)

Returns

System.Threading.Tasks.ValueTask<Microsoft.Extensions.AI.Evaluation.EvaluationResult>
An Microsoft.Extensions.AI.Evaluation.EvaluationResult containing one or more Microsoft.Extensions.AI.Evaluation.EvaluationMetrics.

Remarks

The Microsoft.Extensions.AI.Evaluation.EvaluationMetric.Names of the Microsoft.Extensions.AI.Evaluation.EvaluationMetrics contained in the returned Microsoft.Extensions.AI.Evaluation.EvaluationResult should match Microsoft.Extensions.AI.Evaluation.IEvaluator.EvaluationMetricNames.

Also note that chatConfiguration must not be omitted if the evaluation is performed using an AI model.