IterationCoherenceEvaluator

NexusLabs.Needlr.AgentFramework.Evaluation

IterationCoherenceEvaluator Class

Deterministic evaluator that scores the iteration coherence of an iterative-loop agent run from the captured NexusLabs.Needlr.AgentFramework.Diagnostics.IAgentRunDiagnostics snapshot carried in an AgentRunDiagnosticsContext.

public sealed class IterationCoherenceEvaluator : Microsoft.Extensions.AI.Evaluation.IEvaluator

Inheritance System.Object 🡒 IterationCoherenceEvaluator

Implements Microsoft.Extensions.AI.Evaluation.IEvaluator

Remarks

This evaluator only produces metrics when NexusLabs.Needlr.AgentFramework.Diagnostics.IAgentRunDiagnostics.ExecutionMode is "IterativeLoop". For any other execution mode (or when the context is missing) the evaluator returns an empty Microsoft.Extensions.AI.Evaluation.EvaluationResult, which callers should treat as "not applicable".

When applicable, the evaluator emits: - Iteration Count — number of LLM iterations, derived from NexusLabs.Needlr.AgentFramework.Diagnostics.IAgentRunDiagnostics.ChatCompletions. - Iteration Empty Outputs — number of iterations whose NexusLabs.Needlr.AgentFramework.Diagnostics.ChatCompletionDiagnostics.ResponseCharCount is 0. - Terminated Coherently — boolean rollup. true when the run succeeded, produced at least one iteration, and the final iteration produced non-empty output. - Iteration Efficiency Ratio — ratio of useful iterations (produced text output or triggered tool calls) to total iterations. - Degenerate Loop Detected — boolean. true when two or more consecutive iterations produced identical text output. Requires NexusLabs.Needlr.AgentFramework.Diagnostics.ChatCompletionDiagnostics.Response to be captured; defaults to false when response data is unavailable. - Max Iterations Hit — boolean. true when the iteration count reached or exceeded the configured maxIterations threshold. Only emitted when maxIterations was provided to the constructor.

Constructors

IterationCoherenceEvaluator(Nullable<int>) Constructor

Creates a new IterationCoherenceEvaluator.

public IterationCoherenceEvaluator(System.Nullable<int> maxIterations=null);

Parameters

maxIterations System.Nullable<System.Int32>

Optional expected iteration limit. When provided, the evaluator emits the MaxIterationsHitMetricName metric. When null, the metric is omitted.

Fields

IterationCoherenceEvaluator.DegenerateLoopMetricName Field

Metric name for the boolean indicating a degenerate (repeated-output) loop.

public const string DegenerateLoopMetricName = "Degenerate Loop Detected";

Field Value

System.String

IterationCoherenceEvaluator.EfficiencyRatioMetricName Field

Metric name for the ratio of useful iterations to total iterations.

public const string EfficiencyRatioMetricName = "Iteration Efficiency Ratio";

Field Value

System.String

IterationCoherenceEvaluator.EmptyOutputsMetricName Field

Metric name for the count of iterations with empty output.

public const string EmptyOutputsMetricName = "Iteration Empty Outputs";

Field Value

System.String

IterationCoherenceEvaluator.IterationCountMetricName Field

Metric name for the iteration count.

public const string IterationCountMetricName = "Iteration Count";

Field Value

System.String

IterationCoherenceEvaluator.IterativeLoopExecutionMode Field

The execution mode value that gates this evaluator.

public const string IterativeLoopExecutionMode = "IterativeLoop";

Field Value

System.String

IterationCoherenceEvaluator.MaxIterationsHitMetricName Field

Metric name for the boolean indicating the iteration count reached maxIterations.

public const string MaxIterationsHitMetricName = "Max Iterations Hit";

Field Value

System.String

IterationCoherenceEvaluator.TerminatedCoherentlyMetricName Field

Metric name for the boolean rollup indicating coherent termination.

public const string TerminatedCoherentlyMetricName = "Terminated Coherently";

Field Value

System.String

Properties

IterationCoherenceEvaluator.EvaluationMetricNames Property

Gets the Microsoft.Extensions.AI.Evaluation.EvaluationMetric.Names of the Microsoft.Extensions.AI.Evaluation.EvaluationMetrics produced by this Microsoft.Extensions.AI.Evaluation.IEvaluator.

public System.Collections.Generic.IReadOnlyCollection<string> EvaluationMetricNames { get; }

Implements EvaluationMetricNames

Property Value

System.Collections.Generic.IReadOnlyCollection<System.String>

Methods

IterationCoherenceEvaluator.EvaluateAsync(IEnumerable<ChatMessage>, ChatResponse, ChatConfiguration, IEnumerable<EvaluationContext>, CancellationToken) Method

Evaluates the supplied modelResponse and returns an Microsoft.Extensions.AI.Evaluation.EvaluationResult containing one or more Microsoft.Extensions.AI.Evaluation.EvaluationMetrics.

public System.Threading.Tasks.ValueTask<Microsoft.Extensions.AI.Evaluation.EvaluationResult> EvaluateAsync(System.Collections.Generic.IEnumerable<Microsoft.Extensions.AI.ChatMessage> messages, Microsoft.Extensions.AI.ChatResponse modelResponse, Microsoft.Extensions.AI.Evaluation.ChatConfiguration? chatConfiguration=null, System.Collections.Generic.IEnumerable<Microsoft.Extensions.AI.Evaluation.EvaluationContext>? additionalContext=null, System.Threading.CancellationToken cancellationToken=default(System.Threading.CancellationToken));

Parameters

messages System.Collections.Generic.IEnumerable<Microsoft.Extensions.AI.ChatMessage>

The conversation history including the request that produced the supplied modelResponse.

modelResponse Microsoft.Extensions.AI.ChatResponse

The response that is to be evaluated.

chatConfiguration Microsoft.Extensions.AI.Evaluation.ChatConfiguration

A Microsoft.Extensions.AI.Evaluation.ChatConfiguration that specifies the Microsoft.Extensions.AI.IChatClient that should be used if one or more composed Microsoft.Extensions.AI.Evaluation.IEvaluators use an AI model to perform evaluation.

additionalContext System.Collections.Generic.IEnumerable<Microsoft.Extensions.AI.Evaluation.EvaluationContext>

Additional contextual information (beyond that which is available in messages) that the Microsoft.Extensions.AI.Evaluation.IEvaluator may need to accurately evaluate the supplied modelResponse.

cancellationToken System.Threading.CancellationToken

A System.Threading.CancellationToken that can cancel the evaluation operation.

Implements EvaluateAsync(IEnumerable<ChatMessage>, ChatResponse, ChatConfiguration, IEnumerable<EvaluationContext>, CancellationToken)

Returns

System.Threading.Tasks.ValueTask<Microsoft.Extensions.AI.Evaluation.EvaluationResult>
An Microsoft.Extensions.AI.Evaluation.EvaluationResult containing one or more Microsoft.Extensions.AI.Evaluation.EvaluationMetrics.

Remarks

The Microsoft.Extensions.AI.Evaluation.EvaluationMetric.Names of the Microsoft.Extensions.AI.Evaluation.EvaluationMetrics contained in the returned Microsoft.Extensions.AI.Evaluation.EvaluationResult should match Microsoft.Extensions.AI.Evaluation.IEvaluator.EvaluationMetricNames.

Also note that chatConfiguration must not be omitted if the evaluation is performed using an AI model.