AI Architecture Evaluator

Overview

CAL-004 explores architecture itself as an input to engineering automation through grounded architecture conformance evaluation. It tests whether an AI evaluator can inspect a Mermaid Architecture-as-Code artifact and determine whether it conforms to authoritative Cloud Architect Lab requirements.

Findings are grounded in retrieved evidence and citations rather than generic model judgment. The evaluator reports what it observes, which governed requirement applies, and why the available evidence passes, fails, or cannot be evaluated. It does not modify architecture, recommend remediation, generate Terraform, or deploy infrastructure.

Project Objective

Demonstrate that an AI-assisted evaluator can perform grounded architecture conformance evaluation by applying authoritative engineering knowledge to a machine-readable architecture artifact in a repeatable, auditable way.

CAL-003 established governed retrieval and knowledge-authority controls. CAL-004 builds on that foundation by asking whether AI can apply the retrieved knowledge to evaluate an architecture artifact.

Evaluation Boundary

• Mermaid source is the only architecture input.
• Governed CAL documents provide requirements and decisions.
• The model compares artifact evidence with retrieved CAL knowledge.
• Findings include status, severity, evidence, rationale, and citations.
• The evaluator remains read-only and cannot take engineering action.

Structured Findings

The architecture artifact answers what exists. Governed CAL knowledge answers what should exist. The model performs the comparison but is not itself the authority.

Severity describes engineering impact rather than model confidence and is limited to low, medium, and high classification metadata.

Validation Approach

The initial validation uses the approved CAL-002 Mermaid architecture as a known-good fixture and a deliberately altered copy as a known-bad fixture. Controlled defect injection tests whether the evaluator detects the resulting nonconformance without access to the human-maintained answer key.

• Detection rate for deliberately introduced defects
• False-positive and false-negative review
• Citation correctness
• Appropriate use of not-evaluable findings
• Adversarial testing with prompt-injection text inside the Mermaid artifact

Project Status

CAL-004 is complete. Requirements, architecture, AI and security design, validation boundaries, engineering decisions, fixtures, and end-to-end evaluation results are documented. The completed evaluation showed where grounded AI analysis adds value and where deterministic conformance checks remain necessary.

Repository

The current design documents, requirements, decisions, validation plan, and lessons learned are available on GitHub.