cc8.1-5-001

CC8.1 · D5 · f1

Difficulty

Gaps

Red Herrings

74%

Avg Score

Task

You are auditing Beacon SaaS's SOC 2 Type II compliance for control CC8.1 (Change Management). Review the provided evidence to assess whether Beacon SaaS has effective change management controls in place, including proper authorization, testing, approval, and deployment practices. Evaluate whether deviations from the documented policy represent genuine control gaps or reasonable operational decisions. Consider the full context when making judgments — some situations may be ambiguous and require nuanced analysis.

Evidence

evidence/change-policy.md — Beacon SaaS change management policy defining peer review, staging testing, CAB approval, deployment windows, and emergency change procedures.
evidence/deployment-log-q4.csv — Complete deployment log for Q4 2025 covering 20 changes with approval chains, test results, rollback events, and deployment outcomes.
evidence/automated-test-results.json — CI/CD automated test results for all Q4 2025 deployments, including pass rates, flaky test tracking, and test suite metadata.
evidence/capacity-planning-q4.md — Infrastructure capacity planning report for Q4 2025. Not relevant to CC8.1 change management controls. [noise]

Findings

ID	Type	Severity	Finding
F-001	gap	medium	CHG-414 deployed to production with 94.2% test pass rate, below the 98% policy threshold Deployment CHG-414 (Real-time analytics pipeline migration to Kafka) proceeded to production with a 94.2% automated test pass rate, which is below the 98% overall pass rate required by Section 4.3 of ...
F-002	gap	medium	CHG-415 approved by Engineering Manager instead of CAB, with no formal delegation authority Deployment CHG-415 (Onboarding wizard flow redesign), classified as Medium risk, was approved by Jared Kim (Engineering Manager) instead of the Change Advisory Board as required by Section 4.4 of the ...
F-003	gap	medium	Two rollbacks in Q4 2025 triggers the escalation threshold defined in policy metrics Q4 2025 saw two production rollbacks: CHG-406 (API rate limiting, rolled back October 29 due to latency threshold breach) and CHG-418 (Kubernetes node pool scaling, rolled back December 17 due to pod ...
F-004	gap	low	Emergency hotfix CHG-409 bypassed staging testing — retrospective CAB review completed within SLA CHG-409 (Critical XSS vulnerability hotfix for CVE-2025-41823) was deployed as an emergency change on November 6 at 22:15 ET, bypassing staging environment testing and the standard deployment window. ...
F-005	gap	medium	Flaky test remediation SLA exceeded by over 100 days for Kafka integration tests The automated test results show that 3 flaky tests (FLAKY-028, FLAKY-029, FLAKY-030) identified on September 14, 2025 have been open for 109 days, far exceeding the 30-day remediation SLA defined in t...

Results

Model	Provider	Score	Recall	Prec.	F1	Gaps	Reported
Sonnet 4.6	Anthropic	83%	100%	71%	83%	5/5	7
Opus 4.7	Anthropic	91%	100%	83%	91%	5/5	6
GPT-5.5	OpenAI	73%	80%	67%	73%	4/5	6
GPT-4.1	OpenAI	71%	100%	56%	71%	5/5	9
Haiku 4.5	Anthropic	43%	100%	28%	43%	5/5	18
GPT-4o	OpenAI	80%	80%	80%	80%	4/5	5