Certified AI Assurance Practitioner

Covers the technical foundations an assurance practitioner needs to critically evaluate ML systems, including model families, training pipelines, and the statistical basis of evaluation. Emphasis is on knowing where technical claims can be overstated or misapplied.

• ▪Supervised, unsupervised, and generative model families
• ▪Training/validation/test splits and cross-validation
• ▪Overfitting, underfitting, and regularization
• ▪Feature engineering and data leakage risks
• ▪Statistical significance and confidence intervals for metrics
• ▪Foundation models, fine-tuning, and transfer learning
• ▪Limitations of benchmark datasets

Focuses on rigorous quantitative evaluation of model performance, including metric selection, calibration, and benchmark design. Practitioners learn to distinguish genuine capability improvements from benchmark leakage or cherry-picked results.

• ▪Classification metrics: precision, recall, F1, AUC-ROC, AUC-PR
• ▪Calibration and reliability diagrams
• ▪Regression and generative model evaluation metrics
• ▪Benchmark contamination and data leakage detection
• ▪Human evaluation protocols and inter-rater reliability
• ▪Out-of-distribution and stress-test evaluation design
• ▪Comparative benchmarking methodology and reporting standards
• ▪Statistical testing for model comparison

Addresses quantitative fairness auditing across the ML lifecycle, from training data representativeness through outcome disparities. Candidates learn the mathematical tradeoffs among competing fairness definitions and how to select metrics appropriate to context.

• ▪Sources of bias: sampling, label, and measurement bias
• ▪Demographic parity, equalized odds, and predictive parity
• ▪Impossibility results among fairness metrics
• ▪Disparate impact analysis and the four-fifths rule
• ▪Bias mitigation: pre-processing, in-processing, post-processing techniques
• ▪Intersectional and subgroup fairness auditing
• ▪Fairness in generative and LLM outputs

Covers adversarial and stress-testing methods for classical models and large language models, including structured red-team exercises. Emphasis is placed on repeatable methodology rather than ad hoc probing.

• ▪Adversarial examples and perturbation-based attacks
• ▪Prompt injection and jailbreak techniques
• ▪Data extraction and membership inference attacks
• ▪Red-team planning, scoping, and reporting frameworks
• ▪Automated red-teaming and adversarial test generation
• ▪Robustness metrics under distribution shift
• ▪Content safety and harmful-output classification testing

Focuses on the documentation practices that make assurance findings durable and auditable, including model cards, system cards, and dataset documentation. Candidates learn to evaluate documentation for completeness against recognized templates.

• ▪Model cards: structure and required elements
• ▪System cards for compound and agentic systems
• ▪Datasheets for datasets and data statements
• ▪Intended use, out-of-scope use, and known limitations disclosure
• ▪Version control and change logs for models and datasets
• ▪Documentation for third-party and vendor-supplied models

Combines operational monitoring practices with formal governance frameworks. Candidates must connect technical drift-detection pipelines to the accountability structures required by NIST AI RMF, ISO/IEC 42001, and the EU AI Act.

• ▪Data drift and concept drift detection methods
• ▪Production performance monitoring and alerting thresholds
• ▪Incident response and model rollback procedures
• ▪NIST AI RMF 1.0: Govern, Map, Measure, Manage functions
• ▪ISO/IEC 42001 AI management system requirements
• ▪EU AI Act risk tiers and conformity assessment obligations
• ▪US executive-branch AI guidance and federal agency requirements
• ▪Third-party/vendor AI risk assessment