Risk & ML Models: Stress, Scenario Testing & Evaluation

The use of AI and machine learning in finance has grown significantly in the last few years. As more and more AI and ML applications are being deployed in enterprises, concerns are growing about the increased complexity of models, the growing ecosystem of untested frameworks and products, potential for AI accidents, model and reputation risk. As the debate about explainability, fairness, bias, and privacy grows, there is increased attention to understanding how the models work and whether the models are designed and thoroughly tested to address potential issues.

The growth of data-driven applications have changed the financial industry. AI and ML models have accelerated business transformation, reduced turn-around times and have enabled applications that weren’t feasible just a few years ago. Institutions have ramped up the adoption of ML models and are seeing significant benefits through the growing portfolios of ML based decision making models. While the interest is huge, the challenges of comprehensively testing and evaluating ML models remain. AI accidents and the risks associated with algorithmic decision making is challenging enterprises to innovate and adopt risk management techniques factoring the new realities!

In this course, learn:

• The key aspects of risks in ML models
• Key techniques in stress, scenario testing and evaluation of machine learning models
• Through examples and case studies, learn state-of-the-art in testing and evaluation of ML-based models and how to comprehensively address risk when developing, deploying, and monitoring ML applications

By the end of the course, participants will have a clear idea of the challenges, best practices, and pragmatic tools that can be used to address risks in machine learning models.

Hands-on examples and case studies through QuSandbox will be provided to reinforce concepts.

This course is a part of the QuantUniversity Machine Learning and AI Risk Certificate Program.

 Take advantage of additional discounts by enrolling in the Certificate program.

Introduction to Machine Learning, AI and Risk

•  Machine Learning In Finance : A Tour Key Methods Used In Machine Learning
• Defining Risk In ML Models
• Concept Drift, Data Drift, Model Drift
• Stress, Scenario Testing & Evaluation
• Key Metrics
• The Role Of Algorithmic Auditors For ML Models
• Motivation: Case Study Covid -19
• Scenario Generation And Testing With Synthetic Data

Stress Testing and Scenario Generation

• How Are AI/ML Models Different From Traditional Models?
• Scenario Stress TestingReverse Stress Testing
• Scenario GenerationThe ML Life Cycle And Risks

• Case Study: Stress Testing Of An ML-Based Forecasting Model Under Different Regimes

 Module 3

Metrics and Evaluation for Risk in ML Models

• Metrics For Quantifying Risks In ML Models
• Working With Sensitive Data
• Detecting Data Leakage
• Quantifying Risk & Metrics For ML Models
• Monitoring And Retuning/Retraining
• ML Risk Reporting

• Case Study: A Dashboard For Measuring And Evaluating Risk In ML Models

Anomolies and Outliers

• Detecting And Addressing Anomalies
• Explainability & Outlier Analysis
• Methods For Generating And Testing For Anomalies
• Checks For Plausibility
• Data Techniques And Ensemble Methods To Address Anomalies
• Case Study: Anomaly Detection In Time-Series Datasets Using GANS

Model Validation and ML Models

• Verification vs Validation of ML Models
• Benchmarking ML Models
• Challenger Models
• Backup Models
• Issues When Adopting Machine Learning Models
  • Model Selection Challenges
  • Interpretability And Explainability
• Case Study: Validating An ML Model For Credit Risk
  
  

 Module 6

 Frontier Topics and Wrap-up

• Operationalizing Evaluation Of Risk In ML Models
  • Real-Time & Near-Realtime Risk Evaluation
  • Architecture Choices for Scaling Risk Calculations
  • Issues With Integrating Traditional and ML Models
  • Governance Mechanisms to Address Risk in ML Models
  • Algorithm Auditing & Issues of Bias and Fairness
  • Adversarial Attacks, Sensitive Data and Unknown Risks
• Frontier Topics
  • Deep Learning and Other ML Innovations
  • Technologies and Trends to Watch

• Risk professionals
• Model Validators
• Model Auditors
• ML engineers and Software engineers involved in ML and AI deployment
• Data Scientists

Sri Krishnamurthy, CFA, CAP is the founder of QuantUniversity, a data and quantitative analysis company. His experience includes analytics, quantitative analysis, statistical modeling and software development. He is a quantitative specialist with significant experience in designing data mining and analytic systems for some of the world’s largest asset management and financial companies.

Sri has worked at MathWorks as a Computational Finance Consultant where he worked with customers providing asset management, energy analytics, risk management and trading solutions. Prior to that, Sri was a consultant at Endeca (now Oracle) in their Analytics Group and at Citigroup in their Fixed-Income Group building large-scale analytical and trading systems.

Sri is a Charted Financial Analyst and a Certified Analytics Professional.  He is an active member of the Boston Security Analysts society and QWAFAFEW. He is the creator of the Fintech Certificate Program & Analytics Certificate Program and teaches graduate courses in Quantitative methods, Data science and Analytics and Big Data at Babson College, Northeastern University and Hult International Business School.