Risk & ML Models: Stress, Scenario Testing & Evaluation

Institutions have ramped up the adoption of ML models and are seeing significant benefits through the growing portfolios of ML based decision making models. While the interest is huge, the challenges of comprehensively testing and evaluating ML models remain. 

 

Register Now
Note: You will leave PRMIA.org and be taken
to QuantUniversity's registration site. 
Agenda

Course Experts

CRL Credits

 Course Access:
90-day course access from date of purchase

Time:
Self-study, self-paced

Instructor:
Sri Krishnamurthy, CFA, CAP
Founder & CEO, QuantUniversity


 Length/Duration:
6 Modules, 1.5 hours per module

 


About This Course
 

The use of AI and machine learning in finance has grown significantly in the last few years. As more and more AI and ML applications are being deployed in enterprises, concerns are growing about the increased complexity of models, the growing ecosystem of untested frameworks and products, potential for AI accidents, model and reputation risk. As the debate about explainability, fairness, bias, and privacy grows, there is increased attention to understanding how the models work and whether the models are designed and thoroughly tested to address potential issues.

The growth of data-driven applications have changed the financial industry. AI and ML models have accelerated business transformation, reduced turn-around times and have enabled applications that weren’t feasible just a few years ago. Institutions have ramped up the adoption of ML models and are seeing significant benefits through the growing portfolios of ML based decision making models. While the interest is huge, the challenges of comprehensively testing and evaluating ML models remain. AI accidents and the risks associated with algorithmic decision making is challenging enterprises to innovate and adopt risk management techniques factoring the new realities!

In this course, learn:

  • The key aspects of risks in ML models
  • Key techniques in stress, scenario testing and evaluation of machine learning models
  • Through examples and case studies, learn state-of-the-art in testing and evaluation of ML-based models and how to comprehensively address risk when developing, deploying, and monitoring ML applications

By the end of the course, participants will have a clear idea of the challenges, best practices, and pragmatic tools that can be used to address risks in machine learning models.

Hands-on examples and case studies through QuSandbox will be provided to reinforce concepts.

 

This course is a part of the QuantUniversity Machine Learning and AI Risk Certificate Program.

 Take advantage of additional discounts by enrolling in the Certificate program.

 
Outline
 Lesson   Topic
 Module 1  

Introduction to Machine Learning, AI and Risk

  •  Machine Learning In Finance : A Tour Key Methods Used In Machine Learning
  • Defining Risk In ML Models
  • Concept Drift, Data Drift, Model Drift
  • Stress, Scenario Testing & Evaluation
  • Key Metrics
  • The Role Of Algorithmic Auditors For ML Models
  • Motivation: Case Study Covid -19
  • Scenario Generation And Testing With Synthetic Data

 Module 2  

Stress Testing and Scenario Generation

  • How Are AI/ML Models Different From Traditional Models?
  • Scenario Stress TestingReverse Stress Testing
  • Scenario GenerationThe ML Life Cycle And Risks
  • Case Study: Stress Testing Of An ML-Based Forecasting Model Under Different Regimes

 Module 3

 

Metrics and Evaluation for Risk in ML Models

  • Metrics For Quantifying Risks In ML Models
  • Working With Sensitive Data
  • Detecting Data Leakage
  • Quantifying Risk & Metrics For ML Models
  • Monitoring And Retuning/Retraining
  • ML Risk Reporting
  • Case Study: A Dashboard For Measuring And Evaluating Risk In ML Models

 Module 4  

Anomolies and Outliers

  • Detecting And Addressing Anomalies
  • Explainability & Outlier Analysis
  • Methods For Generating And Testing For Anomalies
  • Checks For Plausibility
  • Data Techniques And Ensemble Methods To Address Anomalies
  • Case Study: Anomaly Detection In Time-Series Datasets Using GANS

 

 Module 5  

Model Validation and ML Models

  • Verification vs Validation of ML Models
  • Benchmarking ML Models
  • Challenger Models
  • Backup Models
  • Issues When Adopting Machine Learning Models
    • Model Selection Challenges
    • Interpretability And Explainability
  • Case Study: Validating An ML Model For Credit Risk

 Module 6

 

 Frontier Topics and Wrap-up

  • Operationalizing Evaluation Of Risk In ML Models
    • Real-Time & Near-Realtime Risk Evaluation
    • Architecture Choices for Scaling Risk Calculations
    • Issues With Integrating Traditional and ML Models
    • Governance Mechanisms to Address Risk in ML Models
    • Algorithm Auditing & Issues of Bias and Fairness
    • Adversarial Attacks, Sensitive Data and Unknown Risks
  • Frontier Topics
    • Deep Learning and Other ML Innovations
    • Technologies and Trends to Watch


Who Should Attend

 

  • Risk professionals
  • Model Validators
  • Model Auditors
  • ML engineers and Software engineers involved in ML and AI deployment
  • Data Scientists


About Our Expert

  
 

Sri Krishnamurthy, CFA, CAP is the founder of QuantUniversity, a data and quantitative analysis company. His experience includes analytics, quantitative analysis, statistical modeling and software development. He is a quantitative specialist with significant experience in designing data mining and analytic systems for some of the world’s largest asset management and financial companies.

Sri has worked at MathWorks as a Computational Finance Consultant where he worked with customers providing asset management, energy analytics, risk management and trading solutions. Prior to that, Sri was a consultant at Endeca (now Oracle) in their Analytics Group and at Citigroup in their Fixed-Income Group building large-scale analytical and trading systems.

Sri is a Charted Financial Analyst and a Certified Analytics Professional.  He is an active member of the Boston Security Analysts society and QWAFAFEW. He is the creator of the Fintech Certificate Program & Analytics Certificate Program and teaches graduate courses in Quantitative methods, Data science and Analytics and Big Data at Babson College, Northeastern University and Hult International Business School.


Continued Risk Learning Credits:  9

PRMIA Continued Risk Learning (CRL) programs provide you with the opportunity to formally recognize your professional development, documenting your evolution as a risk professional. Employers can see that you are not static, making you a highly valued, dynamic, and desirable employee. The CRL program is open to all Contributing, Sustaining, and Risk Leader members, providing a convenient and easily accessible way to submit, manage, track and document your activities online through the PRMIA CRL Center. To request CRL credits, please email [email protected]g.

Registration
 Membership Type Price
   
PRMIA Network - Use Promo Code: PRMIA100 $ 599.00
General Public $ 699.00

 

QuantUniversity Registration Policies

This course is being delivered by a PRMIA Training Partner, QuantUniversity. All registrations, payments, and course operations will be managed exclusively by QuantUniversity.

When registering for the course, you will leave PRMIA.org and use QuantUniversity's registration system, proprietary learning management system, and QuSandbox for labs.

Questions and requests should be directed to QuantUniversity at: [email protected].

Please review QuantUniversity's cancellation and refund policy at the time of purchase, as their policies supersede PRMIA's policies.

  

Need Support?

Contact our training partner for questions about the course, group registrations, and technical support at: [email protected]

Access

Immediate access to the course is granted for 90 days after your purchase.

If this is your first time accessing the PRMIA website you will need to create a short user profile to register. Save on registration by becoming a member.

 

Register Now

 

Thank you to our sponsors, including:


Questions?

Contact Us


Looking to further your career?

Become a Member

Sign Up for Mailing List