Risk & ML Models: Stress, Scenario Testing & Evaluation

Risk & ML Models: Stress, Scenario Testing & Evaluation
NEW COURSE! We are entering a time where the speed at which decisions are made is critical. Ensuring your models work with comprehensive testing and evaluation is key for successful ML projects.  Join PRMIA and QuantUniversity for this 6-week program geared towards practitioners!

Register Now
(You will be redirected to our training partner's registration system.)

Course Experts

CRL Credits

Course Period: March 30 - May 4, 2021
Lessons Launch: 
Each Tuesday beginning March 30, 2021
Instructor Access: 
March 30 - May 11, 2021


Presented by:
Sri Krishnamurthy, CFA
Chief Data Scientist, QuantUniversity

 Session Length:
90 minutes +Labs


About This Course

The use of AI and machine learning in finance has grown significantly in the last few years. As more and more AI and ML applications are being deployed in enterprises, concerns are growing about the increased complexity of models, the growing ecosystem of untested frameworks and products, potential for AI accidents, model and reputation risk.  As the debate about explainability, fairness, bias, and privacy grows, there is increased attention to understanding how the models work and whether the models are designed and  thoroughly tested to address potential issues. 

The growth of data-driven applications have changed the financial industry. AI and ML models have accelerated business transformation, reduced turn-around times and have enabled applications that weren’t feasible just a few years ago. Institutions have ramped up the adoption of ML models and are seeing significant benefits through the growing portfolios of ML based decision making models. While the interest is huge, the challenges of comprehensively testing and evaluating ML models remain. AI accidents and the risks associated with algorithmic decision making is challenging enterprises to innovate and adopt risk management techniques factoring the new realities!

In this QuantUniversity Course, we will discuss the key aspects of risks in ML models and discuss key techniques in stress, scenario testing and evaluation of machine learning models. Through examples and case studies, we will discuss the state-of-the-art in testing and evaluation of ML-based models and how to comprehensively address risk when developing, deploying and monitoring ML applications. By the end of the course, participants will have a clear idea of the challenges, best practices and pragmatic tools that can be used to address risks in machine learning models

Hands-on examples and case studies through QuSandbox will be provided to reinforce concepts.
 Lesson   Topic
 Lesson 1
  Introduction to Machine Learning, AI & Risk
• Machine Learning In Finance: A tour of key methods used In Machine Learning
• Define risks in ML models
• Concept drift, data drift, model drift
• Stress, scenario testing and evaluation
• Key metrics
• The role of algorithmic audits for ML models

 Lesson 2
  Stress Testing and Scenario Generation
• How are AI/ML models different from traditional models?
• Scenario stress testing
• Reverse stress testing
 o Identifying and assessing tail risk scenarios
 o Scenario generation
 o Role of synthetic data and data augmentation
• The ML life cycle and risks

 Lesson 3
  Metrics and Evaluation for Risk in ML Models
• Metrics for quantifying risks In ML models
• Working with sensitive data
• Detecting data leakage
• Quantifying risk and metrics for ML models
• Monitoring and retuning/retraining
• ML risk reporting
Case Study: A Dashboard for Measuring and Evaluating Risk In ML Models

 Lesson 4    Anomalies and Outliers
• Detecting and addressing anomalies
• Explainability and outlier analysis
• Methods for generating and testing for anomalies
• Checks for plausibility
• Data techniques and ensemble methods to address anomalies
Case Study: Anomaly Detection in Time-series Datasets Using GANS

 Lesson 5    Model Validation of Machine Learning Models
• Verification vs Validation of ML models
• Benchmarking ML models
• Challenger models
• Backup models
• Issues when adopting ML models:
  o Model selection challenges
  o Interpretability and explainability
Case Study: Validating a ML Model for Credit Risk

 Lesson 6    Frontier Topics & Wrap-up
• Operationalizing Evaluation Of Risk In ML Models.
    o Real-time and near-real time risk evaluation 
    o Architecture choices for scaling risk calculations
    o Issues with integrating traditional and ML models
    o Governance mechanisms to address risk in ML models
    o Algorithm auditing and issues of bias and fairness
    o Adversarial attacks, sensitive data and unknown risks
• Frontier Topics
    o Deep learning and other ML innovations
    o Technologies and trends to look out for

Who Should Attend

Risk Professionals, Model Validators, Model Auditors, Data Scientists, ML Professionals, ML Ops and Software professionals.

About Our Experts

  Sri Krishnamurthy is the founder of QuantUniversity, a data and Quantitative Analysis Company and the creator of the Analytics Certificate program and Fintech Certificate program. Sri has more than two decades of experience in analytics, quantitative analysis, statistical modeling and designing large-scale applications.

Prior to starting QuantUniversity, Sri has worked at Citigroup, Endeca, MathWorks and with more than 25 customers in the financial services and energy industries. He has trained more than 1000 students in quantitative methods, analytics and big data in the industry and at Babson College, Northeastern University and Hult International Business School.

Sri earned an MS in Computer Systems Engineering and another MS in Computer Science, both from Northeastern University and an MBA with a focus on Investments from Babson College.

  About Our Training Partner: QuantUniversity is a quantitative analytics advisory focusing on the intersection of Data science, Machine learning and Quantitative Finance. We take a practitioner’s approach to working with pragmatic applications of frontier topics to real-world financial and energy problems. QuantUniversity advises various companies in Quant Finance application development, validation and in algorithmic auditing. We also run data science and machine learning workshops in the United States and online in its Explore-Experience-Excel series through QuAcademy. QuantUniversity is pioneering the next generation platform for Algorithmic auditing that supports anonymization, model escrow and tracking, synthetic data generation and experimentation through the QuSandbox.


Continued Risk Learning Credits:9

PRMIA Continued Risk Learning (CRL) programs provide you with the opportunity to formally recognize your professional development, documenting your evolution as a risk professional. Employers can see that you are not static, making you a highly valued, dynamic, and desirable employee. The CRL program is open to all Contributing, Sustaining, and Risk Leader members, providing a convenient and easily accessible way to submit, manage, track and document your activities online through the PRMIA CRL Center. To request CRL credits, please email learning@prmia.org.

 Membership Type Price
 PRMIA Network
 Non Member $799.00

QUANTUNIVERSITY Registration Policies
This course is being delivered by a PRMIA Training Partner, QuantUniversity. All registrations, payments, and course operations will be managed exclusively by QuantUniversity. When registering for the course, you will leave PRMIA.org and use QuantUniversity's registration system, proprietary learning management system, and QuSandbox for labs. Questions and requests should be directed to QuantUniversity at info@qusandbox.com.

Please review QuantUniversity's cancellation and refund policy at the time of purchase, as their policies supersede PRMIA's registration policies.  

Need Support? 
Contact our training partner for questions about the course, group registrations and technical support info@qusandbox.com.

Analytics for a cause initiative - Thank you QuantUniversity for supporting future risk leaders!
QuantUniversity is a proud sponsor and contributor to educational scholarships, valued at $30,000, to students from eight countries and 12 chapters, participating in the PRMIA Risk Management Challenge for 2020 and 2021!  Learn more about this and other outreach supported by QuantUniversity.

If this is your first time accessing the PRMIA website you will need to create a short user profile to register. Save on registration by becoming a member.

PRMIA Network Fee: $699.00 using discount code: PRMIA100
(You will be redirected to our training partner's registration system.)

Register Now

3/30/2021 - 5/4/2021
Virtual Course

Sign In to Register for Event


Contact Us

Looking to further your career?

Become a Member

Sign Up for Mailing List


Contact Us

Looking to further your career?

Become a Member

Sign Up for Mailing List