Model Risk Management for Machine Learning Models

The financial industry has been adopting AI and machine learning at a rapid pace. Alternative datasets including text analytics, cloud computing, and algorithmic trading are game changers for many firms exploring novel modeling methods to augment their traditional investment and decision workflows. In this course, we aim to bring clarity to some of the model risk management and validation challenges with data science and machine learning models in the enterprise.


 

Register Now
Note: You will leave PRMIA.org and be taken
to QuantUniversity's registration site. 
Use code: PRMIA100

Agenda

Course Experts

CRL Credits

 Course Access:
90-day course access from date of purchase

Time:
On-demand, Self-study

Instructor:
Sri Krishnamurthy, CFA, CAP
Founder & CEO, QuantUniversity


 Length/Duration:
7 Modules, 1.5 hours per module

 


About This Course
 

The financial industry has been adopting AI and machine learning at a rapid pace. Alternative datasets including text analytics, cloud computing, and algorithmic trading are game changers for many firms exploring novel modeling methods to augment their traditional investment and decision workflows. As more and more open-source technologies penetrate enterprises, quants and data scientists have a plethora of choices for building, testing and scaling models. While there is significant enthusiasm, model risk professionals and risk managers are concerned about the onslaught of new technologies, programming languages, and data sets that are entering the enterprise. With little formal guidance from regulators on how to validate models and quantify model risk, organizations are developing their own home-cooked methods to address model risk management challenges.

In this course, we aim to bring clarity to some of the model risk management and validation challenges with data science and machine learning models in the enterprise. We will discuss key drivers of model risk in today’s environment and how the scope of model risk management is changing. We will introduce key concepts and discuss aspects to be considered when developing a model risk management framework incorporating data science techniques and machine learning methodologies in a pragmatic way.

Learning Objectives 

Upon completion of this course, you will be able to:

  • Describe the role of Machine Learning and AI in financial services
  • Discuss Model Risk Management challenges and best practices for machine learning models
  • Validate machine learning models: Quantifying risk, best practices and templates
  • Understand the regulatory guidance and the future
  • Experience practical case studies with sample code

 

This course is a part of the QuantUniversity Machine Learning and AI Risk Certificate Program.

 Take advantage of additional discounts by enrolling in the Certificate program.

 
Outline
 Lesson   Topic
 Module 1              
       
 

Machine Learning and AI: A Model Risk Perspective

  • Drivers of Model Risk in the age of data science and AI
  • Machine Learning vs Traditional quant models
  • How has the world changed?
  • A tour of Machine Learning and AI methods
  • Supervised vs Unsupervised Learning (Regression, Neural Networks, XGBoost, PCA, Clustering)
  • Deep Learning & Reinforcement Learning (Keras, Tensorflow, PyTorch)
  • Automatic Machine Learning & Machine Learning APIs (Google, Comprehend, Watson)
  • ML on the cloud vs On-prem
  • Models redefined: Data, Modeling environment, Modeling tools, Modeling process 
 Module 2  

Model Risk Management for Machine Learning Models-Part 1

  • ML Life cycle management
  • Tracking
  • Metadata management
  • Scaling
  • Reproducibility
  • Interpretability
  • Testing
  • Measurement

The Decalogue: Ten key aspects to factor when developing your model risk management framework when integrating Machine Learning models

  •  Models redefined: It’s not just input, process and output
  • Governing the Machine Learning process
  • Model Verification and Validation for Machine Learning Models
  • Performance Metrics and Evaluation criteria
  • Model Inventory and tracking
 Module 3  

Model Risk Management for Machine Learning Models-Part 2

The Decalogue (cont’d)

  • Integrating Data Governance and Model Governance
  • Development Models vs Production Models
  • Fairness, Reproducibility, Auditability, Explainability, Interpretability & Bias
    • How do we objectively measure these?
    • Review of the Apple-Goldman Sachs credit card debacle
  • Machine Learning options and considerations
    • AutoML (Data Robot, H20.ai, etc.), ML as a service (Google, Comprehend, Watson) and home-cooked custom models
  • ML and Governance: Roles and Responsibilities redefined

Managing models in the day of Covid19

  • Perspectives on point-forecasts, validation and fat-tail
 Module 4  

Pragmatic Model Risk Management for AI/ML models

  • Challenges and best practices for pragmatic model management within the enterprise
  • Working with open source projects
  • Working with vendor models and machine learning APIs
  • Quantifying model risk for machine learning models
  • Model risk management for deep-learning models
  • Validation criteria and best practices
  • Templates for Model Validation for machine learning models
  • Synthetic data for Model Risk Management
  • Use of Synthetic datasets
 Module 5  

Hands-on Case Study

Learn from the past: How does Supervised machine learning work?

  • Validating a Credit-risk machine learning model: A case study illustrating a model validation of a credit risk model involving machine learning
  • Working with Regression, Neural Networks, and Random Forest models
  • Development models vs Production models
    • Sample templates and worksheets will be provided
  • Roadmap for the MRM team to upskill and keep abreast of changes in the AI and ML landscape
  • Training, education, and expectation setting - Future outlook: Regulation, Sandboxes, Frameworks
  • Review of recent regulatory efforts
  • How should companies proactively plan for changes and the future?

 Optional
 Module 6

   Capstone Project-Part 1: Scoping and design
Put your newly learned skills to practice while being mentored through the process. Participants will go through an exercise to perform model validation on a machine learning model of their choice.
 Optional
 Module 7
   Capstone Project-Part 2: Demonstrate your skills
Participants will demonstrate their findings and obtain feedback from instructors and/or industry participants.


Who Should Attend

 

  • Fundamental and quantitative analysts, risk and investment professionals, portfolio managers new to data science and machine learning
  • Financial professionals new to data-driven methodologies
  • Machine learning enthusiasts interested in use cases in fintech and financial organizations


About Our Expert

  
 

Sri Krishnamurthy, CFA, CAP is the founder of QuantUniversity, a data and quantitative analysis company. His experience includes analytics, quantitative analysis, statistical modeling and software development. He is a quantitative specialist with significant experience in designing data mining and analytic systems for some of the world’s largest asset management and financial companies.

Sri has worked at MathWorks as a Computational Finance Consultant where he worked with customers providing asset management, energy analytics, risk management and trading solutions. Prior to that, Sri was a consultant at Endeca (now Oracle) in their Analytics Group and at Citigroup in their Fixed-Income Group building large-scale analytical and trading systems.

Sri is a Charted Financial Analyst and a Certified Analytics Professional.  He is an active member of the Boston Security Analysts society and QWAFAFEW. He is the creator of the Fintech Certificate Program & Analytics Certificate Program and teaches graduate courses in Quantitative methods, Data science and Analytics and Big Data at Babson College, Northeastern University and Hult International Business School.


Continued Risk Learning Credits:  7.5

PRMIA Continued Risk Learning (CRL) programs provide you with the opportunity to formally recognize your professional development, documenting your evolution as a risk professional. Employers can see that you are not static, making you a highly valued, dynamic, and desirable employee. The CRL program is open to all Contributing, Sustaining, and Risk Leader members, providing a convenient and easily accessible way to submit, manage, track and document your activities online through the PRMIA CRL Center. To request CRL credits, please email [email protected].

Registration
 Membership Type Price
   
PRMIA Network - Use Promo Code: PRMIA100 $ 599.00
General Public $ 699.00

 

QuantUniversity Registration Policies

This course is being delivered by a PRMIA Training Partner, QuantUniversity. All registrations, payments, and course operations will be managed exclusively by QuantUniversity.

When registering for the course, you will leave PRMIA.org and use QuantUniversity's registration system, proprietary learning management system, and QuSandbox for labs.

Questions and requests should be directed to QuantUniversity at: [email protected].

Please review QuantUniversity's cancellation and refund policy at the time of purchase, as their policies supersede PRMIA's policies.

  

Need Support?

Contact our training partner for questions about the course, group registrations, and technical support at: [email protected]

Access

Immediate access to the course is granted for 90 days after your purchase.

If this is your first time accessing the PRMIA website you will need to create a short user profile to register. Save on registration by becoming a member.

 

Register Now

 

Thank you to our sponsors, including:


Questions?

Contact Us


Looking to further your career?

Become a Member

Sign Up for Mailing List