Case Study: Evaluating training effectiveness of Managerial Development programs for Capability Development Function

Client’s Pain Points

The Client’s Capability Development function wanted to evaluate the training effectiveness of the Managerial Development program focused of shaping emerging leaders & building managerial & leadership pipeline across the organization. (Also see, Why Leadership training fails & what to do about it, HBR, 2016)

Training Effectiveness evaluation & Return on Investment

The Managerial development programs served the purpose to develop a defined set of competencies, a critical success factor in the Client’s organization for emerging leaders to succeed & thrive in their managerial roles. Being a strategic part of overall talent management effectiveness, the organization has invested significant resources to this flagship program, & though it had received positive feedback & strong validations from participants, the HR Leadership wanted to quantify its training effectiveness and value.

The key objectives that were mandated to our team from HR Leadership were:

• Using Evidence based approach & HR Analytics (also see, HR Analytics Case Studies), objectively determine the training effectiveness and impact of program – with an estimated training roi of the program.
• Ascertain a prescriptive strategy to maximize the impact, & target the program to those who would benefit most from it.

The Approach

Hypothesis Refinement, Strategic Alignment & Internal Benchmarking

To get to the above objectives with precision, it is critical that we understand & internalize the critical success factors, it’s taxonomies & standardized definitions, that were most critical to the Managerial development program. Hence, we started deliberate conversations & interviews with the Business Leadership, the GMs & with executives from Human Capital and Learning & Development functions.  

To fine tune the hypothesis, we profiled & interviewed more than critical mass of senior leaders – this helped us to benchmark & refine the critical managerial success factors, most important for promotion & success in the organization. In addition to this, we administered focus group discussions across 4 critical buckets – Designated Managers who came from the MDP pipeline, Designate Managers who were lateral hires & did not take the MDP program, Emerging Managers who are already enrolled in the program & Alumni of MDP program who have not been designated as managers. These sessions helped to highlight issues and questions specific to organization and to customize language to better ‘resonate’ with participants.

The key is Self – Efficacy: Self-Efficacy Evaluation Measure

Self-efficacy, a validated psychological construct that gauges the confidence, a person has in their ability to do a particular thing. Self-efficacy has consistently been shown in research to be, after intelligence, the best predictor of achievement within many domains.

Based on our research, we defined a number of critical factors that could reasonably predict the likelihood of potential managers applying for, and successfully achieving, senior roles in organization. Basis this construct, we developed a questionnaire to measure changes in participants’ self-efficacy during and after the program.

The Pilot

To improve the validity & reliability of the developed questionnaire & hypothesis, we conducted a pilot exercise with a sample subset of the target group. Not only did this add to the validation of rigorous processes we deployed, but it as well helped gather valuable feedback & improvise the measurement tools.

Evaluation & Control groups

To gauge the effectiveness of developed measurement tools, we ran a pre-program measurement exercise (with approx. 45 participants), with an identical sized control group to ascertain the self-efficacy scores. A similar exercise was administered on these groups – 6 months post the completion of MDP program.

The Result

There were 3 critical findings from the statistical analysis of Training Effectiveness evaluation:

• Our analysis indicated that 55% of the participants benefited significantly (very high training effectiveness) & around 28% of participants benefited at baseline acceptable levels (moderate training effectiveness)
• Participants in the program generally increased their self-efficacy on some of the critical factors (Bucket A – Designated Managers who came from the MDP pipeline) compared to those in the control group, however this only partly met the organization’s expectations
• The most beneficiary segment was potential managers with less than 2 years’ tenure in the organization; and potential managers who were very low in Self-Efficacy score before the program commenced.
• Both, Potential & Designated Managers outside of the categories mentioned above benefited only minimally or not at all from participating.