Skip to main content
European Commission logo
EURAXESS Researchers in motion

CD Policy Recommendations - Measuring impact

Measuring the impact of researchers career development actions


Traditionally, within the researcher development community, most developers tend to be very much focused on developing innovative and successful activities for researchers within their institutions, and have little time to focus on evaluation activities.

The outcomes and impact of trainings and professional development actions addressing different skills and capabilities are very varied and range from very direct impact (e.g., satisfaction with the content of the course of participants) to more indirect, meaningful and medium or long-term impact (e.g. actual change in behaviour leading to better professional performance and outcomes), which, to make things more complex, may even be affected by many other factors.

 

 

Why Measure Impact?

Why would it be useful for the different stakeholders to share a common impact framework project?

For Researchers

Researchers have a leading role in successful professional development strategies. There is a general agreement that researchers need to take an active role and become responsible for developing their own professional development and careers. This implies there are a number of drivers for researchers supporting impact frameworks:

  • By participating in evaluations of research development activities and strategies, researchers can influence future policies put in place after the evaluation of impact data.
  • Assuming impact analysis in their own self–assessments researchers can monitor their own progress against their personal professional goals.
 

 

 

 

 

This model based in four levels of impact was designed by Donald Kirkpatrick in the 50´s in order to structure the way in which impact could be measured in a sequenced way.

Level 1 – Reaction: solicits opinions of the learning experience following a training event or course. This includes questions such as:

  • Did the trainees feel that the training was worth their time?
  • Did they think that it was successful?
  • What were the biggest strengths of the training, and the biggest weaknesses?
  • Did they like the venue and presentation style?
  • Did the training session accommodate their personal learning styles?

Level 2 – Learning: measures the degree to which participants acquired the intended knowledge, skills and attitudes as a result of the training:

  • Identifying what you want to evaluate (i.e., the things that could change: knowledge, skills, or attitudes.)
  • Measuring these areas identified both before and after training.
  • Once training is finished, test your trainees a second time to measure what they have learned, or measure learning with interviews or verbal assessments.

Level 3 – Behavior: measures the degree to which participants’ behaviors change as a result of the training – basically whether the knowledge and skills from the training are then applied on the job:

  • Did the trainees put any of their learning to use?
  • Are trainees able to teach their new knowledge, skills, or attitudes to other people?
  • Are trainees aware that they've changed their behavior?

Level 4 – Results: seeks to determine the tangible results of the training such as:

  • Reduced cost
  • Improved quality and efficiency
  • Increased productivity
  • Employee retention
  • Increased sales
  • Higher morale

 

 

 

The TOTADO framework establishes four basic levels at which impact needs to be assessed, differing in the unit of analysis

Individual outcomes: the trainee is the unit of analysis, and the dimensions to be measured include:

  • Affective (e.g. participant reactions to training, motivation, self-efficacy, attitudes, mental well-being)
  • Cognitive (e.g., verbal knowledge, knowledge representation)
  • Behavioural (e.g., off-the job task performance, on-the-job task performance)
  • Physical (e.g. health, fitness, injuries)
  • Instrumental (e.g., events, actions or status changes resulting from participation in training and development activities that are work intrinsic (increased job control), work extrinsic (pay rise, promotion) or work relational (forming new relationships or networks)

Team outcomes: the team or group is the unit of analysis, and the dimensions to be measured include:

  • Affective (e.g. changes in average team identity or trust)
  • Cognitive (verbal knowledge, knowledge representation in terms of shared cognition)
  • Behavioural (team task processes, intra-team processes)
  • Instrumental (events, actions or status changes for the team as a whole such as increased team autonomy or gaining team bonuses)

Organizational outcomes: although many organizational outputs can be obtained from aggregating the individual and team level outputs, some dimensions can only be measured at the level of the organization:

  • Financial (e.g., turnover, profit, share price)
  • Outputs (e.g., quantity, quality, variety of components, products or services)
  • Processes (e.g., time to complete tasks, communication system efficiency, or any other aspect informing how well the organization works)
  • Resources (human or non-human)

Societal outcomes: impact of activity beyond the organizational level, and thus, refers to dimensions related to an area or group outside the organization:

  • Economic (e.g., R&D investment in the region, sector or country);
  • Health and Welfare (e.g., work absenteeism in the region, sector or country)
  • Educational (e.g., scientific output in the region, sector or country)
  • Law and Order (e.g., local crime rates in the region, sector or country)
  • Environmental (e.g., pollution levels)

 

 

Read the recommendations, or case studies, or

Back to main policy recommendations page