Case Study - US Legal Firm

Associate Position

Background

The purpose of the case study using Harrison Assessments' was to determine the ability of the Harrison Assessments' software to predict success for this position. The legal firm provided 49 profiles of current employees. Three were eliminated due to consistency scores of less than zero. The low consistency score indicates that the individual either attempted to fool the questionnaire or tried to answer the questions too quickly. Therefore, the total sample population was 46.

All the employees in the sample were rated according to their job performance by supervisors.

Each employee in the sample was rated or each criteria according to the following scale:

90-100
Excellent to Best
80-89
Very Good
70-79
Average to Good
60-69
Below average
50-59
Poor
Below 40
Failed or will soon fail in the position

It was intended to do a blind study in which we split the subjects into two groups. However Group A consisted only of 26 subjects and Group B consisted only of 20 subjects. Thus the sample sizes were too small to conduct such a blind study. However, just for the exercise, Group A was initially used to develop a template and predictions were given for Group B. Despite the very small sample, the predictive results were 65% accurate. The two groups were then used together (an adequate sample size) to develop the template. The original template was reasonably close, however the negative points were too strong, creating too many false negatives. The larger sample enabled adjustments that brought the overall accuracy level up to 89%. The results can be improved by eliminating consistency scores that are below 30. This increased the accuracy level to 93%.

The aim of the research was to determine the exact traits that make Associates successful at the legal firm and to determine the relative accuracy of the system to predict success based upon the 'job template' that incorporates those traits. The template formulates those traits into 'traits to have' as well as 'traits to avoid'. The template is then used to measure future applicants and serve as a developmental guide for current employees. The template is shown by a graph that indicates how a person scores against each of the required traits and then offers a final 'bottom line' score between zero and one hundred that represents the individual's level of 'total suitability' for a particular position. A score of one hundred represents a person who is completely suitable for that position. Assuming the person is eligible for a position (has the education, experience and technical skills), a suitability score of 75 or greater represents a person who has a good probability of performing effectively in that position. A score of 74 or less represents a person who is considered to be unlikely to perform well in that position.

Accuracy level

The results showed a very high predictive accuracy and a strong correlation between the Harrison Assessments' suitability score and the actual job performance. This indicates that the template includes a comprehensive set of traits related to suitability for this position.

1) Predictive accuracy

A prediction is considered accurate if one or more of the following conditions are met:

a) The suitability score is 75 or greater and the performance score is 75 or greater.

b) The suitability score is less than 75 and the performance score is less than 75.

c) The suitability score is within 6 points of the performance score.

The logic behind this definition - If the suitability score were 75 or above and the person were eligible for the position, it would indicate a prediction that the person would probably succeed in the position. If the suitability score were less than 75 and the person was moderately eligible (not highly eligible) for the position then this would indicate a prediction of below average performance. Also, if the suitability score were within 6 points of the performance score it would indicate a very close prediction and thus should be considered accurate. The suitability score for each person appears on the template in the far right-hand column of the table.

From the research sample, 41 of the 46 employees (89%) showed a correlation between the Harrison Assessments' suitability score and performance to be accurate, according to the definition above. A further analysis showed that some of the inaccuracies were related to individuals with low consistency scores. Besides indicating truthfulness, the consistency score also indicates an ability to think clearly. This is probably related to success in this position. It was determined that the cut-off point for consistency for this position should be a minimum of 30 on the consistency score. When this cut-off point was used, the accuracy level rose to 93%.

2) Variation between the suitability ratings and the performance ratings

The degree of variation between the suitability score and the performance ratings also provides an effective means of understanding the accuracy level of the methodology. The variation was examined in two ways: the average variation between the predictive suitability rating and the performance rating for each group and the percentage of predictions that fall within different ranges of closeness.

The average variation between the suitability score and the performance rating was 8.2 points. This indicates a close relationship between the suitability score and the performance ratings.

Next we examined the percentage of employees who's two scores (performance rating and predicted suitability score) were within 5 points, 8 points, 10 points and 15 points. The table below shows these percentages for each group.

Variation between performance rating and predictive suitability score.

Within 5 points
Within 8 points
Within 10 points
Within 15 points
39%
59%
72%
87%

If the two scores were within five points, the prediction would be considered to be extremely accurate. If within 8 points, the prediction would be considered to be quite accurate. If within 10 points, the prediction should still be considered to be reasonably accurate. If the two scores are greater than 10 points apart, but within 15 points, it could not be considered to be accurate, but at the same time could not be considered to be very inaccurate. If the two scores are greater than 15 points apart, they should be considered to be inaccurate.

Conclusion

The results show the template developed is able to predict job success with a high level of accuracy. The sample size was sufficient to assume reliable results for future applicants.

The research indicates that the strongest success factor for this position is the willingness to take initiative, that is to solve problems on one's own by taking the necessary actions. In addition, the individual must be willing to make decisions on his/her own using logic and analytical capabilities. A motivation to earn high pay is also a positive trait. In addition the individual should comfortable trying to influence others and be able to manage stress effectively.

There are numerous factors that would deter success in this position that relate to interpersonal skills, task orientation and motivational factors. The counter-productive traits related to interpersonal skills include: the tendency to be blunt, evasive, harsh, dominating, dogmatic, lack diplomacy, expecting diplomacy (overly sensitive), disliking working in a team, or overly introverted. In addition, trying to be overly helpful is also counter-productive. Task requirements are such that a person will have difficulty in this position if he/she dislikes doing research, structure, analytical tasks, working quickly, or is disinterested in business or finance. As related to motivation, the position requires someone who will work autonomously, not shy away from a challenge and is motivated a least to some degree to have a stable career.