Program Evaluation
of
The Alumni Career Network
Office of Alumni Relations
The Johns Hopkins University
July 1986 through July 1991
Prepared by Ricky Fine
July 15, 1991
The Alumni Career Network
Office of Alumni Relations
The Johns Hopkins University
Executive Summary
The evaluation of the Alumni Career Network was conducted to determine the effectiveness of this program in providing career guidance and advice to students and alumni of The Johns Hopkins University. All participants, both mentors and advisees, were sent separate, large questionnaires as the measurement instrument.
Findings from this formative evaluation will help the staff of the Office of Alumni Relations assess the strengths and weaknesses of the program for both mentors and advisees, improve the current operation, and provide data for policy and recommendations for the future.
The evaluation revealed that:
- The mentors like the program and want to be called upon more often to provide career advice.
- The advisees want the Alumni Career Network to be a job placement service for graduates of the University.
- Participants need more education and support to effectively use the system.
- There is a need for a full service Career Planning and Placement Center at the University.
A complete description of the program, the evaluation and the results follows.
BACKGROUND INFORMATION
The Johns Hopkins University Alumni Career Network was created by the Office of Alumni Relations to bond the alumni and the University in mutual service and to promote good will by assisting alumni with career planning. To this end, the Alumni Career Network provides graduates seeking career advice the opportunity to contact fellow alumni who have volunteered as mentors. This unique, University-wide network affords the Hopkins family an international link for career advice, guidance and direction.
Discussions regarding establishing a career service for alumni began during the summer of 1986. Because each Division of the University traditionally serves its own alumni, a committee was established with representation from the Development/Alumni staff assigned to the different Schools in order to assess current programs, services and needs. This process was very valuable in that it enabled the committee to learn about the different Divisional initiatives and to share important information. At that time, only the School of Advanced International Studies had a strong, established network with its alumni for career planning. Other Schools offered ad hoc programs such as Career Days for their students and maintained informal lists of people who could be contacted regarding job information. In addition, students at Homewood were able to use the services of the Placement and Student Employment Office, which heretofore found its greatest usefulness in placing engineering graduates.
As a result of these meetings and the recommendation of the Alumni Task Force, it was agreed that the Office of Alumni Relations would manage a University-wide computer network to serve all graduates of all the Divisions of the University seeking career advice. For the first time, for example, a graduate from the School of Arts and Sciences would be able to investigate New York's financial arena with an alumnus from the School of Continuing Studies, Business Division; while a SAIS grad might discuss government employment with a School of Public Health physician working in Rio de Janeiro.
To further focus and define services the University could hope to offer its alumni, the directors of career programs at thirteen similar universities were interviewed in September and October of 1987. A network incorporating exceptional characteristics of other systems was designed based upon these data.
The Alumni Career Network began serving alumni in April, 1988. As a result of a marketing campaign that included promotion through advertisements in the Johns Hopkins Magazine and information dissemination at local alumni chapter meetings, the number of people utilizing the Alumni Career Network continues to grow. At present, between five and ten telephone inquiries are received on a daily basis. In addition, a special marketing letter and a brochure were sent in December, 1988, to 5,500 young alumni nationwide as part of an effort to increase their participation in the Annual Fund and to inform them of this new program. Official expansion of the Alumni Career Network service for students dramatically increased participation.
Students began using the network to explore career options during the fall of 1988. To date, over 100 undergraduates have utilized the system. As of March 1, 1989, 50 students had registered as advisees.
As of March 1, 1989, the total number of alumni utilizing the services of the Alumni Career Network was as follows:
Mentors
886
Persons Seeking Career Advice
184
The breakdown of the 184 people seeking career advice by Division included:
Division
Arts and Sciences
86
Continuing Studies
45
Engineering
18
Johns Hopkins Hospital
2
School of Medicine
5
Peabody
3
School of Public Health
11
SAIS
12
Bologna
2
Not surprisingly, graduates from the most recent classes comprised the greatest number of people seeking career mentors. In 1989, the highest participation among the 184 alumni included:
Class (all schools)
Number of Participants
1988
37
1987
18
1986
22
1985
11
1984
11
Practically every class had at least one person who had utilized the service. The oldest alumnus to date requesting assistance is a graduate of the class of 1949, and the youngest are current students. Mentors in virtually every industry and profession have been requested, including in the field of electronics, publishing, retail, aerospace, finances, banking, medicine, education, the military, and local, state and federal government, etc. Fortunately, with the strong database of 983 mentors, we have a full complement of professionals ready to offer their assistance and support. We have, however, identified a problem with a lack of resource people in remote geographic regions; for example, one young man wished to speak to someone who was working in the scuba diving business in the Caribbean. There are no known Hopkins graduates doing this work at present, fortunately. We were also unable to fulfill the request of one sophomore who wished either to teach history at the university level or to open a Hungarian pastry shop. Why not do both? While we have no bakers specializing in rugelach, the printout of mentors did include people who own small businesses as well as a number of professors.
As of 1989, review of the breakdown by Division of the then 886 mentors worldwide indicated that people had a great need for this program and were most interested in volunteering their services:
Division
Number
Applied Physics Lab
1
Arts and Sciences
465
Continuing Studies
117
Engineering
127
Health Services
1
Johns Hopkins Hospital
9
School of Medicine
31
Peabody
5
School of Public Health
22
SAIS
99
Bologna
10
University
1
It is most gratifying to see that the call for mentors was taken seriously by Hopkins University alumni. With one exception, the Class of 1932, every single year since 1931 has at least one member who is participating as a mentor for their fellow graduates of The Johns Hopkins University. As may be expected, the greatest number of volunteers by class are from the most recent decade. As of 1989, of the 886 total, the participation by the top five classes was:
Class (all schools)
Number of Mentors
1978
57
1981
47
1982
53
1985
61
1986
53
Follow-up phone interviews with twelve randomly selected people who had requested information were conducted during March, 1989. The consensus is that the Alumni Career Network “provides a much needed service to individuals, enabling [them] to tap into Hopkins' vast resources in a new way.” It also demonstrates a growing sense of community and community responsibilities, much needed in our world today." (Laureen Pepersack, School of Public Health '80, New Mexico)
Eleven of the twelve respondents were very pleased with the contacts they had with their mentors and found them to be helpful, receptive and supportive. The mentors asked good questions and appeared sincere in their desire to provide meaningful assistance. Evaluation of this program is not based upon job outcome, since a significant goal of this program is to provide the opportunity for career exploration in the context of the University.
One young man stressed that his mentors were very "high quality people," and the conversations were "on target." He plans to follow up with one of his mentors for additional contacts, and also plans future work with the Florida Fisheries Industry upon relocation.
The twelfth person interviewed felt, when reading the list of mentors, that these people would not be able to help him because the fields given were "not specific enough." This 54 year old manager in production wants to make a career move but has recently had bypass surgery; he has placed his career search "on the back-burner for the time being," but requested another printout during the course of our conversation.
DESCRIPTION OF THE EVALUATION RESEARCH
The purpose of this evaluation research is twofold. Overall, a formative research evaluation was conducted to assess the effectiveness of the program. Several questions were formulated to help measure whether there was a different outcome for mentors and advisees based upon using this system. A questionnaire instrument was written to
- Determine strengths and weaknesses of the system for both mentors and advisees.
- Identify gaps in the system.
- Pinpoint weaknesses in the process that can be eliminated.
- Identify the impact of the mentoring relationship.
- Evaluate data for policy and recommendations.
Questions around advisee impact include the following:
- Are they using the system?
- Are they contacting the mentors?
- Is the system useful?
- Was the match on target (advisee to mentor)?
Two questionnaires were designed to measure these data. The first step involved determining the specific information needed for the formative design. The instruments (see appendix) were mailed in January, 1991: 983 to mentors and 550 to advisees. A separate and different cover letter explained to each group was part of the mailing, as was a stamped, return envelope. The response rate was quite high: 440 mentor returns (65% return rate) and 148 returned questionnaires from advisees (27% return rate). However, a number of mentors and advisees had not yet been contacted or made contact with those people whose names were sent to them. For the purpose of this evaluation, therefore, data analyses have been confined to only the population that contacted or was contacted by others on the system. Information about the exchange between mentors and advisees provides other bases for the outcome measure. Data were analyzed for each question on the instrument. Frequency distributions for the demographic and descriptive information, as well as for the Likert scale, were computed and analyzed (see appendix).
Page 6
RESULTS
The evaluation of the Alumni Career Network measured the cumulative effectiveness of this system from July, 1986 to June, 1991. Before 1986 there was no mentoring service for alumni of the Johns Hopkins University. Assessment of the effectiveness of the system by year is impossible because there are no data to track over the five year time frame of its existence. There was zero coverage prior to this program; since its creation over 550 advisees have been served.
The Alumni Career Network is not a job placement program, and no objective data regarding the use of this service exist. For example, whether use of the ACN affects outcome regarding its use cannot be ascertained. Furthermore, there are no comparative data on a similar population.
There are, however, valuable conclusions to be drawn from the following graphs computed from the frequency distributions. Mentor questions one, four and 10 (see questionnaire, Likert Scale) and Table 1 clearly demonstrate that the mentors are enthusiastic in their support for the network, they strongly agree that it is a worthwhile service, and it is personally satisfying for them to assist members of the Hopkins family.
While the advisees agreed that the mentors were professional and provided good information, they did not think the network helped them find a new job. They would, however, recommend the service to a friend or colleague. Examination of all the data provides a strong sense that the users want the ACN to be a job placement service, and see the immediate outcome as obtaining a new professional position. Unfortunately, this is an unrealistic goal, and is not the purpose of this program.
The modal case, i.e., the most frequently occurring response for the questions addressing the two features the mentors like most about the network, was the opportunity to help others associated with Hopkins (38/112). The second most common measure was the chance to meet new people (16/112). The modal case for improvements they would like implemented pointed to a more aggressive marketing campaign (13/87). The second most common measure was to increase the number of requests for their services (9/87).
Interestingly, the modal case for the question addressing the two features advisees like best about the ACN was the fact that it exists (15/76). The second most common measure (12/76) was the opportunity to meet with Hopkins alumni who genuinely wish to help them. Data regarding the 2 features they would like improved indicated the following: a greater number of mentors on each printout (9/73), a request for quicker turn around times for the print out (8/73), and requests to update the mentor list on a regular basis (8/73).
DISCUSSION OF RESULTS
The results obtained in this study reflect subjective data based on program participants' evaluation of the system. The results are valuable because they are user-oriented. The evaluation focuses on personal and organizational dynamics, group information needs, program history, and intended uses of information.
TABLE 1: MENTORS' ATTITUDES TOWARD PROGRAM
| LABEL | NUMBER | MEAN | STANDARD DEVIATION |
|---|---|---|---|
| *Mentor Question 1 | 238 | 1.58325329 | 0.76761194 |
| Mentor Question 2 | 182 | 2.85164835 | 0.98887320 |
| *Mentor Question 3 | 215 | 1.87441860 | 0.83051977 |
| *Mentor Question 4 | 123 | 1.95934959 | 0.94433925 |
| *Mentor Question 5 | 94 | 1.69148936 | 0.81684675 |
| *Mentor Question 6 | 130 | 1.64615385 | 0.75597402 |
| Mentor Question 7 | 119 | 2.89915966 | 1.15270469 |
| *Mentor Question 8 | 124 | 4.37096774 | 0.94969179 |
| Mentor Question 9 | 118 | 3.07627119 | 1.40910822 |
| *Mentor Question 10 | 134 | 1.82089552 | 0.81204047 |
| Mentor Question 11 | 118 | 3.50847458 | 1.26589560 |
| Mentor Question 12 | 113 | 3.14159292 | 1.15625053 |
| Mentor Question 13 | 122 | 3.81147541 | 1.11586580 |
| Mentor Question 14 | 99 | 4.10101010 | 1.02514259 |
- The following conclusive statements may be made regarding the data listed above:
Mentor Question 1 Mentors agree that the Alumni Career Network is a worthwhile service.
Mentor Question 3 Mentors agree that the Alumni Career Network offers an important facet - networking - to network job seekers.
Mentor Question 4 Mentors agree that advisers were professional.
Mentor Question 5 Mentors agree that advisers were prompt in keeping appointments.
Mentor Question 6 Mentors agree that advisers were courteous.
Mentor Question 8 Mentors disagreed that advisers seemed disinterested.
Mentor Question 10 Mentors agreed that the experience of helping a fellow alumnus(a) of Johns Hopkins gave them great satisfaction.
Mentor Question 1
The Alumni Career Network is a worthwhile service.
Response
Frequency
Str. Agree
130
Agree
[unclear: approximately 85]
Uncertain
20
Disagree
[unclear: approximately 5]
Str. Disagree
10
FREQUENCY
0 10 20 30 40 50 60 70 80 90 100 110 120 130
Mentor Question 4
The people I spoke with were professional.
Response
Frequency
Str. Agree
40
Agree
60
Uncertain
10
Disagree
7
Str. Disagree
3
FREQUENCY
0 · 10 · 20 · 30 · 40 · 50 · 60 · 70
Mentor Question 10
Helping a fellow JHU alumnus gave me great satisfaction.
Frequency
Response
Frequency shown
Str. Agree
approximately 50
Agree
approximately 62
Uncertain
approximately 12
Disagree
approximately 6
The horizontal axis is labeled MENTOR QUESTION 10. The frequency scale runs from 0 to 70.
TABLE 2: ADVISEES' ATTITUDES TOWARD PROGRAM
| LABEL | NUMBER | MEAN | STANDARD DEVIATION |
|---|---|---|---|
| *Advisee Question 1 | 36 | 1.88888889 | 0.94951951 |
| Advisee Question 2 | 36 | 3.55555556 | 1.34045716 |
| *Advisee Question 3 | 37 | 2.02702703 | 0.95703496 |
| Advisee Question 4 | 39 | 3.02564103 | 1.34726193 |
| Advisee Question 5 | 36 | 2.38888889 | 1.22539271 |
| Advisee Question 6 | 35 | 2.94285714 | 1.32715656 |
| Advisee Question 7 | 19 | 3.68421053 | 1.15723001 |
| Advisee Question 8 | 30 | 3.20000000 | 1.27035157 |
| *Advisee Question 9 | 36 | 4.22222222 | 1.07201042 |
| Advisee Question 10 | 50 | 3.22000000 | 1.43270830 |
| Advisee Question 11 | 48 | 3.02083333 | 1.43644928 |
| Advisee Question 12 | 47 | 3.10638298 | 1.43305758 |
| Advisee Question 13 | 49 | 3.73469388 | 1.22092025 |
| Advisee Question 14 | 45 | 3.84444444 | 1.20520756 |
| *Advisee Question 15 | 46 | 4.39130435 | 1.04303533 |
| Advisee Question 16 | 45 | 2.86666667 | 1.28982028 |
| Advisee Question 17 | 53 | 2.67924528 | 1.55382388 |
| Advisee Question 18 | 64 | 2.06250000 | 1.18019369 |
| Advisee Question 19 | 60 | 3.00000000 | 1.39003719 |
- The following conclusive statements may be made regarding the data listed above:
Advisee Question 1 — Advisees agree that mentors listened to questions and concerns.
Advisee Question 3 — Advisees agree that mentors were experienced in their fields.
Advisee Question 9 — Advisees agree that mentors were knowledgeable in their professions.
Advisee Question 15 — Advisees agree that the Alumni Career Network was not helpful in finding a new job.
Advisee Question 5
Mentor(s) provided good information on their profession.
Frequency
Response
Frequency
Str. Agree
6
Agree
22
Uncertain
3
Disagree
1
Str. Disagree
5
Advisee Question 15
The ACN was helpful in finding a new job.
Response
Frequency
Str. Agree
2
Agree
1
Uncertain
5
Disagree
9
Str. Disagree
30
FREQUENCY: 0, 10, 20, 30
Advisee Question 18
I would recommend the Network to a fellow alumnus.
Frequency
Response
Frequency
Str. Agree
[unclear: approximately 22]
Agree
[unclear: approximately 25]
Uncertain
[unclear: approximately 15]
Disagree
[unclear: approximately 8]
Str. Disagree
[unclear: approximately 10]
The frequency axis is marked 0, 10, 20, and 30.
CONCLUSIONS, RECOMMENDATIONS AND OPTIONS
This evaluation of the Alumni Career Network demonstrates the need for continuing efforts to serve the career development needs of the students and alumni of The Johns Hopkins University.
Although the Office of Alumni Relations provides advisees with educational information regarding the Network and how to use it to explore career options, it is clear that the advisees want a more proactive program for career exploration and change. The original purpose, however, of the Alumni Career Network was to provide career advice. The University's responsiveness to the graduates' needs and how it can best address those needs benefits both the University and its graduates.
Certainly, the mentors are very willing to serve the University by providing career counseling, a satisfying experience that could be enhanced by sending more people to them. Users should be reminded that the Network cannot place them in a new job; rather, it can help empower them to make thoughtful decisions and choices with the expertise of others. Advisees need further encouragement to help them achieve their goals. The current educational materials that are sent out to help advisees address the job search process, and the usefulness of a support system, interestingly, a majority of advisees felt that using the network was an enjoyable and useful experience, and would refer their colleagues to it.
Recommendations for improving the system include:
- Recruiting the alumni to take advantage of new software features.
- Updating the mentor database.
- Conducting an annual evaluation to learn whether the program is increasing or decreasing in usage.
The specific results outlined in the evaluation should prove useful in guiding the University in its decisions to provide continuing education, support, and service to alumni and students.
THE JOHNS HOPKINS UNIVERSITY
ALUMNI CAREER NETWORK QUESTIONNAIRE
The purpose of this questionnaire is to learn if the Alumni Career Network (ACN) is fulfilling its purpose of providing a useful mechanism for students and alumni to contact mentors for career advice, guidance, and direction. All information is confidential. No address will be made available to match you with advisees. Please provide your work address (street is not necessary).
City ____________________ State ____________________ County ____________________
- When and from which Johns Hopkins program did you graduate (or plan to graduate)?
Year(s) ____________________ / Degree(s) ____________________
Division(s) ____________________ / Major(s) ____________________
- What is your profession?
- In what areas have you provided advice?
- How did you learn about the Alumni Career Network?
- Why did you choose to participate as a mentor?
Instructions: Please answer the following questions based on your experiences with the mentors with whom you have had the most contact.
- How many advisees have you spoken with to date?
- On average, how many times did you meet with each of your advisees?
- On average, how much total time did you spend with each of your advisees?
[handwritten: W]
THE JOHNS HOPKINS UNIVERSITY
ALUMNI CAREER NETWORK QUESTIONNAIRE
The purpose of this questionnaire is to learn if the Alumni Career Network (ACN) is fulfilling its purpose of providing a useful mechanism for students and alumni to contact mentors for career advice, guidance, and direction. All information is confidential. No attempt will be made to match your name with your answers. Please provide your work address (street is not necessary).
City ____________________ State ____________________ County ____________________
- When and from which Johns Hopkins program did you graduate (or plan to graduate)?
Year(s) ____________________ / Degree(s) ____________________
Division(s) ____________________ / Major(s) ____________________
- In what types of professions are you interested? (i.e. law, medicine, investment banking, engineering, etc.)
- In what position, and in what business, are you currently employed?
- How did you learn about the Alumni Career Network?
- Why did you choose to use the Alumni Career Network?
Instructions: Please answer the following questions based on your experiences with the mentors with whom you have had the most contact.
- How many mentors have you contacted to date?
- On average, how many times did you meet with each of your mentors?
- On average, how much total time did you spend with each of your mentors?
- Did you feel comfortable using the ACN?
- What motivated you to use the ACN? (i.e. exploring careers, the job market, graduate school)
Instructions
Again, please answer the following questions based on your experiences with Johns Hopkins mentors with whom you have had the most contact.
Circle the appropriate number on a scale of 1 to 6 whether you agree or disagree with the statements that follow.
1=strongly agree 2=agree 3=uncertain 4=disagree 5=strongly disagree 6=not applicable
My mentor(s)
Statement
Scale
listened to my questions and concerns.
1 2 3 4 5 6
was able to give me the time I needed.
1 2 3 4 5 6
was experienced in his or her field.
1 2 3 4 5 6
provided the information I needed.
1 2 3 4 5 6
provided good information on his or her profession.
1 2 3 4 5 6
gave me good ideas about pursuing my chosen field.
1 2 3 4 5 6
helped me with my resume.
1 2 3 4 5 6
was organized in serving my needs.
1 2 3 4 5 6
lacked knowledge of his or her profession.
1 2 3 4 5 6
The Alumni Career Network
Statement
Scale
was useful for learning about career opportunities.
1 2 3 4 5 6
was useful for exploring options.
1 2 3 4 5 6
stimulated my thinking about careers.
1 2 3 4 5 6
was an integral part of my career search.
1 2 3 4 5 6
helped me to learn about myself.
1 2 3 4 5 6
was helpful in finding a new job.
1 2 3 4 5 6
was an enjoyable experience.
1 2 3 4 5 6
was not useful.
1 2 3 4 5 6
1=strongly agree 2=agree 3=uncertain 4=disagree 5=strongly disagree 6=not applicable
I would recommend the Network to a fellow alumnus. 1 2 3 4 5 6
If it were appropriate, I would recommend the Network to a non-alumnus. 1 2 3 4 5 6
Which two features do you like most about the Network?
Which two features could be improved to provide better service?
Additional Comments:
Sources
IMG_2683.HEIC · IMG_2684.HEIC · IMG_2685.HEIC · IMG_2686.HEIC · IMG_2687.HEIC · IMG_2688.HEIC · IMG_2689.HEIC · IMG_2690.HEIC · IMG_2691.HEIC · IMG_2692.HEIC · IMG_2693.HEIC · IMG_2694.HEIC · IMG_2695.HEIC · IMG_2696.HEIC · IMG_2697.HEIC · IMG_2698.HEIC · IMG_2699.HEIC · IMG_2700.HEIC · IMG_2701.HEIC · IMG_2702.HEIC · IMG_2703.HEIC · IMG_2704.HEIC
