Problem-solving treatment (PST) offers a promising approach to the depression care; however, few PST training opportunities exist. A computer-guided, interactive media program has been developed to deliver PST electronically (
Effective treatments have been developed for depression, including antidepressant medications and psychotherapies. Many patients prefer counseling or psychotherapy to taking medications, if it is available [
Problem-solving treatment (PST; also known as problem-solving therapy) has emerged as an effective treatment for depression [
PST has great potential for dissemination because it can be delivered by mental health as well as nonmental health specialists in a wide range of healthcare settings, including primary care [
Automating PST training could improve the ability to further disseminate PST and potentially result in cost savings. Although the supervision portion of the training requires the involvement of a live clinician, it is conceivable that some or all of the workshop could be automated. Self-instructional training is not a new concept; self-instructional books have existed for centuries and self-instructional videos for decades. These could be good starting points to learn PST. However, it has been demonstrated that the more interactive a training experience is, the more effective it is [
An interactive media-based computer program has been developed to provide PST electronically (
Because the program models best practices for providing PST, it may be useful not only as an intervention but also as a teaching tool, to demonstrate how to deliver PST. The purpose of this study was to obtain pilot data on the feasibility and efficacy of using this program to teach PST to therapists unfamiliar with the intervention. Primary research questions addressed were as follows. Is use of Is live training associated with change in skill at performing PST, for persons who have already been trained via
Secondary research questions were as follows. Is use of How acceptable and easy to use is
The subject population consisted of clinical staff and trainees from a human services organization in Framingham, MA, with no prior training in PST. All participants volunteered for the study and received no compensation for participation. Training was offered to all clinicians and trainees working at the site.
The study was approved by the Massachusetts General Hospital’s Institutional Review Board, and consent to participate in the study was implied by voluntary completion of the study questionnaires. Receiving training was not contingent on participation in the evaluation component, and participants were not required to participate as part of their employment.
Training consisted of participants using the
Assessments were conducted at three time points and trainings were conducted at two points. At baseline (before beginning training), participants completed a test of their knowledge of PST and a measure of self-efficacy to do counseling (regarding both general and PST-specific skills). Following completion of their training using
This was a primary measure, which consisted of one open-ended essay-format question: “Please describe the process of problem-solving therapy in detail, including all steps and the criteria for successfully completing each one.” Essays were scored using criteria developed for the study (see Appendix
The standardized patient roleplay was a primary measure. The audiotaped roleplay sessions were scored using the Problem-Solving Treatment in Primary Care Adherence and Competence Scale (PST-PAC) [
Scoring was performed by J. A. Cartreine and J. L. Seville, who were blind to the time point of each roleplay. One of the raters (J. L. Seville) had been trained in the use of the PST-PAC previously; the other (J. A. Cartreine) was trained to rate roleplays by rating audio recordings of sample cases.
The
The Program Acceptability Questionnaire (PAQ) was written for this study and consisted of six questions on various aspects of the acceptability of the PST computer program, such as how much the program helped the trainee learn PST and whether the subject would recommend its use to other prospective trainees (see Appendix
The System Usability Scale (SUS) [
Scores for the PST-PAC were obtained using the following procedures: for each standardized patient interview, all eight scores given by each rater were averaged, yielding two scores (one for each rater). For items where the raters’ scores differed by more than one point, the raters discussed the item and arrived at a consensus score for that item. Then, the average was taken of the two raters’ scores, yielding one final score per standardized patient interview.
PST knowledge essays were scored on 27 items that map onto seven subscales, which correspond to the steps of PST. Because the seven subscales contain different numbers of items, an average of the scores for each subscale, for each rater, was calculated. The seven scores were then summed, yielding one score per essay for each rater. Finally, an average was calculated between scores awarded by each rater, to obtain a full-scale score for each participant on each essay written.
Self-efficacy scores were calculated by summing the responses to individual items (with some questions reverse scored). Additionally, subscales were calculated by summing the items measuring general self-efficacy and those measuring PST-specific self-efficacy.
Due to the small sample size, nonparametric analyses were used to answer the research questions. The Wilcoxon signed ranks test was used to compare skill levels between Assessment 2 and 3 (no skill test was administered in Assessment 1). The same test was also used to gauge change in knowledge and self-efficacy at Assessment 1 compared to 2, and Assessment 2 compared to 3. Because multiple comparisons were made, a Bonferroni correction was used, which set the
Descriptive statistics were used to characterize the sample and results of the SUS and PAQ. Because the PAQ was created for this study, an insufficient quantity of data has been collected to support a factor analysis and the reporting of a composite score. Therefore, results on this questionnaire are reported by item.
Thirteen participants (11 female and 2 male) enrolled, with a mean age of 37.5 (±12.9); 11 self-identified as Caucasian and the other 2 as racial minorities. Six were licensed clinical social workers, with an average of 2 years in practice (±1.4, range 2 to 6 years); the other 7 were social work graduate students. No participants who inquired about the study subsequently refused to participate or dropped out.
Interrater reliability for both the essay and roleplay scoring was calculated using percent agreement between raters. For the skill measure, interrater agreement per item (defined as agreement on the rating plus or minus 1 point) between the two raters averaged 79.6%. To increase concordance between raters, consensus scores (as described in the methods section) were obtained for items in which raters differed by >1 point. For the knowledge (essay) measure, the percent agreement (within one point) between the raters across items was 87.5%. Because agreement between raters on the knowledge measure was high from the outset, a consensus process was not needed to improve concordance.
Summary results of the knowledge, skill, self-efficacy, and usability measures are presented in Table
Assessments of knowledge, skill, self-efficacy, and
Pre-training | Post- |
Post-live training | |||||||
---|---|---|---|---|---|---|---|---|---|
Median (range) | Mean ± SD | Scaled score |
Median (range) | Mean ± SD | Scaled score |
Median (range) | Mean ± SD | Scaled score |
|
Knowledge of PST | 0.9 |
1.53 ± 2.04 | 2.57 | 14.86b |
14.06 ± 6.74 | 42.46 | 16.09 |
16.74 ± 5.17 | 45.97 |
Skill implementing PST | — | — | — | 20.5 |
19.65 ± 6.70 | 51.25 | 26 |
25.23 ± 7.16 | 65.00 |
Self-efficacy (Composite) | 126 |
129.54 ± 14.64 | 64.00 | 133 |
134.31 ± 13.81 | 67.33 | 145c |
143.31 ± 17.36 | 76.67 |
Self-efficacy (General) | 63 |
63.31 ± 6.29 | 70.00 | 65 |
64.23 ± 6.10 | 70.00 | 66 |
66.54 ± 8.23 | 74.29 |
Self-efficacy (PST-specific) | 65 |
66.23 ± 9.59 | 61.25 | 68 |
70.08 ± 9.21 | 65.00 | 77d,e |
76.77 ± 10.52 | 76.25 |
Usability of |
— | — | — | 67.5 |
69.23 ± 7.8 | — | — | — | — |
dCompared to pretraining (
eCompared to post-
Knowledge scores significantly increased from pretraining to post-using
To facilitate comparisons between knowledge and skill levels, scaled scores of 0 to 100 were calculated by converting the median to the percentage of the maximum score possible on each scale (see Figure
Scaled scores for skill, knowledge, and self-efficacy. Scaled Score = Median converted to percent of maximum possible score.
Five items were unanswered in the PCSEI data set. Values for these missing data points were imputed by calculating the mean of the participant’s responses to the other items from the same subscale on that administration of the PCSEI. Composite self-efficacy scores, which included all items on the PCSEI, remained stable from pretraining to post-
The median SUS score was 67.5 (range 60 to 87.5; mean =
Program Acceptability Questionnaire (scale range 0 to 6).
Item | Mean ± SD | Median |
---|---|---|
I felt comfortable using the PST program for training | 5.23 ± 0.73 | 5 (4 to 6) |
Doing training using this program was acceptable to me | 4.92 ± 0.76 | 5 (4 to 6) |
Using the program helped me understand how to do PST | 4.69 ± 0.75 | 5 (4 to 6) |
I enjoyed using the program to learn PST | 4.62 ± 1.12 | 4 (3 to 6) |
I would rather do training in a live workshop than with the computer | 4.54 ± 1.05 | 5 (3 to 6) |
I would recommend this program to a colleague who is interested in learning how to do PST | 4.54 ± 1.13 | 5 (2 to 6) |
PST is an evidence-based intervention for depression; however, few providers have training in it. A novel approach to teaching PST is the use of a computer-automated treatment,
Thirteen persons with no prior experience delivering PST (6 licensed clinical social workers and 7 social work graduate students) used the
It was found that live training does not add knowledge or skill beyond what
PST skill was not assessed at baseline due to logistical constraints; however, it is likely that it would also have been very low before training, since none of the subjects had received prior training. After completing 4 sessions of
The only specific areas of improvement found after completing the workshop were knowledge and implementation of decision-making guidelines (i.e., evaluation of pros and cons in problem solving). Both skill and knowledge regarding decision-making guidelines improved following the workshop, suggesting that
A key ingredient in clinical training is supervision while treating actual patients, and it is unrealistic to expect that use of a self-instructional training program or a one-day workshop would produce fully competent PST clinicians. However, after using
Regarding the usability of
Acceptability of
This study advances the literature on self-instructional training of therapists, and of clinicians in general, by using an e-therapy program as a training tool to teach clinicians an evidence-based treatment. Research on self-directed/computer-automated/online training has reported mixed training outcomes, likely because the pedagogies of the tools vary widely, from passively viewed didactics to written manuals to interactive programs [
This study has several limitations, including the mixed educational background of the sample and the lack of a comparison group design, which limit generalizability. Without a comparison group, it is impossible to say for certain what trainees learned from
Regarding modifications to the
Several national and statewide initiatives are implementing evidence-based treatments for depression and could benefit from cost-effective training, such as those by the National Network of Depression Centers [
James A. Cartreine, Brigham and Women’s Hospital/Harvard Medical School, Boston, MA.
Janette L. Seville, Geisel School of Medicine at Dartmouth, Hanover, NH.
Trina E. Chang, Massachusetts General Hospital/Harvard Medical School, Boston, MA.
Mark T. Hegel, Geisel School of Medicine at Dartmouth, Hanover, NH.
In the space below, please answer the following essay question. You have 10 minutes.
Assess whether the written response addresses each element below, on a scale of 0–5.0 1 2 3 4 5 Very Poor Poor Borderline Satisfactory Good Very Good
Number
Element
Subscale
(1)
Problem should be observable
Problem
(2)
Problem should be under client’s control
Problem
(3)
Goal should be behavioral (something client does)
Goal
(4)
Goal should be general (more than one way to reach it)
Goal
(5)
Goal should be observable (or countable)
Goal
(6)
Goal should be achievable (or short term, i.e., 2 weeks)
Goal
(7)
Goal should not just be the converse of the problem
Goal
(8)
Brainstorming should include multiple ideas/solutions
Brainstorming
(9)
Do not prejudge solutions
Brainstorming
(10)
Pros are identified for each idea/solution (or what makes each unique)
Solutions
(11)
Multiple cons are assessed for each idea/solution
Solutions
(12)
Effort to implement solution is assessed
Solutions
(13)
Time to implement solution is assessed
Solutions
(14)
Cost of solution is assessed
Solutions
(15)
Need for other people to implement solution is assessed
Solutions
(16)
Negative impact on others is assessed
Solutions
(17)
One or more solutions must be chosen
Solutions
(18)
Action plan can be implemented soon
Action plan
(19)
Action plan is step by step, detailed
Action plan
(20)
Action plan includes Who, What, Where, and When are the steps to be taken
Action plan
(21)
Anticipate obstacles and plan around them
Action plan
(22)
Plan B’s (backup plans) are created
Action plan
(23)
Enjoyable activities should be scheduled for each week
Pleasant events
(24)
Clinician checks whether action plan was implemented and goal was reached
Progress check
(25)
Clinician checks client’s satisfaction with his or her effort
Progress check
(26)
Clinician helps clients troubleshoot how to improve their problem solving
Progress check
Version 8-13-09
Mark T. Hegel, Ph.D., Geisel School of Medicine at Dartmouth, Hanover, NH.
Laurence Mynors-Wallis, M.D., University of Southampton, UK.
For each item, assess the therapist on a scale of 0–5 and record the rating on the line next to the item number. Evaluating the outcome Review of all current tasks Praise success Exploration of failure Rate Satisfaction and Mood Reinforce PST-PC Model Review previous problem areas Defining the problem Specific, feasible problem chosen Described in objective terms Problem explored, clarified Identify barriers Establishing a realistic goal Goal is objective Described in behavioral terms Goal is achievable Goal is general Follows directly from problem statement Addresses barriers identified in problem Generating solutions Prime for brainstorming Brainstorming facilitated Solutions from patient Withhold judgment Implementing Decision-Making Guidelines and Choosing the Solution(s) Consider “pros” and “cons” for one’s self/others Rate each theme Compare solutions Solution(s) satisfies the goals Negative impact is limited Implementing the preferred solution(s) Specific tasks identified Realistic behavior requirements Plan B Plan pleasant activities for the week Process tasks Clear demarcation of PST-PC stages Kept session near a 30-minute timeframe Cue and review for stages Summarize process at end of session Facilitate a positive problem orientation Facilitate independence in guiding PST process Communication and interpersonal effectiveness Facilitates communication (supportive vocalizations/nonverbals) Use of patient’s own language and phrases Warm/confident/professional Tactful limiting of peripheral and unproductive discussion Global rating How would you rate the problem-solving therapist overall in this session? (does not need to approach a mathematical average of previous eight items)0 1 2 3 4 5 Very Poor Poor Borderline Satisfactory Good Very Good
Based on Larson et al. [
This is not a test. There are no right or wrong answers. Rather, this is an inventory that attempts to measure how you think you behave doing problem-solving therapy. Please respond to the items as honestly as you can to most accurately portray how you think you behave as a PST therapist. Do not respond with how you wish you could perform each item, or think you might in the future. Rather, answer in a way that reflects your actual estimate of how you perform as a counselor at the present time.1 2 3 4 5 6 Strongly Disagree Strongly Agree
Item
Subscale
(1)
I can effectively redirect clients who choose to work on problems over which they have limited control
PST
(2)
I am likely to impose my values on the client during the interview
General
(3)
When I initiate the end of a session, I am positive it is in a manner that is neither abrupt nor brusque, and I end sessions on time
General
(4)
I can help clients construct action plans that meet the criteria of problem-solving therapy
PST
(5)
I feel that I will not be able to respond to the client in a nonjudgmental way with respect to the client’s values, beliefs, and so forth
General
(6)
I feel that I can respond to the client in an appropriate amount of time (neither interrupting the patient nor waiting too long to respond)
General
(7)
I anticipate that the type of response I use at a particular time may not fit with the problem-solving therapy approach
PST
(8)
I can help clients with the brainstorming step of problem-solving therapy
PST
(9)
I feel confident that I have resolved conflicts in my personal life so that they will not interfere with my counseling abilities
General
(10)
I feel that I have enough fundamental knowledge to do effective problem-solving therapy
PST
(11)
I am able to respond in a helpful way when clients report that they have not worked on a problem since last session
PST
(12)
I may not be able to maintain the intensity and energy level needed to produce client confidence and active participation
General
(13)
I am not sure that when doing problem-solving therapy I will express myself in a way that is natural without deliberating over every response or action
PST
(14)
I am confident that I can conduct problem-solving therapy, adhering to the guidelines set out in the PST therapy manual
PST
(15)
My assessments of client problems may not be as accurate as I would like them to be
PST
(16)
I am confident that I can help patients define their problems in a manner suitable for problem solving
PST
(17)
I do not feel I possess a large enough repertoire of techniques to deal with the different problems my client may present
General
(18)
I am uncomfortable about dealing with clients who appear unmotivated to work toward their goals
PST
(19)
I have difficulty dealing with clients who do not verbalize their thoughts during the counseling session
General
(20)
I am unsure how to deal with clients who appear noncommittal and indecisive
General
(21)
I am an effective counselor with clients of a different socioeconomic status
General
(22)
I am unsure how to lead my client to work toward concrete goals
PST
(23)
I am confident that I can assess my client’s readiness and commitment to work on a given problem
PST
(24)
When working with ethnic minority clients I am confident that I will be able to bridge cultural differences in the counseling process
General
(25)
I am confident that I can help my patients evaluate the pros and cons of their solutions and choose a feasible solution in an efficient and helpful manner
PST
(26)
I am confident that I can support the client in choosing his or her own problems on which to work
PST
(27)
I feel I may give advice
General
(28)
In working with culturally different clients, I have a difficult time viewing situations from their perspective
General
(29)
I anticipate having difficulty helping clients write goal statements that meet all of the problem-solving therapy criteria
PST
(30)
I am afraid that I may not be able to effectively relate to someone of lower socioeconomic status than me
General
James A. Cartreine, Brigham and Women’s Hospital/Harvard Medical School, Boston, MA.
Please read each item carefully and circle a number to show how much you agree or disagree with the statement.
I felt comfortable using the PST program for training Doing training using this program was acceptable to me Using the program helped me understand how to do PST I enjoyed using the program to learn PST I would rather do training in a live workshop than with the computer I would recommend this program to a colleague who is interested in learning how to do PST
1
2
3
4
5
6
Strongly Disagree
Strongly Agree
The
Development of the