Research shows that implementation success is associated with fidelity to the evidence-based practice. But it seems like actually measuring the degree of fidelity in your implementation is a little trickier. One article I read recommended a mix of chart reviews, observations of team meetings, surveys of clients and staff, interviews with staff, and fidelity checklists or scales. But where do I fine the resources to help me do all those things? Like the fidelity checklists or scales? How do you go about assessing fidelity?
Without some measure of fidelity one runs the risk of having "paper implementation", i.e., agency records document that appropriate staff members have been trained, but there is no evidence that they are actually using the EBP. This happens a lot with motivational interviewing (MI), where more than a two day workshop is typically required for staff to achieve competence. With MI I don't think there is any substitute for evaluating recorded client sessions. Digital recorders are relatively cheap and unobtrusive and there are several well-established instruments for assessing fidelity. These include the Motivational Interviewing Assessment: Supervisory Tools for Enhancing Proficiency (MIA:STEP) and the Motivational Interviewing Treatment Integrity (MITI) coding instrument. In addition to providing fidelity checks, these tools are very useful for coaching. The hard part seems to be changing the clinical culture so that session review becomes a routine part of practice.
Digital session review is a great way to assess fidelity of practice. In my experience it is likely that considerable resistance to that will develop across the spectrum of practitioners when that policy is implemented. Are there any suggestions for breaking down that resistance or at least in successfully beginning the process of cultural change? How common is the practice of digital recording techniques?
One way might be to use contingency management. Positive incentives for achieving verified competence in an EBP might encourage clinicians to cooperate with digital session reviews. Using the reviews as a constructive means for helping clinicians to learn and enhance new skills might also facilitate acceptance.
Love this idea! Use an EBP to enhance adoption of an EBP.
Such a great question! I work with various programs and the time and resources to do a through examination of program fidelity. An early study by Barnoski and Aos 2004 is a great demonstration of how participants are adversely affected by an improperly implemented EBP. In my research, we either use tools created by the EBP developer or we look for adherence to best practices in that area of service. We collect data using the same methods you noted above.
Are there alternatives to recorded session for measuring the effectiveness of the counselor session? I have found that most clients do not want their sessions recorded due to legal concerns, shame and trust issues and other extremely personal issues.
One of the ways our facility has addressed this issue is through "Efforts To Outcomes"(ETO), an on-line database used to document the whole spectrum of treatment with a client, similar to electronic health records(ETH), but more in depth. It provides for clinical, case management, and peer support aspects of the treatment plan. Each area has EBP that is documented along with other resources such as 12 step, community supports, etc... The modality we use is similar to Critical Time Intervention(CTI) and it seems to be working well. The data from the ETO goes to entities that help with funding such as DPH, SAMHSA, and BSAS (of course all clients sign releases for providing info). But because of the reporting going to the funding orginizations and may have an effect on future funding, fidelity certainly has to be somewhat of a priority, this may be an incentive for agencies and management to be on board.
This is a good question. We use cognitive-behavioral curricula that do not include a fidelity instrument. All I have been able to find is something for MI.