RESEARCH STATEGIC PLAN AND LOGIC MODEL FINAL 05172022LM

PINE RIVER INSTITUTE

STRATEGIC RESEARCH PLAN
2021 - 2025

Laura Mills, Director, Research & Evaluation
[email protected]

1

2

PRI RESEARCH PURPOSE

Research & Evaluation activities will support PRI’s mission by evaluating our treatment approach,
learning about factors that foster success, and sharing knowledge about youth & family treatment.
These activities will also serve our goals of broadening our impact and fostering evaluation-informed
best practices, all with the ultimate goal of helping more youths and families.

PRI Mission: We are dedicated to helping adolescents struggling with addictive
behaviours, frequently combined with other mental health challenges. By
fostering maturity in a family-centered program, we help young people get
their lives, their relationships, and their futures back on track. We are dedicated
to treatment excellence, research and evaluation, continuous improvement,
and the exchange of knowledge.

RESEARCH & EVALUATION LOGIC MODEL

www.pineriverinstitute.com

3

PRI RESEARCH & EVALUATION STRATEGY

Guided by our Logic Model, our strategic plan is detailed in this document. Specific actions for 2021
include:

1. Wider dissemination of all research products
2. Increased connection between research and clinical team
3. Decide whether to invest in province-wide evaluation capacity building
4. Launch ‘PRI v TAU’ Research Project
5. Launch ‘Calamities & Keepers of COVID-19’ Project

LONG TERM IMPACT: HELP MORE FAMILIES

LONG-TERM IMPACT: HELP MORE YOUTHS & FAMILIES
The ultimate goal of PRI Research & Evaluation is to help more families. We plan to do this through
knowledge exchange and fostering uptake of evaluation and evaluation-informed treatment at PRI,
among colleagues, and beyond.
INTERMEDIARY OUTCOMES: BROADEN KNOWLEDGE & EVALUATION-INFORMED PRACTICE

INTERMEDIARY OUTCOME: BROADENING KNOWLEDGE
There is a paucity of knowledge about residential youth treatment, and thus an opportunity to
advance scientific findings about the profiles of youths who attend residential treatment, the impact
of treatment, and factors that foster success. Parent engagement with youth residential treatment is of
particular interest, due to its importance and lack of available literature.
From 2021 – 2025 we will increase the impact of R&E by disseminating our findings more broadly than in
the past. For any project we undertake, we aim to share our findings in the following:

Published Manuscripts in Peer-Reviewed Journals
Public Media (i.e., newspaper, press reliease)
Social Media
Bulletin to Colleagues
Website Snapshot with Infographic
International Conference Presentation
Clinical : Research Team Workshops
PRI Board of Directors Bulletin
Staff Newsletter
Parent Workshop or Conference
Youth Workshop

www.pineriverinstitute.com

4

INTERMEDIARY OUTCOME: FOSTER EVALUATION & EVALUATION-INFORMED PRACTICE
A dedicated Research & Evaluation capacity is not common practice among community-based
treatment agencies in Ontario. It is not mandated by governments and nor is it considered a core
program element for funding. PRI is fortunate to have leadership that supports evaluation, and this has
led to our capacity to provide evaluation and outcome measurement for many of our initiatives and
thus information and accountability to our stakeholders. This commitment to measuring key initiatives
has often translated into grant and funding acquisition from funders who appreciate this ability.
Evaluation-Informed Treatment Decisions. PRI is poised to pioneer innovative work in the area of
evaluation-informed treatment decisions. We have developed the tools and have the capacity to
foster clinician-client evaluation feedback loops. This evidence-based practice is well known to
benefit clinical work but is novel in the area of youth intensive treatment and virtually unheard of in the
context of parents whose youth is in treatment.
Evaluation Capacity among Colleagues. If PRI can help other agencies to conduct ongoing evaluation, it
benefits everyone. The province would better understand which youth get treated where and what
outcomes can be expected from different treatment approaches. Eventually, this knowledge could
lead to evaluation-informed treatment allocation and foster optimal outcomes for youths and families.
From about 2012-2017, we led a project to increase capacity for evaluation among youth-serving
agencies across Ontario. The project had great momentum, some valuable outputs, and all agencies
were poised to start implementing their evaluation plan. These outputs are still available, and agency
readiness is still strong, but the project was passed onto Addictions & Mental Health Ontario (AMHO) in
2017 and has since seen no progress. It would be highly valuable for this work to come back to life. By
the end of 2021, we will decide whether investing in this endeavor is strategic and possible.

www.pineriverinstitute.com

5

IMMEDIATE OUTCOMES: VALIDATING EFFECTIVENESS & UNDERSTANDING IMPACT

Evaluation and the exchange of related learning is the primary role of the research department and
the only way for us to validate our approach to therapy and understand how we impact our families.
Every year we do this with an evaluation report and additional special projects.

IMMEDIATE OUTCOMES: ANNUAL REPORT & SPECIAL PROJECTS
Annual Evaluation Report. Our evaluation report highlights the impact PRI has on the health, behaviour,
and relationships of our clients and families. Wherever possible, we use standardized measurement
tools and match clients scores from pre- to post-treatment and again at follow-up. We report parent
observed scores, as parent response rate is more than double that of youths. Youth and clinician
scores and testimonials supplement the parent findings.

Pine River versus Treatment as Usual. Another way to evaluate our treatment is to compare outcomes
of PRI families to those of a comparison group. We partnered with UofT’s Dr. Amanda Uliaszek to
publish a study on PRI versus applicants who did not attend, with data from 2006-2012. In 2021, again
with Dr. Uliaszek and other colleagues, we will replicate this study with data from thousands of
applicants and explore more questions about the landscape of treatment seekers, the trajectories of
their health, the type of treatment they may have received, and the differences between PRI and the
Treatment as Usual group.

Factors that Foster Success. Now that we have established that PRI treatment is associated with benefits
to youth health, behaviour, and relationships, it is time to dig deeper and try to understand the factors
that optimize success. We will meet with the clinical team in 2021 to clarify a list of research questions
that will have direct clinical impact. This list will serve as the ‘menu’ from which research partners can
select their area of study.

COVID-19’s Costs and Keepers. We will explore the possibilities of a project that determines the impact
of COVID-19 across the PRI teams and clients. We will learn the negative impacts of the pandemic
and give PRI information to learn for future situations, as well as any positive impacts of the changes
made necessary by COVID-19.

IMMEDIATE OUTCOMES: UNDERSTAND FACTORS THAT FOSTER SUCCESS
After establishing the impact of treatment, it’s vital to try to explore what factors optimize treatment for
youths and families. We have demonstrated that treatment completion is one such factor and have
(informally) not found that outcomes differ by gender. We have also found that parent engagement
in treatment is a predictor of youth treatment completion.

We have a wealth of data to explore client, family, and treatment factors that foster success. We will
collaborate with the clinical team to determine the most important for PRI to explore and develop a
project list to share with research partners. This line of inquiry is helps us answer the question ‘for whom
do we best work, and for whom should be explore different approaches to helping’, and as such is a
learning ground to understand how to better help all clients at PRI.

www.pineriverinstitute.com

6

IMMEDIATE OUTCOMES: CLOSER COLLABORATION BETWEEN RESEARCH & PRACTICE
Research is often seen as tangential to clinical work, and despite effort from all teams, PRI is not an
exception. Our goal is to better integrate these two functions, such that research decisions are always
developed in collaboration with the clinical team and that practice is informed by evidence.
Clinician-Research Collaborative Meetings. Clinical impact will drive all research activities, and research
findings will inform clinical decisions. Both teams will engage in monthly meetings to discuss:

a. what we measure and why (with appropriate changes)
b. prospective projects and research questions whose findings promise best value for clinical team
c. clinical perspective on research findings from individual cases
Progress Monitoring & Feedback-Informed Treatment. Our evaluation feedback loops are user-friendly
feedback forms developed from the evaluation surveys completed by youths, parents, and clinicians
before, during, and after treatment. Each summarizes key health, behaviour, and relationship changes
throughout and after treatment. These are intended to support clinical dialogue, treatment planning,
and program decisions.
Feedback loops are well documented to benefit to
clients but are relatively new in the area of residential
treatment for youth. Parent feedback loops are virtually
unheard of in the area of youth treatment. This is an
opportunity for PRI to be the innovator of this practice
and share knowledge about its value.
Our protocol for sharing these forms with the clinical team
is in development. Currently we book 10-minute review
calls for each youth completed feedback form and we
are brainstorming ways to increase clinician engagement
with this process. We hope to increase the response rate
of clinicians to their evaluation surveys, get approval for
research to review forms with parents, and embed youth
feedback forms as regular tools for clinical planning.

www.pineriverinstitute.com

7

IMMEDIATE OUTCOMES: KNOWLEDGE EXCHANGE
Publications & Conferences. All research activities should have the goal of publishing in peer-reviewed
journals for the purposes of advancing the science of youth and family treatment. For the same
reason, we endeavor to share knowledge at conferences in Canada and the U.S.
Public Knowledge Contributions. Our goals for 2021-2025 are to broaden the reach of research findings
by sharing knowledge in the media outlined above (i.e., website, social, etc) and consider delivering
presentations outside the U.S. and Canada.
Internal Knowledge Exchange. All research products should be shared internally at presentations at staff
meetings, newsletter spotlights, parent workshops, and board of director presentations. Each of these
should be considered an opportunity to engage immediate PRI stakeholders with evaluation and to
learn about their curiosity, which is an excellent foundation for future research topics.

IMMEDIATE OUTCOMES: EVALUATION OF CENTRE FOR FAMILY INITIATIVES PROJECTS
In 2021, PRI launched the Centre for Family Initiatives, with three fundamental services: Parent Waitlist
Support Group, Community-Based Parent Workshops, and Clinical Training Programs. It is incumbent
on the research team to evaluate the impact of these initiatives and we will co-create a plan for this
with the Director of the Centre for Family Initiatives and other PRI stakeholders.

IMMEDIATE OUTCOMES: TRI-COUNCIL REB APPROVAL
Dr. Debra Pepler (YorkU) submits PRI’s REB application each year in late spring. The application is an
‘umbrella’ application, covering all ongoing evaluation and planned special projects for the year. The
application must be prepared with all relevant appendices by the end of May.

IMMEDIATE OUTCOMES: PROVINCIAL DATA COMPLIANCE
As a program funded by the Ministry of Health and Long-Term Care, PRI is obligated to submit client
data to a provincial client record management system called DATIS. The Research Associate is
responsible for this work and we have always been data contributor in good standing.

www.pineriverinstitute.com

8

STRATEGIES: PARTNERSHIP DEVELOPMENT AND MAINTENANCE

Partnerships with university researchers is a key strategy for the research department. These mutually
beneficial relationships build capacity for PRI to broaden our impact and we in turn provide highly
coveted clinical data.

ACTIVITIES

DATA COLLECTION, MANAGEMENT, & PREPARATION
Several activities contribute to data collection and management. A comprehensive process
description is available in the Research Associate Process Manual. An overview is depicted below,
from the first point of contact to disengagement with research & evaluation. At any time, a family
member can refuse to contribute to evaluation and will be removed from our contact list.
Survey Administration Timing & Flow Chart

Progress Tracking. The Research Associate tracks the flow of all families who have engaged with
treatment at PRI. Using this guide, the RA updates a calendar that indicates and notifies when youths,
parents, or clinicians should be administered research surveys.
Connecting with Families. For each respondent at every timepoint, respondents have the opportunity to
consent or decline to contribute to research. In the case of parents of youths younger than 16,
additional consents are required in order for PRI to access data provided by youths.

www.pineriverinstitute.com

9

Parent & Youth Evaluation Administration. The protocol for contacting our youths, their families, and
clinicians is detailed in the Research Associate Manual.
Data Entry & Preparation. Surveys that are submitted electronically are warehoused in OutcomeTools
and all surveys that have been completed by pen-to-paper are entered by the RA into
OutcomeTools. Data preparation is completed annually starting in January, with specific protocol
outlined in Appendix A.

TANGIBLE INPUTS

Bestnotes CRM. BestNotes is the electronic and online Client Record Management system serves as the
central hub for all client and program information, housing such information as case files, agency
contacts, team notes, incidents, etc.
OutcomeTools. OutcomeTools is the program used to house all research and evaluation survey data.
SPSS. PRI holds a subscription to the base SPSS package, with a need for the Advanced Analyses and
Custom Tables, which we have free until Feb 2022.
Phone & Internet. At the time of this document update, the Research Associate was working from home
due to COVID-19. Her own phone plan and internet is being used. Internet costs are also incurred
personally by the research team.
Zoom Access. PRI provides the Director, R&E access to zoom.
Library Access. Dr. Debra Pepler arranges with YorkU for access to the university library.
NATSAP Membership. Our membership at NATSAP is a valuable asset for networking and project
development. The Director, R&E also serves as chair of the Research Committee at NATSAP and has
been a guest editor for their journal, the Journal of Therapeutic Schools & Programs.

www.pineriverinstitute.com

10

Appendix A: Data Preparation Protocol

Preparation of all surveys for research or evaluation activities is done each year, starting January 2.
Each survey should be organized with identifying information removed, in an excel file with 4
worksheets: Raw data, All Data, Longitudinal Data, and BestNotes.

A master BestNotes file is created in the ‘Reports – Report Builder’ of BestNotes, selecting all relevant
variables and choosing ‘Student’. Stage at Departure should be reviewed each year with the clinical
team and admissions department before considering this file complete. The final BestNotes file should
contain CID, Sex, Year of Admission, Year of Discharge, Stage at Discharge, Phase at Discharge, and
Completed versus Non-completed.

The Raw Data file is downloaded from OutcomeTools and contains all individual responses from the
direct download of each survey. All columns with qualitative responses, and any identifying
information such as name, clinician name, or parent name must be deleted for sharing.

The All Data file is derived from the raw data, eliminating individual responses and retaining subscores
and broadband scores. In some cases, these scores will need to be calculated.

Delete Duplicates: Two variables are created: The relationship to the student is called ‘RELATECODE’,
scored as Youth (0), Mom (1), Dad (2), and Other (3). The timepoints are renamed ‘INSTACODE’ and
coded as below.

PRE-TREATMENT POST-DISCHARGE
3M Post-PRI 4
Initial Screener 1 6M Post-PRI 5
1Y Post-PRI 6
Stage 2 App (Enr. App) 2 2Y Post-PRI 7
3Y Post-PRI 8
Assessment 2.5 4Y Post-PRI 9
5Y Post-PRI 10
Admission 3 6Y Post-PRI 11
7Y Post-PRI 12, etc
2wks in OLE 3.1

Stage 2 3.2

Stage 3 3.3

Stage 4 3.4

Stage 5 3.5

Concatenate CID with RELATECODE and INSTACODE and then conditionally format for duplicates.
Explore why duplicates exist and correct as necessary.

The Longitudinal file is created from the All Data file. Data from each instacode is made into it’s own
worksheet, all column titles are renamed to include the timepoint, and checked for duplicates.

Collapsing Timepoints: Screener, S2App, Assessment, and Admission can be collapsed to ‘PRE’, with
the instacode variable retained (as a potential covariate). Stages 2 and 3 are collapsed and called
‘RES’, with Stage 3 retained in the case of duplicates. Stages 4 and 5 are collapsed to TRANS, stage 5
data are retained. All post-treatment timepoints are collapsed, always retaining the data closest to 1-
year post-tx in the case of duplicates. All collapsed files must have renamed columns (using the
concatenate function). All worksheets can then be merged (using AbleBits Merge Tables wizard). All
individual instance worksheets can be deleted once the Longitudinal file is successful.

Each file is now ready to share, after agreeing to the project plan and receiving a signed data sharing
agreement.

www.pineriverinstitute.com