PSYCHOL 6601OL - Understanding & Synthesising Psychological Evidence

ASSESSMENT 3: REPORT ON META-ANALYSIS AND EMERGING RESEARCH QUESTIONS

Due: End of Week 6, Sunday 11.59 pm

Scenario
You are working for the Australian Government Department of Health, which has, over the
past few months, provided funding and personnel for a meta-analysis into the effectiveness
of various interventions in improving culturally responsive communication skills among
healthcare providers. A team comprised of department personnel and collaborating
university researchers extracted the data and ran the main analyses. You are working in a
team asked to report on the results and consider their policy implications. You have been
provided with a set of questions to answer, and your write-up for the questions will be
circulated to all the participants of an upcoming meeting on the best format and workflow for
publishing the meta-analysis.

Your tasks
Your task is to prepare a report on a meta-analysis and associated directions for future
qualitative and quantitative research. You are required to answer the following six questions
pertaining to the dataset described below.

Dataset
The numeric cells contain post-intervention means on variables measuring culturally
responsive communication skills among healthcare providers or students undertaking health-
related degrees. There are means (and SDs) for participants in the treatment group (trt) and a
control group (ctrl). In all rows, higher values imply better communication skills. Values
represent the longest term follow-up available in the study. Thus, for example, if
communication skills were measured immediately following the intervention and six months
later, the scores from six months later are taken.

The following links are available in MyUni:
• A reference list for all the studies included in the meta-analysis, complete with links to
the course reading list. See 'References for meta-analysis'.
• An explanation for all variables in the dataset. See 'Links to meta-analysis data files:
Dataset spreadsheet: Variables'.
• A full dataset in csv format. It is not expected that you will conduct any analyses in
JASP, but the dataset is available should you wish to. See: 'Links to data files: Full
dataset and Partial dataset with outlier removed'.
The following variables are the critical ones apart from the means, SDs, and the Hedges’ g
effect size measures (see 'Links to data files: Dataset spreadsheet: Variables'):
• years_ago—i.e., age of the study
– a meta-regression was conducted to determine whether effect sizes have increased
over time, perhaps reflecting growing understanding of the need for culturally
sensitive communication in health settings and beyond.
• patient_rated—whether the outcome is a rating obtained from patients (1), rather than a
self-reported indicator (0).
– a subgroup analysis was conducted to examine whether intervention effects are
stronger when outcomes are self-reported.
• student_participants—whether the participants in the study were students: 0 = no, 1 =
yes.
– a subgroup analysis was conducted to examine whether intervention effects are
stronger in more controlled settings—i.e., when participants are students.
• implicit_focus—whether the measure comes from a study in which the intervention
focused on reducing implicit bias, and where an implicit association test score was
therefore the primary outcome: 0 = no, 1 = yes.
– none of the outcomes in this meta-analysis are implicit association test scores –
instead we’re working with self-report-based or patient-report-based measures of
communication competence in cross-cultural settings.
– a subgroup analysis was conducted to examine whether intervention effects are
smaller on measures not intended to be the primary outcome measures in the
associated studies.
• long_term_follow_up—whether the measure was obtained immediately after
intervention (0) or as part of a longer-term follow-up of at least a month’s duration (1).
– a subgroup analysis was conducted to examine whether intervention effects are
smaller when measured longer term.

Questions to answer
1. Draw conclusions about the effectiveness of interventions aimed at improving the cultural
competence of health professionals. In your answer, reference:
• any features of the dataset you consider relevant
• the reported pooled effect sizes and heterogeneity statistics
• the results of the outlier and influence analyses
[250 words; refer to meta-analysis Steps 2, 3 and 4 in Module 3]
Forest plot, pooled effect size, and heterogeneity estimates
Estimator: DerSimonian-Laird.
Knapp-Hartung adjustment used for confidence intervals around the pooled effect.

Outlier analysis
Identified outliers (random-effects model)
------------------------------------------
"harmsen2005"
Results with outliers removed
-----------------------------
SMD 95%-CI %W(random) exclude
castillo2007 -0.4190 [-0.8519; 0.0139] 11.3
chapman2018 1.4412 [ 0.7915; 2.0909] 9.2
gutierrez2014 -0.0811 [-0.4328; 0.2706] 12.0
devine2012 -0.2541 [-0.6724; 0.1642] 11.4
berlin2010 0.2230 [-0.3286; 0.7746] 10.1
smith2001 0.8228 [ 0.3991; 1.2464] 11.4
wade1991 1.8897 [ 0.1420; 3.6374] 3.0
thom2006 -0.0897 [-0.6332; 0.4537] 10.2
harmsen2005 3.8596 [ 2.7688; 4.9504] 0.0 *
cooper2011 0.0588 [-0.5552; 0.6728] 9.5
chang2017 0.0451 [-0.3209; 0.4110] 11.9
Number of studies combined: k = 10
SMD 95%-CI t p-value
Random effects model 0.2207 [-0.2317; 0.6731] 1.10 0.2985
Prediction interval [-0.9548; 1.3962]
Quantifying heterogeneity:
tau^2 = 0.2199 [0.0964; 1.2228]; tau = 0.4689 [0.3105; 1.1058]
I^2 = 78.4% [60.5%; 88.1%]; H = 2.15 [1.59; 2.90]
Test of heterogeneity:
Q d.f. p-value
41.58 9 < 0.0001
Details on meta-analytical method:
- Inverse variance method
- DerSimonian-Laird estimator for tau^2
- Jackson method for confidence interval of tau^2 and tau
- Hartung-Knapp adjustment for random effects model

Influence analysis

2. Subgroup analyses and a meta-regression described in the ‘Dataset’ section produced the
following results. Interpret the findings, draw connections between this analysis and the
analysis in the previous section, and suggest directions for future research. Note that the
analyses were conducted for 10 rather than 11 studies (i.e., with an outlier removed).
[200 words; refer to meta-analysis Step 4 in Module 3]
Subgroup analyses
patient_rated
Coefficients
95% Confidence Interval
Estimate Standard Error z p Lower Upper
intercept
0.220 0.244
0.903 8.000
0.393 -0.342
patient_rated 0.026 0.508
0.051 8.000
0.961 -1.146
Note. Wald tests.
Residual Heterogeneity Estimates
95% Confidence Interval
Estimate Lower Upper
τ²
0.248 0.111 2.023
τ
0.498 0.333 1.422
I² (%)
80.736 65.271 97.158
H²
5.191 2.879 35.191
student_participants
Coefficients
95% Confidence Interval
Estimate Standard Error z p Lower Upper
intercept
0.289 0.283 1.020 8.000 0.338 -0.364
student_participants -0.143 0.429 -0.333 8.000 0.748 -1.132
Note. Wald tests.
Residual Heterogeneity Estimates
95% Confidence Interval
Estimate Lower Upper
τ²
0.250 0.107 2.033
τ
0.500 0.328 1.426
I² (%)
79.791 62.963 96.983
H²
4.948 2.700 33.146
Page 8 of 21
implicit_focus
Coefficients
95% Confidence Interval
Estimate Standard Error z p Lower Upper
intercept
0.313 0.280
1.117 8.000
0.296 -0.333
implicit_focus -0.205 0.423
-0.484 8.000
0.642 -1.181
Note. Wald tests.
Residual Heterogeneity Estimates
95% Confidence Interval
Estimate Lower Upper
τ²
0.233 0.103 2.001
τ
0.482 0.321 1.415
I² (%)
78.623 62.016 96.937
H²
4.678 2.633 32.646
long_term_follow_up
Coefficients
95% Confidence Interval
Estimate Standard Error z p Lower Upper
intercept
0.234 0.302 0.775 7.000 0.463 -0.480
long_term_follow_up -0.113 0.404 -0.280 7.000 0.788 -1.067
Note. Wald tests.
Residual Heterogeneity Estimates
95% Confidence Interval
Estimate Lower Upper
τ²
0.243 0.091 1.494
τ
0.493 0.301 1.222
I² (%)
81.289 61.820 96.387
H²
5.344 2.619 27.675

Meta-regression
years_ago
Coefficients
95% Confidence Interval
Estimate Standard Error z p Lower Upper
intercept
-0.004 0.430 -0.010 8.000 0.993 -0.996
years_ago 0.021 0.034 0.613 8.000 0.557 -0.058
Note. Wald tests.
Residual Heterogeneity Estimates
95% Confidence Interval
Estimate Lower Upper
τ²
0.248 0.105 1.736
τ
0.498 0.325 1.318
I² (%)
79.916 62.808 96.531
H²
4.979 2.689 28.823
3. Given the analysis results below, what would be your conclusions as to whether the
pooled effect size reflects publication bias? In your answer, comment on:
• whether analyses to correct for publication bias are needed.
• whether analyses involving the p-curve can be conducted.
[200 words; refer to meta-analysis Step 5 in Module 3]

Regression test for Funnel plot asymmetry ("Egger's test")
t p
sei
2.122
0.067

4. Cite the results of the meta-synthesis below, as well as the results of this meta-analysis, in
discussing the health policy implications of the current evidence base on interventions
aimed at improving the cultural responsiveness of healthcare providers. Cite at least one
additional reference in your answer. This can be a single study, review, or government
report.
[500 words; refer to content from Modules 4 and 5—i.e., content on qualitative
aggregation and practice-and-policy implications]
Krstić, C., Krstić, L., Tulloch, A., Agius, S., Warren, A., & Doody, G. A. (2021). The experience
of widening participation students in undergraduate medical education in the UK: A
qualitative systematic review. Medical Teacher.
Links to the article and associated online supplement are available on MyUni (see
'References for question on meta-synthesis')
5. What implications do the results of this meta-analysis have for research on interventions
against implicit and explicit bias against racial minorities? Write your answer as though it is
an excerpt from the ‘limitations and future directions’ section of the Discussion in a
journal article describing the meta-analysis. In your answer, cite at least two references
(studies, reviews, or government reports) that you have not cited elsewhere, and at least
four references in total. [300 words; refer to Module 6 content]
6. In preparation for a preliminary informal meeting with other researchers in the field who
are potentially looking to begin a collaborative project with your team, justify and describe
a research question that can be addressed using data from Project Implicit. The research
question does not have to follow from the meta-analysis results or the results of a
reference (study, review, or report) you have cited so far. You can use ‘I’ and ‘my’ in your
answer to explain your decisions, and use the headings ‘(a) Justification’, ‘(b) Hypotheses’,
and ‘(c) Challenges’.
a) Your justification should be presented first, and should describe the study or meta-
analysis result you would be following up.
b) Your description should then specify which Project Implicit dataset you would use, and
the conceptual and operational research questions (or hypotheses) you would
investigate.
c) Finally, your description should contain a paragraph on possible challenges to using
the data in this way. Mention at least one practical challenge and one theoretical
challenge based on the results of performing Steps 3 and 7 in the video on dataset
scoping in Module 6. Cite at least one reference from the list of Project Implicit
publications in your answer.
[750 words; refer to Module 6 content on the open data case study; note that you do not
need to perform Step 4 described in the video for the purposes of this assessment]