forked from CognitionOpenDataProject/set_OvGDB
-
Notifications
You must be signed in to change notification settings - Fork 0
/
pilotReport.Rmd
168 lines (126 loc) · 6.5 KB
/
pilotReport.Rmd
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
---
title: "COD Reproducibility Report"
output:
html_document:
toc: true
toc_float: true
---
#### Article ID: OvGDB
#### Pilot: Richie Lenne
#### Co-pilot: Tom Hardwicke
#### Start date: 05/23/2017
#### End date: 05/23/2017
-------
#### Methods summary:
Use a within-subject design participants read two versions of a story about a person, Adams, who failed to show up to a scheduled meeting. Language in the story was manipulated such that Adams was personally at fault for missing the meeting in one version (high blame condition), and less personally at fault in the second version (low blame condition). After each version of the story participants rated the extent to which Adams ought to have met their friend.
------
#### Target outcomes:
For this article please focus on findings reported for Experiment 1 in Section 2.2. Specifically, you should attempt to reproduce all descriptive and inferential analyses reported in the text below and associated tables/figures:
> 2.2. Results and discussion
> Participants were more likely to say that an agent ought to keep a promise they can't keep in the high blame condition (M = 8.90, SD = 39.16) than in the low blame condition (M = -17.84, SD = 33.31), t(79) = -4.62, p < 0.001, d = 0.74. Importantly, the judgments in the high blame condition were significantly above the midpoint, t(79) = 2.03, p = 0.045, d = 0.65. On the whole, 31% of participants in the low blame condition and 60% of subjects in the high blame condition gave answers above the midpoint. To check for order effects, we compared the ratings of participants who read low blame first (n = 42) and high blame first. There were no significant order effects for whether participants read low blame first (M = -22.05, SD = 32.89) or second (M = -13.18, SD = 33.59; p = .24) or high blame first (M = 9.57, SD = 40.96) or second (M = 8.16, SD = 37.61; p = .87).
**Note**
Make sure to use the original article for additional context and information about any necessary pre-processing steps. Also check for additional supplementary materials that may provide supporting documentation for analysis procedures.
------
```{r global_options, include=FALSE}
knitr::opts_chunk$set(echo=TRUE, warning=FALSE, message=FALSE)
```
## Step 1: Load packages
```{r}
library(tidyverse) # for data munging
library(knitr) # for kable table formating
library(haven) # import and export 'SPSS', 'Stata' and 'SAS' Files
library(readxl) # import excel files
library(CODreports) # custom report functions
library(psych) # descriptive
```
## Step 2: Load data
```{r}
dat_original<-read_excel("data/data1.xls")
```
## Step 3: Tidy data
```{r}
# remove variable descriptions from first row.
dat <- dat_original[-1,]
# removed Ps who fail attention check, row = c(6,11,72)
dat <- dat[-c(6,11,72),]
# make DVs numeric
dat$`Low Blame` <- as.numeric(dat$`Low Blame`)
dat$`High Blame` <- as.numeric(dat$`High Blame`)
# make display order a factor
dat$`Display Order` <- as.factor(dat$`Display Order`)
dat<-data.frame(dat)
```
## Step 4: Run analysis
### Descriptive statistics
> low blame condition (M = -17.84, SD = 33.31)
high blame condition (M = 8.90, SD = 39.16)
**correct**
```{r}
dat %>% select(Low.Blame, High.Blame) %>% describe()
```
> On the whole, 31% of participants in the low blame condition and 60% of subjects in the high blame condition gave answers above the midpoint.
**error in low blame, they reported percent greater than or equal midpoint, instead of greater than**
```{r}
# scale midpoint is 0
dat %>% select(Low.Blame) %>% filter(Low.Blame>0) %>% nrow() / dat %>% select(Low.Blame) %>% nrow()
dat %>% select(High.Blame) %>% filter(High.Blame>0) %>% nrow() / dat %>% select(High.Blame) %>% nrow()
compareValues(reportedValue = .31, obtainedValue = 0.2625)
```
### Inferential statistics
> Participants were more likely to say that an agent ought to keep a promise they can't keep in the high blame condition (M = 8.90, SD = 39.16) than in the low blame condition (M = -17.84, SD = 33.31), t(79) = -4.62, p < 0.001, d = 0.74.
**error in D**
```{r}
# paired sampel t-test
(tx <- t.test(dat$Low.Blame, dat$High.Blame, paired = T))
# Cohen's D = t*sqrt(2/n)
(d <- -tx$statistic*sqrt(2/nrow(dat)))
compareValues(reportedValue = 0.74, obtainedValue = 0.731039)
```
> Importantly, the judgments in the high blame condition were significantly above the midpoint, t(79) = 2.03, p = 0.045, d = 0.65.
**Error in D**
```{r}
(tx <- t.test(dat$High.Blame))
# one sample Cohen's D = mean/sd
mean(dat$High.Blame)/sd(dat$High.Blame)
compareValues(reportedValue = 0.65, obtainedValue = 0.2272631)
```
> There were no significant order effects for whether participants read low blame first (M = -22.05, SD = 32.89) or second (M = -13.18, SD = 33.59; p = .24) or high blame first (M = 9.57, SD = 40.96) or second (M = 8.16, SD = 37.61; p = .87).
**correct**
```{r}
#t-test
low1 <- as.numeric(dat[dat$Display.Order==1, c("Low.Blame")])
low2 <- as.vector(dat[dat$Display.Order==2, c("Low.Blame")])
t.test(low1,low2)
# descriptives
dat %>%
group_by(Display.Order) %>%
summarize(n=length(Low.Blame),
mean=mean(Low.Blame,na.rm=T),
sd=sd(Low.Blame,na.rm=T))
# t-test
high1 <- as.numeric(dat[dat$Display.Order==1, c("High.Blame")])
high2 <- as.vector(dat[dat$Display.Order==2, c("High.Blame")])
t.test(high1,high2)
# descriptives
dat %>%
group_by(Display.Order) %>%
summarize(n=length(High.Blame),
mean=mean(High.Blame,na.rm=T),
sd=sd(High.Blame,na.rm=T))
```
## Step 5: Conclusion
```{r}
codReport(Report_Type = 'pilot',
Article_ID = 'OvGDB',
Insufficient_Information_Errors = 0,
Decision_Errors = 0,
Major_Numerical_Errors = 2,
Minor_Numerical_Errors = 1)
```
The first large numerical error is related to the percent of people in the Low Blame condition who were above DV midpoint.
The likely cause is that the authors accidently reported the percent of people greater than **or equal to** the midpoint for Low Blame, instead of those greater than the midpoint.
There are errors in the two effect sizes reported. The likely cause is not apparent.
There is an anomoly in the order of presentation. The original reports Low Blame was displayed first for n=42, which requires that High Blame be first for n = 38, however looking at these data High Blame was also presented first for n=42. Not sure how to summarize this for our purposes because the numerical value reported is accurate.
```{r session_info, include=TRUE, echo=TRUE, results='markup'}
devtools::session_info()
```