Skip to content

Latest commit

 

History

History
99 lines (90 loc) · 9.32 KB

Simulations_results.md

File metadata and controls

99 lines (90 loc) · 9.32 KB
Raw

This file contains supplementary material on the simulation experiments reported in Ghasempour, Moosavi and de Luna (2023, Convolutional neural networks for valid and efficient causal inference; arXiv version).

Architecture and hyperparameters

For FNN and CNN we need to choose an architecture (number of layers and number of neurons per layers, and for CNN filter size $s$ as well) and hyperparameters (epoch size, batch size, learning rate). Number of layers was arbitrarily set to two (deeper NN would imply heavy computational costs in a simulation study). Number of neurons per layers for CNN was chosen to optimize prediction performance of the nuisance models on training data for one replication (kept then fixed for all replications). For FNN, the number of neurons per layers was chosen to make sure that the number of free parameters is the same for FNN and CNN. Filter size for CNN was fixed to $s=4$ for first layer and 3 for the second layer, which somehow corresponds to the filters defining the DGP of setting two although importantly the latter are non-linear.
Finally, hyperparameters were chosen as number of neurons per layers, i.e. by to optimize prediction performance of the nuisance models on training data for one replication (kept then fixed for all replications).

Simulation results

Here we display the results of the simulations described in ReadMe and in the manuscript. First, results with normally distributed errors for the outcome models, followed by results with heavy-tailed Gamma distribution for the same error terms. Results are commented in the manuscript.

Normally distributed error terms

DGP1 SS 5000
est_ATT lower_bound upper_bound bias coverage montCarlo_sd Mean_SD
cnn -6.07033 -8.74736 -3.3933 0.184982 0.947 1.378719 1.365855
fnn -6.31148 -8.92061 -3.70235 0.426133 0.94 1.353171 1.331214
OR_ds -5.88626 -8.61459 -3.15793 0.000911 0.947 1.403742 1.392029
DR_xy -5.8888 -8.62247 -3.15512 0.003447 0.947 1.40511 1.394758
OR_xy -5.88794 -8.61625 -3.15964 0.002596 0.947 1.404614 1.392018
farrell -5.88766 -8.62133 -3.15398 0.002307 0.948 1.403854 1.394758
DR_ds -5.88824 -8.62193 -3.15454 0.00289 0.948 1.404323 1.394768
tmle -5.9294 -8.64277 -3.21604 0.044055 0.944 1.39612 1.384396
DGP1 SS 10000
cnn -5.896 -7.80054 -3.99146 0.089426 0.958 0.971114 0.971723
fnn -6.02934 -7.90753 -4.15115 0.222763 0.949 0.960799 0.958279
OR_ds -5.80536 -7.73039 -3.88032 0.001219 0.955 0.980653 0.982178
DR_xy -5.80664 -7.73522 -3.87806 6.45E-05 0.955 0.981272 0.983987
OR_xy -5.80636 -7.73135 -3.88137 0.000216 0.955 0.981023 0.982157
farrell -5.80588 -7.73447 -3.87729 0.000694 0.955 0.980953 0.983993
DR_ds -5.80587 -7.73447 -3.87728 0.000701 0.955 0.980973 0.983995
tmle -5.83415 -7.75189 -3.9164 0.027571 0.957 0.980767 0.97846
DGP2 SS 5000
est_ATT lower_bound upper_bound bias coverage montCarlo_sd Mean_SD
cnn -1.5271 -1.72529 -1.32891 0.046949 0.937 0.09383 0.101118
fnn -1.55636 -1.75128 -1.36144 0.076213 0.89 0.094887 0.09945
OR_ds -1.71287 -1.87972 -1.54603 0.232723 0.206 0.082722 0.085127
DR_xy -1.71886 -1.89826 -1.53947 0.238715 0.225 0.081974 0.091529
OR_xy -1.71715 -1.88341 -1.55088 0.236998 0.182 0.082289 0.08483
farrell -1.71328 -1.89356 -1.53299 0.233127 0.26 0.082675 0.091985
DR_ds -1.71474 -1.89472 -1.53476 0.234591 0.25 0.082568 0.091829
tmle -1.71327 -1.8807 -1.54583 0.233118 0.204 0.08286 0.085428
DGP2 SS 10000
cnn -1.50465 -1.64627 -1.36303 0.024949 0.945 0.070034 0.072255
fnn -1.51598 -1.65634 -1.37562 0.036278 0.928 0.070095 0.071613
OR_ds -1.70372 -1.82246 -1.58498 0.224016 0.04 0.061793 0.060582
DR_xy -1.70887 -1.83649 -1.58125 0.229165 0.044 0.061058 0.065114
OR_xy -1.70686 -1.82539 -1.58834 0.227163 0.037 0.061493 0.060474
farrell -1.70399 -1.83196 -1.57601 0.224285 0.054 0.061758 0.065293
DR_ds -1.7058 -1.83362 -1.57799 0.2261 0.045 0.061453 0.065213
tmle -1.70397 -1.82302 -1.58493 0.22427 0.041 0.061736 0.060738

Gamma distributed error terms

DGP1 SS 5000
est_ATT lower_bound upper_bound bias coverage montCarlo_sd Mean_SD
cnn -5.94027 -8.68325 -3.19728 0.19163 0.952 1.357459 1.399506
fnn -6.20015 -8.86918 -3.53112 0.451516 0.944 1.324664 1.361774
OR_ds -5.74284 -8.51278 -2.9729 0.005797 0.962 1.380977 1.413261
DR_xy -5.76312 -8.56358 -2.96266 0.014484 0.965 1.381555 1.42883
OR_xy -5.76104 -8.52987 -2.99221 0.012401 0.962 1.382306 1.412695
farrell -5.75503 -8.55618 -2.95388 0.006397 0.966 1.380812 1.429184
DR_ds -5.7566 -8.5584 -2.9548 0.007964 0.964 1.38076 1.429517
tmle -5.7987 -8.5548 -3.0426 0.050064 0.963 1.368193 1.406199
DGP1 SS 10000
cnn -5.888 -7.84491 -3.93108 0.1127 0.94 1.009305 0.998446
fnn -6.04585 -7.97311 -4.11858 0.270552 0.936 0.988616 0.983317
OR_ds -5.77682 -7.73786 -3.81578 0.001521 0.94 1.020658 1.00055
DR_xy -5.7819 -7.76486 -3.79894 0.006606 0.944 1.020083 1.011733
OR_xy -5.78174 -7.74245 -3.82102 0.006442 0.94 1.019874 1.000383
farrell -5.7815 -7.76467 -3.79832 0.006199 0.941 1.020483 1.011844
DR_ds -5.78139 -7.76475 -3.79803 0.00609 0.941 1.020473 1.011938
tmle -5.80749 -7.76186 -3.85312 0.032194 0.934 1.023058 0.997148
DGP2 SS 5000
est_ATT lower_bound upper_bound bias coverage montCarlo_sd Mean_SD
cnn -2.93178 -3.1301 -2.73345 0.047603 0.936 0.097309 0.101188
fnn -2.95734 -3.15074 -2.76394 0.07317 0.891 0.096676 0.098675
OR_ds -3.13088 -3.2934 -2.96837 0.246711 0.159 0.082904 0.082918
DR_xy -3.13328 -3.30223 -2.96432 0.249103 0.164 0.082468 0.086201
OR_xy -3.13332 -3.29536 -2.97129 0.249151 0.145 0.082274 0.082674
farrell -3.13011 -3.29981 -2.96042 0.24594 0.181 0.083138 0.086581
DR_ds -3.13031 -3.29983 -2.96078 0.246134 0.178 0.083093 0.086494
tmle -3.13009 -3.29321 -2.96697 0.245917 0.161 0.083118 0.083224
DGP2 SS 10000
cnn -2.90537 -3.0481 -2.76263 0.022512 0.945 0.071362 0.072826
fnn -2.91384 -3.05509 -2.77259 0.030986 0.924 0.071389 0.072065
OR_ds -3.12497 -3.24035 -3.0096 0.24212 0.012 0.05954 0.058866
DR_xy -3.12767 -3.24779 -3.00754 0.244812 0.014 0.059416 0.061291
OR_xy -3.12756 -3.24275 -3.01237 0.244703 0.008 0.059314 0.058771
farrell -3.12461 -3.24502 -3.0042 0.241758 0.018 0.05973 0.061434
DR_ds -3.12503 -3.24539 -3.00467 0.242175 0.018 0.059587 0.06141
tmle -3.12467 -3.2404 -3.00895 0.24182 0.012 0.059778 0.059044