Visualize DTox visible neural network model training, performance, and comparison with other methods
-
First, the evolution of training/testing loss over epoches during DTox learning process was visualized using line charts for each dataset. Overall, the optimal DTox model can be reached within 100-200 epoches for the implementation task on Tox21 datasets. After the optimal point, training loss keeps decreasing while testing loss remains fluctuant or increases.
-
In addition, the relative efficiency and performance of DTox under alternative settings of early stopping criterion were visualized using line charts. On average, the run time of DTox can be saved by 30% with a 2% sacrifice in model performance. Further, the run time of DTox can be cut in half with a 5% sacrifice in model performance. These statistics offer DTox users some flexibility in balancing model efficiency and performance, especially when implementing the model on larger datasets.
-
Next, model statistics of optimal DTox and matched matched multi-layer perceptron (MLP) were visualized using barplots for each dataset. On average, an optimal DTox visible neural network model contains ~400 hidden pathway modules and ~40,000 learned parameters. The average ratio between training samples and number of DTox parameters is ~0.15, ~40 times of the average ratio for VNN parameters.
-
Then, normalized DTox model performance of Tox21 datasets across root pathway settings were visualized using heatmap and upsetplot. The optimal root pathway setting varies by dataset. Overall, including more pathways in the root of DTox visible neural network results in better model performance.
-
Finally, validation performance of DTox visible neural network model (DTox) was compared against other method implementations. The comparison was visualized using barplots for each dataset. Three other methods were considered: i) fully connected MLP neural network model, otherwise with the same number of hidden layer/neuron as the matched DTox model (MLP), ii) optimal random forest model after hyperparameter tuning (RF), and iii) optimal gradient boosting model after hyperparameter tuning (GB). In general, DTox model achieved the same level of predictive performance as these well-established classification algorithms.
-
In addition, the validation performance of optimal DTox visible neural network model was also compared against alternative models built upon shuffled layouts. The comparison was visualized using barplots. Three alternative layouts were considered: i) alternative DTox visible neural network model with the same setting as optimal model but built under shuffled Reactome pathway hierarchy (ontology), ii) alternative DTox visible neural network model with the same setting as optimal model but trained with input data of shuffled feature profile (feature), and iii) alternative DTox visible neural network model with the same setting as optimal model but trained with input data of shuffled outcome (outcome). In general, shuffled feature profiles significantly impacted the predictive performance of DTox, as the resulting models exhibited random performance resembling negative controls from the shuffled outcome. By contrast, shuffled ontology hierarchy moderately impacted the predictive performance of DTox.
-
First, the similarity of significant DTox paths under different hyperparameter settings of layer-wise relevance propagation rule was visualized using heatmap. The heatmap shows an average summary across results on 15 Tox21 datasets. Overall, the hyperparameter setting does not have strong influence on the detected significant DTox paths. The median Jaccard Index among hyperparameter settings is ~0.7.
-
Next, the comparison between proportion of differentially expressed DTox paths among significant DTox paths vs all background paths was visualized using scatterplot. The scatterplot shows the comparison for significant DTox paths identified using γ = 0.001 and Ɛ = 0.1. Comparison was shown separately for results from four Tox21 assays (from left to right: aromatase, mitochondria toxicity, PXR agonist, HepG2 cell viability) and three dose-time combinations (from top to bottom 10uM-6h, 10um-24h, 1.11-24uM). The differentially expressed DTox paths that recurrently appear across compounds in aromatase assay were visualized using barplot. The paths relevant to 'Transcriptional Regulation by TP53' is colored in red. In two assays (aromatase and PXR agonist), proportion of differentially expressed DTox paths among significant DTox paths is significantly higher than proportion among all background paths.
-
Then, the comparison between observed and expected proportion of validated compounds was visualized using density plots and barplots. Each plot shows the comparison for significant DTox paths identified using γ = 0.001 and Ɛ = 0.1. Comparison was shown separately for results from four Tox21 assays: AR, ER, RAR, and ROR. In each plot, the density plot on top shows the distribution of sampled expected proportions while the barplot on bottom shows the comparison between observed and expected proportion. In three assays (ER, RAR, and ROR), the observed proportion of validated compounds is at least twice of expected proportion of validated compounds. The validation performance was also compared against LIME (with strict and lax threshold) and Read-across (with source information from DrugBank and ComptoxAI) in line charts.
-
In addition, the relationship map between compounds and nine viability-related pathways in the context of HepG2 viability assay dataset was visualized using heatmap. The enrichment of drug-induced liver injury adverse events or ATC drug classes among compounds identified with viability-related pathways was visualized using heatmap. In each heatmap, cells that represent significant enrichment between column and row pairs were annotated with stars. The comparison of DTox pathway module relevance scores between active and inactive compounds was visualized using survival plots. Each plot shows the comparison for nine viability-related DTox paths identified using γ = 0.001 and Ɛ = 0.1. Comparison was shown separately for two Tox21 assays: CASP3/7 apoptosis and mitochondria toxicity. And the flow of relevance along DTox viability-related paths between query compound (mifepristone), hidden pathway modules, and the HepG2 cell viability outcome was visualized using visNetwork plot. For cytotoxic compounds not linked to the viability-related pathways by DTox, the top 30 most prevalent target proteins and lowest level pathways were visualized using barplots.
-
The distributions of predicted HepG2 and HEK293 cytotoxicity scores among EPA/DrugBank lists were shown in boxplots.
-
The comparison of predicted cytotoxicity scores between positive and negative compounds of DILI phenotypes or DIKI phenotypes was shown in boxplot. As comparison, the association odds ratio between HepG2 cytotoxicity and DILI phenotypes and between HEK293 cytotoxicity and DIKI phenotypes were visualized in barplot. Of all DILI and DIKI phenotypes, only 'hepatic cyst' has an OR significantly greater than 1 (lower bound of 95% confidence interval > 1 as shown in the barplot). And it was shown that the predicted HepG2 cytotoxicity scores of positive compounds associated with hepatic cyst is significantly higher than that of negative compounds (P < 0.05 as shown in the boxplot with a red asterisk along the boxes).