cvms 1.0.1

• rsvg and ggimage are now only suggested and plot_confusion_matrix() throws warning if either are not installed.

• Additional input checks for evaluate().

cvms 1.0.0

Breaking changes

Changed arguments

• In cross_validate() and validate(), the models argument is renamed to formulas. This is a more meaningful name that was recently introduced in cross_validate_fn(). For now, the models argument is deprecated, will be used instead of formulas if specified, and will throw a warning.

• In cross_validate() and validate(), the model_verbose argument is renamed to verbose. This is a more meaningful name that was recently introduced in cross_validate_fn(). For now, the model_verbose argument is deprecated, will be used instead of verbose if specified, and will throw a warning.

• In cross_validate() and validate(), the link argument is removed. Consider using cross_validate_fn() or validate_fn() instead, where you have full control over the prediction type fed to the evaluation.

• In cross_validate_fn(), the predict_type argument is removed. You now have to pass a predict function as that is safer and more transparent.

• In functions with family/type argument, this argument no longer has a default, forcing the user to specify the family/type of the task. This also means that arguments have been reordered. In general, it is safer to name arguments when passing values to them.

• In evaluate(), apply_softmax now defaults to FALSE.
  Throws error if probabilities do not add up to 1 row-wise (tolerance of 5 decimals) when type is multinomial.

Changed metrics

• multinomial MCC is now the proper multiclass generalization. Previous versions used macro MCC. Removes MCC from the class level results. Removes the option to enable Weighted MCC.

• multinomial AUC is calculated with pROC::multiclass.roc() instead of in the one-vs-all evaluations. This removes AUC, Lower CI, and Upper CI from the Class Level Results and removes Lower CI and Upper CI from the main output tibble. Also removes option to enable "Weighted AUC", "Weighted Lower CI", and "Weighted Upper CI".

• multinomial AUC is disabled by default, as it can take a long time to calculate for a large set of classes.

• ROC columns now return the ROC objects instead of the extracted sensitivities and specificities, both of which can be extracted from the objects.

• In evaluate(), it's no longer possible to pass model objects. It now only evaluates the predictions. This removes the the AIC, AICc, BIC, r2m, and r2c metrics.

• In cross_validate and validate(), the r2m, and r2c metrics are now disabled by default in gaussian. The r-squared metrics are non-predictive and should not be used for model selection. They can be enabled with metrics = list("r2m" = TRUE, "r2c" = TRUE).

• In cross_validate_fn(), the AIC, AICc, BIC, r2m, and r2c metrics are now disabled by default in gaussian. Only some model types will allow the computation of those metrics, and it is preferable that the user actively makes a choice to include them.

• In baseline(), the AIC, AICc, BIC, r2m, and r2c metrics are now disabled by default in gaussian.
  It can be unclear whether the IC metrics (computed on the lm()/lmer() model objects) can be compared to those calculated for a given other model function. To avoid such confusion, it is preferable that the user actively makes a choice to include the metrics. The r-squared metrics will only be non-zero when random effects are passed. Given that we shouldn't use the r-squared metrics for model selection, it makes sense to not have them enabled by default.

Changes in functionality

• validate() now returns a tibble with the model objects nested in the Model column. Previously, it returned a list with the results and models. This allows for easier use in magrittr pipelines (%>%).

• In multinomial baseline(), the aggregation approach is changed. The summarized results now properly describe the random evaluations tibble, except for the four new measures CL_Max, CL_Min, CL_NAs, and CL_INFs, which describe the class level results. Previously, NAs were removed before aggregating the one-vs-all evaluations, meaning that some metric summaries could become inflated if small classes had NAs. It was also non-transparent that the NAs and INFs were counted in the class level results instead of being a count of random evaluations with NAs or INFs.

• cv_plot() is removed. It wasn't very useful and has never been developed properly. We aim to provide specialized plotting functions instead.

Additions

New functions

• validate_fn() is added. Validate your custom model function on a test set.

• confusion_matrix() is added. Create a confusion matrix and calculate associated metrics from your targets and predictions.

• evaluate_residuals() is added. Calculate common metrics from regression residuals.

• summarize_metrics() is added. Use it summarize the numeric columns in your dataset with a set of common descriptors. Counts the NAs and Infs. Used by baseline().

• select_definitions() is added. Select the columns that define the models, such as Dependent, Fixed, Random, and the (unnested) hyperparameters.

• model_functions() is added. Contains simple model_fn examples that can be used in cross_validate_fn() and validate_fn() or as starting points.

• predict_functions() is added. Contains simple predict_fn examples that can be used in cross_validate_fn() and validate_fn() or as starting points.

• preprocess_functions() is added. Contains simple preprocess_fn examples that can be used in cross_validate_fn() and validate_fn() or as starting points.

• update_hyperparameters() is added. For managing hyperparameters when writing custom model functions.

• most_challenging() is added. Finds the data points that were the most difficult to predict.

• plot_confusion_matrix() is added. Creates a ggplot representing a given confusion matrix. Thanks to Malte Lau Petersen (@maltelau), Maris Sala (@marissala) and Kenneth Enevoldsen (@KennethEnevoldsen) for feedback.

• plot_metric_density() is added. Creates a ggplot density plot for a metric column.

• font() is added. Utility for setting font settings (size, color, etc.) in plotting functions.

• simplify_formula() is added. Converts a formula with inline functions to a simple formula where all variables are added together (e.g. y ~ x*z + log(a) + (1|b) -> y ~ x + z + a + b). This is useful when passing a formula to recipes::recipe(), which doesn't allow the inline functions.

• gaussian_metrics(), binomial_metrics(), and multinomial_metrics() are added. Can be used to select metrics for the metrics argument in many cvms functions.

• baseline_gaussian(), baseline_binomial(), baseline_multinomial() are added. Simple wrappers for baseline() that are easier to use and have simpler help files. baseline() has a lot of arguments that are specific to a family, which can be a bit confusing.

New datasets

• wines dataset is added. Contains a list of wine varieties in an approximately Zipfian distribution.

• musicians dataset is added. This has been generated for multiclass classification examples.

• predicted.musicians dataset is added. This contains cross-validated predictions of the musicians dataset by three algorithms. Can be used to demonstrate working with predictions from repeated 5-fold stratified cross-validation.

New metrics

• Adds NRMSE(RNG), NRMSE(IQR), NRMSE(STD), NRMSE(AVG) metrics to gaussian evaluations. The RMSE is normalized by either target range (RNG), target interquartile range (IQR), target standard deviation (STD), or target mean (AVG). Only NRMSE(IQR) is enabled by default.

• Adds RMSLE, RAE, RSE, RRSE, MALE, MAPE, MSE, TAE and TSE metrics to gaussian evaluations. RMSLE, RAE, and RRSE are enabled by default.

• Adds Information Criterion metrics (AIC, AICc, BIC) to the binomial and multinomial output of some functions (disabled by default). These are based on the fitted model objects and will only work for some types of models.

• Adds Positive Class column to binomial evaluations.

New function arguments

• Adds optional hyperparameter argument to cross_validate_fn().
  Pass a list of hyperparameters and every combination of these will be cross-validated.

• Adds optional preprocess_fn argument to cross_validate_fn(). This can, for instance, be used to standardize the training and test sets within the function. E.g., by extracting the scaling and centering parameters from the training set and apply them to both the training set and the test fold.

• Adds Preprocess column to output when preprocess_fn is passed. Contains returned parameters (e.g. mean, sd) used in preprocessing.

• Adds preprocess_once argument to cross_validate_fn(). When preprocessing does not depend on the current formula or hyperparameters, we might as well perform it on each train/test split once, instead of for every model.

• Adds metrics argument to baseline(). Enable the non-default metrics you want a baseline evaluation for.

• Adds preprocessing argument to cross_validate() and validate(). Currently allows "standardize", "scale", "center", and "range". Results will likely not be affected noticeably by the preprocessing.

• Adds add_targets and add_predicted_classes arguments to multiclass_probability_tibble().

• Adds Observation column in the nested predictions tibble in cross_validate(), cross_validate_fn(), validate(), and validate_fn(). These indices can be used to identify which observations are difficult to predict.

• Adds SD column in the nested predictions tibble in evaluate() when performing ID aggregated evaluation with id_method = 'mean'. Th...

cvms 0.3.2

• Fixes bug in evaluate(), when used on a grouped data frame. The row order in the output was not guaranteed to fit the grouping keys.

cvms 0.3.1

• Fixes documentation in cross_validate_fn(). The examples section contained an unreasonable number of mistakes :-)

• In cross_validate_fn(), warnings and messages from the predict function are now included in
  Warnings and Messages. The warnings are counted in Other Warnings.

cvms 0.3.0

• cross_validate_fn() is added. Cross-validate custom model functions.

• Breaking change: In evaluate(), when type is multinomial, the output is now a single tibble. The Class Level Results are included as a nested tibble.

• Breaking change: In baseline(), lmer models are now fitted with REML = FALSE by default.

• Adds REML argument to baseline().

• Bug fix: the control argument in cross_validate() was not being used. Now it is.

• In cross_validate(), the model is no longer fitted twice when a warning is thrown during fitting.

• Adds metrics argument to cross_validate() and validate(). Allows enabling the regular Accuracy metric
  in binomial or to disable metrics (will currently still be computed but not included in the output).

• AICc is now computed with the MuMIn package instead of the AICcmodavg package, which
  is no longer a dependency.

• Adds lifecycle badges to the function documentation.

cvms 0.2.0

• evaluate() is added. Evaluate your model's predictions with the same metrics as used in cross_validate().

• Adds "multinomial" family/type to baseline() and evaluate().

• Adds multiclass_probability_tibble() for generating a random probability tibble.

• Adds random_effects argument to baseline() for adding random effects to the Gaussian baseline model.

• Adds Zenodo DOI for easier citation.

• In nested confusion matrices, the Reference column is renamed to Target, to use the same naming scheme as in the nested predictions.

Patch

• Bug fix: p-values are correctly added to the nested coefficients tibble. Adds tests of this table as well.
• Adds extra unit tests to increase code coverage.
• When argument "model_verbose"" is TRUE, the used model function is now messaged instead of printed.
• Adds badges to README, including travis-ci status, AppVeyor status, Codecov, min. required R version, CRAN version and monthly CRAN downloads. Note: Zenodo badge will be added post release.

Minor Patch

• Unit tests have been made compatible with R v. 3.5

First CRAN release

News (relative to previous development versions):

• Adds optional parallelization.
• Results now contain a count of singular fit messages. See ?lme4::isSingular for more information.
• Argument "positive" changes default value to 2. Now takes either 1 or 2 (previously 0 and 1). If your dependent variable has values 0 and 1, 1 is now the positive class by default.
• AUC calculation has changed. Now explicitly sets the direction in pROC::roc.
• Unit tests have been updated for the new random sampling generator in R 3.6.0. They will NOT run previous versions of R.
• Adds baseline() for creating baseline evaluations.
• Adds reconstruct_formulas() for reconstructing formulas based on model definition columns in the results tibble.
• Adds combine_predictors() for generating model formulas from a set of fixed effects.
• Adds select_metrics() for quickly selecting the metrics and model definition columns.
• Breaking change: Metrics have been rearranged and a few metrics have been added.
• Breaking change: Renamed argument folds_col to fold_cols to better fit the new repeated cross-validation option.
• New: repeated cross-validation.
• Created package :)