Transferring Python optimizer 'AdaMod' to Keras in R

[faltinl]

In An Adaptive and Momental Bound Method for Stochastic Learning, a new optimizer AdaMod is being reported which is able to smooth out unexpected large learning rates throughout the training process. For details regarding its use in Python environments see https://github.com/lancopku/AdaMod. A recommended setting under these conditions is

optimizer = optim.AdaMod(
    m.parameters(),
    lr= 1e-3,
    betas=(0.9, 0.999),
    beta3=0.999,
    eps=1e-8,
    weight_decay=0,
)

Since I am working in RStudio with Keras in R, I tried to transfer AdaMod to my working reticulate/conda environment by following steps:

library(keras)  
reticulate::py_install(c('adamod'), pip = TRUE)

which produced

Requirement already up-to-date: adamod in c:\minico~1\envs\r-reti~1\lib\site-packages (0.0.3)
Requirement already satisfied, skipping upgrade: torch>=0.4.0 in c:\minico~1\envs\r-reti~1\lib\site-packages (from adamod) (1.4.0)

from which I gained some confidence to try AdaMod in one of my deep learning programs, configuring my model as I am used to do in case of Keras standard optimizers by specifying

...
losslist <- list(output = 'kullback_leibler_divergence') 
metlist <- list(output = 'accuracy')
model %>% compile(AdaMod(m.parameters(), 
                        lr= 1e-3, 
                        betas=c(0.9, 0.999), 
                        beta3=0.999, 
                        eps=1e-8, 
                        weight_decay=0),
                  loss = losslist,
                  weighted_metrics = metlist)  

However, the disappointing result was

error in AdaMod(m.parameters(), lr = 0.001, betas = c(0.9, 0.999), beta3 = 0.999, ...:
   could not find function "AdaMod" 
Called from: compile.keras.engine.training.Model(., AdaMod(m.parameters(), 
    lr = 0.001, betas = c(0.9, 0.999), beta3 = 0.999, eps = 1e-08, 
    weight_decay = 0), loss = losslist, weighted_metrics = metlist)

The question obviously is, how could I proceed to get Keras in R to find and successfully apply this function?

Note: This problem was raised about a year ago in

1. How to install optimizer AdaMod into Keras/Reticulate virtual environment? #2
2. as well as in Transfer of a new Python optimizer to Keras in R, #1049

but, unfortunately, is still pending unresolved. Any help would be highly appreciated.

[t-kalinowski]

Hi, Thanks for filing.

a) The link you provide to AdaMod is for a pytorch implementation. When you run pip install adamod, that also pulls down the pytorch implementation. AFIK, it will not work with a tensorflow/keras backend, regardless of if you're in R or python.

b) In one of the issues you cross-posted, someone helpfully pointed you to a tensorflow implementation.

c) Unfortunately, the tensorflow implementation is not "packaged" up such that it will install nicely using pip or conda. It's a standalone python script that implements the optimizer.

If you are interested in trying out the tensorflow implementation of the AdaMod optimizer, implemented by githhub user evanatyourservice, instead of installing it through pip or conda, you can clone it and then import it like this.

git clone https://github.com/evanatyourservice/AdaMod-tf

Then in the R session, you can import it like so:

AdaMod <- reticulate::import_from_path("AdaMod", "/path/to/where/you/cloned/AdaMod-tf")$AdaMod

[faltinl]

Tnx a lot. That sounds great. Sorry, I've never before worked with GitHub repositories.

However, when I push the "Code"-button right above evanatyourservice beta 3 , there is an option to get a ZIP download of the files contained in this repo. Is it not possible to import AdaMod from there as well? And if so, how would I proceed from thereon?

When I did this, the command

AdaMod <- reticulate::import_from_path("AdaMod", "D:/R/WP_keras/AdaMod/AdaMod.py")$AdaMod

was perfectly accepted in the R session, but compiling the (existing) model with

model %>% compile(AdaMod(lr= 1e-3, 
                         betas=c(0.9, 0.999), 
                         beta3=0.9995, 
                         eps=1e-8, 
                         weight_decay=0),
                   loss = losslist,
                   weighted_metrics = metlist) 

produced the error message

Error in AdaMod(lr = 0.001, betas = c(0.9, 0.999), beta3 = 0.9995, eps = 1e-08, , eps = 1e-08, weight_decay=0): could not find function "AdaMod"

So, apparently, that's not the right way to proceed...

[t-kalinowski]

reticulate::import_from_path takes a path to the directory to where the module exists, not a path to the module itself. I think if you modify the above like this it should work:

reticulate::import_from_path("AdaMod", "D:/R/WP_keras/AdaMod/")

[faltinl]

Thanks again for your continued interest in my problem. I am sorry to say, however, the suggestion from above didn't solve it. The command

AdaMod <- reticulate::import_from_path("AdaMod", "D:/R/WP_keras/AdaMod/")

was again accepted smoothly, but executing the compile command model %>% compile(AdaMod... produced exactly the same error message as before,

Error in AdaMod(lr = 0.001, betas = c(0.9, 0.999), beta3 = 0.9995, eps = 1e-08, , eps = 1e-08, weight_decay=0): could not find function "AdaMod",

although AdaMod has correctly identified the .py-module and somehow Keras had an inkling of the expected status of AdaMod being a module, too, as you can see from a look at its properties in RStudio's environment pane:

Among the many other attempts to get AdaMod going I tried a command similar to your very first proposal, namely

AdaMod <- reticulate::import_from_path("AdaMod", "D:/R/WP_keras/AdaMod/")$AdaMod

This command was, again, accepted and according to the RStudio environment pane produced a class object,

but trying to compile the existing Keras model as above led to the following error message.

Error in py_call_impl(callable, dots$args, dots$keywords) : 
  TypeError: Unexpected keyword argument passed to optimizer: betas

Detailed traceback:
  File "D:\R\WP_keras\AdaMod\AdaMod.py", line 19, in __init__
    super(AdaMod, self).__init__(name, **kwargs)
  File "C:\MINICO~1\envs\R-RETI~1\lib\site-packages\tensorflow\python\keras\optimizer_v2\optimizer_v2.py", line 269, in __init__
    "passed to optimizer: " + str(k))
Called from: py_call_impl(callable, dots$args, dots$keywords)

I'm really at a loss how to proceed...