Propose really easy reinforcement learning API for v3

[robertleeplummerjr]

What is wrong?

Where does it happen?

How do we replicate the issue?

How important is this (1-5)?

Expected behavior (i.e. solution)

Other Comments

[robertleeplummerjr]

Proposal for basic feed forward style class:

const net = new brain.Reinforce();

Proposal for "episodes" style class:

const net = new brain.RNNReinforce();
const net = new brain.LSTMReinforce();

Inputs and outputs work like that of a standard net. Reward function:

const net = new brain.Reinforce({
  reward: (?) => {
    return 1;
  }
});
const net = new brain.LSTMReinforce({
  reward: (?) => {
    return 1;
  }
});

[robertleeplummerjr]

In addition to Reinforce, there is another part of reinforcement learning that is super important for winning at games, like Chess, which is used for exploration. As far as I'm aware the state of the art best practice is the "Monte Carlo" algorithm and I'd like to at least consider how to generalize an implementation of this where the net can have this fit in like a puzzle piece that is super simple.

I found an excellent article with step by step implementation here: https://medium.com/@quasimik/implementing-monte-carlo-tree-search-in-node-js-5f07595104df

I want to find a good term for this that highlights the practice, over the algorithm, but will start to ponder over that based on these terms:

• Monte Carlo
• Exploitation
• Tree
• Strategy

[DanielMazurkiewicz]

Just some idea... :

const net = new brain.Reinforce(options)
   .reward((some_args) => {
      return some_value
   });

@robertleeplummerjr Where can I find current V3 API proposal?

[samfultonjr]

Is there any ETA on when this will be a thing? This would be amazing if we could use reinforcement learning in brain.js!

[maximilliangeorge]

As someone who is pretty new to this, would this allow me to create a net that can learn over time (from running operations using net.run())?

According to the docs I need to train before running operations:

Use train() to train the network with an array of training data. The network has to be trained with all the data in bulk in one call to train()

Maybe this isn't even correct? I seem to be able to switch between training and running without problem.

[robertleeplummerjr]

Sorry everyone for my tardiness. I've been getting GPU.js, which is brain.js's v2 engine, ready, and am nearly done. I've been saying this for a while, but all I can say is: it will be right, and it will be revolutionary for javascript. I believe it (GPU.js v2) to be days from release. But I will take a break and answer this thread because it is very important to me, and I think about it frequently.

const net = new brain.Reinforce(options)
 .reward((some_args) => {
     return some_value
  });

This is precisely what I mean.

@robertleeplummerjr Where can I find current V3 API proposal?

v3 has not been fully proposed as of yet, but the latest is here:
https://github.com/BrainJS/brain.js/wiki/Roadmap

Is there any ETA on when this will be a thing? This would be amazing if we could use reinforcement learning in brain.js!

I don't have one yet. Let's (all of us) get busy and start making it happen!

As someone who is pretty new to this, would this allow me to create a net that can learn over time (from running operations using net.run())?

Yes. This proposal would be a continuous learning system that, rather than train towards an error rate of 0, it trains for highest rewards. But keep in mind this is a proposal for v3, so it is not yet even proof of concept. That being said, the GPU.js v2 release, when finally released, will make iteration much faster.

Maybe this isn't even correct? I seem to be able to switch between training and running without problem.

It isn't yet because we are talking about a hypothetical api that doesn't yet exist.

[myisaak]

Well the qlearn reinforcement models currently available on nodejs have simple api to reference from. I suggest taking a look at them

[robertleeplummerjr]

This has gained some traction here: https://github.com/BrainJS/rl
I wanted to improve the initial design before "brainjsing" it.

[BasToTheMax]

Any updates?

[robertleeplummerjr]

Can you go ahead and try the project mentioned above? That will likely be the initial candidate. https://github.com/BrainJS/rl works fairly well and is fairly well tested. I want to find its weak area prior to analyzing if it is even possible for the API to fit here.

[robertleeplummerjr]

TL;DR
You can now install the experimental version from https://www.npmjs.com/package/@brain/rl via:

npm install @brain/rl

TS;DR
We secured the "brain" organization from NPM!!!! This took YEARS of work and patience, and communicating with them through their immense workload, and THEY FINALLY DID IT! The initial release of the reinforcement learning project based on work from Andrej Karpathy is now available from npm as https://www.npmjs.com/package/@brain/rl.

Consider this closed!