Modify examples to be simpler

[diogo149]

I'm curious if other people would be interested in having examples in this fashion: https://github.com/enlitic/lasagne4newbs/blob/master/mnist_conv.py. I've found the existing examples to be a little confusing to people new to Lasagne, and made that version to help them out. I think the biggest downside would be teaching slightly less than optimal practices (specifically not transfering a large amount of input data to the GPU at once).

[benanne]

I'm inclined to agree, we may have gone overboard cleaning them up a bit in the past (they used to be simpler before, e.g. 98f2ee7).

It might be better to dial that back a little and cut some corners in the interest of immediate clarity of the code, since the main purpose of the examples is to quickly show people how to get things done, after all. While all the functions are a good idea in practice, they do obscure the flow a bit.

One downside is that this will complicate testing of the examples. We currently have tests that run each example for 1 epoch to ensure everything still works. With scripts like this one, this is impossible (unless we make it accept command line arguments which in turn complicates the example again). I wonder if there's a way around this.

[aleksaro]

Just thinking out loud here, but wouldn't it be an idea to move usage examples of Lasagne into another repository within the Lasagne organization?

This would further reduce the scope of this library while also simplify test coverage.

[f0k]

I wonder if there's a way around this.

We don't have to cut it back that far. Having a main() function is probably okay, and I think wrapping the model creation in a function is nice as well. What I find confusing are the indirections of create_iter_functions() and train() -- I need to flip between files and read a lot of code to understand what's going on, I need to know about generator functions, and I have to focus hard to keep in mind that despite the name, create_iter_functions doesn't have anything to do with Python's iter.

The only reason why we would want those indirections is to share code between the four examples (one for an MLP, three for CNNs), but the current examples are still a pretty complete failure at sharing code. What if we strip that back to a single mnist.py that has a parameter to choose which build_model() is executed: one for a simple MLP, one for a CNN? That example should have extensive comments similar to Diogo's one, to explain things in a tutorial style rather than just throwing the code at the unsuspecting :)

I think the biggest downside would be teaching slightly less than optimal practices (specifically not transfering a large amount of input data to the GPU at once).

I think a big downside of the current example is that it requires transferring everything at once. If I recall correctly, several users have asked how to bypass this. Your example is both easier to understand and easy to extend to large datasets. In the long term, we'd need to include (or refer to) a real tutorial handling different ways of handling the input data, but for now I'd say your style is suited better.

[f0k]

Just thinking out loud here, but wouldn't it be an idea to move usage examples of Lasagne into another repository within the Lasagne organization?

Hmm, that would be an option. Currently we need it within the repository so we can keep the examples in sync with API-breaking changes, and I think good manners dictate providing an example directly with the library, but we could have a separate repository for reproducing results of some popular papers on some popular datasets, for example.

[benanne]

What if we strip that back to a single mnist.py that has a parameter to choose which build_model() is executed: one for a simple MLP, one for a CNN? That example should have extensive comments similar to Diogo's one, to explain things in a tutorial style rather than just throwing the code at the unsuspecting :)

That sounds like an excellent idea, and it mirrors how I use the library in practice (with config files describing the model architecture that are actually Python modules).

[f0k]

it mirrors how I use the library in practice (with config files describing the model architecture that are actually Python modules).

Same here :)

[dnouri]

On Mo, Apr 20, 2015 at 12:35 , Jan Schlüter notifications@github.com
wrote:

I wonder if there's a way around this.

We don't have to cut it back that far. Having a main() function is
probably okay, and I think wrapping the model creation in a function
is nice as well. What I find confusing are the indirections of
create_iter_functions() and train() -- I need to flip between files
and read a lot of code to understand what's going on, I need to know
about generator functions, and I have to focus hard to keep in mind
that despite the name, create_iter_functions doesn't have anything to
do with Python's iter.

The intention behind splitting the example up into these functions
wasn't code sharing, it was trying to get a grasp on what the
dependencies between the different parts are (and it's still hard for
me to understand those sometimes).

I usually try to avoid having functions with a lot of lines: they
become hard to understand because everything potentially depends on
everything else that's happened in the lines above, so that's
confusing. For this reason, my intuition was that it would be helpful
to split that 200+ line no-function example into smaller parts that can
stand on their own, but it turns out this undertaking was only modestly
successful.

I think there's ultimately a lot of DRY code in those examples that
needs to go into the library proper (and there's a ticket for that). I
consider it problematic that we're suggesting newcomers start out by
copying 200+ lines of code to start off their own experiments. It
means that now they're maintaining many lines of code that they have to
understand, instead of being able to start working on their problem
right away.

So I guess the deeper problem behind the examples is that that they are
just too verbose and daunting to new users. Of course you've seen the
Keras examples; I think this is what we want, at least for the
mainstream stuff like MNIST classification. And then there could be
another example where everything is hand-woven and super flexible like
currently.

The only reason why we would want those indirections is to share code
between the four examples (one for an MLP, three for CNNs), but the
current examples are still a pretty complete failure at sharing code.
What if we strip that back to a single mnist.py that has a parameter
to choose which build_model() is executed: one for a simple MLP, one
for a CNN? That example should have extensive comments similar to
Diogo's one, to explain things in a tutorial style rather than just
throwing the code at the unsuspecting :)

Yes, cutting it down to main() and build_model() sounds good!

[benanne]

Agreed, but as long as there is no training loop code in the library there is no way that's going to happen. Coincidentally I spent some time last week setting up a bit of an experimentation framework, parts of which could probably be added to the library for this purpose. But I don't think we should start working on that until after the first release is out.

[f0k]

I think there's ultimately a lot of DRY code in those examples that
needs to go into the library proper (and there's a ticket for that). I
consider it problematic that we're suggesting newcomers start out by
copying 200+ lines of code to start off their own experiments. It
means that now they're maintaining many lines of code that they have to
understand, instead of being able to start working on their problem
right away.

No, I think that's necessary. Lasagne is just a collection of tools to make it easier to setup Theano graphs and update expressions for neural networks, and you have to go through and understand the 200+ lines of example code if you want to apply it to your own problem. Splitting it up into separate functions is good if it helps understanding the example, I'm definitely not against it, but currently I'm afraid it's making things more complicated and should be left to the individual users.

A compromise might be providing an everything-in-one-place example and then an aiming-at-reusability example as a refactored version of the former. That way there's one for people to read and understand (with little cognitive overhead), and one for people to copy as a starting point if they like the suggested refactoring. If there's just a well-engineered usage example, I'd spend half of the time figuring out what the flat code equivalent would be so I can a) see what the example code does and b) understand the software design decisions of the example and whether they're the same for my task or need a different approach.

instead of being able to start working on their problem right away.

This is what's solved by higher-level wrappers such as your nolearn, isn't it? I think Lasagne is aiming at a lower level. But sure, it's possible that some things could be turned into helper functions useful to a broader audience, outside of the usage examples.

Coincidentally I spent some time last week setting up a bit of an experimentation framework

I've got one as well, it's really nice and modular, but not finished enough to publish yet. I'm not sure if we can add bits of that to Lasagne, though, I think that would be somewhat outside the scope.

Yes, cutting it down to main() and build_model() sounds good!

Great! Glad to see you again, by the way!

[dnouri]

On Mi, Apr 22, 2015 at 4:44 , Jan Schlüter notifications@github.com
wrote:

I think there's ultimately a lot of DRY code in those examples that
needs to go into the library proper (and there's a ticket for that).
I
consider it problematic that we're suggesting newcomers start out by
copying 200+ lines of code to start off their own experiments. It
means that now they're maintaining many lines of code that they have
to
understand, instead of being able to start working on their problem
right away.

No, I think that's necessary. Lasagne is just a collection of tools
to make it easier to setup Theano graphs and update expressions for
neural networks, and you have to go through and understand the 200+
lines of example code if you want to apply it to your own problem.
Splitting it up into separate functions is good if it helps
understanding the example, I'm definitely not against it, but
currently I'm afraid it's making things more complicated and should
be left to the individual users.

Yeah I agree that the examples factoring wasn't a particular success.

The question about the scope of Lasagne is an interesting one. Is it a
tool for Theano users that want to build things from scratch? Probably
even seasoned Theano users could benefit from things like training loop
code that allows them to write less code and concentrate on their own
problem faster.

The question remains whether Lasagne also wants to be attractive to
users that aren't familiar with Theano. I get the feeling that a lot
of nolearn.lasagne users have little idea about how to use Theano but
they can still use the tools to good effect. And it doesn't take a lot
of code to allow them this more gentle intro to Lasagne.

A compromise might be providing an everything-in-one-place example
and then an aiming-at-reusability example as a refactored version of
the former. That way there's one for people to read and understand
(with little cognitive overhead), and one for people to copy as a
starting point if they like the suggested refactoring. If there's
just a well-engineered usage example, I'd spend half of the time
figuring out what the flat code equivalent would be so I can a) see
what the example code does and b) understand the software design
decisions of the example and whether they're the same for my task or
need a different approach.

instead of being able to start working on their problem right away.

This is what's solved by higher-level wrappers such as your nolearn,
isn't it? I think Lasagne is aiming at a lower level. But sure, it's
possible that some things could be turned into helper functions
useful to a broader audience, outside of the usage examples.

Yes, I imagine the examples could be made much smaller by using a
couple of more helper functions in Lasagne (that maybe shouldn't be
called create_iter_funcs ;-). Or a Model class! Those that need to go
deeper can take a look at those helper functions and adjust them
according to their needs.

Yes, cutting it down to main() and build_model() sounds good!

Great! Glad to see you again, by the way!

Thanks. :-)

[benanne]

I've added this to the First Release milestone since I think it would be a good idea to address this in time for the first release.

[diogo149]

Excellent, if no one else wants to do this, I can do it this weekend. My summary of the conversation:

1. examples should be a single script
2. there should be a main method that is run (to make sure tests pass)
3. have a build_model function that takes in a type of model between fully-connected / convolutional / cuda-convnet convolutional / cuDNN convolutional

[f0k]

Great! I'd only include fully-connected and convolutional, the others don't really add anything. If we want to point users at the other convolution implementations, that could just be done in a comment.
I'm also not sure about whether build_model should have a parameter for the model or whether we should just have two build_* functions with the same signature and have main() decide which one is called. There's little code that could be shared between the two (probably just the InputLayer?) and having two functions would open up the possibility of adding a third one (build_custom_mlp) with some simple extra parameters (depth, width) to illustrate how the Python definition is a lot more flexible than a config file.
Note that the example should just use the same inputs for both, it's not necessary to flatten them for the MLP as done now.

[benanne]

I think the current cuda-convnet / cudnn examples are actually somewhat valuable, at least until we have a guide explaining how to use the different convolution implementations (the dimshuffle argument for cuda-convnet for example). A guide would definitely be a better format for this, but for now the examples is all we have and removing this information probably isn't such a good idea.

[f0k]

A guide would definitely be a better format for this, but for now the examples is all we have and removing this information probably isn't such a good idea.

But wouldn't it be better to document that in the comments of the example? I.e., when creating the Conv2DLayer, explain that you can force alternative implementations to be used via one of the other classes? Giving three examples that look almost the same could be more confusing than enlightening.

[benanne]

That also works :)

[benanne]

Now that IPython notebooks are rendered inline on GitHub, maybe we should consider this format for our examples instead? Or at least in addition to :)

The idea is to eventually have a ton of examples implementing all kinds of different layer types / nonlinearities / architectures from literature (like the highway layer @ebenolson just wrote: https://gist.github.com/ebenolson/4c223b8e2d72b0e35bde), and I guess notebooks would be a great format for that.

[ebenolson]

I think it's a good idea. Notebooks are great for tutorials since you can have output and plots interspersed with the code.

[ebenolson]

A few negatives:

1. Notebooks and git don't mesh perfectly - for example reviewing pull requests for notebooks may be a bit annoying.
2. It may take some thinking to be able to automatically test notebooks the way we test examples now. I ran into this problem with another deep learning library - their only docs were notebooks which had fallen behind the library and hardly functioned at all.

[benanne]

The testing is a great point. I wonder if there are any tools for that yet, because it should be perfectly possible to test a notebook (at least to test that no errors are thrown and that certain cells give the correct output, or something).

EDIT: relevant reading: http://stackoverflow.com/questions/20483313/testing-ipython-notebooks

[kadarakos]

This package might be a possible solution: https://github.com/zonca/pytest-ipynb

[benanne]

That looks like it helps you write tests in IPython notebooks though, which is not exactly what we want. Rather, we want to write examples in IPython notebooks, and then test whether they can run without errors (which is something we already do with our current, file-based examples).

I think this may be closer to what we need: https://gist.github.com/minrk/2620735

[benanne]

I made an IPython notebook example of my highway networks implementation: https://github.com/Lasagne/Lasagne/blob/highway_example/examples/Highway%20Networks.ipynb

I actually started working on this when the paper appeared on r/machinelearning, and a little later @ebenolson posted an implementation in Lasagne as well. My version actually didn't work at the time because of #104, but luckily it does now :)

I put it in a branch but I haven't made a PR, I'm not sure what to do with it. Maybe it's a little too specialized to have it among the examples bundled with the library, but I like how it showcases the extensibility of the library.

[f0k]

Maybe it's a little too specialized to have it among the examples bundled with the library

Haven't taken a look at the code yet, but I think we could well create some subdirectories under examples and fill them with both introductory code and some chosen paper reproductions.

[benanne]

By the way, @diogo149 are you still up for doing this? If not, does anyone else want to take a stab at it?

[ebenolson]

I'd be interested in working on this, but I won't have a chance till sometime next week.

-----Original Message-----
From: "Sander Dieleman" notifications@github.com
Sent: ‎5/‎16/‎2015 10:35 AM
To: "Lasagne/Lasagne" Lasagne@noreply.github.com
Cc: "Eben Olson" eben.olson@gmail.com
Subject: Re: [Lasagne] Modify examples to be simpler (#215)

By the way, @diogo149 are you still up for doing this? If not, does anyone else want to take a stab at it?
—
Reply to this email directly or view it on GitHub.

[benanne]

Sure, we can wait a few more days :) As long as we don't have any double work happening, that would be unfortunate! I'd wait to hear first if @diogo149 is still interested in doing it, since he proposed it.

By the way, I did another one: https://github.com/Lasagne/Lasagne/blob/highway_example/examples/Hidden%20factors.ipynb

This is a reproduction of one of the experiments from Discovering Hidden Factors of Variation in Deep Networks by Cheung et al., from ICLR 2015. It's a nice demonstration of how to use get_output() in a few different ways, and it shows the value of being able to define networks as graphs of layers.

Unfortunately I'm not able to achieve the same classification accuracy as in the paper (there's probably some detail I'm missing), but the resulting reconstructions do look very similar to what's in the paper. Scroll all the way down for some cool images!

[dnouri]

Maybe it's a little too specialized to have it among the examples
bundled with the library

Haven't taken a look at the code yet, but I think we could well
create some subdirectories under examples and fill them with both
introductory code and some chosen paper reproductions.

How about a different repository inside the Lasagne org for notebooks?
This way not everyone would have to pay the price of downloading the
binaries / images, and you wouldn't need to worry about too specialized
or not.

[benanne]

I guess 'recipes' works for me (especially if we're keeping the basic examples in the main repo and we're not renaming that subdirectory :p).

It's probably more inviting for users also to send us their 'recipes' rather than their 'examples'.

[f0k]

especially if we're keeping the basic examples in the main repo and we're not renaming that subdirectory

Sure, that should still be examples, that's a common convention.

It's probably more inviting for users

Yes, don't underestimate that. I think our package name was quite important as well :)

[benanne]

I'm going to go ahead and set this up. If there is any further discussion about the name we can always rename it later, but I think "recipes" is probably a winner :)

EDIT: here we go https://github.com/Lasagne/Recipes

now to have a think about the directory structure, I guess. Also the notebooks I've done so far depend on the mnist.py example code, which will remain in the main repository. So I'll need to figure something out for that. Duplicating the code across both repositories seems like a bad idea.

[f0k]

Also the notebooks I've done so far depend on the mnist.py example code, which will remain in the main repository.

On all of the example code or just a small part of it? I guess we can have some shared folder in Recipes for MNIST-related code and whatever else might be shared between different recipes. We shouldn't have the recipes rely on anything in the Lasagne repository except for the lasagne module, as that would only complicate the setup.

[benanne]

Agree. I can probably just strip out the dependency as well, but it was nice to not have to include the data loading etc. in the notebooks.

[ebenolson]

Agree. I can probably just strip out the dependency as well, but it was nice to not have to include the data loading etc. in the notebooks.

I think it's a good idea to make the basic examples self-contained. It's easier to understand, and the interdependence creates potential for confusion (for example, currently LEARNING_RATE is defined in mnist_conv.py etc, but changing it there will not affect anything because create_iter_functions uses the value from mnist.py.)

[f0k]

I think it's a good idea to make the basic examples self-contained.

Yes, the basic example in the Lasagne repository should be self-contained and "flat". The notebooks in the Recipes repository may externalize things like (down)loading the data so they can be shared across recipes.

[benanne]

Just a thought, maybe for Recipes we can accept both example code as well as implementations of new techniques (layers, update rules, ...) that aren't necessarily "executable" by themselves. That way it can also double as a repository for contributed code that doesn't fit in the main library.

We could create a separate repo for this, but I have a feeling that there will be a lot of overlap and people will be confused about which repo their stuff belongs in. Lots of examples will also implement custom layers anyway.

As for the organisation of the repo: having a subdirectory for each example is probably the safest bet. We could technically put single-file examples in the top level directory, but if those examples are then modified later and grow to multiple files we'd have to move them into subdirectories anyway.

Comments are welcome, in the meantime I'll start populating the repo a bit in the next few days, I think.

[f0k]

We could have two or three different categories that make up the first subdirectory level, and subdirectories under those that are the separate contributions. Possible categories: papers for reimplementations of research papers as runnable code, tutorials for take-you-by-the-hand tutorials and notebooks (possibly also including papers, but with a focus on "how it's done" instead of the results), snippets for bits and pieces as you mentioned -- things that are not runnable on their own, but some of which may be used in papers or tutorials.
I didn't think that through, but it seems a plausible way of categorizing things. We could have fancier names, of course: starters, mains and sides/side_dishes ;)

[benanne]

I guess that works, although I wouldn't overdo it with the names in this case. papers, tutorials and snippets is a lot clearer :)

[f0k]

Okay. We just need to figure out a clean way to import snippets, if we want to use it both for "things to copy/paste into your code if you need them" and "things we need in multiple recipes and don't want to copy/paste".

1. Require people to add the snippets path to their PYTHONPATH? (That's a bit cumbersome.)
2. Have the examples do silly things like sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', '..', 'snippets', 'foo')? (That means they cannot simply copy an example somewhere else and modify it.)
3. Have the examples import the snippets via the imp module? (Same problem as for 2.)
4. Include relative symlinks to the snippet files that are used? (That probably won't work on Windows.)

Other ideas? It's probably just a few things like the MNIST data download/reading code and maybe some layer implementations that could be shared between multiple tutorials or papers and tutorials, but still. I'd try to keep this as a simple collection of directories, not an installable module.

[benanne]

I feel like that MNIST loading code doesn't belong in 'snippets' though... I see that more as a repository for library extensions (new layers, update functions, etc.), data loading boilerplate doesn't really belong there imo.

[ebenolson]

I like this - I have a handful of stuff that could go into snippets/sides.

I feel like there should be a separation though between examples/tutorials which it should be a priority to keep bug-free, and other stuff which might not be so actively maintained.

[benanne]

There will still be a bunch of (tested) examples bundled with the library. This repo would be for contributed content. Which potentially includes examples and tutorials, but it would be okay for those not to be as well-tested.

[ebenolson]

Ah ok so this repo is the distinction :)

So is the plan still to rewrite the mnist.py examples as ipython notebooks?

[benanne]

I'm not sure to be honest. If we have a bunch of recipes in the form of notebooks it might not be necessary. Some people might also not be familiar with the notebook format. Also a requirement would be that we can still test them like we currently do, I don't think that discussion was resolved.

[f0k]

I feel like that MNIST loading code doesn't belong in 'snippets' though... I see that more as a repository for library extensions (new layers, update functions, etc.), data loading boilerplate doesn't really belong there imo.

I think it could be both. We could have yet another subdirectory shared for things shared between examples (like the MNIST loading code), but we will probably also have some of the snippets ending up in tutorials or papers, so we could just as well combine this into one.

So is the plan still to rewrite the mnist.py examples as ipython notebooks?

As Sander said, not for the Lasagne repository, but possibly for the Recipes repository.

[benanne]

Does anybody else want to take the lead on this, and on populating the Recipes repository as well (I'll give you commit rights)? I'm moving abroad in a few weeks and I need to finish up my dissertation, so I don't have much time to spare for Lasagne at the moment.

[f0k]

Does anybody else want to take the lead on this, and on populating the Recipes repository as well (I'll give you commit rights)?

I'm good with co-maintaining the main repo for now. Anybody else interested in maintaining the Recipes? @craffel? @ebenolson?

I'm moving abroad in a few weeks and I need to finish up my dissertation

<offtopic> Shall we try releasing Lasagne before that? </offtopic>

[benanne]

<offtopic> Shall we try releasing Lasagne before that? </offtopic>

Yes please!

[ebenolson]

I'll try to get some of my code ready for Recipes this week, but I'm a bit short on time right now as well. I could take care of responding to PRs there though.

[f0k]

Yes please!

What's the deadline then?

I could take care of responding to PRs there though.

Sounds great!

[benanne]

The deadline is three weeks ago :) I don't know, I guess there is no deadline. Just sooner rather than later, preferably.

There's not a lot left to do, cleaning up the examples and sorting out the regularization docs are the main things. And wrapping up the default nonlinearity discussion, I guess.

[benanne]

@ebenolson you should now have read/write access to the Recipes repository. Let me know if it doesn't work because I'm not sure I set it up correctly...