Revert markdown lint check package vers pinning #209

* Create initial github actions pipeline for building on windows * add the build script referenced in the github actions ci pipeline * comments * commit something that should fail the uncrustify check just to make sure that the pipelines fail * Revert "commit something that should fail the uncrustify check just to make sure that the pipelines fail" This reverts commit 3fee079. Manually verified that this does indeed fail the build pipeline. * tell editors to use 2 spaces instead of 4 by default for yaml files

* cleanup spacing/comments from previous copy/paste/edit a tad * initial attempt at enabling various python checks in the ci pipelines note: here we run them on linux instead of windows for easier scripting, though (hopefully) the results are more or less that same in either spot (excepting for lf/crlf line ending differences which we should eventually fix separately) * various python action pipeline changes - try to run some of the checks on both windows and linux - try to support a couple of different versions of python? - run the lint and unit tests from the same workspace (saves on some runner startup cost) - try to use the "approved" python setup task from github (else the PATH env doesn't seem to work out right) * test failing the python licenseheaders check * attempting to fix running python checks on linux * Revert "test failing the python licenseheaders check" This reverts commit 9ca6b96. Manually confirmed that this does indeed fail the build as expected. * separate and document windows/linux commands for running python checks * add a bash equivalent for the grpc generation cmd while we're at it * disable python 3.8 checks for now It's failing pylint checks with some "useless statement" warnings

…s (After Rebase) There was a bug in partitioning the input space in the homogeneous random forest. As a result the individual decision trees would return a Prediction object even if the features were outside of their input space.

In preparing the search space for the Gaussian process we erroneously ordered the dimensions in the order they were specified in the OptimizationProblem, but the bayes_opt reorders them alphabetically.

Some selections taken from !410291 ```sh git checkout -b bpkroth/some-linux-cpp-fixups master git format-patch --stdout master..bpkroth/oss-linux-build-work -- *.cpp *.h *.inl *.vcxproj | git am ``` - un-inline some MlosPlatform.Std.inl functions - comments referencing MlosPlatform.Std.ink - tweaks to allow SmartCache to build on Linux - partial (incomplete) work on enabling additional C++ projects on Linux - comment fixup - fixups to allow some more compilation on Linux - use the macro we defined - file can't include itself? - more build fixups for clang - more fixups to allow Mlos.Unit to link - precompiled header fixup for building on Windows Related work items: #806033

Version2 of PredictionClass. _From Adam's email:_ **Problems with current implementation** First, returning a Prediction object for each invocation of bayesian_optimizer.predict() means that to draw one heatmap we need to return 10,000 objects (100x100 pixels). This eliminates hope for memory locality, puts a strain on allocators, requires us to reassemble a numpy array for the benefit of matplotlib, etc. Returning a single data frame would be much preferred. (Pandas data frame also opens door for using Arrow down the road.) Secondly, most of the time we don’t care about most of the values contained in the Prediction object. Each utility function cares only about one or two, the tomograph cares only about the mean and nothing else right now. Computing all the extra values in the Prediction class is a waste of CPU and memory. This PR is a restart of the abandoned: https://msdata.visualstudio.com/Database%20Systems/_git/MLOS/pullrequest/413447 Related work items: #743670, #863921

* Fixed Tomograph to work again with the new Prediction class * added a notebook to demonstrate active learning on smart cache in python * cleaned up MLOS SDK and MLOS Infra for Python * Fixed a bunch of bugs * Made sure that BayesianOptimizerProxy once again conforms to OptimizerInterface

Andreas pointed out that to the user of the BayesianOptimizer the interaction between these parameters is confusing and can lead to subtle bugs: {maximize, minimize} x {upper_confidence_bound, lower_confidence_bound}. As one would often use upper_confidence_bound for maximization and lower_confidence_bound for minimization . However, these parameters are set in different places (OptimizationProblem vs. OptimizerConfig) and entirely non-intuitive. So this PR renames them to upper_confidence_bound_on_improvement and lower_confidence_bound_on_improvement and changes the math accordingly. This way, when the user switches from maximizing, to minimizing, she/he/they doesn't/don't have to change the optimizer config. This rename is in-line with the names of other utility functions: Probability of Improvement, Expected Improvement.

Pandas spits out tens of thousands lines of warnings during Unit Tests runs. This PR cleans it up.

Various fixes in C++ projects to enable Linux build. Do not use string_view in Mlos.UnitTest project. Change include order (include template specialization before using them).

Using sample variance instead of mean variance to compute the confidence intervals makes them not shrink as we add more observations. While this measure can still be useful down the road to detect noisy-TV type of problems, using mean variance makes more sense for the problems we are solving now.

This PR ensures default parameters are copied and not accidentally modified by the user. I'm also introducing a long-haul test for BO, which was the original motivation. Happy to split that off into a different PR. I wasn't sure where to put the DefaultConfigMeta helper, lmk if there's a better place. There's two reasons this is tricky: DEFAULT is used on the class, but properties are instance-level, so we need a meta-class. Also, because of the way the dictionaries are instantiated, we can't make copies of nested points easily, so we need to use deepcopy. This solution still has the issue that we're using mutable objects as default parameters. If a user would modify ``_DEFAULT`` for some reason, the default instantiations would change.

… with MLOS

* init website * remove other themes * add basic config * remove non-existant submodules * fix site building script * add empty index, update base url * add generated website to gitignore * fix booksection config * remove unused folder * add link to python api to hugo page * fix link to python api * add docs on python only installation * add generated website to gitignore * remove submodule with book theme * remove generated assets * add code to download book theme to build_site.sh * add if for book checkout * fixup readme a bit * update links in readme, clean up some empty files * remove empty gitmodules * remove generated files * add manual menu, don't lowercase files * try to make links work in hugo and on github * fix up links for hugo * easier way to get python docs into menu * don't have license in hugo menu * fix regex * fix python api doc link again * check out specific tag of book theme (v8) * clarify formulation. * add comment to explain sed commands * update menu, fix link to bayesian optimization * add sed line to change paths for BayesianOptimization notebook

* add sphinx build to gitignore * add apidoc templates * mostly works * slightly cleaner * clean up sphinx apidoc generation * update index * fix source links in python api docs * restructure docs with one more menu level * change link name to main website * add python api index document

* add template for notebook conversion, add links to notebook download * don't keep before file in sed * make download more prominent in template * fix string replace comment

python documentation fixups from microsoft#12 and !422716

Adding goodness of fit measurements for the training (and some validation) datasets, along with a way to query these statistics via gRPC.

Implement support for Posix Named Shared Memory and Named Semaphores. SmartSharedChannel builds on Ubuntu16.04

Added r_squared

In preparation for the demo I'm running the code over and over and I've accumulated a lot of small improvements. The fixes include: * Point.to_pandas() -> Point.to_dataframe() rename * New: Point.from_dataframe() * New: RegressionModelFitState.to_dataframe() * SmartCacheWorkloadLauncher takes duration_s as parameter * RandomSearchOptimzier utilizes Hypergrid.random_dataframe() rather than rolling its own * Tracer collects data about dataframes passed as function arguments (hence all of the args->kwargs changes) * Unskipped a couple of long haul tests * Added some convergence state tests to bayesian optimzier tests * Added: assert suggested_config in self.optimization_problem.parameter_space to BayesianOptimizer * Setting attributes on Point works as expected now. * Cleaned up a broken Unit Test

# What's changing? Second step in simplifying the Hypergrids. In this PR I eliminated the CompositeHypergrid entirely, and all of its functionality is now present in the SimpleHypergrid. All unit tests are passing and I tested all the notebooks too. The main change occurs in Hypergrids.py - most other changes are related to testing code and suitable renames. The one exception is the HypergridJsonEncoderDecoder.py where there is a functional change. This change is backwards compatible, so old code (in C#) will continue working. # Effects on client code None - CompositeHypergrid was almost entirely internal and never instantiated directly by the user. # Reasons for the change 1. It's much simpler to write the recursive functions required for the Hypergrids when there is only one class, rather than two classes. 2. Greg is implementing Hypergrids in C# and it makes no sense to duplicate something complex, if we can duplicate something simple. # Next Steps There will be some renames: GuestSubgrid -> JoinedSubgrid, etc. I will coordinate them tightly to make sure people's branches continue to work well.

This allows users to run start_optimizer_microservice as a script once the package is installed

Some suggestions from Yiwen and some warm-up tasks for the students

Lots of work-in-progress to try and make the Linux build/test/dev environment more natural. - Provide `Makefile` wrappers around `cmake` and `dotnet` so that developers can "just" run `make` - Consolidate some `cmake` directives into files in `build/` so that they can be reused for additional projects. - Also apply a handful of fixups to allow building/linking additional C++ projects. - Provide `cmake` functions to allow invoking `dotnet build` to generate `SettingsProvider` files for C++ files. - This allows running `make` within a given source directory to get just what you need updated and avoid race conditions by having multiple `dotnet build` processes running in parallel for different C++ projects. - Allow invoking tests via `ctest` (along with wrappers for `dotnet test` as well) - Provide a `ctags` db generation rule (to allow easier source code navigation in cli editors like `emacs` and `vim`) - Improve the `dotnet` and `cmake` fetch utilities. - Add scripts to allow setting up the shell environment to reference them - But also don't require that - Add support for both `Release` and `Debug` builds - Requires different `cmake` output directories - Disable Windows support (for now) - Some documentation (probably needs improvement) - Also added a `Dockerfile` and some docs to allow more easily testing this across different versions of Ubuntu since a number of the issues we found were due to features in Ubuntu 20.04 not being available in 16.04 (which needs to be supported for SqlServer). - Various Open Issues - `make dotnet-test` fails on the `Mlos.ModelServies.UnitTests` project at the moment due to wrong Python version being referenced - `MlosTestRun_MlosAgentServer_MlosUnitTest` is currently commented out in `Mlos.UnitTest/CMakeLists.txt` since it launches but just hangs which causes the build to hang. It also doesn't fail the test when you `pkill dotnet` (which it should). Related work items: #806033, #815235, #815236, #815237, #836531

Preparing the smart cache example to be more self-contained and ready for class.

* add environment.yml * fix typo * update readme instructions * update enironment.yml * formatting fixes

* clean up notebooks a bit * remove reference to StartHere notebook

* start by reorganizing the files also use relative links for the other files * add some text abount contributing to the top level file remove the nested file

* add website build to CI * try to build website on gh actions * make a change * fix actions script * fix slashes * install nbconvert * install nbconvert correctly * figure out paths * fix path issues in website scripts * try to start nbconvert lol * path issues once more * install sphinx_rtd_theme * add numpydoc to requirements * try to push to github pages * only deploy on main * fix typo * use apt-get * two heads are better than one * another typo

* decrease max degree of polynomial to reduce time spent in unit tests * fixups for python scripts to explicitly look for python3.7 for now fixes some issues when runing the unit tests in an environment that has both 3.7 and 3.8 available with 3.8 as the default * add explicit timeouts for our CI steps * tweak the timeouts to give just a bit more splay * be consistent on use of apt vs apt-get

* Adding jupyter launch instructions to README * Regrouping Prerequisite docs by OS instead of by step * wip * wip * some more initial work on fixingup the docs * more work on reorging the prereqs document to be more OS oriented * slight tweaks on the build instructions * fix bad path Co-authored-by: Carlo Curino <carlocurino@ChristyiMac.lan> Co-authored-by: Brian Kroth <bpkroth@microsoft.com> Co-authored-by: Brian Kroth <bpkroth@users.noreply.github.com>

) * minor typos on documentation * use __repr__ instead of __str__ everywhere * change test output

* Clarified options for the build tools installations for Linux * Update documentation/01-Prerequisites.md Co-authored-by: Brian Kroth <bpkroth@users.noreply.github.com> * Update documentation/01-Prerequisites.md Co-authored-by: Brian Kroth <bpkroth@users.noreply.github.com> * Update documentation/01-Prerequisites.md Co-authored-by: Brian Kroth <bpkroth@users.noreply.github.com> * Updated toc * Reduced redundancy with publicly available documentation * Simplified (?) Windows instructions * Update documentation/01-Prerequisites.md Co-authored-by: Brian Kroth <bpkroth@users.noreply.github.com> * Update documentation/01-Prerequisites.md Co-authored-by: Brian Kroth <bpkroth@users.noreply.github.com> * Fixed the block quotes Co-authored-by: Brian Kroth <bpkroth@users.noreply.github.com>

* Redrawn MLOS Architecture Adding some details on User Input and Code Gen * Updating per feedback * Iterating on image per feedback * Removing old images * Updating link in docs and minor line fix Co-authored-by: Carlo Curino

A bunch of work for fixing up some broken documentation links and improving the pipelines to try and prevent that in the future. Uses a fake anchor hack discussed previously to identify which links, when rendered in github pages, should be sent back to the main github repo page, and which should remain relative, though suitably fixed up. Adds a Makefile for building the site both locally (for testing) as well as in the pipelines. This also fixes a an issue with the python_api code being copied to the wrong location so it wasn't being updated. Adds a workflow to test for broken links in the plain markdown files as well. None of them check for faulty anchors. Partial work on providing the ability to execute the notebooks prior to markdown rendering as well (there's a bug that seems to be related to the recent shell=True fixup from #46 I haven't addressed there yet though). For that we also needed the ability to install all of the necessary pip modules. I tried to add a script to avoid use of requirements.txt and instead use the conda environment.yml without actually making use of conda (kind of a hack for the moment). Eventually I intend to replace that with proper docker container build image reuse.

* Increase python unit test CI timeout * crank up the python unit test timeouts a ton to workaround some random slowness issues with the github runners

Checking in all of the demo code, which includes the Tensorboard writer.

…sed in the request Fixed the bug

Pumped all observations data through gRPC and added tests

… regression model to deal... Pull Request 425209: Added ability for Regression-Enhanced Random Forest regression model to deal with categorical dimensions The categorical dimensions are enabled in the RERF regression model via the use of dummy variables derived from a OneHotEncoder, opting to drop the first level so as to avoid the DummyVariableTrap. See references below for details - best to start with the CMU statistics class materials. Other code touched: 1. Moved the Prediction enum used by RERF (and it's unit tests) to PREDICTION_VALUE, PREDICTION_VALUE_VARIANCE, and PREDICTED_VALUE_DEGREES_OF_FREEDOM as these were previously set at the corresponding SAMPLE based fields. 2. Added RERF unit test fitting four distinctly different quadratic surfaces associated with categorical dimensions. **References** 1. Very thorough and readable notes documenting linear regression using basis functions, categorical, and ordinal features: a) http://www.stat.cmu.edu/~cshalizi/mreg/15/lectures/lecture-16.pdf - intro to encoding categorical & ordinal dimensions, and b) http://www.stat.cmu.edu/~cshalizi/mreg/15/lectures/19/lecture-19.pdf - most closely describes what's been added to the RERF regression model. 1. https://en.wikipedia.org/wiki/Categorical_variable#Dummy_coding 1. https://en.wikipedia.org/wiki/Dummy_variable_(statistics) 1. https://www.algosome.com/articles/dummy-variable-trap-regression.html 1. https://en.wikipedia.org/wiki/Design_matrix since the code references a design matrix in some comments. Related work items: #889101

…tion This run revealed an issue in variance computation: https://msdata.visualstudio.com/Database%20Systems/_build/results?buildId=21291699&view=results Specifically, a variance value of -7.450581e-09 made the sqrt throw. This PR corrects it by adding a small constant to the variance in the Random Forest. <s>At the moment this PR also includes a beefed up test that revealed the problem. Once that test passes we will have good evidence that the problem is resolved. I will restore the test to its original state and check in.</s> The test ran all night redoing the computation on every iteration. It did not throw so I conjecture that the bug is fixed.

Move to new optimizer, remove support for old simple Bayesian optimizer, remove SQLRPC, implement smart cache example (Cpp implementation of Python SmartCache demo), implement Hypergrid deserialization

Include smart cache in the unit tests.

…oft#63) * add usage of optimum method * Apply suggestions from code review line breaks for github review Co-authored-by: Brian Kroth <bpkroth@users.noreply.github.com> Co-authored-by: Brian Kroth <bpkroth@users.noreply.github.com>

* add website related targets to top level Makefile for convenient full project clean * restore a file revision controlled file after we clean the rest of the sphinx output FIXME: however, running the sphinx build still alters that file

* make sure we generate the grpc code with the right version of python * let us pass extra params to the python unittest module For instance "-k some_test_name" * add some more rules to cleanup after the python code * add a fixup to http_proxy environment variable interaction issues with grpc * raise a warning instead of messing with the user environment * actually, just print the warning - don't interrupt * also document the issue logged for the 3.8 bug

The main portion of this PR is the implementation of the Glow Worm Swarm Optimization algorithm as an alternative to the RandomSearchOptimizer. You will find its implementation in: ```\mlos\Optimizers\ExperimentDesigner\UtilityFunctionOptimizers\GlowWormSwarmOptimizer.py``` The first part of this post has a good description of the algorithm: https://www.hindawi.com/journals/mpe/2016/5481602/ Note that our implementation is aware of hierarchical hypergrids and essentially will split up the glow worms so that each group operates on a single, consistent space (i.e. glow worms will not in general move from LRU to MRU implementation except for random restarts). There is still plenty of low-hanging fruit (like watermelon size) in the implementation. We should get a big improvement by remembering the best parameters from previous calls to GlowWormSwarmOptimizer.suggest() and reusing them as the starting positions for subsequent calls to suggest (with some randomness of course). Other possible improvements include: varying the step size, varying the number of iterations, implementing better stopping criteria. But this PR is already big, so we can do that later. In order to plug in this optimizer I needed to standardize the UtilityFunctionOptimizer interface, hence the base classes: UtilityFunctionOptimizer and UtilityFunction were born. I also needed an adapter to translate points into a unit continuous search space. As I was going along, I also fixed a few small bugs. Lastly, the new adapters and the existence of the glow worm optimizer make it pretty trivial to implement a random-near-incumbent strategy that Ed says is hard to beat. That would be really something - having 3 utility function optimizers, 2 surrogate models, 2 utility functions would allow us to instantiate 12 distinctly different optimizers and this number will grow super-linearly with every new implementation.

Following Andreas' feature request I added SimpleHypergrid.__getitem__() API. While I was at it I also cleaned up a bit as outlined below. I still need to run the linter and carefully peruse the code since the last merge was quite big and I had a lot of conflicts. However, I wanted you to be able to start checking out the work if you'd like. * Removed the DecisionTreeRegressionModelConfig instances for consistency - more such changes to come. * Made Point.__getitem__() throw a KeyError instead of returning None to make it consistent with the dictionary behavior. * Added a Point.get() method to still allow getting a default value if the dimension is not in Point * Renamed guest_subgrids to joined_subgrids * Separated Hypergrid and SimpleHypergrid into their respective files * Made dimension an Abstract Base Class (ABC) and enforced it * Made SimpleHypergrid pretty print like so: ![image (2).png](https://msdata.visualstudio.com/5603b9e3-ece0-4518-b9c8-1a36054d9970/_apis/git/repositories/e9fd4787-d7af-4833-beb4-34d130373fe8/pullRequests/428076/attachments/image%20(2).png)

Since the HypergridAdapters actually project points from one space to another, I thought a name change would be good. Especially that translation means "moving a point by some vector". ALSO: I added a little default behavior of being able to automatically project points if the authors of a HypergridAdapter derived class decides it's not worth their while to implement it by hand. The usage of HypergridAdapters as a wonderful data pre-processing tool has not caught up yet, so this change affects only my code and I have tested it :) But I encourage you to start using the adapters, because they are a reusable, clean and effective way of translating configurations from what the user wants to see, to what the models can compute :) Here are some ideas for useful features: * a PCA adapter - to preprocess all of those context values we are about to start pumping into our models * a one-hot-encoding adapter - as a reusable way to deal with categorical dimensions * a bootstrapping adapter * a column-dropping adapter * a logit adapter - to preprocess the objectives that have finite ranges

…fects Fixed RERF instantiation and prediction interval defects. Related work items: #894559, #894585

…ctory Note: there are other instances of this, but we'll resolve those separately

* some relative link sed fixups * document an issue I ran into * perform local anchor link checking as well * address comments about extra seds to remove '/./' patterns * consolidate some pip commands

* reorganize and update conda/python install docs * remove instructions to get different miniconda and force python=3.7 in environment * Update documentation/01-Prerequisites.md Co-authored-by: Brian Kroth <bpkroth@users.noreply.github.com> * Update README.md Co-authored-by: Brian Kroth <bpkroth@users.noreply.github.com> * remove nb_conda_kernels for now. weird? * skip conda packages in pip conversion * don't try to install python with pip * consolidate special cases * typo lol Co-authored-by: Brian Kroth <bpkroth@users.noreply.github.com>

* add base implementation of optimum in OptimizerInterface * rename OptimizerInterface to OptimizerBase * simplify optimum, return points * rename file OptimizerInterface * fix tests * test optimum in BayesianOptimizerProxy * add pylint disable unused argumetn * compute optimum after registering * Update source/Mlos.Python/mlos/Optimizers/OptimizerBase.py Co-authored-by: Brian Kroth <bpkroth@users.noreply.github.com> * Update source/Mlos.Python/mlos/Optimizers/OptimizerBase.py Co-authored-by: Brian Kroth <bpkroth@users.noreply.github.com> * fix extra whitespace * fix OptimizerBase docstring * throw nicer error when calling optimum before register * update type hint for optimum * remove unused dict from typing * add some more tests * more verbose logging * typos, more prints * fix fstrings * more typo * typo * indentation merge issue * undo merge mistake Co-authored-by: Brian Kroth <bpkroth@users.noreply.github.com>

* remove some debug output * include a menu on the notebooks index page if you land it from the main README.md

* include cmake test target in cake build for linux Note: cake doesn't currently provide a ctest target, so we just call the virtual "test" target that cmake generates to do the same. * stub in a cmake test definition for running SmartCache with Mlos.Agent.Server FIXME: as with Mlos.UnitTest - it currently hangs * no need to hook that particular dotnet test into cmake it's pure c# and called from the "dotnet test" target. * add infrastructure for skipping certain dotnet tests by default temporarily disabling since they're broken * tweaks to vscode settings for easier initial debugging experience * make sure that when we debug build, we define DEBUG for the compiler Typically, clang/gcc only define NDEBUG for Release builds, but don't define DEBUG for other (e.g. Debug) builds. This is to replicate MSVC behavior which defines DEBUG for Debug builds. Without this, some UnitTests which were only supposed to run for Retail builds due to failing asserts otherwise were getting build unintentionally. * quick hack to also treat files without any extension as executables for Linux * just assume any file that doesn't end in a .json is an executable The previous method didn't work because for a binary named "Mlos.UnitTest" standard path parsing tools think the extension is ".UnitTest" * enabling the C++ unit tests via the agent server via ctest * some tweaks to try and get the output from the target process quicker * initial work to add some linux build/test work to the pipelines (just dotnet for now) * reorg and include cmake-build as well * make sure some apt dependencies are installed * also run the cake build script for testing * more vscode debugger improvements * improvements to allow searching for the settings registry assembly in Linux environments plus some other comments about eventual other fixups * initial work on letting cmake run the c++ tests via ctest * comments * work around test fixtures only executing once This is a hack around the fact that we currently need to cleanup all of the shared memory regions after ourselves before we can do another (or different) run. * add cmake test to the CI runs * address some comments

* minor fixups to existing docs * start some instructions and explanations on running the SmartCache end to end example * add the docker build/run commands in explicitly * Update source/Examples/SmartCache/README.md Co-authored-by: Andreas Mueller <t3kcit@gmail.com> * Update source/Examples/SmartCache/README.md Co-authored-by: Andreas Mueller <t3kcit@gmail.com> * Update source/Examples/SmartCache/README.md Co-authored-by: Andreas Mueller <t3kcit@gmail.com> * link tweaks * reorg to address some comments * allow browsing some details about the examples on the github io pages as well Co-authored-by: Andreas Mueller <t3kcit@gmail.com>

Leaving the output in so that it's easier to review. I recommend having one window with a raw output and one window with a preview open simultanously.

This is the first of the PRs introducing the optimizer evaluation toolkit. Much of the toolkit's description is in the OptimizerEvaluationTools notebook so please refer to it for details. Broadly speaking, I put together a notebook that demonstrates how the optimizer evaluation tools can be used. So far we only keep track of goodness of fit, but I plan to add optima, and utility function optimziers to the toolkit too. I mostly try to replicate what Ed and Andreas do to understand the optimizers' performance and automate it as much as possible. I even used a bunch of Ed's visualizations :) **About the framework**: most of the synthetic functions we have used so far, now derive from SyntheticFunctionBase - basically exposing an interface to evaluate_point(), evaluate_dataframe(), and expose: parameter_space, output_space, and context. These functions are produced by the SyntheticFunctionFactory. Ideally, we will build up a portfolio of interesting functions, and @<Andreas Mueller> said he could point us to already existing libraries. Additionally, this PR introduces a ComponentConfigStore class, which we discussed in the Mlos channel. It is meant as a replacement for all of the XYZConfig classes. So far I have only instantiated two stores: ObjectiveFunctionConfigStore, and DecisionTreeConfigStore and once we are happy with the interface, and functionality, I'll replace all occurences of XYZConfig.

It has served its purpose and will live on in the git history.

… Jupyter (#59) * minor typos on documentation * use __repr__ instead of __str__ everywhere * change test output This version just resolves some merge conflicts in ADO and then brings it back to Github.

Restart shared channel after the failure. Fix a bug when receiving a frame with incorrect type index. Missing: Restart when there is a sequence correct frame, partial, correct. Detect inactive channel. Add unit-test.

* finish writing a comment * spacing

* Add end to end example docs to the github pages menu * addressing some comments Remove the less polished SmartSharedChannel example from the website view

This change improves runtime of the DecisionTreeRegressionModel.predict() function. The main idea is to use column-wise operators so that pandas and numpy can vectorize them. Tracing reveals 2x-200x improvement in filtering out invalid rows depending on the complexity of the hypergrid and the number of rows in the dataframe.

* initial work to binplace binaries on linux This provides "make install" style targets to binplace binaries to target/bin/{Release,Debug}/ for easier execution. Note that SettingsRegistry assemblies were already being placed there. With the current Mlos.Agent lookup behavior, this means that we can now run ```sh cd target/bin/Release ./Mlos.Agent.Server ./Mlos.UnitTest ``` Instead of having to specify 1. the path to the dotnet tool 2. the paths to the Mlos.Agent.Server assembly 3. the path to the target executable 4. the MLOS_SETTINGS_REGISTRY_PATH environment variable To do this, we provide an Mlos.NetCore.Publishing.targets file that provides a new target "PublishProjectBinary" that in turn runs an appropriate "dotnet publish" command for the given platform. This method requires the fewest changes to the existing build system since we can specify the RuntimeIdentifier in the command line instead of in all of the build files. To allow recursive operation for projects that don't include that target, we let that target be a no-op most of the time. Next, we hook that into the existing Makefile and cmake wrappers as well. TODO: - tests - documentation * back out 'dotnet publish' commands * add the rules necessary to binplace dotnet dlls This is so we can use 'dotnet target/bin/Release/foo.dll' to run either the codegen tool (for external projects) or the Mlos.Agent.Server The main changes: - add binplace rules for all the dependencies - especially grpc - also add the *.runtimeconfig.json file for our two executables * include debug symbols for release builds Per discussion with grlap * addressing comments - consolidate GrpcVersion references into a property - make Mlos.NetCore projects binplace by default * with netcore projects binplacing by default this isn't necessary * Change binplacing logic for settings registry code gen to accommodate external projects

Added a class enumerating all supported definitions of "optimum" and implemented three of them: * PREDICTED_VALUE_FOR_OBSERVED_CONFIG * LOWER_CONFIDENCE_BOUND_FOR_OBSERVED_CONFIG * UPPER_CONFIDENCE_BOUND_FOR_OBSERVED_CONFIG Users can select which of the definitions to use based on their risk profiles. For the confidence bound based optima, the users can specify the probability of type 1 error (alpha) to indicate their required level of certainty.

It is currently quite difficult to follow what's going on across three different languages. This PR attempts to heavily comment the SmartCache C++ example to help inform new users how the flow works across each component. It also links some of the other component markdown documentation into the documentation linking structure for easier browsing on the web page. * wip: documenting smart cache example * some initial work on a troubleshooting faq for starters, we just document a workaround to using the dotnet vscode plugins when in vscode (fixes microsoft#98) * unnecessary - it's included in stdafx.h * make vscode consistent with our Makefile methods for generating cmake output * add some comments * fixup a lack of null check and include remarks along with summary comments * don't line wrap * add a link to the troubleshooting documentation so it shows up in the webpages view * add more docs * mark some fields as private * more comments * comment about and disable the agent termination messages * Revert "comment about and disable the agent termination messages" This reverts commit e806def. * some docs for the Mlos.Agent.Server * add some more links in the overview readme * reduce some duplicate text * add some comments about the optimizer property for the C# agent MlosContext * update the overview comments in that file * simplify the execution commands * Revert "mark some fields as private" This reverts commit 55e85ad. This change needs to be done internally on ADO to handle some additional classes first. * Update source/Examples/SmartCache/Main.cpp Co-authored-by: Andreas Mueller <t3kcit@gmail.com> * Update source/Examples/SmartCache/Main.cpp Co-authored-by: Andreas Mueller <t3kcit@gmail.com> * Update source/Examples/SmartCache/SmartCache.SettingsRegistry/Codegen/SmartCache.cs Co-authored-by: Andreas Mueller <t3kcit@gmail.com> * Update source/Mlos.Agent.Server/MlosAgentServer.cs Co-authored-by: Andreas Mueller <t3kcit@gmail.com> * Update source/Mlos.Agent.Server/MlosAgentServer.cs Co-authored-by: Andreas Mueller <t3kcit@gmail.com> * Update source/Mlos.Agent.Server/MlosAgentServer.cs Co-authored-by: Andreas Mueller <t3kcit@gmail.com> * addressing feedback, trying to improve some comments * remove a 'using' statement that somehow got added * fixup a botched merge that broke the stylecop checks * cleanup and link in various other in-tree markdown documentation - add a reference the the more complete settings system codegen markdown documentation in the SmartCache.cs comments - cleanup that markdown file (both linting and some language) - rename it so that we can include it in the webpage - do the same for several other files - link them from the repo organization page so we can browse to them from the generated web pages * fixups for website link checking * address comments * simplify some commands, fix some links * comments to help readers see where the external agent message handlers are * addressing comments Co-authored-by: Andreas Mueller <t3kcit@gmail.com>

This replaces the XYZConfig classes with ComponentConfigStore objects.

Fixing the crash found in the CDPx pipeline. Agent disposed the shared memory while there was a still active reader thread. Link to the failure: https://msdata.visualstudio.com/Database%20Systems/_build/results?buildId=22386882&view=logs&j=fd490c07-0b22-5182-fac9-6d67fe1e939b&t=dcf93f35-c1ba-5dce-0078-8d628250da3a

In active learning mode Mlos.Agent hangs, when there is an unhandled exception. Agent waits for the target process termination and target process waits for the message from the agent.

Optimizer returns an integer when expected. This simplfies parsing parameters (JsonElement to int).

…ts (microsoft#106) * setup a commandline parser for Mlos.Agent.Server for configuring the endpoints * listen on any ip address for incoming grpc connections * check for the executable * update msbuild invocation of mlos.agent.server for new cli syntax as well

* added feature imputation to RERF predict() * ensured unit test exercised single row x prediction; retained more info from fit to use for prediction Co-authored-by: Ed Thayer <edthaye@microsoft.com>

All tests in TestBayesianOptimizer now test both a local optimizer and an optimizer created via gRPC. These tests ended up being pretty slow since we didn't have an API to register observations in bulk over gRPC. So I added that, along with a bunch of tracing to diagnose the problem and validate the improvement.

* improve and add debugging targets * add a 'dotnet-build-quick' target for easier access to faster rebuilds * spelling * remove an environment variable that was breaking unencrypted grpc connections This overrode and made the following line useless: AppContext.SetSwitch("System.Net.Http.SocketsHttpHandler.Http2UnencryptedSupport", true); The resulting error was: Unhandled exception. Grpc.Core.RpcException: Status(StatusCode=Internal, Detail="Bad gRPC response. Response protocol downgraded to HTTP/1.1.") Further details in: #4 grpc/grpc-dotnet#654 dotnet/runtime#988 * make one of the unit tests also invoke the optimizer * build the mlos.agent.server when building projects that need it for unit tests This makes it easier to run "make all test" in one project directory. * need to make sure that the mlos modules is already installed and available

... so there are no duplicate binplace issues for certain build systems

…eanup shared memory in Active Learning mode Implement managed MlosContext, NetCore shared channel can be instantiated with different channel policies. Mlos.Agent.Sevice unlinks shared memory when running in active learning mode. GlobalMemoryRegion has a counter with number of active processes, the last process will cleanup OS resources (unlink shared memory on Linux). We are not detecting the crashes, when the terminated process did not decrease usage counter.

* start on CPP example notebook for smart cache

* add instructions for lab2.5 * address Brian's comments * Update source/Mlos.Notebooks/SmartCacheCPP.ipynb Co-authored-by: Brian Kroth <bpkroth@users.noreply.github.com> * fix docker invocation, clean run * unrelated typo from walkthrough Co-authored-by: Brian Kroth <bpkroth@users.noreply.github.com>

* tweaks to default debugger setup - don't require optimizer * tweak a notebook link to use the github pages rendered link * Revert "tweak a notebook link to use the github pages rendered link" This reverts commit c289504. * look for extra args and output more useful help text * a few more usage help tweaks * reformat the strings in the code, and make it work for non "Errors" as well like having ExtraArgs * addressing comments - move cli parsing to a separate file - change error message for unknown extra arguments * add a simple test target to check for proper parsing of bad arguments note that parsing of the right arguments already happens in other unit tests This invokation method appears to work on both windows and linux * Revert "add a simple test target to check for proper parsing of bad arguments" This reverts commit 2ec178a.

* tweaks to vscode plugins to reflow comments at a width that matches the rest of the code style * addressing comments

* make sure to include the wide character string functions * Replace older __in SAL notation with current _In_ to avoid libstc++-10 issues GNU libstdc++ 10 uses the token `__in` in some generated code, so we need to avoid it.

Binplace Net.Core applications. To binplace the application, include following line in the csproj: <Import Project="$(BaseDir)\build\Mlos.NetCore.Binplace.targets" /> VS builds fine.

… !439762

related fixup for clang script as well

…y should include standard... Enable rule SA1642 Constructor documentation summary should include standard text Initializes a new instance of the <see cref="... "/> struct (or class). Enforcing rule, text is autogenerated. Check-in first as an effort to minimize amount of changes in PR "Only the target process creates the config shared memory".

Rename shared memory methods Create to CreateNew, Open to OpenExisting

Only the target process creates the config shared memory. Global shared memory now contains the information about registered memory regions and registered settings assembly. Now the target process creates the config shared memory, where it create it with the desired size and name. (Configuration is not exposed yet, but changing it does not require changes in the agent and the target process). We do support only one shared config memory region, but now this can be extended. Fix a bug when shared maps are not cleanup in case of initialization failure. Implementation of dictionary lookups is moved to SharedConfigDictionaryLookup struct. Next PR: Remove static fields from MlosContext.

Lots of build/pipeline cleanup with the goal of making and publishing standard Docker images the for Linux developers and the CI pipelines (microsoft#36). - Most implementation notes are in `.github/workflows/main.yml`, but here's a brief summary: - Nightly we build fresh docker images for Ubuntu {16.04, 18.04, 20.04}. They are only published if all of the usual build/test jobs pass (using the same logic as for PRs). That also helps prime the local caches for the following: - On each PR, we 1. (re)build the image from a local buildx cache that's keyed on the sha of the PR, and optionally will also pull from the latest "main" build (since it presumably worked). That is initially only stored in a local cache. Part of the goal here is to avoid the need to push unvetted docker images (e.g. tagged by sha) to the public registry and then have to clean them up later (which is not always even possible). 2. For each build/test job, we again "rebuild" the image, using that cache from step 1, so that it goes quickly. Then we load the image into the local runner's docker daemon and start an instance to run our tests. This reload is necessary because individual jobs are assigned to new runners so we can't reuse state between jobs. Jobs are organized into dependency graphs to account for this cache reuse. At the moment, all of the tests run on all versions of Ubuntu, which makes the CI pipeline a bit slow. - DONE: We could consider only running 16.04 and 20.04 in the CI pipelines and then running 18.04 in the nightly scheduled run or else just drop support for it entirely. 3. Once all of the tests pass, and the PR is merged, we execute a final job to update the published version of the job by retagging it and pushing it to various registries. The current registries are all Github services (for convenience to get started): - docker.pkg.github.com: this is nice because it supports the built in `GITHUB_TOKEN` secret so doesn't require any additional work for a forked repo to use for testing (they push to their own path though). However, it doesn't support anonymous pulls and is being deprecated by ... - ghcr.io: this the newly announced Github Container Registry service. It requires a special set of secrets to be assigned to the repo, so needs some additional work to make use of. Also, as of now, the Microsoft org doesn't seem to have it enabled, so for now I've set it to push to the Microsoft-CISL org. Additional work: - Reorg the Dockerfile to use multi-stage builds to allow skipping copying the source code into it so that we only distribute the build tools. We can also optionally build slimmer "python-only" images this way. In a future PR we'll take on the work to switch from straight pip to conda (microsoft#83) - TODO: Could optionally prime the `/src/MLOS` folder in the image so that one could `git pull` immediately to get the latest code if they wanted to without a local copy as a bind mount. - Added a `make docker-image` target to simplify repeated local docker image build invocations. - Switch to using tools (cmake, dotnet) installed in the system rather than fetched during build (microsoft#109) - These scripts are also nice reusable elements for users to install in their own environments if they wish. - DONE: Update the documentation to reference them instead of relying on the old "fetch" method. - DONE: There is work in ADO that provides a similar `install.python.sh` script to setup the `python3.7` install. Will port that over later as well. - Cleanup _most_ of the build system to match (I have not yet removed the old setup scripts, but they are no longer automatically called). - Added vscode integration support for developing inside the container (microsoft#105, microsoft#2) - Added support for running as a non-root user so that bind mounts of the repo source tree work without re-chowning files on the host system as root. This required a fair bit of fixups, but I think should cut down on some issues with students/others and makes the vscode integration easier as well. - Add nightly run scheduling of Python long haul tests (microsoft#113) - TODO: Fixup docker related documentation (microsoft#87)

* Added leveDB tuning notebook * Added brief explanatino * added reference * added leveldb installation instructions and run_workload function * checkpoint * added run optimizer functionality to leveldb_tuning.ipynb * PR changes

… folder name Add Mlos.Cpp.Binplace.targets for to binplace native projects. Resolve property BinplaceRootDir after including Microsoft.Cpp or Microsoft.NET.Sdk, so the Plaftorm propery is always set. Add option SkipBinplaceSettingsRegistry (do not binplace Mlos.NetCore). CakeBuild, when creating nuget spec, verify if there are files in the provided path. Include missing .inl file in vcxproj.

Fix Linux warnings (except unknown pragmas)

Remove static fields from MlosContext class. Create a static Mlos Instance field (workaround until we have Experiment class).

Remove usage of auto from c++ files, remove Mlos.Agent.Server from nuget lib folder (it is present in bin folder)

…est for CoreXT builds Set correctly SettingsAssemblyBinplaceDir in Mlos.Test for CoreXT builds

Remove ApplicationFilePath from SettingsAssembly config. Mlos.Agent.Server is using env variable to locate settings assembly registry.

Remove unused file

Some followup work to PR !444061 and additional minor fixups. - Update the docs to reference the new paths - Make `cmake` aware of the `target/bin/` placed items - Fixup some `cmake` dotnet parallel build issues by changing which target references the dependencies (`dotnet build` has issues when invoked in parallel) - Start removing references to the `MLOS_SETTINGS_REGISTRY_PATH` environment variable - Make sure the `Mlos.Agent.Server` exits non-zero if one of the components or `Mlos.Agent` fails

Adding standard comments in Net.Core projects Removing the following exceptions from the analyzer's rules. SA1623 The property's documentation should begin with: Gets (or Sets) SA1633 The file header is missing or not located at the top of the file CA1801 Parameter of method is never used

Continuation of Brian's work bpkroth/more-cpp-compiler-warning-fixups. Enable clang compilation flags: -Wall -Wextra -Wpedantic -Werror. Remove macros from Mlos.Core. Rename variable name from elementLength to elementDataSize.

Works around some CI issues with "dotnet tool" commands for Cake build with older dotnet versions on older Ubuntu 16.04 ADO runners and similar issues for cmake. TODO: This will need additional fixes in the future. We still have the setup-cmake.sh, setup-dotnet.sh type of scripts which can install local copies, but we've started the process of stripping use of those out of the build system. One benefit of those is that they also allow us to fetch (pin) an explicit version (though that was probably a security issue) and use that version even if the default system repo lists don't have a newer version for us. A more complete fix will be to include the logic to check for an appropriate version in the existing system repos (not just existence of the package), and if it doesn't exist add the necessary package repos as well.

* work to add external project notebooks to the website menus * tweak some of the formatting a tad * copy the leveldb notebook images into place * convert them to markdown style links

- run build.cake for the right config - run cmake in parallel inside cake - run/test the cmake-install target

* fixup some markdown errors * add a reference to the class in the README * fixup notebook links following microsoft#135

…crosoft#139)

* increase longhaul job step timeout beyond default previously we only set the overall job timeout beyond the default of 360 minutes (6 hours) with this change we also explicitly set the timeout for the longhaul execution step in the job (in case that was also defaulting to 360) still not sure if this will fix things, but worth a try

* initial commit for OneHotEncoder HypergridAdapter * Cleaned up project/unproject; dropped point project/unproject methods; corrected some defects in Point, HypergridAdapter; Added more unit tests * Addressing PR feedback * Addressing pylint * Addressing PR feedback Co-authored-by: Ed Thayer <edthaye@microsoft.com>

Co-authored-by: Adam Śmiechowski <adsmiech@microsoft.com>

* A bunch of bug fixes * More bug fixes Co-authored-by: Adam Śmiechowski <adsmiech@microsoft.com>

Co-authored-by: Adam Śmiechowski <adsmiech@microsoft.com>

* Fixed how we compute the number of samples per tree * Fixed bug in TestBayesianOptimizer Co-authored-by: Adam Śmiechowski <adsmiech@microsoft.com>

…crosoft#149) Co-authored-by: Adam Smiechowski <adsmiech@microsoft.com>

Co-authored-by: Adam Smiechowski <adsmiech@microsoft.com>

* reorg docs, fixup links, etc. Follow ups to microsoft#101 (comment) * link fixup * Update source/Mlos.Agent.Server/README.md Co-authored-by: Andreas Mueller <t3kcit@gmail.com> * Update source/Mlos.Agent.Server/README.md Co-authored-by: Andreas Mueller <t3kcit@gmail.com> * addressing comments * add a component documentation menu to the website Co-authored-by: Andreas Mueller <t3kcit@gmail.com>

…oft#152) * Added nested polynomial function * Bug fix in ComponentConfigStore * Added a better comment for NestedPolynomialObjective * Fixed linter * Resolved my own comments Co-authored-by: Adam Smiechowski <adsmiech@microsoft.com>

Co-authored-by: Adam Śmiechowski <adsmiech@microsoft.com>

…ndom Forest Model (microsoft#148) * intial commit to confirm branch rename worked * found and fixed defects in CategoricalToOneHotEncodingAdapter for cases without categorical dims; worked through changes in RERF and RERF tests * removed unneeded comment/code * removed test for gradient; will replace this when the path for using this gradient is clear * added sklearn StandardScaler to RERF to reduce hat matrix condition number; eliminated unit tests for exact polynomial and gradient coeffs as these are relative to scaled X and are not knowable from unit tests; eliminated specific random seeds used in unit tests * addessing PR feedback * increased number of lasso fit iterations to address prediction inaccuracies when fitting heirarchical categorical objective * expect to have resolved flakey test by increasing training set size * addressing pylint * still trying to understand prediction accuracy miss Co-authored-by: Ed Thayer <edthaye@microsoft.com>

Optimzier Evaluation Tools The bulk of this PR lives in two files: - OptimizerEvaluationReport.py - OptimizerEvaluator.py Their usage is presented in the OptimizerEvaluationTools.ipynb Notebook. The goal of this work is to produce a framework to efficiently compare optimizer configurations across a variety of synthetic objective functions.

* use pytest to run python tests * install pytest in windows python checks CI * install pytest in docker image * Update scripts/run-python-tests.sh Co-authored-by: Brian Kroth <bpkroth@users.noreply.github.com> * some review comments * add back some options from the previous invocation * allow waiting for the grpc server to finish stopping and cleanup the client socket as well * rewrite unittest patterns to pytest patterns * no camelcase in teardown in pytest * more unittest to pytest conversion * remove empty inheritance * actually safe file * reviews by adam Co-authored-by: Brian Kroth <bpkroth@users.noreply.github.com> Co-authored-by: Brian Kroth <bpkroth@microsoft.com>

…icrosoft#174)

This PR is a first step towards multi-objective optimization. It's really simple: the user can now define an objective space that's multi-dimensional. Subsequently the user can register observations that contain values for one or more of those objectives. Still, only one objective is optimized. Effectively, all those extra objectives are remembered, but ignored.

@pytest

…ee continuous dimensions (microsoft#172) * Adding hypergrid adapter that explodes continuous dimensions to polynomial basis functions * Added point project/unproject tests; fixed some defects these tests exposed * added more tests; fixed more defects * addressing PR feedback; Recommendation to use @pytest not yet working, so that snippet is commented out - to be fixed before final checkin * cleaned up unit tests w/ @pytest.mark.parameterize; found one defect in the _test_polynomial_feature_values_are_as_expected Co-authored-by: Ed Thayer <edthaye@microsoft.com>

* Remove clang-tidy target It doesn't work as is since it attempts to redefine the target upon every sub-project include. At the moment we use uncrustify anyways. * Initial work to enable gcc support Started as a hack to workaround cmake's FetchContent not appearing to pass the CMAKE_COMPILER_ID through when building with CXX=clang++-10 This also fixes a few minor issues that gcc complained about. TODO: - [ ] Enable CI pipeline tests * allow overriding the compiler choice on the cli * remove cdecl * add gcc testing to ci pipeline * tweaks * include the cxx compiler in the name * exclude gcc from Ubuntu 16.04 * remove some old debugging code * remove need for -fdeclspec from clang also shorten the name of the attr macro * porting some changes from grlap over * change the style of initializer we use for std::array for reflection ... ... to something that gcc in Ubuntu 16.04 can use * enable gcc on 16.04 again * uncrustify fixups * tweak the job names * default to clang, but allow it to be overridden Co-authored-by: Greg Lapinski <grlap@microsoft.com>

This is the first step in a line of work aimed towards making MLOS more natively consumable by external projects. The basic idea here is that we use the standard .NET.Sdk method of packaging each of the relevant `.csproj`s as their own `.nupkg`. This let's us rely on `dotnet` to automatically track `PackageReference` and `ProjectReference` dependencies and generate them into the `.nuspec` that produces the `.nupkg`. Since `.nupkg` files are considered immutable, we have to create unique version numbers for each of them. To do this, I added [`GitVersionTask`](https://github.com/GitTools/GitVersion), but only for the OSS builds. Typically, GitVersion expects one to use GitFlow or GithubFlow workflows with develop and release branches. We are currently only using a Mainline workflow model, so the `GitVersion.yml` file is configure as such. The versions produced use SemVer by default, so when building on a branch, we get a version like the following: `Mlos.SettingsSystem.CodeGen.0.1.1-codegen-nuget-for-external-project-integration.33.nupkg` Where - the `0.1.1` portion is taken as the number of commits on the mainline branch since the last v0.1.0 tag found in the repo - the `-codegen-nuget-for-external-project-integration` is the name of the branch - and the `.33` is the number of commits since that branch point. In effect, we can use `git tag` to version our `x.y.z` releases, at least until we start adding `release/0.1.x` style servicing branches. For the moment, the `.nupkg` files are not signed and not published (e.g. to nuget.org). Instead, we produce them into the `target/pkg/$(Configuration)/` tree whenever either the new `Mlos.NetCore.Components.Packages.proj` is build or else when `dotnet pack` is run on some `.csproj`. The `ExternalIntegrationExample.SettingsRegistry.csproj` (which otherwise uses no Mlos build rules and acts as a standard `.NET.Sdk` example) is used for testing and is configured to pull from that tree (for local testing) as well. The next stage of work (separate PR) is to configure a set of `cmake` rules using `FetchContent` to grab the C++ components, build these packages and reference them when building the rest of a C++ project and its dependencies. TODO: - [x] Enable the test project and package building in CI - [x] for Windows - [x] for Linux - [x] Test VisualStudio build - [x] Verify CDPX/CoreXT builds (!473121) WONTFIX: - Leaving actual publishing/signing to nuget.org for a future PR

* recent msbuild auto imports those build/*.targets from PackageReferences * make sure to tell msbuild that we aren't attempting to reference the c# settings registry dll from c++ projects

…soft#185) * update comment * minor tweaks * rejectlist a couple of cmake package versions that are causing problems due to lacking dependencies

This PR introduces a simple algorithm for computing a pareto frontier, given an objectives dataframe and a corresponding OptimizationProblem. You will find the algorithm in ParetoFrontier. This class will later be extended to maintain a cache of the latest pareto, and to iteratively add to it as more observations roll in. It will be owned by a BayesianOptimizer.

…t#186)

* start on context features * add draft of context usage * add draft of context in BO notebook * line length lint * use Optimizer.feature_space * allow computation of optimum in BayesianOptimization.optimum * be backwards compatible for now * add wrapper around surrogate model * add example of BO with context * fix sign issue, raise on name conflict * get dimension names for comparison * try to pacify pylint * update optimizer interface * one more for the interface * rework context notebook * whoops safe failure * refactor suggest into OptimizerBase, add maximize to public interface * intermediate checkin * update to mirror refactoring * minor fixups, use different function in notebook * go back to old join for feature space, remove maximize * slightly slimmer feature space * add context to BayesianOptimizer.register * move feature dataframe construciton to optimization problem * add greedy utility function to maximize utility on speculative optimum * fix use us sign in greedy utility function * undo spurious change * ensure register is consistent in OptimizerBase * Update source/Mlos.Python/mlos/Optimizers/OptimizationProblem.py Co-authored-by: Adam Śmiechowski <51930563+byte-sculptor@users.noreply.github.com> * Update source/Mlos.Python/mlos/Optimizers/OptimizationProblem.py Co-authored-by: Adam Śmiechowski <51930563+byte-sculptor@users.noreply.github.com> * rename best_predicted_within_context to best_speculative_within_context * rename config_values to feature_values * fix merge issues * check in changes for context notebook * make sure BayesianOptimizerProxy adheres to Optimizer interface * rename GreedyUtilityFunction to PredictedValueUtilityFunction * don't explicitly set n_new_samples_before_refit if using default * actually rename greedyutilityfunction * minor fixes for rename * pylint * change interface of predict to use parameter_values * typo * pyflake whitespace * fix suggest calls in grpc * undo support for context in gRPC and cheat a bit * hack around context in gRPC * fix some tests for change of get_all_observations * store parameters and context separately in BayesianOptimizer * remove some more weird projections * create features of concatenated observations lol * pass feature df to compute_goodness_of_fit * indentation fix * remove unused variable * start on testing * fix and test some error messages * add shape tests * add test for optimization problem * pylint * add tests for actual optimization * BayesianOptimizerProxy has no ExperimentDesigner * Update source/Mlos.Python/mlos/Grpc/BayesianOptimizerProxy.py Co-authored-by: Brian Kroth <bpkroth@users.noreply.github.com> * Update source/Mlos.Python/mlos/Optimizers/BayesianOptimizer.py Co-authored-by: Brian Kroth <bpkroth@users.noreply.github.com> * Update source/Mlos.Python/mlos/Optimizers/unit_tests/TestBayesianOptimizer.py Co-authored-by: Adam Śmiechowski <51930563+byte-sculptor@users.noreply.github.com> * Update source/Mlos.Python/mlos/Optimizers/unit_tests/TestOptimizationProblem.py Co-authored-by: Adam Śmiechowski <51930563+byte-sculptor@users.noreply.github.com> * Update source/Mlos.Python/mlos/Grpc/BayesianOptimizerProxy.py Co-authored-by: Brian Kroth <bpkroth@users.noreply.github.com> * Update source/Mlos.Python/mlos/Grpc/BayesianOptimizerProxy.py Co-authored-by: Adam Śmiechowski <51930563+byte-sculptor@users.noreply.github.com> * line breaks in context notebook * remove trailing whitespace * remove some default values, fix for new interface for predict * add todo to handle context in OptimizerMicroservice * fix error message matching * fix phrasing of errors * some day... Co-authored-by: Adam Śmiechowski <51930563+byte-sculptor@users.noreply.github.com> Co-authored-by: Brian Kroth <bpkroth@users.noreply.github.com>

…rrogate model. (microsoft#182) * integrated polynomial features adapter into RERF * cleaned up some lint catches * fixed defects in ContinuousToPolynomialFeatures Adapter plus improved unit tests for this adapter * addressing some PR feedback; changes to remedy flaky tests * lowered test threshold based on new observered ranges for unexplained_variance Co-authored-by: Ed Thayer <edthaye@microsoft.com>

…crosoft#188)

* first shot at adding coveralls integration * coverage output as xml * use github action * use codecov not coveralls * install pytest-cov * don't comment on PRs * add badge for codecov * Update .github/workflows/main.yml Co-authored-by: Brian Kroth <bpkroth@users.noreply.github.com> * Update .github/workflows/main.yml Co-authored-by: Brian Kroth <bpkroth@users.noreply.github.com> Co-authored-by: Brian Kroth <bpkroth@users.noreply.github.com>

Should fix this error that recently showed up in our docker/build-push-action step: "Error: Invalid Opening Quote: a quote is found inside a field at line 2" Co-authored-by: Andreas Mueller <andreasmuellerml@gmail.com>

fixes microsoft#191

More comments in Net.Core projects

More comments in Net.Core projects.

Move AssemblyInitialize to new ExperimentSession class. This is not complete implementation of ExperimentSessionManager. Only SmartCache project is updated. Small fixes related to shared memory cleanup.

Annotate C++ code using SAL 2.0

Chanage SAL annotation _Check_return_ to _Must_inspect_result_

Removing unused file

Remove unused method declaration

Correct CodeGen namespace

…r issues). Rename StaticSingle to AlignedInstance (and fix ctor issues). Set ResolveAssemblyWarnOrErrorOnTargetArchitectureMismatch to None to avoid warning when building in VS.

Workaround build issues in VisualStudio

Exchange FileDescriptors using Unix domain socket

and other minor cleanup

…icrosoft#195) Specifically, fix the dependency logic for ctest fixtures to avoid some periodic random failures when multiple optimizers tried to start at once (colliding on the use of the port) and remove unnecessary dependencies (Mlos.UnitTest didn't need any of the optimizer fixtures).

Work to make it easier/possible to integrate the C++ components of MLOS with an external C++ project (e.g. LevelDB, MySql, etc.) via `cmake`. - Fixes some issues with our own `cmake` config assumptions. - Adds examples patterns and documentation for using in an "external project" (currently just a minimal example). - Hooks those into the CI pipelines. This branch was split off of the work from microsoft#177 (since merged) to build NuGet packages for some of the C# components, so many of the original commits are shared.

* Tweaks to Python Test CI pipelines - Increase the timeouts on the pytest runs. There have been some timeouts recently, due in part to the addition of code coverage tracking (microsoft#187). - Separate the pylint checks (short) from the pytest runs (long, flaky) so that we can have the docker image publish task proceed in nightly runs even if the pytest run was flaky. * Avoid duplicate ctest runs during Github CI runs Fixes microsoft#196 * pass the GITHUB_WORKFLOW env var through the docker process * fixup argument order

Attempting to fix the following error: ``` -> Pinging Codecov https://codecov.io/upload/v4?package=github-action-20201130-cc6d3fe&token=secret&branch=main&commit=27c39e71a07b063cc2c59a3eb72609a722d17510&build=397032896&build_url=http%3A%2F%2Fgithub.com%2Fmicrosoft%2FMLOS%2Factions%2Fruns%2F397032896&name=&tag=&slug=microsoft%2FMLOS&service=github-actions&flags=unittests&pr=&job=&cmd_args=Q,f,n,F,Z HTTP 400 ERROR: Tokenless uploads are only supported for public repositories on GitHub Actions that can be verified through the Actions API. Please use an upload token if your repository is private and specify it via the -t flag. You can find the token for this repository at the url below on codecov.io (login required): Repo token: https://codecov.io/gh/microsoft/MLOS/settings Documentation: https://docs.codecov.io/docs/about-the-codecov-bash-uploader#section-upload-token ``` See Also: https://github.com/microsoft/MLOS/runs/1489105180?check_suite_focus=true

Avoids the following error: ``` Traceback (most recent call last): File "/opt/hostedtoolcache/Python/3.9.0/x64/bin/licenseheaders", line 8, in <module> sys.exit(main()) File "/opt/hostedtoolcache/Python/3.9.0/x64/lib/python3.9/site-packages/licenseheaders.py", line 612, in main arguments = parse_command_line(sys.argv) File "/opt/hostedtoolcache/Python/3.9.0/x64/lib/python3.9/site-packages/licenseheaders.py", line 304, in parse_command_line known_extensions = [ftype+":"+",".join(conf["extensions"]) for ftype, conf in typeSettings.items()] File "/opt/hostedtoolcache/Python/3.9.0/x64/lib/python3.9/site-packages/licenseheaders.py", line 304, in <listcomp> known_extensions = [ftype+":"+",".join(conf["extensions"]) for ftype, conf in typeSettings.items()] KeyError: 'extensions' Error: Process completed with exit code 1. ```

* Make sure cmake generated makefiles target the right build type config when invoked through cake * remove unnecessary dependency on netcore build for cake builds (it's handled internally by cmake as well)

* avoid broken markdown-link-check action version The following upstream change adds quotes around our list of folders to check which results in passing a bad path to find: gaurav-nelson/github-action-markdown-link-check@53c1dda

* Started moving hyperspheres to a separate objective function * Created a Hypersphere ObjectiveFunction and the tests are passing * Linter Co-authored-by: Adam Smiechowski <adsmiech@microsoft.com>

Upstream now has the fix for the issue we were running into: gaurav-nelson/github-action-markdown-link-check#88 https://github.com/gaurav-nelson/github-action-markdown-link-check/releases/tag/1.0.10 https://github.com/gaurav-nelson/github-action-markdown-link-check/releases/tag/v1

1. Invokes pytest with a `-n auto` argument to make multiple tests run in parallel (num parallel jobs = num cores) 2. Installs pytest-xdist in all our pipelines to make the above possible 3. Shortens the duration of TestOptimizerEvaluator 4. Modifies the tests that use gRPC to attempt to start the service on 100 different ports before giving up. This is the easiest way to make sure that all tests requiring gRPC can run in parallel. The alternative would be to have them all talk to a single instance, but it would turn an easy parallelization problem into a hard one as we'd need to manage the lifetime of that single instance. 5. Relaxes the check in TestSmartCacheWithRemoteOptimizerV3.py as the current one was a bit too ambitious and lead to some flakiness. 6. Puts a band-aid on test_optimization_with_context. microsoft#207 hints at a long-term fix. Co-authored-by: Adam Śmiechowski <adsmiech@microsoft.com> Co-authored-by: Brian Kroth <bpkroth@users.noreply.github.com>

* fixup glob for running long haul tests fixes microsoft#191 * need to wrap wildcards in shell

This PR introduces a multi-objective random forest model, which is essentially a collection of single-objective random forests - each predicting a different objective. This model is not yet exposed as part of the optimizer. Co-authored-by: Adam Śmiechowski <adsmiech@microsoft.com>

…ns (microsoft#197) * Comparison of two different ways to summarize decision tree predictions * removed two obsolete helper functions; corrected typo in formula in intro markdown cell * modified what's considered true observation summary stats to reflect duplication of obs across leaves Co-authored-by: Ed Thayer <edthaye@microsoft.com>

…#211) * Workaround an issue introduced by recent versions of sphinx With sphinx 3.4.0-3.4.2 (released in late December 2020), sphinx generates asserts by the Point module due to invalid keys (e.g. __sphinx_mock__) in the Point. Exception occurred: File "/src/MLOS/source/Mlos.Python/mlos/Spaces/Point.py", line 75, in __getitem__ raise KeyError(f"This Point does not have a value along dimension: {dimension_name}") KeyError: 'This Point does not have a value along dimension: __sphinx_mock__' * A better fix to the sphinx hasattr issue * fixup: raise an appropriate error, instead of returning None https://docs.python.org/3.7/reference/datamodel.html?highlight=__getattr__#object.__getattr__

microsoft#212) This is the next batch of code getting us towards multi-objective optimization. PAST: Previously we have been able to register multiple objectives with the optimizer, but the optimizer wouldn't use them at all. PRESENT: With this PR the optimizer will instantiate a regression model for each of the objectives, but it doesn't yet use the predictions for anything. FUTURE: The next steps will be to write a few multi-objective utility functions so that the optimizer can use them to suggest new configurations. The chief goal of this PR is to integrate the MultiObjectiveHomogeneousRandomForest into the framework. Note that there is no change to the Optimizer's interface. The prediction of the first objective is returned, and all the others are thrown out.

* add manual CI run support * disable use of old github docker package registry

This PR enables multi-objective optimization based on multi-objective homogeneous random forest surrogate model and a probability of improvement utility function. The core of this PR is in `MultiObjectiveProbabilityOfImprovementUtilityFunction.py` which uses Monte Carlo simulation to estimate the probability that a given configuration expands the pareto frontier. To support the above functionality, we need a way of determining whether a point is dominated or not. ParetoFrontier.is_dominated(...) does that trick. One way to track optimizer's progress is to track the change in the volume of the pareto frontier. Measuring that volume precisely appears hard, so we settle for a probably approximately correct answer produced by ParetoFrontier.approximate_pareto_volume() for now. All of the other changes revolve around: 1) Integrating the new UtilityFunction into the framework. 2) Persisting the ParetoFrontier within the Optimizer (needed by the utility function). 3) Testing. 4) Small fixes. **Testing** I added new unit tests for new functionality. I also ran multi-objective optimization on 3 different problems over the weekend to compare it to random search (10 repeats for both guided and random search). Firstly, nothing crashed. Secondly, the results are very promising even without any tuning. The graphs below show the comparison between average pareto volume vs. number of iterations for across 10 runs. 2 objectives: ![image](https://user-images.githubusercontent.com/51930563/105795949-952a7500-5f42-11eb-91c7-e4887ec9267f.png) ![image](https://user-images.githubusercontent.com/51930563/105795961-9d82b000-5f42-11eb-85cb-12730e081eea.png) 4 objectives: ![image](https://user-images.githubusercontent.com/51930563/105795872-6b714e00-5f42-11eb-83e2-8532e4444eff.png) ![image](https://user-images.githubusercontent.com/51930563/105795902-7a580080-5f42-11eb-84a8-299a01278f96.png) More work will be required to: 1) Update the OptimizerEvaluator and OptimizerEvaluationReport to capture multi-objective statistics. 2) Update the gRCP interface to support multi-objective predictions 3) Visualizations, more tests and usability improvements. 4) Other utility functions. But practically, this PR enables multi-objective optimization.

…#216) OptimizerEvaluator now tracks pareto over time, pareto volume over time and multi-objective goodness of fit metrics. OptimizerEvaluationReport can now both write itself to disk and read itself back into memory.

Adding EnvelopedWaves and MultiObjectiveEnvelopedWaves as two new synthetic functions with many useful properties. These are sine waves enveloped by another function, either linear, quadratic or another sine wave. An enveloped sine wave produces complexity for the optimizer that allows us evaluate its behavior on non-trivial problems. Simultaneously, sine waves have the following advantages over polynomials: 1. They have well known optima - even when we envelop the function with another sine wave, as long as we keep their frequencies harmonic, we can know exactly where the optimum is. 2. They cannot be well approximated by a polynomial (Taylor expansion is accurate only locally). 3. For multi-objective problems, we can manipulate the phase shift of each objective to control the shape of the pareto frontier. The way multi-objective enveloped waves work is that we pass the same parameters through 1 or more single-objective enveloped waves functions. One useful property is that we not only know where the optima for individual functions are (maxima of sine are easy to find), but we can also know and control the shape of the pareto frontier, by controlling the phase difference between the individual objectives. For example: a phase difference of 0, means that that the objective functions are overlaid on top of each other and their optima are exactly on top of each other, so the pareto frontier is a single, optimal point Alternatively, the phase difference of quarter-period, introduces a trade-off between the objectives where y0 = sin(x) and y1 = sin(x - math.pi / 2) = -cos(x) which yields a pareto frontier in a shape of a quarter-cirle. Yet another option is to use a phase difference of math.pi. This yields a trade-off between the objectives where: y0 = sin(x) and y1 = sin(x - math.pi) = -sin(x) = -y0 which yields a pareto frontier where a gain in one objective results in an equal loss in the other objective, so the shape of that frontier is a diagonal of a square.

This PR makes it possible to create the new objective function via the ObjectiveFunctionFactory. Existing tests cover this addition.

This really simple change adds a start time and an end time to the optimizer evaluation report, thus allowing us to also know how long the optimization process took.

Adding another factory to the project. We already have a BayesianOptimizerFactory, and an ObjectiveFunctionFactory, and with this PR we add a UtiliyFunctionOptimizerFactory. Some subsequent PRs will contain the remaining UtilityFunctionFactory, and RegressionModelFactory. This will help us unify our testing, make sure that all derived classes expose the same interfaces and exhibit sufficiently similar behavior.

I was working on Random Near Incumbent Optimizer and I noticed that GlowWormSwarmOptimizer was doing something really wrong with the context features. I don't really understand how/if it ever worked with context. So this is a quick fix, where we have the worms keep track of the parameter values (not feature values) and only constructing the features dataframe when calling the utility function. This way, the glow worm swarm optimizer can suggest optimal parameters for a given context, instead of suggesting optimal parameters and context. I really don't know how it ever worked. I suspect it was never able to compute the utility value, and always bailed on the first 'if' producing a de-facto random suggestion.

Commits on Aug 18, 2020

Initial MLOS import

bpkroth committed Aug 18, 2020

Configuration menu

View commit details

Copy full SHA for 44c65e7

Browse repository at this point

Copy the full SHA

44c65e7 View commit details

Browse the repository at this point in the history

Commits on Aug 19, 2020

removing empty file (#10 )

bpkroth authored Aug 19, 2020

Configuration menu

View commit details

Copy full SHA for 1f40fca

Browse repository at this point

Copy the full SHA

1f40fca View commit details

Browse the repository at this point in the history

Commits on Aug 25, 2020

Fix minor error in README.md (#13 )

bpkroth authored Aug 25, 2020

Configuration menu

View commit details

Copy full SHA for f1443b8

Browse repository at this point

Copy the full SHA

f1443b8 View commit details

Browse the repository at this point in the history

Revert markdown lint check package vers pinning #209

Revert markdown lint check package vers pinning #209

Commits on Apr 6, 2020

Commits on Aug 18, 2020

Commits on Aug 19, 2020

Commits on Aug 21, 2020

Commits on Aug 24, 2020

Commits on Aug 25, 2020

Commits on Aug 28, 2020

Commits on Aug 31, 2020

Commits on Sep 1, 2020

Commits on Sep 2, 2020

Commits on Sep 3, 2020

Commits on Sep 4, 2020

Commits on Sep 8, 2020

Commits on Sep 9, 2020

Commits on Sep 10, 2020

Commits on Sep 11, 2020

Commits on Sep 12, 2020

Commits on Sep 14, 2020

Commits on Sep 15, 2020

Commits on Sep 16, 2020

Commits on Sep 17, 2020

Commits on Sep 21, 2020

Commits on Sep 22, 2020

Commits on Sep 23, 2020

Commits on Sep 24, 2020

Commits on Sep 25, 2020

Commits on Sep 27, 2020

Commits on Sep 28, 2020

Commits on Sep 29, 2020

Commits on Oct 1, 2020

Commits on Oct 6, 2020

Commits on Oct 12, 2020

Commits on Oct 13, 2020

Commits on Oct 14, 2020

Commits on Oct 15, 2020

Commits on Oct 17, 2020

Commits on Oct 19, 2020

Commits on Oct 20, 2020

Commits on Oct 22, 2020

Commits on Oct 23, 2020

Commits on Oct 26, 2020

Commits on Oct 27, 2020

Commits on Oct 28, 2020

Commits on Oct 31, 2020

Commits on Nov 4, 2020

Commits on Nov 7, 2020

Commits on Nov 12, 2020

Commits on Nov 13, 2020

Commits on Nov 14, 2020

Commits on Nov 17, 2020

Commits on Nov 18, 2020

Commits on Nov 19, 2020

Commits on Nov 20, 2020

Commits on Nov 23, 2020

Commits on Nov 30, 2020

Commits on Dec 1, 2020

Commits on Dec 2, 2020

Commits on Dec 3, 2020

Commits on Dec 9, 2020

Commits on Dec 10, 2020

Commits on Dec 11, 2020

Commits on Dec 15, 2020

Commits on Jan 5, 2021

Commits on Jan 14, 2021

Commits on Jan 15, 2021

Commits on Jan 28, 2021

Commits on Jan 29, 2021

Commits on Feb 18, 2021

Commits on Feb 19, 2021

Commits on Feb 25, 2021

Commits on Mar 5, 2021