Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

a biggish revamp of the toplevel documentation #65

Merged
merged 3 commits into from
Dec 13, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ wheels/
*.egg-info/
.installed.cfg
*.egg
public/

# PyInstaller
# Usually these files are written by a python script from a template
Expand Down
78 changes: 61 additions & 17 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
![charmory logo](docs/assets/charmory.png)
![charmory logo](docs/assets/armory-logo.png)

---

Expand All @@ -12,15 +12,10 @@

# Overview

Charmory is a scaffolding name as we rework code coming from the `armory.` namespace.
It is slated to be renamed to `armory` once we adapt all legacy code that needs
to be adapted. We expect the `charmory.` namespace to be disappear by the end of 2023.

Presently, working use of armory-library, as shown in the `examples/` directory
imports symbols from both `armory` and `charmory` namespaces. Soon a global substitution
in user code from `charmory` to simply `armory` will be needed. We'll announce
in the release notes when this is needed.

Armory-library is a pure Python library which allows the measurement of ML systems in
the face of adversarial attacks. It takes the years of experience gained and techniques
discovered under the [DARPA GARD program][gardproject] and makes it available to the
general user.


# Installation & Configuration
Expand All @@ -29,18 +24,62 @@ in the release notes when this is needed.
pip install armory-library
```

Will make the `armory` and `charmory` namespaces available to your Python environment.
This is all that is needed to get a working Armory installation. However, Armory-library
is a library and does not contain any sample code. We provide examples in the
`armory-examples` repository which is released concurrently with Armory-library.

## Example programs

To install the examples, run:

```bash
pip install armory-examples
```

This will install the example code in your current Python environment and will add
a command `armory-examples`. This will list the available examples and allow you
to run them. For example:

```bash
$ armory-examples
mnist-vit-pgd
$ armory-examples mnist-vit-pgd
# runs the example mnist-vit-pgd evaluation
```

The [example source code][example-src], along with the [Armory-library
documentation](docs/index.md) is a good place to learn how to construct your own
evaluations using armory-library.

# Quick Look

We have provided an sample notebook using Armory to evaluate an MNIST classifier
in the presence of a Project Gradient Descent (PGD) attack. The notebook can be
run for free on Google Colab to get a taste of how Armory works.

[![Open In Colab][colab-badge]][colab-url]

# Usage
See the documentation in the [armory-library docs](https://armory-library.readthedocs.io/en/latest/).
# Documentation

The Armory-library documentation is [published on Read the Docs][docs-url] or
can be viewed directly in [the docs directory](docs/index.md) of this repository.

# The historic GARD-Armory

Armory-library is the successor to the [GARD-Armory research program run under
DARPA][GARD-Armory]. As that program is nearing its conclusion, that repository
will be archived sometime in 2024 and there will be no further development in
GARD-Armory by the time you are reading this sentence. The development teams
for both GARD-Armory and Armory-library can be reached at <armory@twosixtech.com>

# Acknowledgment

This material is based upon work supported by the Defense Advanced Research Projects
Agency (DARPA) under Contract No. HR001120C0114. Any opinions, findings and
conclusions or recommendations expressed in this material are those of the author(s)
and do not necessarily reflect the views of the Defense Advanced Research Projects
Agency (DARPA).
Agency (DARPA) under Contract No. HR001120C0114 and US Army (JATIC) Contract No.
W519TC2392035. Any opinions, findings and conclusions or recommendations expressed in
this material are those of the author(s) and do not necessarily reflect the views of the
Defense Advanced Research Projects Agency (DARPA) or JATIC.



Expand All @@ -57,3 +96,8 @@ Agency (DARPA).
[docs-url]: https://readthedocs.org/projects/armory/
[style-badge]: https://img.shields.io/badge/code%20style-black-000000.svg
[style-url]: https://github.com/ambv/black
[gardproject]: https://www.gardproject.org
[colab-badge]: https://colab.research.google.com/assets/colab-badge.svg
[colab-url]: https://colab.research.google.com/github/twosixlabs/armory-library/master/examples/notebooks/armory_colab_example.ipynb
[example-src]: https://github.com/twosixlabs/armory-library/tree/master/examples/src/armory/examples
[GARD-Armory]: https://github.com/twosixlabs/armory
114 changes: 36 additions & 78 deletions docs/CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -1,91 +1,49 @@
Contributing to Armory
======================
Contributions to Armory are welcomed and highly encouraged! Armory contains a complex suite of tools that both configure the execution
environment and compose a set of objects (from an `experiment` file) to be executed in said environment.
# Contributing to armory-library

Primarily, Armory has two main modes of operation:
- Native (also known as `--no-docker`) mode - This uses a pre-set python environment to execute the configuration file.
- Docker - This uses docker to compose and launch docker images, and executes the armory experiments
within the container.
Thank you for your interest in contributing to armory-library, a library of tools and
resources for adversarial machine learning evaluation. We welcome and appreciate all
kinds of contributions, such as bug reports, feature requests, code, documentation,
tests, etc. Please read this guide before making your contribution and follow the steps
below.

For more details, including how to set up your development environment for either mode of operation see: [Setting up Development Environment](#Setting-up-the-Development-Environment)
## How to Contribute

Armory Development follows the [GitHub Standard Fork & Pull Request Workflow](https://gist.github.com/Chaser324/ce0505fbed06b947d962).
Pull-requests can be freely submitted but we suggest that you conform to our own
mechanical standards before submitting.

Armory uses GitHub Actions to test contributions, for more details see [Armory CI](/.github/ci_test.yml). Generally it will be most
useful to set up the Armory pre-commit hooks. For more information see the [Armory Style Guide](/docs/developers/style.md#pre-commit-hooks).
1. Your work should be based on the current `master/HEAD` commit.
2. Our static code analysis and formatting tools are opinionated and mechanized.
Please run `pre-commit run --all-files` before submitting your pull-request.
3. New code should have accompanying unit tests.
4. Ensure that all unit tests pass before submitting your pull-request with
`task test`.
3. Contact us at <armory@twosixtech.com> if you have any questions or to discuss
your contribution.

## Setting up the Development Environment
Armory follows the [GitHub Standard Fork & Pull Request Workflow](https://gist.github.com/Chaser324/ce0505fbed06b947d962) and therefore, to
get started with contributing to armory, you will first need to head over to [https://github.com/twosixlabs/armory](https://github.com/twosixlabs/armory)
and fork the repo. Once forked, clone that fork to your computer and cd into the forked repo location (herein refered to as `YOUR_ARMORY_REPO`).

From here, you will need to setup your python virtual environment and, depending on your use case, other applications such as Docker. The following
section will describe the details here in a bit more detail.

### Native Operation Mode
Armory can run natively within a python virtual environment on a `host` machine. To get setup you will need to
create a [virtual environment](https://docs.python.org/3/library/venv.html). Once created and activated, you will need
to install some additional requirements. Typically, it is useful to use the `-e` flag with the `armory` pip so that it
will point to your local directory, therefore utilizing code edits without requiring follow-on installs. To accomplish
this run:
```bash
cd YOUR_ARMORY_REPO
pip install -e .[developer]
```
Now that you have the environment setup, kickoff the baseline tests to make sure its all good:
```bash
pytest -s tests/test_host
```
depending on you `$PATH`, pytest might refer to a pytest outside your virtualenv, which can cause issues. As
an alternative you can use (make sure your virtualenv is active):
```bash
python -m pytest -s tests/test_host
```

If this is successful you are off to the races! If you would like to run armory in `--no-docker` mode, see:
[Armory No Docker Setup](/docs/no_docker_mode.md).

### Docker Operation Mode
Armory can utilize [docker](https://www.docker.com/) to launch containers for execution of armory experiments.
For information on how to install docker on your machine see: [Docker Installation](https://docs.docker.com/get-docker/).

Once docker is installed, armory downloads and launches containers based on the `__version__` string found in `armory.__init__`.

Note: only release versions of armory will be published to [Dockerhub](https://hub.docker.com/), therefore,
development branch images much be built locally using:
```bash
cd YOUR_ARMORY_REPO
bash docker/build.sh <tf2|pytorch|pytorch-deepspeech|all> dev
```
## Documentation

## Style Guide
Armory enforces code / file styling using [Flake8](https://flake8.pycqa.org/), [black](https://github.com/psf/black),
[yamllint](https://yamllint.readthedocs.io/en/stable/), etc. For more information about
how we configure these tools, see [Armory Style Guide](/docs/style.md).
We assume that contributors will be making small changes to the documentation that
are in the [code repository `docs` directory](docs/). If you are making larger
changes that would require more substantial changes to the documentation, you'd
likely want to talk with us first to discuss your plans.

## Pull Requests
## Code of Conduct

We gladly welcome [pull requests](
https://help.github.com/articles/about-pull-requests/).
By participating in this project, you agree to abide by these principles. Please report
any unacceptable behavior to armory@twosixtech.com.

If you've never done a pull request before we recommend you read
[this guide](http://blog.davidecoppola.com/2016/11/howto-contribute-to-open-source-project-on-github/)
to get you started.
We welcome everyone who wants to contribute to our project, regardless of their
background, identity, or experience. We expect you to be respectful, courteous, and
constructive in your interactions with others. We do not tolerate any form of
harassment, discrimination, or abuse.

Before making any changes, we recommend opening an issue (if it
doesn't already exist) and discussing your proposed changes. This will
let us give you advice on the proposed changes. If the changes are
minor, then feel free to make them without discussion.
We are part of a larger open-source community. We want to create a positive and healthy
environment for everyone. Please be mindful of how your actions and words affect others,
and uphold the values of openness, diversity, and inclusion.

## Test Cases
When adding new features please add test cases to ensure their correctness. We use
pytest as our test runner.
## Contact

For running `pytest`, users should follow `.github/workflows/ci_test.yml`.
This has tests for docker and native modes as well as formatting.
If you have any questions, suggestions, or feedback, please feel free to contact us at
<armory@twosixtech.com>.

## Documentation
When adding new functionality or modifying existing functionality, please update documentation.
Docs are all markdown (`.md`) files located in [docs](/docs/) directory or its subdirectories.
If doc files are added or removed, please also update the [markdown yaml](/mkdocs.yml)
Thank you for your contribution!
File renamed without changes
26 changes: 3 additions & 23 deletions docs/index.md
Original file line number Diff line number Diff line change
@@ -1,24 +1,4 @@
# Welcome to Armory Testbed
# Armory-library Documentation Overview

## Overview

ARMORY is a test bed for running scalable evaluations of adversarial defenses.
Models, datasets, and evaluation scripts can be pulled from external
repositories or from the baselines within this project.

Our evaluations are created so that attacks and defenses may be
interchanged. To do this we standardize all attacks and defenses as subclasses of
their respective implementations in IBM's [adversarial-robustness-toolbox](https://github.com/IBM/adversarial-robustness-toolbox)

## Developers
Check out our [developer documentation](developers/README.md) for information on how to contribute to ARMORY.

## Dataset licensing
See [dataset_licensing.md](dataset_licensing.md) for details related to the licensing of datasets.

## Acknowledgment
This material is based upon work supported by the Defense Advanced Research Projects
Agency (DARPA) under Contract No. HR001120C0114. Any opinions, findings and
conclusions or recommendations expressed in this material are those of the author(s)
and do not necessarily reflect the views of the Defense Advanced Research Projects
Agency (DARPA).
- [getting-started](docs/getting-started.md)
- [experiment-tracking](docs/experiment-tracking.md)
1 change: 1 addition & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,7 @@ src_paths = ["library/src"]


[tool.taskipy.tasks]
# the keys on the left are arguments for the `task` command, for example `task lint`
docs = "python -m mkdocs build --verbose --config-file ./tools/mkdocs.yml"
lint = "python -m pre_commit run --all-files"
test = "python -m pytest -m unit"