Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce lint job in CI #277

Merged
merged 19 commits into from
Nov 3, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 12 additions & 1 deletion .circleci/config.yml
Original file line number Diff line number Diff line change
@@ -1,6 +1,16 @@
# CircleCI 2.0 configuration file. See <https://circleci.com/docs/2.0/language-python/>.
version: 2
jobs:
lint:
docker:
- image: node:lts
steps:
- checkout

- run:
name: Lint Markdown files
command: make lint

build:
docker:
- image: python:3.7
Expand Down Expand Up @@ -28,7 +38,7 @@ jobs:

- run:
name: Test doc
command: make test
command: make test-build

- run:
name: Build doc
Expand Down Expand Up @@ -62,6 +72,7 @@ workflows:
version: 2
build:
jobs:
- lint
- build
- deploy:
requires:
Expand Down
3 changes: 3 additions & 0 deletions .editorconfig
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
[*.md]
trim_trailing_whitespace = true
indent_style = space
23 changes: 23 additions & 0 deletions .markdownlint.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
# These rules are documented on https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md

header-style:
style: atx # use the hash style headers instead of the underlined one

ul: # unordered list style
style: sublist

line_length: false # do not check for line length; rely on editor wrapping

no-inline-html:
allowed_elements:
- abbr
- i

no-duplicate-header:
siblings_only: true

emphasis-style:
style: underscore

strong-style:
style: asterisk
10 changes: 9 additions & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -14,9 +14,17 @@ install:
@pip install --upgrade pip
@pip install -r requirements.txt --use-deprecated=legacy-resolver

test:
test: lint test-build

test-build:
@${MAKE} dummy SPHINXOPTS="-q -W"

lint: # requires Node and NPM to be installed
@npx --yes markdownlint-cli "**/*.md"

format: # requires Node and NPM to be installed
@npx --yes markdownlint-cli --fix "**/*.md"

# Serve the documentation in dev mode.
dev:
@rm -Rf $(BUILDDIR)
Expand Down
46 changes: 27 additions & 19 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,22 +8,22 @@ This documentation is built with [Sphinx](https://www.sphinx-doc.org/), a Python

In order to avoid conflicting dependencies with other projects on your local machine, it is recommended to install its dependencies in a virtual environment. To create a virtual environment, run:

```
```sh
python3 -m venv .venv
source .venv/bin/activate
```

To install dependencies, run:

```
```sh
make install
```

## Build

To build the HTML documentation, run:

```
```sh
make html
```

Expand All @@ -33,7 +33,7 @@ The HTML output will be generated in the `build/html` directory.

To serve the documentation in dev mode, run:

```
```sh
make dev
```

Expand All @@ -43,49 +43,57 @@ The documentation will be served on `http://127.0.0.1:8000`

To test the documentation, run:

```
```sh
make test
```

This will also lint the source files using [Markdownlint](https://github.com/DavidAnson/markdownlint), for which you will need [Node](https://nodejs.org) and NPM.

## Autoformat

If `make lint` gives you errors, you can try running the following command to automatically format your contributions according to the existing conventions:

```sh
make format
```

## Fixing the doc

If the tests fail, here's what you can do:

1. If the errors also concern OpenFisca-Core, please take a look at the [README](https://github.com/openfisca/openfisca-core/blob/master/README.md).
If the errors also concern OpenFisca-Core, please take a look at the [README](https://github.com/openfisca/openfisca-core/blob/master/README.md).

2. If not, clone & install the documentation:
If not, clone & install the documentation:

```
```sh
git clone https://github.com/openfisca/openfisca-doc
make install
```

3. create a branch to correct the problems:
Create a branch to correct the problems:

```
```sh
git checkout -b fix-doc
```

4. Fix the offending problems.
Fix the offending problems. You can test-drive your fixes by checking that each change works as expected:

You can test-drive your fixes by checking that each change works as expected:

```
```sh
make test
```

5. Commit at each step, so you don't accidentally lose your progress:
Commit at each step, so you don't accidentally lose your progress:

```
```sh
git add -A && git commit -m "Fixed missing doctree"
```

6. Once you're done, push your changes:
Once you're done, push your changes:

```
```sh
git push origin `git branch --show-current`
```

7. Finally, open a [pull request](https://github.com/openfisca/openfisca-doc/compare/master...fix-doc).
Finally, open a [pull request](https://github.com/openfisca/openfisca-doc/compare/master...fix-doc).

That's it! 🙌
21 changes: 11 additions & 10 deletions source/coding-the-legislation/10_basic_example.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,27 +25,28 @@ Let's explain in details the different parts of the code:
### The variable attributes

All variables have a set of attributes.

* `value_type` defines the type of the formula output. Possible types are the basic Python types.
Note however that OpenFisca uses NumPy to [run calculations vectorially](25_vectorial_computing.md),
so the actual type of data may be slightly different from the built-in Python ones.
Available types are:
- `bool`: boolean
- `date`: date
- `Enum`: discrete value (from an enumerable). [See details](20_input_variables.md) in the next section.
- `float`: float (Note that to reduce memory usage, float are stored on 32 bits using NumPy's `float32`)
- `int`: integer
- `str`: string
- `bool`: boolean
- `date`: date
- `Enum`: discrete value (from an enumerable). [See details](20_input_variables.md) in the next section.
- `float`: float (Note that to reduce memory usage, float are stored on 32 bits using NumPy's `float32`)
- `int`: integer
- `str`: string
* `entity` defines who or what group the variable concerns, e.g. individuals, households, families.
* `definition_period` defines the period on which the variable is calculated. It can be `MONTH` (e.g. salary), `YEAR` (e.g. income taxes), or ETERNITY (e.g. date of birth).
* `label` is a human friendly way to describe the variable.
* `reference` is a list of relevant legislative reference for this variables (usually URLs the text of the law or another trustworthy source).

### The formula

- `def formula(person, period):` declares the formula that will be used to calculate the `flat_tax_on_salary` for a given `person` at a given `period`. Because `definition_period = MONTH`, `period` is constrained to be a month.
- `salary = person('salary', period)` calculates the salary of the person, for the given month. This will, of course, work only if `salary` is another variable in the tax and benefit system.
- `return salary * 0.25` returns the result for the given period.
- [Dated Formulas](40_legislation_evolutions.md) have a start and/or an end date.
* `def formula(person, period):` declares the formula that will be used to calculate the `flat_tax_on_salary` for a given `person` at a given `period`. Because `definition_period = MONTH`, `period` is constrained to be a month.
* `salary = person('salary', period)` calculates the salary of the person, for the given month. This will, of course, work only if `salary` is another variable in the tax and benefit system.
* `return salary * 0.25` returns the result for the given period.
* [Dated Formulas](40_legislation_evolutions.md) have a start and/or an end date.

## Testing a formula

Expand Down
19 changes: 10 additions & 9 deletions source/coding-the-legislation/20_input_variables.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,10 +12,10 @@ class salary(Variable):
definition_period = MONTH
```


The only difference is that we do **not** have a formula to calculate the value of a variable.

If we ask the value of `salary` for a given month, the returned result will be:

* The **input** that was provided when initializing the simulation if it exists.
* The **default value** of the Variable if no input has been provided.

Expand All @@ -34,17 +34,18 @@ class french_citizen(Variable):

If you do not explicitly define a default value, the following will be used:

- For numeric variables: `0`.
- For boolean variables: `False`.
* For numeric variables: `0`.
* For boolean variables: `False`.

## Advanced example: enumerations (enum)

### Usecases

Enumerations are variables that have a limited set of possible values. For instance:
- Highest academic level: high school, associate degree, bachelor's degree, master's degree, doctorate…
- A household housing occupancy status: owner, tenant, free-lodger, homeless…
- The main occupation of a person: employee, freelancer, retired, student, unemployed…

* Highest academic level: high school, associate degree, bachelor's degree, master's degree, doctorate…
* A household housing occupancy status: owner, tenant, free-lodger, homeless…
* The main occupation of a person: employee, freelancer, retired, student, unemployed…

### Defining and using an enumeration variable

Expand All @@ -63,8 +64,9 @@ class HousingOccupancyStatus(Enum):
```

> OpenFisca enums are based on Python 3 native enums. Each enum item (for instance `HousingOccupancyStatus.tenant`) has:
> - a `name` attribute that contains its key (e.g. `tenant`)
> - a `value` attribute that contains its description (e.g. `"Tenant or lodger who pays a monthly rent"`)
>
> * a `name` attribute that contains its key (e.g. `tenant`)
> * a `value` attribute that contains its description (e.g. `"Tenant or lodger who pays a monthly rent"`)

Then, create an OpenFisca variable `housing_occupancy_status`:

Expand All @@ -78,7 +80,6 @@ class housing_occupancy_status(Variable):
label = u"Legal housing situation of the household concerning their main residence"
```


You can now use the enum in variable formulas!

For instance, assuming the enumeration and the formula using it are defined in the same file:
Expand Down
11 changes: 3 additions & 8 deletions source/coding-the-legislation/25_vectorial_computing.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,6 @@ OpenFisca calculation are all **vectorial**. That means they operate on arrays r

The practical benefit is that computations are almost as expensive for one entity as they are for hundred thousands. This is how datasets can be analysed and how reforms can be modelled accurately. However, to support this feature, you will need to apply some constraints on how you write formulas.


## Formulas always return vectors

Each [formula](../key-concepts/variables.md#formulas) computation in OpenFisca must return a vector.
Expand All @@ -20,7 +19,7 @@ def formula(persons, period, parameters):

will print `array([41, 42, 45])`.

This formula code will work the same if there is one Person or three or three million in the modelled situation. Formulas always receive as their first parameter an array of the [entity](./50_entities.md) on which they operate (e.g. *n* Person, Household…) and they should return an array of the same length.
This formula code will work the same if there is one Person or three or three million in the modelled situation. Formulas always receive as their first parameter an array of the [entity](./50_entities.md) on which they operate (e.g. _n_ Person, Household…) and they should return an array of the same length.

Most of the time, formulas will refer to other variables and NumPy will do the appropriate computation without you even noticing:

Expand Down Expand Up @@ -53,12 +52,10 @@ In a similar fashion, if you expect a formula to return a boolean and forget tha

The rest of this page gives practical replacements for situations in which you get such errors.


## Control structures

Some usual control structures such as `if...else`, `switch`, and native Python logical operators such as `or` and `not` do not work with vectors. Semantically however, they all have alternatives, and the only change is in syntax.


### `if` / `else`

Let's say you want to write that logically reads as:
Expand Down Expand Up @@ -104,6 +101,7 @@ This `where` function is provided directly by NumPy. There are many other [NumPy
### Multiples conditions

Let's consider a more complex case, where we want to attribute to a person:

- `200` if their salary is less than `500`;
- `100` if their salary is strictly more than `500`, but less than `1000;`
- `50` if their salary is strictly more than `1000`, but less than `1500;`
Expand All @@ -130,7 +128,7 @@ If no [NumPy function](https://docs.scipy.org/doc/numpy/reference/routines.math.

For instance, let's consider that a person will be granted `200` if either:

- they are more than 25 *and* make less than `1000` per month;
- they are more than 25 _and_ make less than `1000` per month;
- or they are disabled.

```py
Expand All @@ -144,7 +142,6 @@ def formula(person, period):

> You should always use [NumPy function](https://docs.scipy.org/doc/numpy/reference/routines.math.html#sums-products-differences) such as [`where`](https://docs.scipy.org/doc/numpy/reference/generated/numpy.where.html) and [`select`](https://docs.scipy.org/doc/numpy/reference/generated/numpy.select.html) when they are relevant: logical operations using arithmetic operators should be used as last resort as they are not very readable.


## Arithmetic operations

Basic arithmetic operations such as `+` or `*` behave the same way on vectors than on numbers, you can thus use them in OpenFisca formulas. However, some operations must be adapted.
Expand All @@ -155,7 +152,6 @@ Basic arithmetic operations such as `+` or `*` behave the same way on vectors th
| `max` | `max_(x,y)` |
| `round` | `round_(x,y)` |


## Boolean operations

| Scalar (won't work) | Vectorial alternative |
Expand All @@ -164,7 +160,6 @@ Basic arithmetic operations such as `+` or `*` behave the same way on vectors th
| `and` | `x * y` |
| `or` | `x + y` |


## String concatenation

The `+` operator, as well as formatted `%s` strings for concatenation should be replaced by a call to `concat(x, y)`.
Loading