Merge pull request #129 from e10v/dev

Update docs and readme

.markdownlint.yaml

@@ -2,3 +2,4 @@ MD007:
   indent: 4
 
 MD013: false
+MD046: false

README.md

@@ -31,26 +31,24 @@ pip install tea-tasting
 
 ## Basic example
 
-```python
-import tea_tasting as tt
-
-
-data = tt.make_users_data(seed=42)
-
-experiment = tt.Experiment(
-    sessions_per_user=tt.Mean("sessions"),
-    orders_per_session=tt.RatioOfMeans("orders", "sessions"),
-    orders_per_user=tt.Mean("orders"),
-    revenue_per_user=tt.Mean("revenue"),
-)
-
-result = experiment.analyze(data)
-print(result)
-#>             metric control treatment rel_effect_size rel_effect_size_ci pvalue
-#>  sessions_per_user    2.00      1.98          -0.66%      [-3.7%, 2.5%]  0.674
-#> orders_per_session   0.266     0.289            8.8%      [-0.89%, 19%] 0.0762
-#>    orders_per_user   0.530     0.573            8.0%       [-2.0%, 19%]  0.118
-#>   revenue_per_user    5.24      5.73            9.3%       [-2.4%, 22%]  0.123
+```pycon
+>>> import tea_tasting as tt
+
+>>> data = tt.make_users_data(seed=42)
+>>> experiment = tt.Experiment(
+...     sessions_per_user=tt.Mean("sessions"),
+...     orders_per_session=tt.RatioOfMeans("orders", "sessions"),
+...     orders_per_user=tt.Mean("orders"),
+...     revenue_per_user=tt.Mean("revenue"),
+... )
+>>> result = experiment.analyze(data)
+>>> print(result)
+            metric control treatment rel_effect_size rel_effect_size_ci pvalue
+ sessions_per_user    2.00      1.98          -0.66%      [-3.7%, 2.5%]  0.674
+orders_per_session   0.266     0.289            8.8%      [-0.89%, 19%] 0.0762
+   orders_per_user   0.530     0.573            8.0%       [-2.0%, 19%]  0.118
+  revenue_per_user    5.24      5.73            9.3%       [-2.4%, 22%]  0.123
+
 ```
 
 Learn more in the detailed [user guide](https://tea-tasting.e10v.me/user-guide/). Additionally, see the guides on [data backends](https://tea-tasting.e10v.me/data-backends/), [power analysis](https://tea-tasting.e10v.me/power-analysis/), [multiple hypothesis testing](https://tea-tasting.e10v.me/multiple-testing/), and [custom metrics](https://tea-tasting.e10v.me/custom-metrics/).

docs/custom-metrics.md

@@ -15,7 +15,6 @@ First, let's import all the required modules and prepare the data:
 
 ```pycon
 >>> from typing import Literal, NamedTuple
-
 >>> import numpy as np
 >>> import pyarrow as pa
 >>> import pyarrow.compute as pc
@@ -26,7 +25,6 @@ First, let's import all the required modules and prepare the data:
 >>> import tea_tasting.metrics
 >>> import tea_tasting.utils
 
-
 >>> data = tt.make_users_data(seed=42)
 >>> data = data.append_column(
 ...     "has_order",
@@ -92,14 +90,12 @@ Let's define the metric and discuss each method in details:
 ...         self.correction = tea_tasting.utils.auto_check(correction, "correction")
 ...         self.method = tea_tasting.utils.check_scalar(
 ...             method, "method", typ=str, in_={"g-test", "pearson"})
-... 
 ...     @property
 ...     def aggr_cols(self) -> tea_tasting.metrics.AggrCols:
 ...         return tea_tasting.metrics.AggrCols(
 ...             has_count=True,
 ...             mean_cols=(self.column,),
 ...         )
-... 
 ...     def analyze_aggregates(
 ...         self,
 ...         control: tea_tasting.aggr.Aggregates,
@@ -185,11 +181,9 @@ Metric should have the following methods and properties defined:
 ...             if alternative is not None
 ...             else tea_tasting.config.get_config("alternative")
 ...         )
-... 
 ...     @property
 ...     def cols(self) -> tuple[str]:
 ...         return (self.column,)
-... 
 ...     def analyze_granular(
 ...         self,
 ...         control: pa.Table,

docs/data-backends.md

@@ -17,11 +17,13 @@ This guide:
 
 ## Demo database
 
-This guide uses [DuckDB](https://github.com/duckdb/duckdb), an in-process analytical database, and [Polars](https://github.com/pola-rs/polars) as example data backends. To be able to reproduce the example code, install **tea-tasting**, Ibis with DuckDB extra, and Polars:
+???+ note
 
-```bash
-pip install tea-tasting ibis-framework[duckdb] polars
-```
+    This guide uses [DuckDB](https://github.com/duckdb/duckdb), an in-process analytical database, and [Polars](https://github.com/pola-rs/polars) as example data backends. To be able to reproduce the example code, install Ibis with DuckDB extra and Polars in addition to **tea-tasting**:
+
+    ```bash
+    pip install ibis-framework[duckdb] polars
+    ```
 
 First, let's prepare a demo database:
 
@@ -30,7 +32,6 @@ First, let's prepare a demo database:
 >>> import polars as pl
 >>> import tea_tasting as tt
 
-
 >>> users_data = tt.make_users_data(seed=42)
 >>> con = ibis.duckdb.connect()
 >>> con.create_table("users_data", users_data)

docs/index.md

@@ -34,16 +34,13 @@ pip install tea-tasting
 ```pycon
 >>> import tea_tasting as tt
 
-
 >>> data = tt.make_users_data(seed=42)
-
 >>> experiment = tt.Experiment(
 ...     sessions_per_user=tt.Mean("sessions"),
 ...     orders_per_session=tt.RatioOfMeans("orders", "sessions"),
 ...     orders_per_user=tt.Mean("orders"),
 ...     revenue_per_user=tt.Mean("revenue"),
 ... )
-
 >>> result = experiment.analyze(data)
 >>> print(result)
             metric control treatment rel_effect_size rel_effect_size_ci pvalue

docs/multiple-testing.md

@@ -2,6 +2,14 @@
 
 ## Multiple hypothesis testing problem
 
+???+ note
+
+    This guide uses [Polars](https://github.com/pola-rs/polars) as an example data backend. To be able to reproduce the example code, install Polars in addition to **tea-tasting**:
+
+    ```bash
+    pip install polars
+    ```
+
 The [multiple hypothesis testing problem](https://en.wikipedia.org/wiki/Multiple_comparisons_problem) arises when there is more than one success metric or more than one treatment variant in an A/B test.
 
 **tea-tasting** provides the following methods for multiple testing correction:
@@ -19,7 +27,6 @@ As an example, consider an experiment with three variants, a control and two tre
 >>> import polars as pl
 >>> import tea_tasting as tt
 
-
 >>> data = pl.concat((
 ...     tt.make_users_data(
 ...         seed=42,
@@ -67,7 +74,6 @@ Let's calculate the experiment results:
 ...     orders_per_user=tt.Mean("orders"),
 ...     revenue_per_user=tt.Mean("revenue"),
 ... )
-
 >>> results = experiment.analyze(data, control=0, all_variants=True)
 >>> print(results)
 variants             metric control treatment rel_effect_size rel_effect_size_ci  pvalue
@@ -178,10 +184,8 @@ In the examples above, the methods `adjust_fdr` and `adjust_fwer` received resul
 ```pycon
 >>> data1 = tt.make_users_data(seed=42, orders_uplift=0.10, revenue_uplift=0.15)
 >>> data2 = tt.make_users_data(seed=21, orders_uplift=0.15, revenue_uplift=0.20)
-
 >>> result1 = experiment.analyze(data1)
 >>> result2 = experiment.analyze(data2)
-
 >>> print(tt.adjust_fdr(
 ...     {"Experiment 1": result1, "Experiment 2": result2},
 ...     metrics,

docs/power-analysis.md

@@ -11,15 +11,13 @@ In this example, **tea-tasting** calculates statistical power given the relative
 ```pycon
 >>> import tea_tasting as tt
 
-
 >>> data = tt.make_users_data(
 ...     seed=42,
 ...     sessions_uplift=0,
 ...     orders_uplift=0,
 ...     revenue_uplift=0,
 ...     covariates=True,
 ... )
-
 >>> orders_per_session = tt.RatioOfMeans("orders", "sessions", rel_effect_size=0.1)
 >>> print(orders_per_session.solve_power(data, "power"))
 power effect_size rel_effect_size n_obs
@@ -63,7 +61,6 @@ You can analyze power for all metrics in the experiment. Example:
 ...         orders_per_user=tt.Mean("orders", "orders_covariate"),
 ...         revenue_per_user=tt.Mean("revenue", "revenue_covariate"),
 ...     )
-
 >>> power_result = experiment.solve_power(data)
 >>> print(power_result)
             metric power effect_size rel_effect_size n_obs

docs/user-guide.md

@@ -15,16 +15,13 @@ Begin with this simple example to understand the basic functionality:
 ```pycon
 >>> import tea_tasting as tt
 
-
 >>> data = tt.make_users_data(seed=42)
-
 >>> experiment = tt.Experiment(
 ...     sessions_per_user=tt.Mean("sessions"),
 ...     orders_per_session=tt.RatioOfMeans("orders", "sessions"),
 ...     orders_per_user=tt.Mean("orders"),
 ...     revenue_per_user=tt.Mean("revenue"),
 ... )
-
 >>> result = experiment.analyze(data)
 >>> print(result)
             metric control treatment rel_effect_size rel_effect_size_ci pvalue
@@ -230,9 +227,7 @@ Example usage:
 ```pycon
 >>> import tea_tasting as tt
 
-
 >>> data = tt.make_users_data(seed=42, covariates=True)
-
 >>> experiment = tt.Experiment(
 ...     sessions_per_user=tt.Mean("sessions", "sessions_covariate"),
 ...     orders_per_session=tt.RatioOfMeans(
@@ -244,7 +239,6 @@ Example usage:
 ...     orders_per_user=tt.Mean("orders", "orders_covariate"),
 ...     revenue_per_user=tt.Mean("revenue", "revenue_covariate"),
 ... )
-
 >>> result = experiment.analyze(data)
 >>> print(result)
             metric control treatment rel_effect_size rel_effect_size_ci  pvalue
@@ -275,13 +269,11 @@ Example usage:
 ```pycon
 >>> import tea_tasting as tt
 
-
 >>> experiment = tt.Experiment(
 ...     orders_per_user=tt.Mean("orders"),
 ...     revenue_per_user=tt.Mean("revenue"),
 ...     sample_ratio=tt.SampleRatio(),
 ... )
-
 >>> data = tt.make_users_data(seed=42)
 >>> result = experiment.analyze(data)
 >>> print(result)
@@ -326,7 +318,6 @@ Use [`get_config`](api/config.md#tea_tasting.config.get_config) with the option
 ```pycon
 >>> import tea_tasting as tt
 
-
 >>> print(tt.get_config("equal_var"))
 False
 
@@ -343,14 +334,13 @@ Use [`set_config`](api/config.md#tea_tasting.config.set_config) to set a global
 
 ```pycon
 >>> tt.set_config(equal_var=True, use_t=False)
-
 >>> experiment = tt.Experiment(
 ...     sessions_per_user=tt.Mean("sessions"),
 ...     orders_per_session=tt.RatioOfMeans("orders", "sessions"),
 ...     orders_per_user=tt.Mean("orders"),
 ...     revenue_per_user=tt.Mean("revenue"),
 ... )
-
+>>> tt.set_config(equal_var=False, use_t=True)
 >>> print(experiment.metrics["orders_per_user"])
 Mean(value='orders', covariate=None, alternative='two-sided', confidence_level=0.95, equal_var=True, use_t=False, alpha=0.05, ratio=1, power=0.8, effect_size=None, rel_effect_size=None, n_obs=None)
 
@@ -359,16 +349,13 @@ Mean(value='orders', covariate=None, alternative='two-sided', confidence_level=0
 Use [`config_context`](api/config.md#tea_tasting.config.config_context) to temporarily set a global option value within a context:
 
 ```pycon
->>> tt.set_config(equal_var=False, use_t=True)
-
 >>> with tt.config_context(equal_var=True, use_t=False):
 ...     experiment = tt.Experiment(
 ...         sessions_per_user=tt.Mean("sessions"),
 ...         orders_per_session=tt.RatioOfMeans("orders", "sessions"),
 ...         orders_per_user=tt.Mean("orders"),
 ...         revenue_per_user=tt.Mean("revenue"),
 ...     )
-
 >>> print(tt.get_config("equal_var"))
 False
 
@@ -382,6 +369,14 @@ Mean(value='orders', covariate=None, alternative='two-sided', confidence_level=0
 
 ### More than two variants
 
+???+ note
+
+    This guide uses [Polars](https://github.com/pola-rs/polars) as an example data backend. To be able to reproduce the example code, install Polars in addition to **tea-tasting**:
+
+    ```bash
+    pip install polars
+    ```
+
 In **tea-tasting**, it's possible to analyze experiments with more than two variants. However, the variants will be compared in pairs through two-sample statistical tests.
 
 Example usage:
@@ -390,21 +385,18 @@ Example usage:
 >>> import polars as pl
 >>> import tea_tasting as tt
 
-
 >>> data = pl.concat((
 ...     tt.make_users_data(seed=42, return_type="polars"),
 ...     tt.make_users_data(seed=21, return_type="polars")
 ...         .filter(pl.col("variant").eq(1))
 ...         .with_columns(variant=pl.lit(2, pl.Int64)),
 ... ))
-
 >>> experiment = tt.Experiment(
 ...     sessions_per_user=tt.Mean("sessions"),
 ...     orders_per_session=tt.RatioOfMeans("orders", "sessions"),
 ...     orders_per_user=tt.Mean("orders"),
 ...     revenue_per_user=tt.Mean("revenue"),
 ... )
-
 >>> results = experiment.analyze(data, control=0, all_variants=True)
 >>> print(results)
 variants             metric control treatment rel_effect_size rel_effect_size_ci pvalue

mkdocs.yml

@@ -79,6 +79,8 @@ plugins:
 
 markdown_extensions:
   - _strip_doctest_artifacts
+  - admonition
+  - pymdownx.details
   - pymdownx.superfences
   - toc:
       permalink: "#"

src/tea_tasting/aggr.py

@@ -56,8 +56,9 @@ def with_zero_div(self) -> Aggregates:
         """Return aggregates that do not raise an error on division by zero.
 
         Division by zero returns:
-            - `inf` if numerator is greater than `0`,
-            - `nan` if numerator is equal to or less than `0`.
+
+        - `inf` if numerator is greater than `0`,
+        - `nan` if numerator is equal to or less than `0`.
         """
         return Aggregates(
             count_=None if self.count_ is None else tea_tasting.utils.Int(self.count_),
@@ -69,9 +70,6 @@ def with_zero_div(self) -> Aggregates:
     def count(self) -> int:
         """Sample size (number of observations).
 
-        Raises:
-            RuntimeError: Count is `None` (if it was not defined during initialization).
-
         Returns:
             Sample size (number of observations).
         """