v0.11.0

Added

Rust SDK xaynet-sdk

xaynet-sdk contains the basic building blocks required to run the Privacy-Enhancing Technology (PET) Protocol. It consists of a state machine and two I/O interfaces with which specific Xaynet participants can be developed that are adapted to the respective environments/requirements.

If you are interested in building your own Xaynet participant, you can take a look at xaynet-sdk, our Rust participant which we use primarily for testing or at xaynet-mobile our mobile friendly participant.

A Mobile friendly Xaynet participant xaynet-mobile

xaynet-mobile provides a mobile friendly implementation of a Xaynet participant. It gives the user a lot of control on how to drive the participant execution. You can regularly pause the execution of the participant, save it, and later restore it and continue the execution. When running on a device that is low on battery or does not have access to Wi-Fi for instance, it can be useful to be able to pause the participant.

C API

Furthermore, xaynet-mobile offers C bindings that allow xaynet-mobile to be used in other programming languages ​​such as Dart.

Python participant SDK xaynet-sdk-python

We are happy to announce that we finally released xaynet-sdk-python a Python SDK that consists of two experimental Xaynet participants (ParticipantABC and AsyncParticipant).

The ParticipantABC API is similar to the old one which we introduced in v0.8.0. Aside from some changes to the method signature, the biggest change is that the participant now runs in its own thread. To migrate from v0.8.0 to v0.11.0 please follow the migration guide.

However, we noticed that our Participant API may be difficult to integrate with existing applications, considering the code for the training has to be moved into the train_round method, which can lead to significant changes to the existing code. Therefore, we offer a second API (AsyncParticipant) in which the training of the model is no longer part of the participant.

A more in-depth explanation of the differences between the Participant APIs and examples of how to use them can be found here.

Multi-part messages

Participant messages can get large, possibly too large to be sent successfully in one go. On mobile devices in particular, the internet connection may not be as reliable. In order to make the transmission of messages more robust, we implemented multi-part messages to break a large message into parts and send them sequentially to the coordinator. If the transmission of part of a message fails, only that part will be resent and not the entire message.

Coordinator state managed in Redis

In order to be able to restore the state of the coordinator after a failure or shutdown, the state is managed in Redis and no longer in memory.

The Redis client can be configured via the [redis] setting:

[redis]
url = "redis://127.0.0.1/"

Support for storing global models in S3/Minio

The coordinator is able to save a global model in S3/Minio after a successful round.

The S3 client can be configured via the [s3] setting:

[s3]
access_key = "minio"
secret_access_key = "minio123"
region = ["minio", "http://localhost:9000"]

[s3.buckets]
global_models = "global-models"

xaynet-server must be compiled with the feature flag model-persistence in order to enable this feature.

Restore coordinator state

The state of the coordinator can be restored after a failure or shutdown. You can find more information about the restore behavior here.

Restoring the coordinator be configured via the [restore] setting:

[restore]
enable = true

xaynet-server must be compiled with the feature flag model-persistence in order to enable this feature.

Improved collection of state machine metrics

In v0.10.0 we introduced the collection of metrics that are emitted in the state machine of xaynet-server and sent to an InfluxDB instance. In v0.11.0 we have revised the implementation and improved it further. Metrics are now sent much faster and adding metrics to the code has become much easier.

Removed

• xaynet_client (was split into xaynet_sdk and xaynet_mobile)
• xaynet_ffi (is now part of xaynet_mobile)
• xaynet_macro

What's next?

Roadmap 2021

v0.10.0

Added

• Preparation for redis support: prepare for xaynet_server to store PET data in redis #416, #515
• Add support for multipart messages in the message structure #508, #513, #514
• Generalised scalar extension #496, #507
• Add server metrics #487, #488, #489, #493
• Refactor the client into a state machine, and add a client tailored for mobile devices #471, #497, #506

Changed

• Split the xaynet crate into several sub-crates:
  • xaynet_core (0.1.0 released), re-exported as xaynet::core
  • xaynet_client (0.1.0 released), re-exported as xaynet::client when compiled with --features client
  • xaynet_server (0.1.0 released), re-exported as xaynet::server when compiled with --features server
  • xaynet_macro (0.1.0 released)
  • xaynet_ffi (not released)

v0.9.0

xain/xain-fl repository was renamed to xaynetwork/xaynet.

The new crate is now published as xaynet: https://crates.io/crates/xaynet

Added

This release introduces the integration of the PET protocol into the platform.

Note: The integration of the PET protocol required a complete rewrite of the codebase and is therefore not compatible with the previous release.

v0.8.0

Added

• New tutorial for the Python SDK (#355)
• Swagger description of the REST API (#345), and is published at https://xain-fl.readthedocs.io/en/latest/ (#358)
• The Python examples now accepts additional parameters (model size, heartbeat period, verbosity, etc.) (#351)
• Publish docker images to dockerhub

Security

• Stop using pickle for messages serialization
  (#355). pickle is insecure
  and can lead to remote code execution. Instead, the default
  aggregator uses numpy.save().

Fixed

• The documentation has been updated at https://xain-fl.readthedocs.io/en/latest/ (#358)
• Document aggregator error on Darwin platform (https://github.com/xainag/xain-fl/pull/365/files)

Changed

• Simplified the Python SDK API (#355)
• Added unit tests for the coordinator and aggregator (#353), (#352)
• Refactor the metrics store (#340)
• Speed up the docker builds (#348)

v0.7.0

On this release we archived the Python code under the legacy folder and shifted the development to Rust.
This release has many breaking changes from the previous versions.
More details will be made available through the updated README.md of the repository.

• PB-584 Update the manifest Cargo.toml with description, license, keywords, repository URL and project homepage (#343) [Ricardo Saffi Marques]
• PB-584 Add LICENSE file at the root of the repository prior to release v0.7.0 (#342) [Ricardo Saffi Marques]
• Update Cargo.lock for release. [Ricardo Saffi Marques]
• Update versions and authors (#335) [Corentin Henry]
• Remove the rustfmt config file (#341) [Corentin Henry]
• Remove nix files (#337) [Corentin Henry]
• Remove caddy (#338) [Robert Steiner]
• Nix-shell: install rust-analyzer (#336) [Corentin Henry]
• Re-use the CHANGELOG file from the legacy codebase. [little-dude]
• Rewrite xain-fl in Rust. [little-dude]
• Merge pull request #64 from xainag/optional-metric-store. [Corentin Henry]
• In docker, compile with influx_metrics [little-dude]
• Silence "unused variable" warning from rustc. [little-dude]
• Add CI jobs for the influx_metrics feature. [little-dude]
• Introduce the influx_metrics feature. [little-dude]
• Disable metrics when running locally. [little-dude]
• Make the metric store optional. [little-dude]
• Move the metric store to xain_fl::common. [little-dude]
• Merge pull request #68 from xainag/ci-build-all-features. [Corentin Henry]
• Fix build for the telemetry feature. [little-dude]
• Ci: add matrix to tests various --features flags. [little-dude]
• Merge pull request #67 from xainag/example-fixes. [Corentin Henry]
• Sdk: participant should exit when a heartbeat fails. [little-dude]
• Sdk: crash instead of calling sys.exit when participant errors out. [little-dude]
• Fix exit condition in dummy example. [little-dude]
• Merge pull request #65 from xainag/less-verbose-logging. [Corentin Henry]
• Make logging more finely configurable. [little-dude]
• Merge pull request #66 from xainag/fix-crash. [Corentin Henry]
• Fix crash after training finishes. [little-dude]
• Merge pull request #63 from xainag/misc. [Corentin Henry]
• Log an error when a heartbeat is rejected. [little-dude]
• Use more reasonable values in docker-release-aggregator.toml. [little-dude]
• Sdk: make the heartbeat frequency configurable. [little-dude]
• Simplify match statement. [little-dude]
• Nix: add some cargo tools. [little-dude]
• Merge pull request #61 from xainag/refactor. [Corentin Henry]
• Refactor: simplify RPC code by using the ServiceHandle. [little-dude]
• Merge pull request #52 from xainag/PB-490-protocol-tests. [Corentin Henry]
• Fix protocol test: by default expect two rounds. [little-dude]
• Document the protocol tests. [little-dude]
• Remove ignore. [Robert Steiner]
• Remove unused function. [Robert Steiner]
• Fix typo. [Robert Steiner]
• Clean up. [Robert Steiner]
• Add full training test case. [Robert Steiner]
• Clean up tests. [Robert Steiner]
• Add end_training and end_aggregation tests. [Robert Steiner]
• Add endtraining test. [Robert Steiner]
• Remove comments. [Robert Steiner]
• Add start training tests. [Robert Steiner]
• Add heartbeat test. [Robert Steiner]
• Fix tests. [Robert Steiner]
• Add heartbeat tests. [Robert Steiner]
• PB-490 Add protocol tests. [Robert Steiner]
• Merge pull request #59 from xainag/opentelemetry. [Corentin Henry]
• Integrate with opentelemetry and jaeger. [little-dude]
• Get rid of log and env_logger dependencies. [little-dude]
• Small refactoring. [little-dude]
• Configure a custom Subscriber for the aggregator. [little-dude]
• Split AggregatorService::poll_rpc_requests() [little-dude]
• Implement Display for RPC requests. [little-dude]
• Switch to tracing for logging. [little-dude]
• Update README. [little-dude]
• Merge pull request #56 from xainag/PB-491-test-python-ffi. [Corentin Henry]
• Add reset, get_global_weights and aggregate tests. [Robert Steiner]
• Add add_weights and aggregate test. [Robert Steiner]
• Add python setup in rust-test. [Robert Steiner]
• Add more tests. [Robert Steiner]
• Add load test. [Robert Steiner]
• Update README. [little-dude]
• Add instructions to run the examples. [little-dude]
• Remove more dependencies. [little-dude]
• Remove unused dependency. [little-dude]
• Bump dependencies. [little-dude]
• Merge pull request #54 from xainag/simplify_rpc_client. [Corentin Henry]
• Simplify the RPC implementation. [little-dude]
• Install docker-compose. [little-dude]
• Merge pull request #53 from xainag/error-handling. [Corentin Henry]
• Aggregator: properly handle CTRL+C. [little-dude]
• Aggregator: when a task finishes cancel the other ones. [little-dude]
• Rename channels to reflect whether they are senders or receivers. [little-dude]
• Improve error handling in the py_aggregator module. [little-dude]
• Add anyhow and thiserror dependencies for error handling. [little-dude]
• Merge pull request #49 from xainag/fix-clippy-warnings. [Corentin Henry]
• Update ci. [Robert Debug]
• Fix clippy warnings. [Robert Debug]
• Deny warnings when compiling rust code. [little-dude]
• Do not run CI for PRs. [little-dude]
• Configure python CI (#50) [Robert Steiner]
• Remove dummy aggregator. [little-dude]
• Cleanup the keras_house_prices example. [little-dude]
• Add vscode workspace settings to gitignore. [Robert Debug]
• Clean up write metrics. [Robert Steiner]
• Merge pull request #37 from xainag/benchmarks. [Corentin Henry]
• Delete dummy tensorflow example. [little-dude]
• Cleanup the keras benchmark. [little-dude]
• Optimize dummy example. [little-dude]
• Fix data_handlers. [little-dude]
• Beef up the .dockerignore to speed up docker. [little-dude]
• Port keras example from the benchmarks repo. [little-dude]
• Remove junk files. [little-dude]
• Beef up the .dockerignore to speed up docker. [little-dude]
• Merge pull request #44 from xainag/add_metric_queue. [Corentin Henry]
• Remove tests. [Robert Steiner]
• Fix config. [Robert Steiner]
• Add metric queue. [Robert Steiner]
• Merge pull request #41 from xainag/valgrind. [Robert Steiner]
• Remove valgrind in dev. [Robert Steiner]
• Clean up. [Robert Steiner]
• Fix rebase. [Robert Steiner]
• Add valgrind. [Robert Debug]
• Merge pull request #43 from xainag/memory-leak. [Corentin Henry]
• Fix python memory leak. [little-dude]
• Fix warning about unused import. [little-dude]
• Configure logging for the weighted average python aggregator. [little-dude]
• In the tf.py example just pretend to train. [little-dude]
• Make participant logging less noisy. [little-dude]
• Merge pull request #38 from xainag/add_rendez_vous_tests. [Robert Steiner]
• Fmt. [Robert Steiner]
• Clean up tests. [Robert Steiner]
• Add rendez_vous tests. [Robert Steiner]
• Rustfmt. [little-dude]
• Fix CI. [little-dude]
• Set working directory in CI. [little-dude]
• Fix typo in github workflow file. [little-dude]
• Add CI for rust. [little-dude]
• Document how to run release builds. [little-dude]
• Update images. [little-dude]
• Add top level .gitignore. [little-dude]
• Move the rust code into its own directory. [little-dude]
• Add instructions for profiling. [little-dude]
• TEST CONFIG. [little-dude]
• Add instructions to run the tf example. [little-dude]
• Sdk: handle start training rejections. [little-dude]
• Fix shebang for nixos. [little-dude]
• Tweak the log levels. [little-dude]
• Use different tags for debug/release docker images. [little-dude]
• Require more participants for release builds. [little-dude]
• Docker: add release build and move configs out of docker dir. [little-dude]
• Fix metric store tests. [Robert Steiner]
• Merge pull request #32 from xainag/add-more-metrics. [Robert Steiner]
• Delete old dashboard. [Robert Steiner]
• Replace String with &'static str. [Robert Steiner]
• Add more metrics. [Robert Steiner]
• Add run participant script. [Robert Steiner]
• Use real aggregator in docker. [little-dude]
• Log why a start training request is rejected. [little-dude]
• Train on smaller models (1M) [little-dude]
• Fix py_aggregator memory leak. [little-dude]
• Fix Dockerfile. [little-dude]
• More profiling tools. [little-dude]
• Nix: add gperftools dependency to debug aggregator memory leak. [little-dude]
• Merge pull request #33 from xainag/aggregators. [Corentin Henry]
• Add tensorflow example. [little-dude]
• Make the sdk more generic. [little-dude]
• Formatting and pylint fixes. [little-dude]
• Add real aggregators. [little-dude]
• Merge pull request #34 from xainag/docker-stats. [Robert Steiner]
• Update caddyfile. [Robert Steiner]
• Remove alertmanager. [Robert Steiner]
• Remove alert manager. [Robert Steiner]
• Add docker stats. [Robert Debug]
• Repo cleanup. [little-dude]
• Make the python aggregator configurable. [little-dude]
• Fix aggregator port. [Robert Steiner]
• Add numpy. [Robert Steiner]
• Merge pull request #30 from xainag/collect_metrics. [Robert Steiner]
• Remove clone impl. [Robert Steiner]
• Collect Metrics. [Robert Steiner]
• Fix ports in config files. [little-dude]
• Sdk: various fixes. [little-dude]
• Some more logging. [little-dude]
• Wait for end aggregation message from the aggregator to enter finish state. [little-dude]
• Don't panic upon un-expected end aggregation message from the aggregator. [little-dude]
• Handle protocol events after polling the aggregation future. [little-dude]
• Move DummyParticipant out of sdk package. [little-dude]
• Working full round. [little-dude]
• Working upload. [little-dude]
• Python run black. [little-dude]
• Some renaming. [little-dude]
• Nix: fix build. [little-dude]
• Basic upload handling. [little-dude]
• Delete outdated examples. [little-dude]
• Implement downloading weights. [little-dude]
• Merge pull request #16 from xainag/metrics_store. [Robert Steiner]
• Applied ...

v0.6.0

• HOTFIX add disclaimer (#309) [janpetschexain]
• PB-314: document the new weight exchange mechanism (#308) [Corentin Henry]
• PB-407 add more debug level logging (#303) [janpetschexain]
• PB-44 add heartbeat time and timeout to config (#305) [Robert Steiner]
• PB-423 lock round access (#304) [kwok]
• PB-439 Make thread pool workers configurable (#302) [Robert Steiner]
• PB-159: update xain-{proto,sdk} dependencies to the right branch (#301) [Corentin Henry]
• PB-159: remove weights from gRPC messages (#298) [Corentin Henry]
• PB-431 send participant state to influxdb (#300) [Robert Steiner]
• PB-434 separate metrics (#296) [Robert Steiner]
• PB-406 ❄️ Configure mypy (#297) [Anastasiia Tymoshchuk]
• PB-428 send coordinator states (#292) [Robert Steiner]
• PB-425 split weight init from training (#295) [janpetschexain]
• PB-398 Round resumption in Coordinator (#285) [kwok]
• Merge pull request #294 from xainag/master. [Daniel Kravetz]
• Hotfix: PB-432 📝 📚 Update test badge and CI to reflect changes. [Daniel Kravetz]
• PB-417 Start new development cycle (#291) [Anastasiia Tymoshchuk, kwok]

v0.5.0

Fix minor issues, update documentation.

• PB-402 Add more logs (#281) [Robert Steiner]
• DO-76 🐳 non alpine image (#287) [Daniel Kravetz]
• PB-401 Add console renderer (#280) [Robert Steiner]
• DO-80 🚑 Update dev Dockerfile to build gRPC (#286) [Daniel Kravetz]
• DO-78 ✨ add grafana (#284) [Daniel Kravetz]
• DO-66 ✨ Add keycloak (#283) [Daniel Kravetz]
• PB-400 increment epoch base (#282) [janpetschexain]
• PB-397 Simplify write metrics function (#279) [Robert Steiner]
• PB-385 Fix xain-sdk test (#278) [Robert Steiner]
• PB-352 Add sdk config (#272) [Robert Steiner]
• Merge pull request #277 from xainag/master. [Daniel Kravetz]
• Hotfix: update ci. [Daniel Kravetz]
• DO-72 🎨 Make CI name and feature consistent with other repos. [Daniel Kravetz]
• DO-47 📰 Build test package on release branch. [Daniel Kravetz]
• PB-269: enable reading participants weights from S3 (#254) [Corentin Henry]
• PB-363 Start new development cycle (#271) [Anastasiia Tymoshchuk]
• PB-119 enable isort diff (#262) [janpetschexain]
• PB-363 💎 Release v0.4.0. [Daniel Kravetz]
• DO-73 💚 Disable continue_on_failure for CI jobs. Fix mypy. [Daniel Kravetz]

v0.4.0

Flatten model weights instead of using lists.
Fix minor issues, update documentation.

• PB-116: pin docutils version (#259) [Corentin Henry]
• PB-119 update isort config and calls (#260) [janpetschexain]
• PB-351 Store participant metrics (#244) [Robert Steiner]
• Adjust isort config (#258) [Robert Steiner]
• PB-366 flatten weights (#253) [janpetschexain]
• PB-379 Update black setup (#255) [Anastasiia Tymoshchuk]
• PB-387 simplify serve module (#251) [Corentin Henry]
• PB-104: make the tests fast again (#252) [Corentin Henry]
• PB-122: handle sigint properly (#250) [Corentin Henry]
• PB-383 write aggregated weights after each round (#246) [Corentin Henry]
• PB-104: Fix exception in monitor_hearbeats() (#248) [Corentin Henry]
• DO-57 Update docker-compose files for provisioning InfluxDB (#249) [Ricardo Saffi Marques]
• DO-59 Provision Redis 5.x for persisting states for the Coordinator (#247) [Ricardo Saffi Marques]
• PB-381: make the log level configurable (#243) [Corentin Henry]
• PB-382: cleanup storage (#245) [Corentin Henry]
• PB-380: split get_logger() (#242) [Corentin Henry]
• XP-332: grpc resource exhausted (#238) [Robert Steiner]
• XP-456: fix coordinator command (#241) [Corentin Henry]
• XP-485 Document revised state machine (#240) [kwok]
• XP-456: replace CLI argument with a config file (#221) [Corentin Henry]
• DO-48 ❄️ 🚀 Build stable package on git tag with SemVer (#234) [Daniel Kravetz]
• XP-407 update documentation (#239) [janpetschexain]
• XP-406 remove numpy file cli (#237) [janpetschexain]
• XP-544 fix aggregate module (#235) [janpetschexain]
• DO-58: cache xain-fl dependencies in Docker (#232) [Corentin Henry]
• XP-479 Start training rounds from 0 (#226) [kwok]

v0.3.0

• XP-337 Clean up docs before generation (#188)

• XP-229 Update Readme.md (#189)

• XP-255 update codeowners and authors in setup (#195)

• XP-265 move benchmarks to separate repo (#193)

• XP-243 remove flags from xain-fl codebase

• XP-255 update codeowners and authors in setup (#195)

• XP-257 cleanup cproto dir (#198)

• XP-261 move tests to own dir (#197)

• XP-168 update setup.py (#191)

• DO-35 📰 ✨ Add placeholder for e2e. Update CI for gitflow

• XP-241 remove legacy participant and sdk dir (#199)

• DO-17 🐳 ✨ Add Dockerfiles, dockerignore and docs (#202)

• XP-354 Remove proto files (#200)

• XP-385 Fix docs badge (#204)

• XP-273 scripts cleanup (#206)

• XP-357 make controller parametrisable (#201)

• XP-384 remove unused files (#209)

• DO-43 docker compose minio (#208)

• XP-374 Clean up docs (#211)

• XP-271 fix pylint issues (#210)

• XP-424 Remove unused packages (#212)

• DO-49 🐳 ✨ Create initial buckets (#213)

• XP-373 add sdk as dependency in fl (#214)

• XP-433 Fix docker headings (#218)

• XP-119 Fix gRPC testing setup so that it can run on macOS (#217)

• XP-422 ai metrics (#216)

• XP-308: store aggregated weights in S3 buckets (#215)

• XP-436 Reinstate FINISHED heartbeat from Coordinator (#219)

• XP-208 model sum aggregator as a mock for testing

• XP-208 add test integrating P and C

• XP-208 add identity controller as a mock for test

• XP-208 small typo in id-controller docstring

• XP-208 add coordinator test fixture with mocked components

• XP-208 test for top-level function start_participant

• XP-308: print debugging information for spurious failure

• XP-208 join monitor_thread to confirm response to terminate_event

• XP-208 change mock patch string to object called rather than defined

• XP-208 model sum aggregator as a mock for testing

• XP-208 add test integrating P and C

• XP-208 add identity controller as a mock for test

• XP-208 small typo in id-controller docstring

• XP-208 add coordinator test fixture with mocked components

• XP-208 test for top-level function start_participant

• XP-436 fix bug: coordinator not advertising FINISHED state

• XP-436 revert logging level

• XP-208 join monitor_thread to confirm response to terminate_event

• XP-208 change mock patch string to object called rather than defined

• XP-436 suggested documentation fixes from review

• XP-208 mock the store (following rebase)

• XP-208 fix import

• XP-208 test_start_participant: mark as slow, comment on aggregation

• XP-480 revise message names (#222)

• XP-499 Remove conftest, exclude tests folder (#223)

• XP-333 Replace numproto with xain-proto (#220)

• XP-505: docstrings cleanup (#224)

• XP-508 Replace circleci badge (#225)

• XP-510 allow for zero epochs on cli (#227)

• XP-498: more generic shebangs (#229)

• XP-505: cleanup docstrings in xain_fl.coordinator (#228)

• Prepare release of v0.3.0 (matching version with xain-sdk and xain-proto) (#230)

Co-authored-by: Anastasiia Tymoshchuk atymoshchuk@icloud.com
Co-authored-by: Daniel Kravetz daniel.kravetz@gmail.com
Co-authored-by: Felix Reichel 31513604+PanicButtonPressed@users.noreply.github.com
Co-authored-by: Anselmo Sampietro ans.samp@gmail.com
Co-authored-by: Robert Steiner robertt.debug@gmail.com
Co-authored-by: Corentin Henry little-dude@users.noreply.github.com
Co-authored-by: janpetschexain 58227040+janpetschexain@users.noreply.github.com
Co-authored-by: kwok kwok.doc@gmail.com

v0.2.0

[0.2.0] - 2019-12-02

Changed

• Renamed package from xain to xain-fl