From 3505872ac6b111242d051eb194e90fdba07e6395 Mon Sep 17 00:00:00 2001 From: Michael Rebello Date: Tue, 2 Jul 2019 15:00:10 -0700 Subject: [PATCH] docs: add ios docs on CPU/battery (#217) - Adds iOS documentation on the CPU/battery usage based on the investigation done in https://github.com/lyft/envoy-mobile/issues/113 - Combines the existing documentation for this into a single file for CPU/battery Resolves https://github.com/lyft/envoy-mobile/issues/113. Signed-off-by: Michael Rebello Signed-off-by: JP Simard --- .../performance/battery_impact.rst | 90 ---------- .../performance/cpu_battery_impact.rst | 155 ++++++++++++++++++ .../development/performance/cpu_impact.rst | 80 --------- .../development/performance/performance.rst | 3 +- 4 files changed, 156 insertions(+), 172 deletions(-) delete mode 100644 mobile/docs/root/development/performance/battery_impact.rst create mode 100644 mobile/docs/root/development/performance/cpu_battery_impact.rst delete mode 100644 mobile/docs/root/development/performance/cpu_impact.rst diff --git a/mobile/docs/root/development/performance/battery_impact.rst b/mobile/docs/root/development/performance/battery_impact.rst deleted file mode 100644 index 557d02eb88b2..000000000000 --- a/mobile/docs/root/development/performance/battery_impact.rst +++ /dev/null @@ -1,90 +0,0 @@ -.. _dev_performance_battery: - -Analysis of battery impact -========================== - -In order to identify how Envoy impacts an application, we have created a control application with modifications to our -current hello world example applications. To see the actual applications we used, you can go to `iOS Envoy example `_, -`Android control app `_, `Android Envoy example `_. - -The current configurations make network requests every 200ms and disable caching. - -* TODO: Fill more details about how Envoy is configured - -Experimentation method -~~~~~~~~~~~~~~~~~~~~~~ - -iOS ---- - -* TODO - -Android -------- - -We're currently using HttpURLConnection to communicate and send requests to Envoy. Envoy in it's current state is run as -a process - -Getting the build: - -1. Build the library using ``bazel build android_dist --config=android`` -2. Control: ``bazel mobile-install //examples/kotlin/control:hello_control_kt`` -3. Envoy: ``bazel mobile-install //examples/kotlin/hello_world:hello_envoy_kt --fat_apk_cpu=armeabi-v7a`` - -Experiment steps: - -1. Set a phone's display to sleep after 30minutes of inactivity -2. Unplug the phone from any power source -3. Open up the demo app -4. Wait for the phone to sleep -5. Look at the battery drain the battery settings in the phone to see the battery usage and drainage - -Alternative profiling methods tried: - -1. `AccuBattery `_ -We were unable to get the running time of a given application on AccuBattery to more accurately identify battery usage per minute -2. `Battery Historian `_ -We were unable to get reliable data using this method. Often times, the battery usage of an application appears to use no batteries - -Results -~~~~~~~ - -iOS ---- - -Android -------- - -Through running the applications for 30minutes, the results are: - -- Envoy : 0.17%/min -- Control : 0.18%/min - -The results of Envoy and Control are very similar - -Analysis -~~~~~~~~ - -iOS ---- - -* TODO - -Android -------- - -The results of this experiment is that there isn't much of a difference between Envoy and Control every 200ms. With the :repo:`CPU analysis `, -we are able to see: - -1. Requests to s3 are being logged in Envoy -2. DNS resolution does happen every 5 seconds -3. Stats are flushed every 5 seconds - -The DNS resolution and stats flush happening every 5 seconds was a concern but updating the frequency to 1 minute, we -did not notice a big change. - -Open issues regarding battery usage ------------------------------------ - -Current status -~~~~~~~~~~~~~~ diff --git a/mobile/docs/root/development/performance/cpu_battery_impact.rst b/mobile/docs/root/development/performance/cpu_battery_impact.rst new file mode 100644 index 000000000000..fc1ce7d87743 --- /dev/null +++ b/mobile/docs/root/development/performance/cpu_battery_impact.rst @@ -0,0 +1,155 @@ +.. _dev_performance_cpu_battery: + +Analysis of CPU/battery impact +============================== + +Modified versions of the "hello world" example apps were used to run these experiments: + +- `Android control app `_ +- `Android Envoy app `_ +- `iOS control app `_ +- `iOS Envoy app `_ + +- **Control** - Made a request every ``200ms`` to an endpoint without Envoy compiled in the app. +- **Envoy** - Made the same request at the same interval, but routed through an instance of Envoy. + +All request/response caching was disabled. + +Results +~~~~~~~ + +iOS +--- + +Valid through SHA `f05d43f `_. + +Envoy: + +- Avg CPU: >= 100% +- Avg memory: 12MB +- Battery: 12/20 Xcode Instruments score + +Control: + +- Avg CPU: 12% +- Avg memory: 6MB +- Battery: 1/20 Xcode Instruments score + +**Based on these results, memory usage is similar. However, CPU (and consequently battery) usage is very high.** +The root cause has been identified and is being tracked in `issue 215 `_. + +Android +------- + +Valid through SHA `8636711 `_. + +TODO(buildbreaker): Update battery percentages + +Envoy: + +- Avg CPU: 33.16075949% +- Avg memory: 2.765822785% +- Battery: 0.17%/min + +Control: + +- Avg CPU: 28.81012658% +- Avg memory: 2.169620253% +- Battery: 0.18%/min + +**Based on these results, control and Envoy are relatively similar.** + +Experimentation method +~~~~~~~~~~~~~~~~~~~~~~ + +iOS +--- + +The original investigation was completed as part of `this issue `_. + +For analysis, the `Energy Diagnostics tool from Xcode Instruments `_ +was used. + +Requests were made using ``URLSession``, with the session's cache set to ``nil`` (disabling caching). +Envoy listened to the data sent over ``URLSession``, proxying it through. + +Both apps were run (one at a time) on a physical device (iPhone 6s iOS 12.2.x) while running Instruments. + +Reproducing the Envoy example app: + +1. Build the library using ``bazel build ios_dist --config=ios --config=fat`` +2. Copy ``./dist/Envoy.framework`` to the example's `source directory `__ +3. Build/run the example app + +Android +------- + +We're currently using ``HttpURLConnection`` to communicate and send requests to Envoy. Envoy in it's current state is run as +a process listening to traffic sent over this connection. + +Getting the build: + +1. Build the library using ``bazel build android_dist --config=android`` +2. Control: ``bazel mobile-install //examples/kotlin/control:hello_control_kt`` +3. Envoy: ``bazel mobile-install //examples/kotlin/hello_world:hello_envoy_kt --fat_apk_cpu=armeabi-v7a`` + +Battery usage experiment steps: + +1. Set a phone's display to sleep after 30 minutes of inactivity +2. Unplug the phone from all power sources +3. Open up the demo app +4. Wait for the phone to sleep +5. Look at the battery drain the battery settings in the phone to see the battery usage and drainage + +Alternative profiling methods tried: + +1. `AccuBattery `_: +We were unable to get the running time of a given application on AccuBattery to more accurately identify battery usage per minute + +2. `Battery Historian `_: +We were unable to get reliable data using this method. Often times, the battery usage of an application appears to use no batteries + +CPU usage experiment steps: + +1. Run ``adb shell top -H | grep envoy`` to get the CPU usage of the application (the ``-H`` flag displays the running threads) +2. Wait 10minutes to gather a sample set of data to analyze +3. Take the average CPU% and MEM% + +Analysis +~~~~~~~~ + +iOS +--- + +Envoy had a reasonable increase in memory usage of a few megabytes compared to control. + +CPU/battery usage, however, was much higher. After some digging, the largest contributor to this usage +was `identified as a poller `_. + +Upon further investigation, the `root cause was determined `_ +to be that ``poll_dispatch`` was being used by ``libevent`` instead of the much more performant ``kqueue``. +Forcing ``libevent`` to use ``kqueue`` reduced the CPU usage **from >= 100% down to ~3%**. +This issue and the subsequent fix are being tracked `here `_. + +`We used Wireshark `_ to validate that +network traffic was flowing through Envoy on the phone every ``200ms``, giving us confidence that there was +no additional caching happening within ``URLSession``. + +Android +------- + +There are minimal differences between Envoy and control. By enabling trace logging within Envoy, +we are able to observe the following: + +1. Requests to S3 are being logged in Envoy +2. DNS resolution does happen every 5 seconds +3. Stats are flushed every 5 seconds + +The DNS resolution and stats flush happening every 5 seconds was originally a concern, +but updating the frequency to 1 minute did not result in a significant change. + +Open issues regarding battery usage +----------------------------------- + +For current issues with CPU/battery, please see issues with the +`perf/cpu label `_. diff --git a/mobile/docs/root/development/performance/cpu_impact.rst b/mobile/docs/root/development/performance/cpu_impact.rst deleted file mode 100644 index a151e9782829..000000000000 --- a/mobile/docs/root/development/performance/cpu_impact.rst +++ /dev/null @@ -1,80 +0,0 @@ -.. _dev_performance_cpu: - -.. _ios_envoy_example_app: https://github.com/lyft/envoy-mobile/tree/ac/envoy-battery-cpu-branch/examples/swift/hello_world -.. _android_envoy_example_app: https://github.com/lyft/envoy-mobile/tree/ac/envoy-battery-cpu-branch/examples/kotlin/hello_world -.. _android_envoy_example_control_app: https://github.com/lyft/envoy-mobile/tree/ac/envoy-battery-cpu-branch/examples/kotlin/control - -Analysis of CPU impact -====================== - -Experimentation method -~~~~~~~~~~~~~~~~~~~~~~ - -iOS ---- - -* TODO - -Android -------- - -We're currently using HttpURLConnection to communicate and send requests to Envoy every 200ms. -Envoy in it's current state is run as -a process - -Getting the build: - -1. Build the library using ``bazel build android_dist --config=android`` -2. Control: ``bazel mobile-install //examples/kotlin/control:hello_control_kt`` -3. Envoy: ``bazel mobile-install //examples/kotlin/hello_world:hello_envoy_kt --fat_apk_cpu=armeabi-v7a`` - -Experiment steps: - -1. Run ``adb shell top -H | grep envoy`` to get the CPU usage of the application (the ``-H`` flag displays the running threads) -2. Wait 10minutes to gather a sample set of data to analyze -3. Take the average CPU% and MEM% - -Results -~~~~~~~ - -iOS ---- - -* TODO - -Android -------- - -Envoy: - -- Avg CPU: 33.16075949% -- Avg MEM: 2.765822785% - -Control: - -- Avg CPU: 28.81012658% -- Avg MEM: 2.169620253% - -Analysis -~~~~~~~~ - -iOS ---- - -* TODO - -Android -------- - -The results of this experiment is that there is minimal difference between Envoy and Control. By enabling trace logging -within Envoy, we are able to observe the following: - -1. Requests to s3 are being logged in Envoy -2. DNS resolution does happen every 5 seconds -3. Stats are flushed every 5 seconds - -Open issues regarding battery usage ------------------------------------ - -Current status -~~~~~~~~~~~~~~ diff --git a/mobile/docs/root/development/performance/performance.rst b/mobile/docs/root/development/performance/performance.rst index f3a2d4879018..31ccd732c4d2 100644 --- a/mobile/docs/root/development/performance/performance.rst +++ b/mobile/docs/root/development/performance/performance.rst @@ -7,8 +7,7 @@ Performance Analysis :maxdepth: 2 binary_size - battery_impact - cpu_impact + cpu_battery_impact Performance analysis can take several shapes in mobile applications. These docs describe the process (tools, analysis, measurements) that the Envoy Mobile team