Apps are too big (in size) #202
Comments
|
We rehash this problem every few months. As you've discovered, there is no
|
|
Every kivy app is not 20MB, look at the showcase. It's 6MB, you need to trim your app. Kivy itself comes under 5-6 mb as evident by the showcase. |
And more, but all the things we currently explored have much more drawbacks than usability. The current state is a mix. It can always be improved, but must not be complex to use as well. |
|
And yes, a minimal APK size is ~5MB. 20MB is already big. And most of the time, it's due to the data in your application, not Kivy. |
|
It is 20MB after extraction. If you ;look in Settings->Apps you can dee
|
|
I see I missed a point there: Please note that the package manager has calculated the size wrong since as far as I can remember. Just try to move an app to/from the SD and look at the total. Sorry for the confusion. |
|
This is my main concern before delving into learning Kivy for mobile app development... I'm currently using JUCE and C++ but would rather be using Python for main part of the app.. however, app size and battery drain are concerns that I've seen raised about Kivy mobile apps. |
|
I do think 6-7Megs are acceptable sizes for an app these days, and a lot more if you have data, you rarely ever see less than one meg apps anymore, imho. For battery drain, i never witnessed anything about kivy that would cause that, i do have a few unity apps that makes my phone burn my hand and empty my battery pretty quick (nexus 4), nothing of that kind happened with any kivy app that i tested. |
|
It may not be much of a problem on some newer phones with more storage, but if you can make it a bit smaller, and in the process support more phones, why not? P.S. I'm not talking about performance. |
|
"Why not?" If we could do it without a bunch of extra work we would. This
|
|
Please note the "if you can" ;) |
|
I'm seeing two sizes stated here - ±20mb and 6-7mb - is the smaller size the compressed installer one would download from the app store? For my needs (musical tuning app), comparable apps are around 500kb on the app store.. so in this case, Kivy is not for me, but I am considering it for building API consuming mobile apps at the company I work for, as in this case size should not be such an issue. |
|
6MB is the smallest size for the APK. Once installed, it will be reported
|
|
A kivy APK is 6MB, once it is run for the first time, libraries are extracted, which ends up as 20MB (Compressed data + estracted data). I can confirm that the android app manager is not the only who reports so, diskusage does too. |
|
@mid-kid. This is a known limitation. Kivy might not be suited for every Expecting the kivy apk to be around 500KB though might not be feasible, yet! We need a good selection mechanism they while packaging figures out all the On Sun, Jul 20, 2014 at 8:38 PM, mid-kid notifications@github.com wrote:
|
|
Can we delete the zip after extraction? On Mon, Jul 21, 2014 at 1:02 AM, Akshay Arora notifications@github.com
|
|
What is it with people thinking I expect it to be under 1M? xD @knappador, no, the zip is contained inside the APK, and as such, it can't be removed. We can try to remove some unneeded libs, but I don't think that would get very far. Some stripping could do something, like @tito said (I haven't looked at this project in some time now, so I don't know about the state of affairs), but I think the key resides in changing the distribution method of the libraries. No option proposed so far is ideal (imo neither is the current one). Funny how much activity after 6 months. |
|
The possibility of not including everything in the .mp3 files is interesting and would be more effective than stripping ever could be. The entire tar/mp3 trick was done specifically to deal with the Android toolchain deleting files that we wanted to include in the APK. Maybe we can switch to only packaging files that the Android toolchain mistakenly deletes. I'm currently looking into a data persistence issue when using --private. I bet the two problems cover the same ground. |
|
what aboat to use nuitka and compile all application into c and then use directly into ndk ? it has some optimazation like pack all used package(in my reading in the site) |
|
If you read the thread, that question is already answered. —Reply to this email directly or view it on GitHub. |
|
Lot of activity here. One more addition to the thread:
I'm currently experimenting a new way to boot Python application (not related to Kivy yet): bootstrap. The minimal bootstrap is based on NDK, extract only one file, setup the path to include zip import, have a mechanism to load CPython extensions library from the libs/ directory (so unpacked during the installation). Boot to python is really faster. BUT i wonder how far we can push this approach with all the python library. the .egg format separate the libraries from the python code. This is interesting to use if we want to separate thoses in the final apk. However, the real issue is the data shipped with any python library. If you take Kivy, it will search for the glsl shaders in its own data/glsl directory. But it except a posix path, not a path within a zip. Same for images. Now try to think about a method that could work for any python packages: i didn't find one yet. |
|
tito your last answer was Great!!! may be it give some hope to use embedding python in widder situations. thanks a lot |
|
i dont know it can be helpfull but if there is a way to use jython in android we can add it as a jar file and we reduce the size of python itself |
|
Jython does not support python3, and afaik, it does not work with native libraries compiled for CPython, such as SDL. |
|
Kivy sounds perfect, being Python and cross-platform, but then all the apps it creates are huge for what they do, which makes me feel like it's not worth learning.
On Android it's 37 MB:
How do you "trim your app"? Is this not automatic? |
|
@endolith: @akshayaurora was referring to the .apk file size (where 6-7MB is the minimum), not the unpacked size (where the ~30MB you see is a normal unpacked size for that minimal APK). By trimming the app, I think @akshayaurora means removing parts of the python installation (and potentially our own app) that aren't necessary. Some such things are done automatically, where modules are unlikely to ever be useful on android, but doing it on a large scale would be extremely difficult as it's hard to inspect a python app and work out what modules it will need. Even doing this manually, I suspect it would be difficult to cut more than 1-2MB from the APK (though that's more difference when extracted). |
|
One utopic solution to this would be to have Python+Kivy commonly preinstaled on Android systems :-3 |
|
make an android virus that automatically spreads and install Kivy , Cordova + crosswalk , etc :P |
|
Is there a way to create apk that contain only python script and additional module (without including python vm + kivy) ? and then have the apk detect if the python+kivy is installed. if python+kivy is not installed then download the python+kivy. Therefore every kivy app apk size will not too big. |
|
Kivy and python-for-android are both actively developed (you can check
the recent commits in either of them), but the technical barriers
described earlier in this thread have not really changed.
…On 28/03/17 03:45, Chris Burchett wrote:
What is the latest on this issue? Have any improvements been made? Is
Kivy still under active development?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#202 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/ABNQm4lkaWD9u70CpdQBP3vR0cHFkkaIks5rqHRTgaJpZM4BeanA>.
|
|
@cdb1019 I have a small list of files&folders that could be removed from the Python packaged into APK archive, but a testing is needed: folders:
files:
maybe:
That kills another ~1.86MB from unpacked Python (~7.5MB for me). The rest is only about cutting off stuff from Kivy that you don't need such as behaviors, complex widgets, kivy logos, keyboard layouts, fonts, effects, extras, recorder(input), some modules, storage (if you don't use it), tools (aren't packaged, I think). For all that you'll need to check if they aren't imported or referenced somewhere else (and that's a matter of change always). We can't really strip the Kivy parts as we don't have a clue what a user might use, but we might be able to strip Python a little bit more in the future. |
|
Anyone has plans to make an Android fork/mod/ROM with Kivy built in? - In such a way that all App developers need to do is package the main.py and main.kivy (obfuscated if necessary), and necessary support files not included with kivy. The install file also comes with a requirements.txt which the internal pip interprets. Just a thought that crossed my mind. I don't see it growing big without going the way of Firefox OS. It is just for fun. |
|
I think what you're saying is too vague,, are you referring to something
like a kivy launcher but can also compile the apk at the same time??
…On Monday, April 10, 2017, Laxminarayan G Kamath A ***@***.***> wrote:
Anyone has plans to make an Android fork/mod/ROM with Kivy built in? - In
such a way that all App developers need to do is package the main.py and
main.kivy (obfuscated if necessary), and necessary support files not
included with kivy. The install file also comes with a requirements.txt
which the internal pip interprets.
Just a thought that crossed my mind.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#202 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AJGdRDhZcfsyqp9Ec-G1hK0uGsxmTqQdks5rue4agaJpZM4BeanA>
.
|
|
Yes, It is purposefully vague at the moment. It is more of a problem statement than the solution. The final aim is to be able to distribute APKs or may be some other installer without the whole load of the Interpreter + libraries in every APK/installer. Also, the required python libraries can be installed centrally to reduce duplication. But on giving it proper thought, I realized the duplication problem can't be correctly solved , as upgrading a library for one app would probably break other apps. |
|
Now this comment has nothing to do with the above idea. Have we tried experimenting with xz/lzma compression instead of zip? I know, when the app is decompressed initially it is finally going to occupy the same space, but at least it can reduce the compressed size? |
|
Perhaps a modified version of cx_Freeze could be used? With cx_Freeze, you tell it which modules are needed for your app (already in the kivy requirements), then it scans all those modules and finds their dependencies (you can add ones it can't find automatically) and packs ONLY the needed python modules and their dependencies into a single library.zip file. Also it renames all .so files (only from required modules / packages) in the format package.name.so and places them in a single folder. Once this is done:
Pros:
Cons:
Edit: Perhaps extraction could be sped up using czipfile, or the library contents could be loaded into memfs or something. |
|
I found out how to have all the Doing it this way you can support multiple arch's in one apk, EVEN when your python requirements has compiled components as only the actual py/pyc/pyo files are in the assets folder. You can also then compress all the python libs into a zip and import them from the zip (although it's slower at the moment) or extract as normal.
'''
Copyright (c) 2017, Jairus Martin.
Distributed under the terms of the MIT License.
The full license is in the file COPYING.txt, distributed with this software.
Created on July 10, 2017
@author: jrm
'''
import os
import sys
import imp
from glob import glob
class AndroidLoader(object):
""" Loads renamed .so files from the app's lib folder"""
def load_module(self, mod):
try:
return sys.modules[mod]
except KeyError:
pass
lib = AndroidFinder.so_modules[mod]
m = imp.load_dynamic(mod, lib)
sys.modules[mod] = m
return m
class AndroidFinder(object):
""" Loads renamed so files from the app's lib folder"""
so_modules = {}
def __init__(self):
#: Find all included so files
lib_dir = os.environ['APK_LIB_DIR']
for lib in glob('%s/lib.*.so'%lib_dir):
name = lib.split("/")[-1] # Lib filename
mod = ".".join(name.split(".")[1:-1]) # Strip lib and so
AndroidFinder.so_modules[mod] = lib
def find_module(self, mod, path = None):
if mod in self.so_modules:
return AndroidLoader()
return None
|
|
@endolith I know I am more than 18months late .. But have you checked if the apk generated previously is also getting included in the app? When my app size started doubling every build , I noticed that the previous generated apk was getting bundled into the current apk .. So, over the last few iterations, my development pattern has morphed somewhat .. I keep all the files that go into the real app in the src directory, and give p4a the src directory as the private folder. The blacklist.txt , project config and other files remain in the project's root directory. I suggest Kivy developers using python-for-android to switch to this pattern. Feels clean. |
|
@frmdstryr Please look into https://developer.android.com/studio/build/configure-apk-splits.html if you are interested in building multiple architectures. I myself haven't tried it, but is definitely on my todo list. |
|
@frmdstryr That's extremely encouraging, thanks for the research! I'll certainly try this |
|
I was able to get about a 15% reduction by using pyminifier. Interestingly, it reduces the py files about about 50%, but when compiled to pyo the difference is only ~15%. Still, took the extracted python from 9.6 MB to 8.2 MB and reduced the compressed python by about 0.5MB (3.2MB to 2.7MB). So about 2MB overall, better than nothing. '''
Copyright (c) 2017, Jairus Martin.
Distributed under the terms of the MIT License.
The full license is in the file COPYING.txt, distributed with this software.
Created on July 21, 2017
@author: jrm
'''
def minifiy_sources(ctx, src):
""" Finds all .py files within a path and runs minification on them.
this should be done BEFORE compileall is run.
Warning: This does IN PLACE minification.
"""
info("Minifying sources, this may take some time...")
from pyminifier import token_utils, minification
class Options(object):
tabs = False #: Replace indentation with tabs (vs single space)
cwd = os.getcwd()
original = 0
minified = 0
passed = 0
total = 0
with current_directory(src):
#: Find all files
for py in shprint(sh.find, '.', '-name', '*.py').split("\n"):
py_file = realpath(join(src, py))
if not exists(py_file) or isdir(py_file):
continue
total +=1
#: Read source code
with open(py_file) as f:
code = f.read()
original += len(code)
try:
#: Minify code
#: The operators seem to have a bug...
minified_code = code
for f in [minification.remove_comments_and_docstrings,
minification.fix_empty_methods,
minification.remove_blank_lines,
minification.reduce_operators,
minification.dedent
]:
minified_code = f(minified_code)
#tokens = token_utils.listified_tokenizer(code)
#minified_code = minification.minify(tokens, Options)
#print("OK",)
except Exception as e:
info("Minifying {}... Failed! Reason: {}".format(py, e))
#raise
continue # Can't minify, leave it as is!
#: Write minified code
with open(py_file, 'wb') as f:
f.write(minified_code)
#: Make sure it compiles
try:
#: Minification compiles to pyo to make sure it's ok
#: Compile pip packages to bytecode
sh.bash('-c',
"source {}/venv/bin/activate && env CC=/bin/false CXX=/bin/false"
"PYTHONPATH={} python -{} -m py_compile {}"
.format(cwd, ctx.get_site_packages_dir(), ctx.optimize, py_file))
minified += len(minified_code)
passed +=1
except Exception as e:
info("Compile {}... Failed! Reason: {}".format(py, e))
#: Undo
with open(py_file, 'wb') as f:
f.write(code)
# #raise
minified += len(code)
#: Optimize original file
sh.bash('-c',
"source {}/venv/bin/activate && env CC=/bin/false CXX=/bin/false"
"PYTHONPATH={} python -{} -m py_compile {}"
.format(cwd, ctx.get_site_packages_dir(), ctx.optimize, py_file))
info_main("Minification complete total={} passed={}({}%) savings={}%!"
.format(total, passed, round(100*passed/total), round(100*minified/original)))
|
|
@frmdstryr If you can, try only the minification on the whole Python's I remember trying |
|
@frmdstryr Has anyone researched on Ultimate Executable Packer ? The code is currently way out of my league .. can it be adapted for python pyc/pyo , etc.. ? Might affect startup performance though. |
|
Every such compression will need decompression on the device. It doesn't really matter how quickly you compress if you end up with the loading screen displayed for a minute. You can't just make the whole interpreter disappear with compressing, you need to remove files from it to actually make a usable difference. Just saying. |
|
@KeyWeeUsr That's what I'm doing, except I'm using crystax python 2. Currently I extract the
Yes, exactly. If it does get added, it should be disabled by default as it's more of a final product optimization anyways. Minify does screw up some files, for instance the @kamathln p4a already compresses everything with |
|
It would be interesting if some sort of bundler could be used like what is commonly done with javascript (webpack for example). react-native does this on release builds. It wouldn't save space (unless you can do minification on the bundle) but you could use the trick to have it extracted during install and avoid the startup extraction. Pros:
Cons:
The only "packer" I found was pinliner. I ran a few tests, and it can't bundle multiple packages and modules at the moment (but does work nice for a single package), and it doesn't pack pyc/pyo files. If these were implemented, the python |
|
@frmdstryr Thanks for correcting my assumption. Had assumed it is plain tar. Tried re-compressing the tar.gz with tar.xz instead. My private.mp3 came down from 6.3mb to 4.7mb . The decompression speed was 3x slower than gz though. (still much faster than bz2) . The comparizon was on a laptop though. Planning to try on my phone using termux . |
|
On my phone, in Termux: |
|
The tar.gz is simply a rename of private.mp3 . tar.xz and tar.bz2 are created with .... ....technique |
|
Can I do it by just deleting blank lines, comments,unused things from kivy??? |
|
No, you won't get much if any gain from that. We already compile and ship bytecode and that doesn't help the size much. |
|
Should I use pyminifier?
…On Wed, 22 Jul 2020, 1:29 am Alexander Taylor, ***@***.***> wrote:
No, you won't get much if any gain from that. We already compile and ship
bytecode and that doesn't help the size much.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#202 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AQGPIPDXL5NJDUETGLZHZVTR4XUAJANCNFSM4ALZVHAA>
.
|
|
the issue is not the size of python code, it's mostly the size of binaries, like python itself, and all the cpython extensions (like kivy, but also others), that can quickly run into the megabyte range. I'm not sure this issue is actionable, it's too broad, there will always be reasons to argue that apps are "too big", i'd rather see someone analyse and point at specific issues that can be fixed to make the apk / installed size smaller, and to have these fixed independently. So i'm going to close it, if another maintainer disagree, feel free to reopen, if an user sees a specific issue to fix to help address the point of this issue, feel free to open another one describing the exact issue you see. |
|
Can this be achieved by removing unused libraries and packages? |
and others
Every separate kivy app installed has an identical copy of the libraries. That makes every kivy app be about 20MB in size when only including the kivy library and it's dependencies. That's very big considering the actual content of the app.
There are several options to solve this problem I can think of:
Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.
The text was updated successfully, but these errors were encountered: