Skip to content

Commit 2992e34

Browse files
glenn20dpgeorge
glenn20
authored and
dpgeorge
committed
tools/mpremote: Add support for relative urls in package.json files.
URLs in `package.json` may now be specified relative to the base URL of the `package.json` file. Relative URLs wil work for `package.json` files installed from the web as well as local file paths. Docs: update `docs/reference/packages.rst` to add documentation for: - Installing packages from local filesystems (PR micropython#12476); and - Using relative URLs in the `package.json` file (PR micropython#12477); - Update the packaging example to encourage relative URLs as the default in `package.json`. Add `tools/mpremote/tests/test_mip_local_install.sh` to test the installation of a package from local files using relative URLs in the `package.json`. Signed-off-by: Glenn Moloney <glenn.moloney@gmail.com>
1 parent 4364d94 commit 2992e34

File tree

4 files changed

+149
-19
lines changed

4 files changed

+149
-19
lines changed

‎docs/reference/packages.rst

Lines changed: 42 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -96,6 +96,18 @@ The ``--target=path``, ``--no-mpy``, and ``--index`` arguments can be set::
9696
$ mpremote mip install --no-mpy pkgname
9797
$ mpremote mip install --index https://host/pi pkgname
9898

99+
:term:`mpremote` can also install packages from files stored on the host's local
100+
filesystem::
101+
102+
$ mpremote mip install path/to/pkg.py
103+
$ mpremote mip install path/to/app/package.json
104+
$ mpremote mip install \\path\\to\\pkg.py
105+
106+
This is especially useful for testing packages during development and for
107+
installing packages from local clones of GitHub repositories. Note that URLs in
108+
``package.json`` files must use forward slashes ("/") as directory separators,
109+
even on Windows, so that they are compatible with installing from the web.
110+
99111
Installing packages manually
100112
----------------------------
101113

@@ -116,12 +128,25 @@ To write a "self-hosted" package that can be downloaded by ``mip`` or
116128
``mpremote``, you need a static webserver (or GitHub) to host either a
117129
single .py file, or a ``package.json`` file alongside your .py files.
118130

119-
A typical ``package.json`` for an example ``mlx90640`` library looks like::
131+
An example ``mlx90640`` library hosted on GitHub could be installed with::
132+
133+
$ mpremote mip install github:org/micropython-mlx90640
134+
135+
The layout for the package on GitHub might look like::
136+
137+
https://github.com/org/micropython-mlx90640/
138+
package.json
139+
mlx90640/
140+
__init__.py
141+
utils.py
142+
143+
The ``package.json`` specifies the location of files to be installed and other
144+
dependencies::
120145

121146
{
122147
"urls": [
123-
["mlx90640/__init__.py", "github:org/micropython-mlx90640/mlx90640/__init__.py"],
124-
["mlx90640/utils.py", "github:org/micropython-mlx90640/mlx90640/utils.py"]
148+
["mlx90640/__init__.py", "mlx90640/__init__.py"],
149+
["mlx90640/utils.py", "mlx90640/utils.py"]
125150
],
126151
"deps": [
127152
["collections-defaultdict", "latest"],
@@ -132,9 +157,20 @@ A typical ``package.json`` for an example ``mlx90640`` library looks like::
132157
"version": "0.2"
133158
}
134159

135-
This includes two files, hosted at a GitHub repo named
136-
``org/micropython-mlx90640``, which install into the ``mlx90640`` directory on
137-
the device. It depends on ``collections-defaultdict`` and ``os-path`` which will
160+
The ``urls`` list specifies the files to be installed according to::
161+
162+
"urls": [
163+
[destination_path, source_url]
164+
...
165+
166+
where ``destination_path`` is the location and name of the file to be installed
167+
on the device and ``source_url`` is the URL of the file to be installed. The
168+
source URL would usually be specified relative to the directory containing the
169+
``package.json`` file, but can also be an absolute URL, eg::
170+
171+
["mlx90640/utils.py", "github:org/micropython-mlx90640/mlx90640/utils.py"]
172+
173+
The package depends on ``collections-defaultdict`` and ``os-path`` which will
138174
be installed automatically from the :term:`micropython-lib`. The third
139175
dependency installs the content as defined by the ``package.json`` file of the
140176
``main`` branch of the GitHub repo ``org/micropython-additions``.

‎tools/mpremote/mpremote/mip.py

Lines changed: 29 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@
77
import json
88
import tempfile
99
import os
10+
import os.path
1011

1112
from .commands import CommandError, show_progress_bar
1213

@@ -64,22 +65,33 @@ def _rewrite_url(url, branch=None):
6465

6566

6667
def _download_file(transport, url, dest):
67-
try:
68-
with urllib.request.urlopen(url) as src:
69-
data = src.read()
70-
print("Installing:", dest)
71-
_ensure_path_exists(transport, dest)
72-
transport.fs_writefile(dest, data, progress_callback=show_progress_bar)
73-
except urllib.error.HTTPError as e:
74-
if e.status == 404:
75-
raise CommandError(f"File not found: {url}")
76-
else:
77-
raise CommandError(f"Error {e.status} requesting {url}")
78-
except urllib.error.URLError as e:
79-
raise CommandError(f"{e.reason} requesting {url}")
68+
if url.startswith(allowed_mip_url_prefixes):
69+
try:
70+
with urllib.request.urlopen(url) as src:
71+
data = src.read()
72+
except urllib.error.HTTPError as e:
73+
if e.status == 404:
74+
raise CommandError(f"File not found: {url}")
75+
else:
76+
raise CommandError(f"Error {e.status} requesting {url}")
77+
except urllib.error.URLError as e:
78+
raise CommandError(f"{e.reason} requesting {url}")
79+
else:
80+
if "\\" in url:
81+
raise CommandError(f'Use "/" instead of "\\" in file URLs: {url!r}\n')
82+
try:
83+
with open(url, "rb") as f:
84+
data = f.read()
85+
except OSError as e:
86+
raise CommandError(f"{e.strerror} opening {url}")
87+
88+
print("Installing:", dest)
89+
_ensure_path_exists(transport, dest)
90+
transport.fs_writefile(dest, data, progress_callback=show_progress_bar)
8091

8192

8293
def _install_json(transport, package_json_url, index, target, version, mpy):
94+
base_url = ""
8395
if package_json_url.startswith(allowed_mip_url_prefixes):
8496
try:
8597
with urllib.request.urlopen(_rewrite_url(package_json_url, version)) as response:
@@ -91,12 +103,14 @@ def _install_json(transport, package_json_url, index, target, version, mpy):
91103
raise CommandError(f"Error {e.status} requesting {package_json_url}")
92104
except urllib.error.URLError as e:
93105
raise CommandError(f"{e.reason} requesting {package_json_url}")
106+
base_url = package_json_url.rpartition("/")[0]
94107
elif package_json_url.endswith(".json"):
95108
try:
96109
with open(package_json_url, "r") as f:
97110
package_json = json.load(f)
98111
except OSError:
99112
raise CommandError(f"Error opening {package_json_url}")
113+
base_url = os.path.dirname(package_json_url)
100114
else:
101115
raise CommandError(f"Invalid url for package: {package_json_url}")
102116
for target_path, short_hash in package_json.get("hashes", ()):
@@ -105,6 +119,8 @@ def _install_json(transport, package_json_url, index, target, version, mpy):
105119
_download_file(transport, file_url, fs_target_path)
106120
for target_path, url in package_json.get("urls", ()):
107121
fs_target_path = target + "/" + target_path
122+
if base_url and not url.startswith(allowed_mip_url_prefixes):
123+
url = f"{base_url}/{url}" # Relative URLs
108124
_download_file(transport, _rewrite_url(url, version), fs_target_path)
109125
for dep, dep_version in package_json.get("deps", ()):
110126
_install_package(transport, dep, index, target, dep_version, mpy)

‎tools/mpremote/tests/test_mip_local_install.sh

Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,67 @@
1+
#!/bin/bash
2+
3+
# This test the "mpremote mip install" from local files. It creates a package
4+
# and "mip installs" it into a ramdisk. The package is then imported and
5+
# executed. The package is a simple "Hello, world!" example.
6+
7+
set -e
8+
9+
PACKAGE=mip_example
10+
PACKAGE_DIR=${TMP}/example
11+
MODULE_DIR=${PACKAGE_DIR}/${PACKAGE}
12+
13+
target=/__ramdisk
14+
block_size=512
15+
num_blocks=50
16+
17+
# Create the smallest permissible ramdisk.
18+
cat << EOF > "${TMP}/ramdisk.py"
19+
class RAMBlockDev:
20+
def __init__(self, block_size, num_blocks):
21+
self.block_size = block_size
22+
self.data = bytearray(block_size * num_blocks)
23+
24+
def readblocks(self, block_num, buf):
25+
for i in range(len(buf)):
26+
buf[i] = self.data[block_num * self.block_size + i]
27+
28+
def writeblocks(self, block_num, buf):
29+
for i in range(len(buf)):
30+
self.data[block_num * self.block_size + i] = buf[i]
31+
32+
def ioctl(self, op, arg):
33+
if op == 4: # get number of blocks
34+
return len(self.data) // self.block_size
35+
if op == 5: # get block size
36+
return self.block_size
37+
38+
import os
39+
40+
bdev = RAMBlockDev(${block_size}, ${num_blocks})
41+
os.VfsFat.mkfs(bdev)
42+
os.mount(bdev, '${target}')
43+
EOF
44+
45+
echo ----- Setup
46+
mkdir -p ${MODULE_DIR}
47+
echo "def hello(): print('Hello, world!')" > ${MODULE_DIR}/hello.py
48+
echo "from .hello import hello" > ${MODULE_DIR}/__init__.py
49+
cat > ${PACKAGE_DIR}/package.json <<EOF
50+
{
51+
"urls": [
52+
["${PACKAGE}/__init__.py", "${PACKAGE}/__init__.py"],
53+
["${PACKAGE}/hello.py", "${PACKAGE}/hello.py"]
54+
],
55+
"version": "0.2"
56+
}
57+
EOF
58+
59+
$MPREMOTE run "${TMP}/ramdisk.py"
60+
$MPREMOTE resume mkdir ${target}/lib
61+
echo
62+
echo ---- Install package
63+
$MPREMOTE resume mip install --target=${target}/lib ${PACKAGE_DIR}/package.json
64+
echo
65+
echo ---- Test package
66+
$MPREMOTE resume exec "import sys; sys.path.append(\"${target}/lib\")"
67+
$MPREMOTE resume exec "import ${PACKAGE}; ${PACKAGE}.hello()"

‎tools/mpremote/tests/test_mip_local_install.sh.exp

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
----- Setup
2+
mkdir :/__ramdisk/lib
3+
4+
---- Install package
5+
Install ${TMP}/example/package.json
6+
Installing: /__ramdisk/lib/mip_example/__init__.py
7+
Installing: /__ramdisk/lib/mip_example/hello.py
8+
Done
9+
10+
---- Test package
11+
Hello, world!

0 commit comments

Comments
 (0)