Skip to content

Commit

Permalink
tools: Check in tools for shrinking ICU size, change default to small…
Browse files Browse the repository at this point in the history
…-icu

* Change configure default to "small-icu" (Intl on, English only)
* add "--without-intl" and "vcbuild without-intl" options, equivalent
to --with-intl=none
* update BUILDING.md with above changes
* Checks in tools that generate the deps/icu-small source directory
from ICU source
* Tools and process for updating ICU documented in tools/icu/README.md

Fixes: #3476
PR-URL: #6088
Reviewed-By: James M Snell <jasnell@gmail.com>
  • Loading branch information
srl295 authored and evanlucas committed May 17, 2016
1 parent ee1e5a2 commit dce6413
Show file tree
Hide file tree
Showing 6 changed files with 388 additions and 110 deletions.
38 changes: 12 additions & 26 deletions BUILDING.md
Original file line number Diff line number Diff line change
Expand Up @@ -117,29 +117,14 @@ $ make

### `Intl` (ECMA-402) support:

[Intl](https://github.com/nodejs/node/wiki/Intl) support is not
enabled by default.
[Intl](https://github.com/nodejs/node/wiki/Intl) support is
enabled by default, with English data only.

#### Default: `small-icu` (English only) support

#### "small" (English only) support

This option will build with "small" (English only) support, but
the full `Intl` (ECMA-402) APIs. With `--download=all` it will
download the ICU library as needed.

##### Unix / OS X:

```text
$ ./configure --with-intl=small-icu --download=all
```

##### Windows:

```text
> vcbuild small-icu download-all
```

The `small-icu` mode builds with English-only data. You can add full
By default, only English data is included, but
the full `Intl` (ECMA-402) APIs. It does not need to download
any dependencies to function. You can add full
data at runtime.

*Note:* more docs are on
Expand All @@ -148,7 +133,8 @@ data at runtime.
#### Build with full ICU support (all locales supported by ICU):

With the `--download=all`, this may download ICU if you don't have an
ICU in `deps/icu`.
ICU in `deps/icu`. (The embedded `small-icu` included in the default
Node.js source does not include all locales.)

##### Unix / OS X:

Expand All @@ -164,19 +150,19 @@ $ ./configure --with-intl=full-icu --download=all

#### Building without Intl support

The `Intl` object will not be available. This is the default at
present, so this option is not normally needed.
The `Intl` object will not be available, nor some other APIs such as
`String.normalize`.

##### Unix / OS X:

```text
$ ./configure --with-intl=none
$ ./configure --without-intl
```

##### Windows:

```text
> vcbuild intl-none
> vcbuild without-intl
```

#### Use existing installed ICU (Unix / OS X only):
Expand Down
58 changes: 47 additions & 11 deletions configure
Original file line number Diff line number Diff line change
Expand Up @@ -303,20 +303,28 @@ parser.add_option('--with-etw',
intl_optgroup.add_option('--with-intl',
action='store',
dest='with_intl',
default='none',
default='small-icu',
choices=valid_intl_modes,
help='Intl mode (valid choices: {0}) [default: %default]'.format(
', '.join(valid_intl_modes)))

intl_optgroup.add_option('--without-intl',
action='store_const',
dest='with_intl',
const='none',
help='Disable Intl, same as --with-intl=none')

intl_optgroup.add_option('--with-icu-path',
action='store',
dest='with_icu_path',
help='Path to icu.gyp (ICU i18n, Chromium version only.)')

icu_default_locales='root,en'

intl_optgroup.add_option('--with-icu-locales',
action='store',
dest='with_icu_locales',
default='root,en',
default=icu_default_locales,
help='Comma-separated list of locales for "small-icu". "root" is assumed. '
'[default: %default]')

Expand Down Expand Up @@ -880,7 +888,7 @@ do_not_edit = '# Do not edit. Generated by the configure script.\n'

def glob_to_var(dir_base, dir_sub, patch_dir):
list = []
dir_all = os.path.join(dir_base, dir_sub)
dir_all = '%s/%s' % (dir_base, dir_sub)
files = os.walk(dir_all)
for ent in files:
(path, dirs, files) = ent
Expand Down Expand Up @@ -968,6 +976,7 @@ def configure_intl(o):
locs = set(options.with_icu_locales.split(','))
locs.add('root') # must have root
o['variables']['icu_locales'] = string.join(locs,',')
# We will check a bit later if we can use the canned deps/icu-small
elif with_intl == 'full-icu':
# full ICU
o['variables']['v8_enable_i18n_support'] = 1
Expand All @@ -994,15 +1003,42 @@ def configure_intl(o):
# this is just the 'deps' dir. Used for unpacking.
icu_parent_path = os.path.join(root_dir, 'deps')

# The full path to the ICU source directory.
icu_full_path = os.path.join(icu_parent_path, 'icu')
# The full path to the ICU source directory. Should not include './'.
icu_full_path = 'deps/icu'

# icu-tmp is used to download and unpack the ICU tarball.
icu_tmp_path = os.path.join(icu_parent_path, 'icu-tmp')

# canned ICU. see tools/icu/README.md to update.
canned_icu_dir = 'deps/icu-small'

# We can use 'deps/icu-small' - pre-canned ICU *iff*
# - with_intl == small-icu (the default!)
# - with_icu_locales == 'root,en' (the default!)
# - deps/icu-small exists!
# - with_icu_source is unset (i.e. no other ICU was specified)
# (Note that this is the *DEFAULT CASE*.)
#
# This is *roughly* equivalent to
# $ configure --with-intl=small-icu --with-icu-source=deps/icu-small
# .. Except that we avoid copying icu-small over to deps/icu.
# In this default case, deps/icu is ignored, although make clean will
# still harmlessly remove deps/icu.

# are we using default locales?
using_default_locales = ( options.with_icu_locales == icu_default_locales )

# make sure the canned ICU really exists
canned_icu_available = os.path.isdir(canned_icu_dir)

if (o['variables']['icu_small'] == b(True)) and using_default_locales and (not with_icu_source) and canned_icu_available:
# OK- we can use the canned ICU.
icu_config['variables']['icu_small_canned'] = 1
icu_full_path = canned_icu_dir

# --with-icu-source processing
# first, check that they didn't pass --with-icu-source=deps/icu
if with_icu_source and os.path.abspath(icu_full_path) == os.path.abspath(with_icu_source):
# now, check that they didn't pass --with-icu-source=deps/icu
elif with_icu_source and os.path.abspath(icu_full_path) == os.path.abspath(with_icu_source):
print 'Ignoring redundant --with-icu-source=%s' % (with_icu_source)
with_icu_source = None
# if with_icu_source is still set, try to use it.
Expand Down Expand Up @@ -1043,7 +1079,7 @@ def configure_intl(o):

# ICU mode. (icu-generic.gyp)
o['variables']['icu_gyp_path'] = 'tools/icu/icu-generic.gyp'
# ICU source dir relative to root
# ICU source dir relative to tools/icu (for .gyp file)
o['variables']['icu_path'] = icu_full_path
if not os.path.isdir(icu_full_path):
print '* ECMA-402 (Intl) support didn\'t find ICU in %s..' % (icu_full_path)
Expand Down Expand Up @@ -1083,14 +1119,14 @@ def configure_intl(o):
'source/data/in',
icu_data_file_l)
# relative to dep..
icu_data_in = os.path.join('../../deps/icu/source/data/in', icu_data_file_l)
icu_data_in = os.path.join('..','..', icu_full_path, 'source/data/in', icu_data_file_l)
if not os.path.isfile(icu_data_path) and icu_endianness != 'l':
# use host endianness
icu_data_path = os.path.join(icu_full_path,
'source/data/in',
icu_data_file)
# relative to dep..
icu_data_in = os.path.join('icu/source/data/in',
icu_data_in = os.path.join('..', icu_full_path, 'source/data/in',
icu_data_file)
# this is the input '.dat' file to use .. icudt*.dat
# may be little-endian if from a icu-project.org tarball
Expand All @@ -1117,7 +1153,7 @@ def configure_intl(o):
# with a list of the src files to use
for i in icu_src:
var = 'icu_src_%s' % i
path = '../../deps/icu/source/%s' % icu_src[i]
path = '../../%s/source/%s' % (icu_full_path, icu_src[i])
icu_config['variables'][var] = glob_to_var('tools/icu', path, 'patches/%s/source/%s' % (icu_ver_major, icu_src[i]) )
# write updated icu_config.gypi with a bunch of paths
write(icu_config_name, do_not_edit +
Expand Down
78 changes: 78 additions & 0 deletions tools/icu/README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,84 @@
Notes about the icu directory.
===

How to upgrade ICU
---

- Make sure your node workspace is clean (clean `git status`) should be sufficient.
- Configure Node with the specific [ICU version](http://icu-project.org/download) you want to upgrade to, for example:

```
./configure \
--with-intl=small-icu \
--with-icu-source=http://download.icu-project.org/files/icu4c/56.1/icu4c-56_1-src.zip
make
```

(the equivalent `vcbuild.bat` commands should work also.)

- (note- may need to make changes in `icu-generic.gyp` or `tools/icu/patches` for
version specific stuff)

- Verify the node build works

```
make test-ci
```

Also running
```js
new Intl.DateTimeFormat('es',{month:'long'}).format(new Date(9E8));
```

…Should return `January` not `enero`.
(TODO here: improve [testing](https://github.com/nodejs/Intl/issues/16))


- Now, copy `deps/icu` over to `deps/icu-small`

```
python tools/icu/shrink-icu-src.py
```

- Now, do a clean rebuild of node to test:

(TODO: fix this when these options become default)

```
./configure --with-intl=small-icu --with-icu-source=deps/icu-small
make
```

- Test this newly default-generated Node.js
```js
process.versions.icu;
new Intl.DateTimeFormat('es',{month:'long'}).format(new Date(9E8));
```

(should return your updated ICU version number, and also `January` again.)

- You are ready to check in the updated `deps/small-icu`.
This is a big commit, so make this a separate commit from other changes.

- Now, fix the default URL for the `full-icu` build in `/configure`, in
the `configure_intl()` function. It should match the ICU URL used in the
first step. When this is done, the following should build with full ICU.

```
# clean up
rm -rf out deps/icu deps/icu4c*
./configure --with-intl=full-icu --download=all
make
make test-ci
```

- commit the change to `configure`.

-----

Notes about these tools
---

The files in this directory were written for the node.js effort. It's
the intent of their author (Steven R. Loomis / srl295) to merge them
upstream into ICU, pending much discussion within the ICU-PMC.
Expand Down
Loading

0 comments on commit dce6413

Please sign in to comment.