Implement data transformer for Script_Extensions map data #1353

echeran · 2021-12-02T01:53:54Z

This PR is progress towards #1271

data struct for Script_Extensions data
data provider for Script_Extensions data for property-value-per-cp + transformer

… VarZeroVec type parameter

Manishearth

looks good so far, with #1357 you won't need the custom ULE type and .get_as_zv() would need to be changed, but also you might just not need it since ZeroVecULE<T> has a lot of useful methods already

Branch 'zv-ule' is not yet merged, but simplifies VZV<[T]> into VZV<ZVULE<T>>

…roVecULE and methods

…ia data provider

utils/codepointtrie/src/codepointtrie.rs

utils/zerovec/src/map/serde.rs

utils/zerovec/src/yoke_impls.rs

provider/blob/src/static_data_provider.rs

provider/blob/src/export/blob_exporter.rs

provider/blob/src/blob_schema.rs

Co-authored-by: Shane F. Carr <shane@unicode.org>

…at were removed before merge to main

…&ZeroSlice w/o referencing fn local data

…he same type

… from Vec<Vec<u16> to VZV<ZS<Script>>

…th short alias instead of long alias

components/properties/src/script.rs

sffc · 2021-12-16T23:11:53Z

utils/zerovec/src/zerovec/slice.rs

@@ -321,3 +321,12 @@ impl<T: AsULE> AsRef<ZeroSlice<T>> for Vec<T::ULE> {
        ZeroSlice::<T>::from_ule_slice(&**self)
    }
 }
+
+impl<T> Default for &ZeroSlice<T>


FYI @Manishearth : We are adding a impl Default for &ZeroSlice

echeran added 6 commits December 1, 2021 13:57

Add initial code for data struct and related types for Script_Extensions

1de0ca5

Random minor Rust doc string fixes

5dbdc99

Add initial code for data provider for Script / Script_Extensions

bf53c8f

Attempt to satisfy ULE trait impl for Script newtype/enum

abe3ad4

Revert Script AsULE impl; Use the associated ULE type directly in the…

8418325

… VarZeroVec type parameter

Code from Iain to implement .get for a VZV<[T]> that returns ZV<T>

4a35625

echeran requested review from Manishearth and sffc December 3, 2021 03:04

Manishearth mentioned this pull request Dec 3, 2021

Add SignedPlainOldULE, add serialize impls for ULE types (also missing Eq impl for VZV) #1356

Closed

Manishearth added 10 commits December 3, 2021 16:45

Add ule module to zerovec

4e80ac4

Add ZeroVecULE

2fceaff

Add VarULE impl and ser/de

52323b3

Add other impls

7ea5b5f

Add Ord impls

11f6675

Move VarULE impls over to ZeroVecULE

255b01e

Add Eq impl to VarZeroVec

4bf4824

docs

05ac46a

docs

2a977a8

import

b56308e

Manishearth reviewed Dec 3, 2021

View reviewed changes

Manishearth and others added 10 commits December 3, 2021 19:17

update blob data provider

a64c7b2

Merge branch 'zv-ule' into script-ext-data

f009e86

Branch 'zv-ule' is not yet merged, but simplifies VZV<[T]> into VZV<ZVULE<T>>

Use VZV<ZeroVecULE<Script>> for Script_Extensions

76e87b1

Apply cargo fmt changes

54ff899

Create skeleton of data provider code; perform some helpful refactorings

ed30f01

Fill out the source data -> VZV conversion for ScriptExtensions

a33c62a

Remove the .get_as_zv(usize) for VZV after being superseded by new Ze…

888cf26

…roVecULE and methods

Add function to get Script value from Script_Extensions data struct v…

6722ead

…ia data provider

Apply cargo fmt changes

cdf3565

Fix bit shifting logic for ScriptWithExt

12cf565

Replace debug_assert! with return of Err(MissingResourceKey)

ad00ecf

sffc reviewed Dec 11, 2021

View reviewed changes

utils/codepointtrie/src/codepointtrie.rs Outdated Show resolved Hide resolved

utils/codepointtrie/src/codepointtrie.rs Outdated Show resolved Hide resolved

sffc reviewed Dec 11, 2021

View reviewed changes

echeran and others added 8 commits December 13, 2021 10:21

Apply suggestions from code review

6b5db19

Co-authored-by: Shane F. Carr <shane@unicode.org>

Revert changes after eager merge of PR branch for unicode-org#1357 th…

3bafe8e

…at were removed before merge to main

Removed unneeded line in examples code

a7d828b

Rename provider_uprops serde module for Script_Extensions

f004b71

Merge branch 'main' into script-ext-data

4009be3

Attempt to replace ZeroVecULE with ZeroSlice, but unsure how to pass …

f1c6000

…&ZeroSlice w/o referencing fn local data

Attempt to improve VarZeroVec construction from Vec<Vec<T>>

1618f47

Convert from Script::ULE to ScriptWithExt::ULE directly since it is t…

dc3543b

…he same type

echeran mentioned this pull request Dec 15, 2021

Restrict Script (and ScriptWithExt) to specific range #1404

Open

3 tasks

echeran added 7 commits December 15, 2021 16:57

Fix code to make get_script_extensions_val() return a ZeroSlice ref

0b3eef9

Apply cargo fmt changes

94aceba

Revert attempt to simplify / make more type-safe the data transformer…

8d7b803

… from Vec<Vec<u16> to VZV<ZS<Script>>

Fix unit tests, apply cargo fmt and clippy changes

fff3be9

Add TODO for transformer simplification

9ee05fa

Fix Script_Extensions source data file after running icuexportdata wi…

a53b78f

…th short alias instead of long alias

WIP (find a From impl for VZV to match Vec<Vec<T>>)

19b9f4d

sffc reviewed Dec 16, 2021

View reviewed changes

components/properties/src/script.rs Outdated Show resolved Hide resolved

echeran added 3 commits December 16, 2021 14:44

Simplify construction of VZV<ZeroSlice<Script>> from Vec<Vec<u16>>

0462c73

Impl Default for ZeroSlice; clean up get_script_extensions_val()

c12315f

Merge branch 'main' into script-ext-data

4137848

echeran marked this pull request as ready for review December 16, 2021 23:10

sffc approved these changes Dec 16, 2021

View reviewed changes

sffc reviewed Dec 16, 2021

View reviewed changes

echeran changed the title ~~Implement data pipeline for Script_Extensions data~~ Implement data transformer for Script_Extensions map data Dec 16, 2021

echeran merged commit 685622e into unicode-org:main Dec 16, 2021

echeran deleted the script-ext-data branch December 16, 2021 23:49

robertbastian mentioned this pull request Mar 15, 2022

Add continuous integration for non-testdata data generation #1698

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement data transformer for Script_Extensions map data #1353

Implement data transformer for Script_Extensions map data #1353

echeran commented Dec 2, 2021 •

edited

Loading

Manishearth left a comment

sffc Dec 16, 2021

Implement data transformer for Script_Extensions map data #1353

Implement data transformer for Script_Extensions map data #1353

Conversation

echeran commented Dec 2, 2021 • edited Loading

Manishearth left a comment

Choose a reason for hiding this comment

sffc Dec 16, 2021

Choose a reason for hiding this comment

echeran commented Dec 2, 2021 •

edited

Loading