Better indexing and iterators for structs to prevent copies

[peppergrayxyz]

I'm currently with Unicode text segmentation and have noticed a few things, which could be improved in general:

• allow structs to be indexed
• array conversions always provide an iterator or index function
• warn about unwanted side effects, probably disallow function calls in combination with indexing (assignment, loops)

Examples:

s := 'a1©★🚀'
b := s.bytes[7..11] // get a range from a type which can be directly indexed
r := s.runes[..3][2] // get an index from a type that must be iterated

s := 'a1©★🚀'
for r in s.runes { // iterate using the avialble method (index or iterator)
    //...
}

The current way...

s := 'a1©★🚀'
b := s.bytes()[7..11] // this should actually already give a warning to use s[7..1].bytes() instead
r := s.runes()[3]

s := 'a1©★🚀'
for r in s.runes() { // currently there is no rune iterator implemented
    //...
}

...will copy the whole string, although it is not needed:

• the assignment only needs a copy of a slice or a specific index
• the loop doesn't need its own copy and may actually break before it reaches the end

I use struct string as example. For string you could just do s[i] instead of s.bytes()[i], but this only works with string. struct types cannot be indexed with [].

indexing

The array syntax can be allowed to be used with structs

b := s.bytes[7..11]

and mapped to different functions:

• s.bytes[1] --> s.bytes.get(i int) u8
• s.bytes[7..11] --> s.bytes.slice(start int, end int) []u8 --> for i /* ... */ get(i)
• s.bytes --> slice(0, s.len)

Some types may not support direct indexing, but only iterating. There should be a warning if they are accessed using and index or a range other than [0] or [..index]. If only an index is needed, the sequence must be generated nevertheless, hence the syntax should be...

b := s.runes[..3][2]

..and s.runes[..3] is translated using the iterator for i /* ... */ s.runes.next(i).

This clearly indicates if an index access is just a cast or if there is an iterator involved. This example looks strange, but it probably also is an antipattern to use an iterator like this.

It's hard to spot what is going wrong here:

for i := 0; i < 5; i++ {
    r := utf8.raw_index(s, i)
    println(r)
} 

but way easier if the syntax already hints at the issue:

for i := 0; i < 5; i++ {
    r := s.runes(..i+1)[i]
    println(r)
} 

General case:

struct MyStructIndexable { len int /* ... */ }
fn (m MyStructIndexable) get(index int) sometype { /* ... */ }
fn (m MyStructIndexable) slice(start int, end int) sometype { /* ... */ }
m := MyStructIndexable { /* ... */ }
n := m[2..4]
o := m[5]

struct MyStructItertable { /* ... */ }
fn (m MyStructItertable) next(index int) ?sometype { /* ... */ }
x := MyStructItertable { /* ... */ }
y := x[..4]
z := x[..5][4]

There should be at least one of get or slice or next. Their return value should be optional except when there is a len field.

iterating

We have .next to iterate structs, but it is not mandatory for implementations that convert all data in one go to a target type (e.g. s.runes()) to provide an iterator. The range part of the loop should not accept function calls, but the name of the target and the loop finds the proper iterator

• random indexable: for b in s.bytes { --> s.bytes.get(i)
• not random indexable: for r in s.runes { --> s.runes.next(i)

Not having these function calls would prevent full copies. If you really want the full copy, you'd have to make it explicit:

rs := s.runes()
for r in rs {
    // ....
}

Currently, .next needs at least an index variable. If we had the possibility to read and modify the iterator variable of a loop, we could do:

for i, r in s.runes {
    // ...
} 

and convert it to

mut i := 0
for j := 0; j < s.len ; { 
    len, r := s.runes.next(i)
    // ....
    j += len
    i++
}

.next would not require storing state, i.e. the iterator could be attached to a struct without increasing the size of the struct. The iterator variable will be provided by the loop.

General case:

for i, n in m { /* ... */ }
for i, y in x { /* ... */ }

mut i := 0
for j := 0; j < m.len ; { 
    len, n := m.get(j)
    // ....
    j += len
    i++
}

mut i := 0
for j := 0; ; { 
    len, y := x.next(j) or { break }
    // ....
    j += len
    i++
}

putting it together

We need a way to attach iterators/index functions

struct string { /* ... */ }
fn (s string.bytes) get(i int) u8 { /* ... */ }
fn (s string.runes) next(i int) ?rune { /* ... */ }
fn (s string.graphemes) next(i int) ?string { /* ... */ }
fn (s string.words) next(i int) ?string { /* ... */ }
fn (s string.sentences) next(i int) ?string{ /* ... */ }

Probably not a good idea to stick anything to everything like in JavaScript, but there probably is an elegant way to handle this setup. Otherwise, this could just be done by convention:

struct string { /* ... */ }
fn (s string) bytes_get(i int) u8 { /* ... */ }
fn (s string) runes_next(i int) ?rune { /* ... */ }
// ...

the conversion functions should be there as well:

fn (s string) bytes() []u8 { /* ... */ }
fn (s string) runes() []rune { /* ... */ }
fn (s string) graphemes() []string { /* ... */ }
// ...

reuse

Most of these things already exists for arrays, but are not generalized so that they can be used with structs. The main motivation to have these possibilities for structs is to access an array in different ways without creating (full) copies.

summary

• make it simpler to work with iterators and indexes
• hide implementation, but not memory allocation
• make the easy way the well performing way
• prevent excessive memory usage from simple syntax