I/O Design Meta-Issue

[mppf]

This issue is just meant to list and track I/O issues.

Priority Group 1

• Improve Channel Formats Improve Channel Formats #7846
• I/O: When should output be machine readable? I/O: When should output be machine readable? #7952
• I/O plugins should be written in Chapel I/O 'plugins' should be written in Chapel #8315

Priority Group 2

• when should an array be resized when it is read? when should an array be resized when it is read? #5053 (also Support variable length arrays in JSON loader #7524)
• error handling: returning errors from writeThis error handling: returning errors from writeThis #7261

Priority Group 3

• buffer and bytes should work with channel reading from string, bytes, or buffer #8314
• some methods on channel begin with _ and we might consider renaming 'advance' to 'skip'
• mark / _mark return an error code but should throw Make channel.mark() return the offset and throw if there is an error #13312
• check for I/O functions that would be more usable if they returned the result (since the return was typically used for error codes but now there are errors for that)
• allow passing DefaultRectangular to channel.readBytes`; possibly replace readBytes with something else.
• allow reading classes with allocation, instead of only reading into allocated classes.
• Supporting I/O of class instance cycles? Supporting I/O of class instance cycles? #7951
• writeThis and inheritance How should compiler-generated writeThis routines interact with user-defined ones in the presence of inheritance? #7118

Priority Group 4

• Remote file local channel Remote file local channel #7953
• parallel file.lines() parallel file.lines() #4959
• Spinwaiting in I/O lock can significantly degrade performance under qthreads Spinwaiting in I/O lock can significantly degrade performance under qthreads #7748 (not expected to impact API)

[bradcray]

Related: @benharsh has been looking at I/O overheads in the context of the revcomp benchmark and expects to put together a list of I/O issues / wishes in response to that exercise, as I understand it.

[benharsh]

Question: does anything in the release/examples/ directory rely on the functionality we intend to deprecate?

[benharsh]

My take on a rough prioritization of these tasks (grouped by coarse prioritization):

1. Improve/add channel formats
   Rationale: probably deprecates "%j", breaking existing code
2. machine readable output
   Rationale: behavior-changing, possibly significantly
3. IO plugins written in Chapel
   Rationale: deprecates HDFS, CURL from stdlib

---

4. Return errors from writeThis
   Rationale: Future programs may break if they do not handle errors.
5. Arrays resized when read (or, "how to read an unknown number of things")
   Rationale: Useful idiom. Note: May depend on whether we keep mutate-through-array functionality.

---

6. Buffer/bytes/strings with channels
   Rationale: Useful idiom.
7. Methods beginning with underscore
   Rationale: Solidify API
8. DefaultRectangular and readBytes
   Rationale: may break existing code
9. allocate-as-read classes
   Rationale: Useful idiom.
10. class instance cycles
    Rationale: Annoying for users
11. writeThis inheritance
    Rationale: Annoying default behavior, but can be overrided

---

12. Remote file, local channel
    Rationale: correctness, productivity > performance. Though probably a common idiom.
13. parallel file.lines()
    Rationale: correctness, productivity > performance. Though this may have design repercussions.
14. Spinwaiting under qthreads
    Rationale: correctness, productivity > performance

[mppf]

thanks @benharsh - I've reordered the issue description to reflect your suggestion.

[e-kayrakli]

It might be good to keep a record of code style issues with the IO module here. See #14646

[lydia-duncan]

I recorded some documentation tasks related to IO on #14395, should I link them here as well?

[mppf]

I think this issue should focus on things that fit into the API stabilization topic. Also this particular issue has things to address in a sorted order, so it might not make sense to try to add a big variety of small things to it.

Perhaps linking other issues to the epic Stabilize core I/O routines #8639 is the right move.