Interactive Design Meeting 4/17/15

[kuhlenh]

Interactive Design Meeting 4/17/15

Design Team

The following individuals are members of the dream team:
Anthony D. Green
Chuck Stoner
Dustin Campbell
Kasey Uhlenhuth
Kevin Halverson
Kevin Pilch-Bisson
Lincoln Atkinson
Manish Jayaswal
Matt Warren
Tomas Matousek

Special Guests
Mads Torgersen

---

Agenda: Async/await, scripting scenarios continued

The purpose of this meeting was to clarify an async/await design issue for our Interactive team as well as continuing our discussion on scripting scenarios from the last couple weeks.

Quotes of the Day

"#seed is right up there with #impregnate"
"scripts manipulating other scripts?...it's incestuous really"

Async/Await in the REPL

Previously, we discussed that a user would not get the next prompt in the REPL while awaiting a Task. There was no technical reason why we couldn't let a user continuing typing, we just thought that it would fit the mental model of a REPL best if we used this design.

Note: this means that if a user types await f(); await g(); as a single submission, he will have a different experience than if he had typed them as separate submissions.

However, what if a user types an await, decides it's taking too long to complete, and tries to cancel the process? Currently, typing Ctrl+C actually kills the entire process, causing the user to lose all his previous context. There may be some solution around allowing a user to add a Cancellation Token as an argument, but we have decided to not prioritize this scenario right now (we'd rather have functional use of async/await and launch Interactive than wait to release until get this design right).

Scripting Scenarios continued...

init.csx file

In the last meeting we discussed allowing NuGet authors to include a init.csx file that could set up references and usings as well as provide some initialization code. However, we realized that our proposed method would require each init.csx-esque file to be language-specific. To remedy the pain of an author having to write 3 different scripts for C#, F#, and VB, we decided to tweak our design. NuGets can have a language-agnostic script (we will call it init.x for now) which will only contain references and usings. Authors can then additionally choose to write some initialization code in an init.csx, init.fsx, or init.vbx script. A developer's script will ignore #load-ed init scripts that are of a different language.

If a NuGet only has a C# init script (or no script at all), other authors can create separate NuGet packages that #load the C#-based NuGet and write their own init.vbx or init.fsx script to set up initialization code.

#u global directive

In our last notes there seemed to be some general confusion around the global directive #u. We started off by taking another look at our solution and we've made some changes:

To avoid having a different directive in every language, we are changing the name of #u to #import. It's use it also more intuitive this way.

The purpose of the #import directive is to allow script and NuGet authors to "export" usings. For example,

Given a .dll "MyLibrary" with namespace N₁ with a class C₁ and a script S₁ that references "MyLibrary":
The author of S₁ exports his using via "#import N₁". The author of a new script S₁₂ uses "#load S₁" to seed his script with context from S₁. S₂ can now directly access class C₁ without declaring any using directives.

This is especially helpful if a NuGet (e.g., WPF) has a lot of usings that will be difficult for users to remember and manually add at the top of each script.

public by default

We readdressed an old issue of whether or not everything in a script should be treated as public. Doing this allows users to #load scripts and directly access all the classes and variables. However, this means any 'temporary' variables a script author created to generate context will litter a developer's IntelliSense if he #load-ed that script.

We discussed several workarounds for this (e.g., placing temporary variables in a block, prefixing temporary variables with "_", allowing private), but in the end decided that this is not a top priority scenario at this time. Should problems come up in the future, we will revisit this problem. For now, using blocks to limit the scope of temp variables seems to be a good enough solution.

'into' keyword

A user can use the keyword into if he wants to isolate things from each other or wants to "discover" methods from a #load via IntelliSense.

For example, a user can do

#load a.csx into Foo
Foo.MethodA();

Instead of having into generate a "wrapper class," we think we should take the "wrapper identifier" provided by the user and essentially pre-pend it to every variable and class within the script. This would prevent creating a lot of duplicated code from arbitrary (transitive) #load-ing. A side-effect of this design is that a variable can be accessed via multiple qualified names (but we decided this is better than having multiple instances of the same variable).

These scenarios might help elucidate our design choices:

Scenario 1: Variable of the same name already declared (this will be an error)

Script a.csx

#load b.csx
var x = 1;

Script b.csx

var x = "x";

Scenario 2: Resolve ambiguity with a wrapper identifier

Script a.csx

#load b.csx into Stark
var x = 1;
//I can use my wrapper variable Stark to access b.csx's 'x' without error
var z = Stark.x;

Script b.csx

var x = "x";

Scenario 3: Access variable with nested dependency

Script a.csx

#load b.csx into Stark
var x = 1;
var z = Stark.x;
//I can access c.csx's variable 'y' via b.csx 
var w = Stark.y;

Script b.csx

#load c.csx
var x = "x";

Script c.csx

var y = 1;

Scenario 4: Multiple access points to variable

Script a.csx

#load b.csx into Stark
#load c.csx
var x = 1;
var z = Stark.x;
//I can access c.csx's variable 'y' via b.csx 
var w = Stark.House.y;
//I can access c.csx's variable 'y' via a.csx 
var v = y;

Script b.csx

#load c.csx into House
var x = "x";

Script c.csx

var y = 1;

Scenario 5: Pre-pend wrapper identifiers to prevent duplication

Script a.csx

#load b.csx into Stark
#load c.csx
var x = 1;
//These will reference the same 'x' as opposed to creating multiple instances of 'x'
var z = Stark.x;
var w = Lannister.x;

Script b.csx

var x = "x";

Script c.csx

#load b.csx into Lannister
var y = 1;

[RichiCoder1]

We readdressed an old issue of whether or not everything in a script should be treated as public. Doing this allows users to #load scripts and directly access all the classes and variables. However, this means any 'temporary' variables a script author created to generate context will litter a developer's IntelliSense if he #load-ed that script.

We discussed several workarounds for this (e.g., placing temporary variables in a block, prefixing temporary variables with "_", allowing private), but in the end decided that this is not a top priority scenario at this time. Should problems come up in the future, we will revisit this problem. For now, using blocks to limit the scope of temp variables seems to be a good enough solution.

Haha, I'm reminded of javscript. Are we going to need to bring over ES6 modules ;-)?

[tmat]

@RichiCoder1 No. We've had much better solution in C# since the beginning - block scopes and classes.

[adamralph]

Nice work guys. I really like the revised into proposal. I have some questions:

1. In var z = Stark.x; what is Stark? A local variable with an anonymous type?
2. Will this work for methods? Stark.Foo();?
3. What about classes defined in the loaded script?
4. Has the decision been taken to discard return values from scripts?

[khellang]

If Stark is a "wrapper class", I'd assume

1. To be a static class.
2. Yes, it'll be a static method.
3. They'd just be nested classes, so new Stark.Bar() would work.

I'm also curious about return values of submissions 😄

[adamralph]

Instead of having into generate a "wrapper class," we think we should take the "wrapper identifier" provided by the user and essentially pre-pend it to every variable and class within the script.

[glennblock]

Great stuff, it is exciting to see this convo happening in such a transparent fashion!

Comments....

• Lang agnostic init.x for namespaces sounds good. Agreed on the lang specific files so that packages can support multiple langs. It's more work on the part of the author, but it is also opt-in if they want to support multiple langs via script.
• #import - Renaming #u makes sense to me. As far as moving to #import, why not just call it #export as you described that it is exporting namespaces to the caller. It seems more accurate to call it #export in that case.
• into - As I mentioned earlier, I really like the idea of this and the scenarios make sense. I do echo a few questions above.
  • What will the target of into be? Is it a static class? Is it an instance? For example in script libraries because there is a wrapper, you can have static methods OR instance methods.
  • What if it there are nested classes defined in the csx file? Do these nested types also become available?
  • What about loose procedural code living in a loaded csx? I am assuming it will only get executed once no many how many times the csx file is loaded via #load in any of the scripts that are loaded within the REPL / or for that matter when not in the REPL.
• Public by default. I am on the fence on this. In the node world, you explicitly hang members that you want to make visible to the caller off of the exports object. In the same way, an alternative approach to public by default could be to have #export be used on members as an explicit way of saying, make this visible to the caller. Or better yet, introduce an "export" keyword ;-)

i.e.

//visible to the caller
export void DoSomething() {
...
}

//not visible
var foo;

[khellang]

in any of scriptcs that are loaded within the REPL

I ❤️ that your muscle memory makes you write scriptcs when you really want to write scripts 😉

[glennblock]

LOL! so true!

[glennblock]

@khellang aren't they synonymous? #ducks

[adamralph]

Ah yes, I also made a note to suggest #export instead of #import but forgot to mention it.

[MgSam]

Agree with #export rather than #import based on the given definition:

The purpose of the #import directive is to allow script and NuGet authors to "export" usings.

Calling it #import makes it seem like it should do the same thing as #load.

[kuhlenh]

@adamralph We are still working through implementation details on this (tbc in future notes)!

[RichiCoder1]

RichiCoder1 No. We've had much better solution in C# since the beginning - block scopes and classes.

Was more referring to #import and #export. Very reminicient of ES6's modules proposal. Javascript rough equivalents would look like:

#load Start from "b"
var x = 1;
//I can use my wrapper variable Stark to access b.csx's 'x' without error
var z = Stark.x;

#load Stark from "b"
#load "c"
var x = 1;
//These will reference the same 'x' as opposed to creating multiple instances of 'x'
var z = Stark.x;
var w = Lannister.x;

and to build on @glennblock's example

//visible to the caller
export void DoSomething() {
...
}

//not visible
var foo;

which might be also:

//visible to the caller
void DoSomething() {
...
}

//not visible
var foo;
#export { DoSomething } // This would probably choke on overloads

[khellang]

I think the F# approach (pretty much what's mentioned in the issue) is nice:

If no access specifier is used, the default is public, except for let bindings in a type, which are always private to the type.

So top-level variables etc. will be public by default, but you can specify private on declaration.

[paulomorgado]

Regarding #import I'd like to reinforce what others have already said. It makes no sense having an #import that exports. #export (or even export) makes a lot more sense.

I'm also in favor of explicit exports. That way producers of scripts don't need to be worried with polluting the caller script and the caller script doesn't need to be aware of the loaded script - just the public façade.

It was not clear to me if the package author needs to author a script for each supported language or if C# scripts can import VB scripts and vice versa.

[tmat]

@paulomorgado #import does not export. It imports the members of the target namespace or type into the current scope, just like using does. The only difference is that the scope is the compilation not the file. Very much like VB project-level imports.

[paulomorgado]

That's what I had understood before and was confused now.

Nevertheless, it makes sense to me to export only what I want to export. And in that sense, it might make sense to export those namespaces instead of importing them into the compilation.

In the end, it's the same thing looked at from different perspectives.

[paulomorgado]

Regarding async-await, what is the execution context of the REPL?

Is it the UI thread? Can one create forms and interact with them?

Is there any type of synchronization context?

Since this is an interactive environment, if the task is not completed, the user could be questioned on what to do. (a)wait or continue.

If the user chooses not to (a)wait, then the whole line will be "ignored". await is supposed to work with hot tasks, but what if they aren't?

[adamralph]

Another thing we discussed in our regular scriptcs core team meeting recently was multi-language capability. I.e. a way for host to allow:

// a.csx
#load b.vbx into Foo

This would require resolution from a file name extension to a language specific implementation, unless some other language feature were introduced to allow specification of the language implementation in code.

.csx and .vbx could have support out of the box, with pluggable support for other languages.

[paulomorgado]

Or have a convention on the first line of the script file to specify the scripting engine.

Or have an argument to #load to specify the scripting engine.

[adamralph]

Or have an argument to #load to specify the scripting engine.

That was what I had in mind with the 'language feature' suggestion. I can't really think of an elegant way of doing it though.

[tmat]

We don't plan on supporting multiple languages in a single script. Note that #load-ing of a script is similar to #include in C++, the target language has to be the same.

[adamralph]

Note that #load-ing of a script is similar to #include, the target language has to be the same.

I understand that that is the case with a plain old fashioned #load, but in the case of #load .. into (perhaps the directive should be something other than #load) we have an opportunity to supply and consume scripts as packages (effectively obsoleting scriptcs script libraries) and build something analogous to other script based platforms, e.g. node, Ruby, etc. This would be way smoother if each package has to be written only once in its language of choice, rather than having to include a version for each target language.

[tmat]

@adamralph #load into as currently designed isn't much different from #load. All it does is prefix the names of the members declared on top-level of the target script with the identifier specified in the into clause.

Correct me if I'm wrong, but I don't think you can import Ruby script from Python and vice versa (unless of course on DLR :))

[adamralph]

@tmat I asked for an explanation of the 'wrapper identifier' some time ago (#2167 (comment)) but no-one has answered yet.

The Ruby/Python case is completely different. Those are two distinct runtimes. In the .NET case we are dealing with a single runtime but multiple languages, a design tenet of .NET from day one. This is illustrated by C# and VB and has encouraged a huge number of other language implementations. For scriptcs we already have several other language engines, some serious (e.g. https://github.com/scriptcs-contrib/scriptcs-fsharp) and some fun (e.g. https://github.com/filipw/ScriptCs.Engine.Brainfuck).

I'm not suggesting that the scripting runtime necessarily needs to support this out of the box, but it would be of enormous benefit if we could hook into the language services (i.e. #load .. into) and provide it ourselves.

[tmat]

@adamralph I'm confused,
The design notes say exactly what I said above:
"Instead of having into generate a "wrapper class," we think we should take the "wrapper identifier" provided by the user and essentially pre-pend it to every variable and class within the script. This would prevent creating a lot of duplicated code from arbitrary (transitive) #load -ing."

Yes, .NET is a single runtime and the common target for languages is IL/metadata and common type system. We don't support creating projects that mix languages because that requires that a) the languages involved have semantically similar top-level declarations that mix well with each other b) there is a common interfaces for all compilers thru which they can cooperate.

That doesn't mean you can't interop between scripts written in different languages. The means for interoperability are going to be metadata references. The current extensibility enables host to interpret the content of #r directive as it wishes to. I can imagine the host having some registry of supported compilers/languages and then allow scripts to reference scripts written in another language via #r. For example, #r foo.vbx from C# script would compile foo.vbx into an in-memory assembly using VB compiler and pass it to the C# script as a metadata reference.

This is different from #load since #load provides source "merging".

[glennblock]

If you are pre-appending, this will mean all references will be
pre-appended as well right?

Ie if I have Foo() and Bar() and Foo calls Bar and is loaded into Baz then
the call to Bar will become Baz.Bar()? Is '.' even valid in a member name?

One thing I dislike about this that the wrapper brings is automatic
encapsulation of loose variables. Sure the variables are pre-appended so
they should not conflict, but there are still able to be accessed and
mucked around with. That is unless you explicitly create your own class
that you include the members in.

Glenn
On Sun, May 17, 2015 at 5:01 PM Tomáš Matoušek notifications@github.com
wrote:

@adamralph https://github.com/adamralph I'm confused,
The design notes say exactly what I said above:
"Instead of having into generate a "wrapper class," we think we should
take the "wrapper identifier" provided by the user and essentially pre-pend
it to every variable and class within the script. This would prevent
creating a lot of duplicated code from arbitrary (transitive) #load -ing."

Yes, .NET is a single runtime and the common target for languages is
IL/metadata and common type system. We don't support creating projects that
mix languages because that requires a) that the languages have structurally
and behaviorally similar top-level declarations that mix well with the
other languages b) there is a common interfaces for all compilers thru
which they can cooperate.

That doesn't mean you can't interop between scripts written in different
languages. The means for interoperability are going to be metadata
references. The current extensibility enables host to interpret the content
of #r directive as it wishes to. I can imagine the host having some
registry of supported compilers/languages and then allow scripts to
reference scripts written in another language via #r. For example, #r
foo.vbx from C# script would compile foo.vbx into an in-memory assembly
using VB compiler and pass it to the C# script as a metadata reference.

This is different from #load since #load provides source "merging".

—
Reply to this email directly or view it on GitHub
#2167 (comment).

[tmat]

@glennblock Yes. '.' is valid in metadata member name. We can also mangle the metadata name arbitrarily if needed.

Correct, top-level variables are also accessible thru dotted name. There are several options how to make them truly local to the script - declare them in a block, method, class, lambda, etc.

[tmat]

@glennblock If the above options for script variable encapsulation turn out to be insufficient we can add another one (e.g. some modifier). For now we think they are good enough.

[glennblock]

Thanks. The lack of encapsulation may not be an issue as if one needs to
they can have a class.
On Sun, May 17, 2015 at 7:38 PM Tomáš Matoušek notifications@github.com
wrote:

@glennblock https://github.com/glennblock If the above options for
script variable encapsulation turn out to be insufficient we can add
another one (e.g. some modifier). For now we think they are good enough.

—
Reply to this email directly or view it on GitHub
#2167 (comment).

[glennblock]

Class, lambda etc
On Sun, May 17, 2015 at 10:35 PM Glenn Block glenn.block@gmail.com wrote:

Thanks. The lack of encapsulation may not be an issue as if one needs to
they can have a class.
On Sun, May 17, 2015 at 7:38 PM Tomáš Matoušek notifications@github.com
wrote:

@glennblock https://github.com/glennblock If the above options for
script variable encapsulation turn out to be insufficient we can add
another one (e.g. some modifier). For now we think they are good enough.

—
Reply to this email directly or view it on GitHub
#2167 (comment).

[adamralph]

@tmat thanks for the info. I hope you understand why some explanation was/is still required. The comment history shows I'm not the only one who requires it. 😉

I still don't understand how this is going to work. Although . is valid in a member name, the only way to refer to the member in C# is by using a unicode escape:

Foo\u002EBar();

[tmat]

@adamralph The name in source does not necessarily need to be the same as metadata name. In source you'll refer to it as Foo.Bar(); in metadata it will be a method Foo.Bar on type <Script>.

[adamralph]

But I can't write Foo.Bar(); in C# where Foo.Bar is a member name. It won't compile. The best I can do is Foo\u002EBar(). Am I missing something?

[glennblock]

@admaralph they own the compiler, they can make it do whatever they want.
:-)
On Sun, May 17, 2015 at 11:56 PM Adam Ralph notifications@github.com
wrote:

But I can't write Foo.Bar(); in C# where Foo.Bar is a member name. It
won't compile. The best I can do is Foo\u002EBar(). Am I missing
something?

—
Reply to this email directly or view it on GitHub
#2167 (comment).

[paulomorgado]

@glennblock, just because they can, doesn't mean they should.

I can understand avoiding the collision of a #load with existing classes and thus not generating a wrapper class.

But if the collision would exist, then a name collision will still exist.

Given a s.csx script file with:

int GetMeAnInt()
{
    ....
}

loaded with:

#load s.csx into X

and a class:

public static class X
{
    public static int GetMeAnInt()
    {
         ...
    }
}

which one would X.GetMeAnInt() in the script refer to?

[tmat]

@paulomorgado You'd get an error that X has already been declared at the declaration of class X.

[adamralph]

Joking aside, I did wonder if a new language feature was being proposed here. I can't imagine that allowing . in an identifier would be an easy, if at all possible. let alone desirable, thing to achieve.

Indeed, from the original proposal above:

Scenario 2: Resolve ambiguity with a wrapper identifier

//I can use my wrapper variable Stark to access b.csx's 'x' without error
var z = Stark.x;

Wrapper identifier, wrapper variable - what is this thing?

Assuming the compiler isn't going to allow Foo.Bar any time soon, where Foo.Bar is an identifier, we're back to my original question:

1. In var z = Stark.x; what is Stark? A local variable with an anonymous type?

[paulomorgado]

@tmat, the class X could have been declare before. But that would only result in a different error.

I'm still missing the point of not having a wrapper class instead.

[glennblock]

Agreed the identifier gives me an uneasy feeling / feels a big hacky.

[CyrusNajmabadi]

Closing out. We don't need active issues for meeting nodes.