Local scope in Julia

[B-rando1]

I've been working some more on the Julia render, and I discovered that it's stricter with local scope than other languages. A minimal example is the following:

a = 0

while a < 5
    println("a = $a")
    a += 1;
end

This throws an error, because the while loop introduces a 'soft local' scope (see here for details), and local scopes cannot refer to global variables without explicitly stating that they are. There are two ways to get the intended behaviour:

1. Using global declaration

a = 0

while a < 5
    global a
    println("a = $a")
    a += 1;
end

2. Using a function to define functions in a local scope

If variables are defined in a local scope, then nested local scopes are free to access them.

function main()
    a = 0

    while a < 5
        println("a = $a")
        a += 1;
    end
end

main()

Analysis

Both options have difficulties.

• Option 1 is more straightforward, and probably slightly more idiomatic as well. We might be able to use static vs dynamic to keep track of which variables are global vs local. I think that this wouldn't work for any logic occurring in the global scope, though.
• Option 2 would make it easy to get rid of global variables in program files, but modules would still have global variables to deal with.
• I think if we combine both options, there might be something there. This will require a bit more investigation into what information GOOL stores about variables and how it uses that information. It will also require understanding how State can be used to add global declarations to the top of a scope, similar to how we do imports currently.

[balacij]

It sounds like option 1 is the 'right' choice in the sense that if we go with option 2, we wouldn't be able to generate purely executable Julia scripts. Am I understanding this correctly?

[B-rando1]

It sounds like option 1 is the 'right' choice in the sense that if we go with option 2, we wouldn't be able to generate purely executable Julia scripts. Am I understanding this correctly?

I'm not quite sure what you mean. As long as we add main() to the bottom of the file, it would still run with the same behaviour, i.e. an outside user wouldn't be able to tell that the program is inside a function.

If you mean that scripts require a main function and won't run in the global scope, then you are correct. This might conflict with #3512, but at the same time it's no different than the C++ target. Let me know if you think this is something we should avoid.

My bigger concern with option 2 is that it doesn't help external modules that define global variables, for example here. That isn't a perfect example, as the Python version uses a class, and neither target seems to actually use the Constants file 🤔. It gets the point across anyway, that even with option 2 we still could have cases where we need to separate the representation of global and local variables in GOOL.

The only benefit of still doing option 2, is that it would allow us to safely generate the global declarations at the function level rather than the level of while, for, function, etc. As an example:

# Option 1
a = 1 # Needs to be global for some reason

while a < 5
    global a # Need to declare at top of while loop, because there's no function to declare at
    a += 1
    println(a)
end

# Option 2
a = 1 # Needs to be global for some reason

function main()
    global a # can safely put this at the top of the function, which might be easier to implement
    while a < 5
        a += 1
        println(a)
    end
end

main()

To be honest I'm not completely sure, but my guess is that we have more infrastructure in place to put global declarations at the top of functions than we do for the top of while-loops, etc.; so putting the main script inside a function might make the global declarations easier to implement.

I hope this clears up my reasoning.

[balacij]

We briefly spoke about this in person, but I think I still stand with option 1. Option 1 better captures the intent of a 'unmodular, unbundled'-style program (e.g., one-shot/use-style scripts, as in #3512), unlike Option 2, which would force some amount of modularization. The issue with the comparison to C++ is that it is not a scripting language first, unlike Python and Julia. That being said, 'unmodular, unbundled'-style programs are rather unserious for large projects, so I think it's a nice feature to support for show and tell, but not necessarily something we would expect to seriously use for any non-trivial project.

[B-rando1]

Sounds good @balacij. I'm not exactly sure, but I think we'll have to add a new field for variables denoting whether they're locally or globally defined. I was hoping that we'd be able to reuse Scope or Permanence from ClassInterface.hs, but those are only associated with class attributes, not general variables.

Here are my current thoughts, then:

• Rename the existing ScopeSym to VisibilitySym, since it has to do with public vs private, and the name's confusing.
• Create a new ScopeSym that has to do with local vs global scope.
• Integrate this into variables and constants.
  • Variables need it for reasons stated above.
  • Constants don't need it specifically for scoping issues. Julia is apparently fine reading from global variables without making that explicit; it's only when you write to them that it assumes you want a new local variable. The reason constants need it is because Julia only allows constants in the global scope, so local constants will have to be variables in Julia. I'll create a separate issue for this, as it has more nuances than this.
  • If we decide to do classes in Julia, this would be tricky, and would be tied to decisions we make about representations about static vs dynamic variable representation.

I think this would be worth briefly discussing in #3815.

[JacquesCarette]

The conclusions in the last post above (about VisibiliySym and ScopeSym make sense. We do want script-like programs with global variables as well as proper programs (and libraries) that don't. So we need to know the difference.

[B-rando1]

I'm trying to make the changes I proposed with adding ScopeSym. I have everything else working in drasil-gool (without Julia so far), but I'm getting weird errors in the C++ renderer.

For context, I added ScopeSym to ClassInterface.hs with this definition:

class ScopeSym r where
  type Scope r
  global :: r (Scope r)
  local  :: r (Scope r)

Then I added it as a dependency to VariableSym, and modified the type signatures for 4 of its methods from:

Drasil/code/drasil-gool/lib/GOOL/Drasil/ClassInterface.hs

Lines 126 to 129 in ea4cef9

	var :: Label -> VSType r -> SVariable r
	staticVar :: Label -> VSType r -> SVariable r
	constant :: Label -> VSType r -> SVariable r
	extVar :: Library -> Label -> VSType r -> SVariable r

to:

  var          :: Label -> VSType r -> r (Scope r) -> SVariable r
  staticVar    :: Label -> VSType r -> r (Scope r) -> SVariable r
  constant     :: Label -> VSType r -> r (Scope r) -> SVariable r
  extVar       :: Library -> Label -> VSType r -> r (Scope r) -> SVariable r

Because none of the current renderers use the new data, it was pretty easy to get most of it to the point where it compiles, even if there are some things I'll need to go back to one Julia is in the picture. With C++ though, I'm having issues with the Pair instances. I changed the definitions of the above methods from:

Drasil/code/drasil-gool/lib/GOOL/Drasil/LanguageRenderer/CppRenderer.hs

Lines 270 to 273 in ea4cef9

	var n = pair1 (var n) (var n)
	staticVar n = pair1 (staticVar n) (staticVar n)
	constant n = pair1 (constant n) (constant n)
	extVar l n = pair1 (extVar l n) (extVar l n)

to:

  var n = pair2 (var n) (var n)
  staticVar n = pair2 (staticVar n) (staticVar n)
  constant n = pair2 (constant n) (constant n)
  extVar l n = pair2 (extVar l n) (extVar l n)

Basically, I just switched the pair1s to pair2s, since each function now has an extra argument.

For some reason though, this doesn't typecheck. I get a bunch of errors, but here's what I get for var:

/home/brandon/Research/Drasil/code/drasil-gool/lib/GOOL/Drasil/LanguageRenderer/CppRenderer.hs:275:11: error:
    • Couldn't match kind ‘*’ with ‘* -> *’
      When matching types
        p :: (* -> *) -> (* -> *) -> * -> *
        transformers-0.5.6.2:Control.Monad.Trans.State.Lazy.StateT :: *
                                                                      -> (* -> *) -> * -> *
      Expected: VSType (p CppSrcCode CppHdrCode)
                -> p CppSrcCode CppHdrCode (Scope (p CppSrcCode CppHdrCode))
                -> SVariable (p CppSrcCode CppHdrCode)
        Actual: PairState GOOL.Drasil.State.ValueState p TypeData
                -> PairState GOOL.Drasil.State.ValueState p b0
                -> PairState GOOL.Drasil.State.ValueState p VarData
    • In the expression: pair2 (var n) (var n)
      In an equation for ‘var’: var n = pair2 (var n) (var n)
      In the instance declaration for
        ‘VariableSym (p CppSrcCode CppHdrCode)’
    • Relevant bindings include
        var :: Label
               -> VSType (p CppSrcCode CppHdrCode)
               -> p CppSrcCode CppHdrCode (Scope (p CppSrcCode CppHdrCode))
               -> SVariable (p CppSrcCode CppHdrCode)
          (bound at lib/GOOL/Drasil/LanguageRenderer/CppRenderer.hs:275:3)
    |
275 |   var n = pair2 (var n) (var n)
    |           ^^^^^^^^^^^^^^^^^^^^^

/home/brandon/Research/Drasil/code/drasil-gool/lib/GOOL/Drasil/LanguageRenderer/CppRenderer.hs:275:18: error:
    • Couldn't match type ‘CppSrcCode’
                     with ‘transformers-0.5.6.2:Control.Monad.Trans.State.Lazy.StateT
                             s0 Data.Functor.Identity.Identity’
      Expected: SrcState GOOL.Drasil.State.ValueState TypeData
                -> SrcState s0 b0 -> SrcState GOOL.Drasil.State.ValueState VarData
        Actual: VSType CppSrcCode
                -> CppSrcCode (Scope CppSrcCode) -> SVariable CppSrcCode
    • In the first argument of ‘pair2’, namely ‘(var n)’
      In the expression: pair2 (var n) (var n)
      In an equation for ‘var’: var n = pair2 (var n) (var n)
    |
275 |   var n = pair2 (var n) (var n)
    |                  ^^^^^

/home/brandon/Research/Drasil/code/drasil-gool/lib/GOOL/Drasil/LanguageRenderer/CppRenderer.hs:275:26: error:
    • Couldn't match type ‘CppHdrCode’
                     with ‘transformers-0.5.6.2:Control.Monad.Trans.State.Lazy.StateT
                             s0 Data.Functor.Identity.Identity’
      Expected: HdrState GOOL.Drasil.State.ValueState TypeData
                -> HdrState s0 b0 -> HdrState GOOL.Drasil.State.ValueState VarData
        Actual: VSType CppHdrCode
                -> CppHdrCode (Scope CppHdrCode) -> SVariable CppHdrCode
    • In the second argument of ‘pair2’, namely ‘(var n)’
      In the expression: pair2 (var n) (var n)
      In an equation for ‘var’: var n = pair2 (var n) (var n)
    |
275 |   var n = pair2 (var n) (var n)
    |                          ^^^^^

From the first error it seems like it might have something to do with Type Families, but I'm not sure. I've tried a bunch of stuff to get it working, but it always gives the same or similar errors. Does anyone know what might be wrong here?

[JacquesCarette]

The problem is that pair2 wants to create a function of 2 state arguments from functions with 2 arguments each. Basically: I don't think this is the right combinator to use.

I suspect that the "right" fix will be to instead add the argument here

var          :: Label -> r (Scope r) -> VSType r -> SVariable r

and then change your calls to

var n sc = pair1 (var n sc) (var n sc)

[B-rando1]

I just realized now, I'm not sure if it actually makes sense to have Scope as something we give GOOL. The reason for this is the main function. In Java, C#, and C++, the main function is a function, i.e. local scope; but in Python, Swift, and Julia, the main function is a script, i.e. global scope. I wish I had caught that earlier.

As far as I can tell, each language has more or less the same semantics of what is 'global' - i.e. if it's inside a function, loop, class, etc. it's local; if it's outside everything it's global. That is good, as I think it means expressing scope consistently across GOOL is possible.

The issue is that different targets enter scopes differently. The obvious case is the main function. That might be the only case we have to worry about, but that's enough.

I can think of three ways we can handle this:

• We could generate a different set of GOOL code for languages where the main function is a function and languages where the main function is a script. I think this should be easy to do. The downsides are that it avoids the issue, so if we find other cases where scoping is not consistent between languages, we'd need to generate different GOOL code on that condition, and so on.
  • Also, we still need drasil-code to know what scope it's in, as entering a new scope for a loop, etc. would cause the generated GOOL code to be different. Doing the next option would simplify drasil-code in that respect.
• The other option is to make Scope internal-only, and have variables assign themselves a scope. This sounds complicated, but it might be feasible.
  • We can hopefully use State to keep track of what scope we're in, and when a new variable is declared (using varDec, varDecDef, etc.) we look up the state, and assign local or global based on what we find.
  • Then to reference variables later, we'd need a hashmap of variable-name : Scope to keep track of the scope of each variable. This is the part I'm not sure about, as it would need to handle name shadowing.
• Lastly, we could ignore the issue and assume that in all languages that care about scope (i.e. Julia currently), the main function is in the global scope. This technically shouldn't cause problems for now, but it means lying to half of the renderers about the scope of the main function. Besides that, if we ever add a language that cares about scope and has its main function in a function, we'll need to deal with the issue then.

I'm not really sure what to do here. Option 1 is relatively simple but not ideal. Option 2 feels to me like how it 'should' be done, but it's hard for me to guess if it's simply difficult or if it's completely infeasible for us to do. Option 3 might be the simplest, but it feels 'wrong'.

---

Edit:

I'm also just not sure if it makes sense at all to have the var keyword require a Scope argument. The reason for this is that often in GOOL we have a helper function for a variable name, e.g. i = var "i" int, and then we use that later on, not necessarily in the same function or even in the same file. So either way we need var to dynamically get its scope from the context into which it is placed.

[JacquesCarette]

If we want "scope-polymorphic variables" (for their convenience) then I think we should introduce 'pre-variables' for that purpose. These would need to have their scope specified 'later' when they are used in a known context.

More generally, the notion of scope should be used with more intent. In other words, rather than tying it to the language, we should tie it to "what we're trying to say" - and then render than appropriately for each language.

The important concepts that need to drive the 'scope' notion are those of scripts, libraries and (modular) programs.

So 'scope' arises in the middle of our translation from the intended code, in an intended form, for a specific programming language.

[B-rando1]

Hmm, I'm not sure if I understand what you're saying.

Do you mean that instead of assigning a variable a local or global scope, we assign it a script, function, loop, etc. scope (the granularity might be different, but hopefully you get what I mean)? Then the target renderer would get to choose which scopes are treated as global, and which as local.

If that's what you mean, then that certainly would help simplify things. drasil-code would still need to know when it's generating something inside a loop, etc., but that seems like a much simpler problem than the one I described yesterday.

Is that what you're getting at? Or did I misinterpret what you said?

[JacquesCarette]

Scripts and functions and loops should know what scope they need their variables to be, so they should control the scope.

I was originally thinking that they would choose between local or global. But your idea might be even better.

[B-rando1]

I've been trying to wrap my head around how to get started again on this issue.

Next Steps

It sounds like the first step is to take the work I've done with ScopeSym, and replace its local and global with something more granular.

• Originally I was thinking we'd want a scope for every body that introduces a new scope, e.g. global, main, function, loop, class/struct, etc.
• I think that might be a bit overkill though. We might only need 3: global, main, and local.
  • global in this case meaning 'true global', separate from main, as in Constants.cpp.
  • We could split local up into softlocal and hardlocal to match Julia's classification, but my understanding is that the scopes act pretty similarly. Plus, it should be fairly easy to make the classification more granular later if we need to.

I'm not sure exactly where to go once we have that working correctly. I think we should be able to follow addExceptions to get started with adding global declarations to the top of a scope. The thing I'm not sure how to handle is that in GOOL, the datatypes used for the various constructs that introduce a new scope are quite heterogenous. Functions are MethodSym, classes are ClassSym, and loops and try/catch are MSStatement. This will make it harder to get the types working correctly.

• Also, I think we might need to add a new datatype to represent constructs that introduce a local scope but are currently represented with something generic like MSStatement. The problem is that MSStatement can be local or global, and as I mention below we need to know when and where a local scope is introduced.

Just to zoom out a bit, our addGlobalDecls function that we create should have the following behaviour (for Julia):

• Work out whether we're in a global, main, or local scope.
  • If the current scope is global or main, do nothing.
  • If the current scope is local, then find the outermost local scope, and add the global x declaration at the top of that scope.

As far as I can tell, this is the behaviour we want to have. I did a few tests of weirder situations, and this way seems to hold up.

Questions

• Does the overall plan sound good? Taking a wrong step could be costly, and if we're not sure about anything it would be better to leave this on the backburner.
• Do you agree that following addExceptions with some modifications will work? I don't have much experience with that kind of thing, so I don't know how smoothly it'll go.

[JacquesCarette]

Is this issue still 'live'? It feels like there has been some progress on it.

[B-rando1]

Right, thanks for reminding me. I've been meaning to come back to this. It is still 'live', as the Julia renderer needs to use the scope of variables in order to add global declarations. Currently HelloWorld.jl is the only generated code that's affected, but we definitely should get it working correctly. Here is the part that needs to be fixed:

Drasil/code/stable/gooltest/julia/HelloWorld/HelloWorld.jl

Lines 185 to 188 in 6c4ac76

	while a < 13
	println("Hello")
	a += 1;
	end

Because a is a global variable, we need to declare global a at the top of the loop. Currently it sees a += 1 and assumes that a is a local variable, then crashes because we're reading from a to set its value.

As far as I know, there are still two things left to do:

• Continue giving the proper scope to variables in drasil-code. It's been a while since I've worked on this, but I was slowly moving up from the functions that directly create variables, to the functions that call those functions, etc., until I find the functions that have a 'concrete' scope. Then I can pass local, mainFn, or global down to create a variable in the proper scope.
  • I was thinking that with how much scope is passed around in drasil-code, it might make sense to add it to DrasilState. Maybe we can revisit this once we have it working correctly.
• We also need to update the Julia renderer to take advantage of the scope given. I'm not exactly sure how to start with that. My previous comment laid out what we need to do, and some of the roadblocks in our way.

[B-rando1]

I finished the first point, and created a PR for it (#3888).

To get started on the second point, I noticed that Julia is pretty lenient about where and how often you declare globals. This means that we could just add a global declaration directly above every use of a global variable. That would be massively overkill and not look very idiomatic, but it would get the code working while we work on a proper fix.

[JacquesCarette]

See my comment on #3888 . The idea is right, but since everything is in the State monad already, we should use that instead of explicitly weaving things around.

[B-rando1]

I'm just trying to think through how we handle scope from drasil-code to drasil-gool.

• In drasil-gool, we need to tell it what scope a variable belongs to, every time the variable is used. This is so that whenever a variable is assigned to in Julia, we can check if the scope of the variable is different from the scope of the statement, and add a global declaration if it is.
• Currently, drasil-code keeps track of what Scope of GOOL code is currently being generated. This allows us to create a variable in the right scope.

The issue is that in drasil-code, variables (to my knowledge) don't have any memory of their scope. This means that every time we use a variable, we have no way of knowing what scope it was created in. In the current implementation, drasil-code just tells GOOL that the variable belongs to the current scope. This is not always the case, of course - exactly when we need it to work right is when it doesn't.

Potential solutions:

• Make drasil-code smarter. Maintain a map from variable name to variable scope into GenState, so that we can look it up and use the proper scope.
• Make drasil-gool smarter.
  • Add a map from variable name to variable scope into GOOLState.
  • Only require a Scope when using varDec, varDecDef, etc., and use those statements to update the map.
  • The rest of the time, when a variable is used just look up its scope in the map.

Of these two solutions, I'm leaning towards the second one. It seems to me like if we tell GOOL the scope of a variable when it is declared, we shouldn't need to keep reminding GOOL what its scope is. This would also lighten the load on drasil-code, so it wouldn't need to keep track of yet another thing.

[B-rando1]

My last comment is still a good summary of where we're at currently, if we don't want to do anything too drastic right away. With that said, I had another thought.

• In my comment above as well as in Used the Scope in generated Julia code #3903, I mentioned that while it is technically enough for us to create some way for variables to know what scope they belong to, the code we can generate with that information alone is not very idiomatic.
  • The reason for this is that if GOOL doesn't know what scope it's currently generating, it needs to do something so simple that it'll work anywhere - i.e. adding a global declaration to every single line assigning to a global variable. That's even if we're already in the global scope, even if we've already declared that variable as a global within the current local scope. Julia doesn't have a problem with us doing it, but it doesn't look very good.
• Because of this, what we really need is for GOOL to keep track of what scope it is currently generating. It can't do that right now, because some constructs such as loops are currently represented by MSStatements, which are too generic to denote scope in their current form.
• The thing is though, once GOOL knows what scope it is currently generating in, it doesn't need us to tell us what scope variables belong to - it can just look up the current scope whenever a varDecDef is called.

I'm sorry that I keep bringing up different ideas - it's just that the problem keeps proving to be more complicated than I thought it was. I guess we have three options now:

• Options 1 and 2 can still work, temporarily. We have a lot of the work in place now for one of them to work, and they should be enough to get Helloworld.jl working. The only issue is that the more permanent fix would make a lot of the changes obsolete.
• We could alternatively undo our work towards options 1 and 2, and focus on the GOOL-only solution that I mentioned here. This would cut out unnecessary steps, but it would take longer to get HelloWorld.jl working.

I'm not quite sure where we should go. In one way option 2 is at least a step in the right direction from our current path. We're running low on time, but we might have time to get it working.

Whatever we do, I hope I can leave drasil-code in a better state than it's currently in. I submitted one or two too many PRs before realizing we had to pivot, so there's currently a large number of -- TODO: get scope from ...'s floating around in main. It would be nice to remove those at least.

[B-rando1]

Just another update: I did a test run of option 2, and it seems doable. I got it generating the same (I think at least; I haven't compared them) code as #3903.

There was a weird issue I ran into. When looking up the scope associated with a variable name for Projectile, it crashes if I don't add a check. For some reason, somewhere in there Julia gets a call to assign a variable with an empty name the value void. Since the empty string isn't in GOOL's scope map, it throws an error when we try to look it up.

• I did a lot of digging, but I'm having trouble figuring out where that's coming from. It seems that it might be coming from one of Julia's implementations, as my similar tests with Java and Python didn't seem to show it happening.
• I will hopefully be able to track down what's causing that empty assign, but for now I found that just checking for the empty string before looking it up is enough to make it work.

[B-rando1]

I just thought I'd give an update, since it's unlikely that I'll get further this summer.

We implemented Option 2 and merged the changes in #3899. This means that the generated Julia code is fully-functional, which is a great step.

What's left is to implement Option 3. To recap:

• Currently, we're keeping track of what scope of program is being generated in drasil-code (with currScope in DrasilState), and keeping track of what scope is tied to a variable in drasil-gool (with varScopes in MethodState). This is sufficient for drasil-code to tell a variable what its scope is at declaration, and to add a global declaration when a global variable is assigned a value.
• If we move the part that keeps track of the scope being created from drasil-code to drasil-gool, there will be a few advantages:
  • We could reduce the global declarations to only those that are needed. Currently there are no less than 77 global declarations in the generated Julia code, and exactly 1 of those is required for the code to run properly. The issue is that GOOL doesn't know what scope it is generating, so it has to assume that the global declaration is needed every time a global variable is assigned to. In reality, it's only needed when a global variable is assigned to in a local scope. By keeping track of the current scope in GOOL, we could be more conservative with our global declarations.
  • We could also use that scope to have GOOL figure out each variable's scope completely on its own. This would simplify drasil-code as well as the GOOL tests, which would be nice.
  • This would also give a more accurate scope for temporary variables that GOOL generates. Currently when GOOL creates a variable in an unclear scope, it needs to 'piggy-back' on an existing variable to get its scope. It's the best we can currently do and shouldn't cause problems per se, but it's less robust than it could be.

One last thing to think about is to move global declarations from being inline with variable assignment to being at the top of the outermost local scope. E.g.

a = 0
b = 0
while a < 10
  while b < 10
    println("a = $a; b = $b")
    global b += 1
  end
  global b = 0
  global a += 1
end

could be rewritten as:

a = 0
b = 0
while a < 10
  global a, b
  while b < 10
    println("a = $a; b = $b")
    b += 1
  end
  b = 0
  a += 1
end

This would need to wait at least until the above changes are made and might require other changes, but I personally think it looks a bit better.

[JacquesCarette]

Eventually, what we'll probably want is for both drasil-code and drasil-gool to keep track of the current scope and tag each variable with the scope it belongs to. Having more information around is often a good thing (unless the information costs too much - but that's not the case here).

Yes, the rewritten code is nicer.