Function defined in global scope has "hard scope", function in any block has "semi-soft scope"

[mauro3]

In Julia, function-blocks and type-blocks introduce a hard scope (see #9955 for an explanation of hard and soft scope). However the hard scope is only introduced if not nested inside another block. If nested then they become "semi-soft scopes". Example:

foo1 = 1
# hard scope, assignment makes a new local
# (function and type)
f1() = foo1 = 5
@assert f1()==5 # 5
@assert foo1==1 # foo1 not changed
# soft scope: assignment to global variable if it exists
# (while loops, for loops, try blocks, catch blocks, let blocks)
# (note begin-end block do not make a new scope)
[foo1 = 5 for  i=[1]]
@assert foo1==5 # foo1 now changed

# functions inside any block have hard-scope with respect to variables
# defined outside the block, but soft-scope with  respect to variables
# defined inside the block.  Thus semi-soft scope

foo1=1
let
    foo2=1
    f2() = (foo1=6; foo2=6)
    @assert f2()==6
    @assert foo1==1 # hard scope
    @assert foo2==6 # soft scope
end

begin # works with begin-end blocks too even though they should not affect scope
    foo2=1
    f3() = (foo1=7;foo2=6)
    @assert f3()==6
    @assert foo1==1 # hard scope
    @assert foo2==6 # soft scope
end
# as begin-end block are not scope blocks, so foo2 actually leaks out
@assert foo2==6

# semi-soft scope applies to type-blocks as well but only works with
# begin-end blocks as type definitions not allowed inside a local
# scope.

foo1=1
begin 
    type A
        foo1=7
        foo2=6
        a
    end
    @assert foo1==1 # hard scope
    @assert foo2==6 # soft scope
end

Some thoughts:

• Semi-soft scope is highly confusing and hard to explain in the documentation (see improve documentation of soft vs. hard scope #9955)
• Semi-soft scope leads to errors as copying a function into a block will make it behave differently. Examples: wrapping above example in a block will make it fail and Support writing to variables in outer, non-global scope #5331 (comment).
• that begin-end blocks and if-end blocks affect scoping is even more confusing as otherwise they do not influence scoping.

I think the main reason for semi-soft scope is to make certain types of closures possible. An idiomatic example from base/multi.jl:

let next_pid = 2    # 1 is reserved for the client (always)
    global get_next_pid
    function get_next_pid()
        retval = next_pid
        next_pid += 1
        retval
    end
end

This relies on the soft-scoping of next_pid. If functions were to get totally hard scope, some way to refer to outer variables but not globals would be needed (see #5331), as writing global next_pid would refer to a next_pid in the scope outside the let-block:

let next_pid = 2    # 1 is reserved for the client (always)
    global get_next_pid
    function get_next_pid()
        nonlocal next_pid
        retval = next_pid
        next_pid += 1
        retval
    end
end

Considering that the current example already needs one global statement, a further nonlocal statement is not bad in exchange for a lot more consistency.

To discuss:

• What should happen for sure it that begin-end and if-end block should not make contained functions to use a semi-soft scope. I think this is a bug. If not, what are the use cases?
• Further what I think should happen is:
  • function and type-blocks should always have a hard-scope irrespective of nesting.
  • then we need a way to refer to outer scopes (see above example, Support writing to variables in outer, non-global scope #5331 and a python 3 example)
• document it, which is issue improve documentation of soft vs. hard scope #9955

This has come up before: #423 (comment), #5331 and #7234. However, in those issues it is not as clearly documented as here, thus a new issue.

[ScottPJones]

👍 I hadn't realized just what was going on, but after reading this, I see that I was bitten by these differences many times already.

[elextr]

@mauro3 I'd describe it differently with the simple rule:

"Functions (and types, didn't check but it makes sense) do not implicitly access globals."

This makes sense, a global defined anywhere could accidentally modify the behaviour of those constructs. It applies to those constructs since they are the only ones whose existence can exceed the lifetime of their containing scope, so a global created at any time could affect them, for and friends don't survive their containing scope.

There is not really any change in the behaviour of function when nested, its just that when at the top level the containing scope is global, so the globals rule applies, but when nested, there is a containing local scope and those names are inherited as normal.

[mauro3]

Thanks @elextr, presumably you mean "Functions (and types, didn't check but it makes sense) do not implicitly modify globals.", because reading global certainly happens. If the begin-end blocks are indeed a bug, then I think your rule is true and would be good to be incorporated into the docs as the motivation for this behavior. (also it would only apply to functions as type-defs can only appear at global scope.)

However, I think all of the above still needs spelling out as deducting this from your rule is far from easy.

What are your thoughts on making this explicit by introducing the nonlocal keyword and have functions always have a hard-scope? At the moment copying a function into an inner scope potentially changes its behavior, which can be very confusing.

[elextr]

Thanks @elextr, presumably you mean "Functions (and types, didn't check but it makes sense) do not implicitly modify globals."

Ahh, I meant to say "assign" not "access", thanks for picking that up.

I am not sure about spelling it out, that makes it look complex, but the actual rule is simple. But I suppose spelling out the effects may be useful if the simple rule is also mentioned (after one of the experts confirms it is indeed correct).

What are your thoughts on making this explicit by introducing the nonlocal keyword and have functions always have a hard-scope?

Personally I always like explicit over implicit, so specifying captures explicitly could be good, but that would make anonymous functions and one line functions incapable of capturing local variables, since their syntax doesn't allow such a declaration. That is likely to break lots of existing code.

The alternative is making anonymous and one-line functions implicitly capture, and functions defined with function behave differently.

Neither of those alternatives is very attractive, so I think the status quo is the best.

[mauro3]

I think, if a simple rule has complex effects, as is the case here, then those effects need describing.

Concerning introducing a nonlocal. I think anonymous functions and one-linear could work just fine. This works at the moment:

julia> a =1
1

julia> f = () -> global a=2
(anonymous function)

julia> f()
2

julia> a
2

So I see no reason a nonlocal couldn't work. (And read access would stay implicit.)

[mauro3]

Rambled on about nonlocal over in #5331 (comment), which is the proper issue for that.

[JeffBezanson]

The description in this issue makes it seem more complicated than it is. There aren't really so many different kinds of scope. All variables are inherited by inner scopes for both reading and writing, unless explicitly overridden with local. The only special case is global variables. Global variables are not implicitly inherited for writing within functions. I believe that's the only exception. So #7234 does indeed look like a bug.

If nonlocal is only needed for functions, it actually adds another difference between functions and other kinds of scope. For example

julia> function foo()
        x = 1
        for i=1:1
         local x = 2
        end
       x
       end
foo (generic function with 1 method)

julia> foo()
1

julia> function foo()
        x = 1
        for i=1:1
         x = 2
        end
       x
       end
foo (generic function with 1 method)

julia> foo()
2

Would the for loop in the second case need nonlocal too?

[ScottPJones]

Ok... I think the bugs brought up in #7234 have just been clouding the issues for everybody, making things seem a lot more complicated. Does it look like it will be difficult to fix?

[mauro3]

Thanks Jeff and Lex, you make it sound so easy... Thanks Jeff for confirming Lex'
s rule. So, I'll close this and try and incorporate the lessons I learned into a docs update.