Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Operator precedence of & and | is surprising as element-wise boolean operators #5187

Closed
mbauman opened this issue Dec 18, 2013 · 89 comments
Closed
Assignees
Labels
kind:breaking This change will break code needs decision A decision on this change is needed parser Language parsing and surface syntax
Milestone

Comments

@mbauman
Copy link
Sponsor Member

mbauman commented Dec 18, 2013

I just got bit by the python-like operator precedence for the operators | and &. As bitor and bitand, it's a wonderful choice (making checking for bit flags much simpler — a common gotcha in C).

But when behaving as their element-wise boolean counterparts to || and &&, it is surprising. A < 1 || A > 2 must be written differently if A is an array: (A .< 1) | (A .> 2). Perhaps all that's needed here is a bit more documentation (i.e., putting https://github.com/JuliaLang/julia/blob/master/src/julia-parser.scm#L1-L19 into the Mathematical Operators page, or as something to mention to users coming from Matlab).

But as I look at the source for |(::StridedArray,::StridedArray), I see that it's actually applying the bitwise operator to all elements. As a radical alternative, what about adding .&& and .|| with similar precedence to && and || that ensures boolean elements? Functionally, & and .&& would behave the same on logical arrays, but they'd each have the precedence one would expect in each context. (Of course, there'd still be some cognitive dissonance here as the elementwise boolean operators couldn't short-circuit like their scalar equivalents).

@JeffBezanson
Copy link
Sponsor Member

There might not be a single precedence ordering perfect for all cases. The main problem here is that the result of < is seldom combined with anything; i.e. (x<1) + 2 is quite rare, so without explicit parens parses as x<(1 + 2).

@StefanKarpinski
Copy link
Sponsor Member

I kind of suspect that our precedences could use a little tweaking, but yeah, every choice is bad in some way.

@mbauman
Copy link
Sponsor Member Author

mbauman commented Dec 18, 2013

Yes, exactly. I definitely think the precedence for bitand is the right choice. That's what led to my brainstorm of the .&& operators.

Perhaps it's just my Matlab-think that causes this confusion, where & truly is the element-wise little brother to scalar &&.

@malmaud
Copy link
Contributor

malmaud commented Dec 22, 2013

+1 for .&& and .||. It seems totally consistent with .*, etc. My understanding is Numpy only ended up using & for element-wise "logical and" since they ran out of infix operators. It's a constant source of confusion (eg, from http://wiki.scipy.org/NumPy_for_Matlab_Users,

"Precedence: Numpy's & operator is higher precedence than logical operators like < and >; Matlab's is the reverse.
If you know you have boolean arguments, you can get away with using Numpy's bitwise operators, but be careful with parentheses, like this: z = (x > 1) & (x < 2). The absence of Numpy operator forms of logical_and and logical_or is an unfortunate consequence of Python's design".)

Also this is anecdotal, but I suspect taking the element-wise "logical and" of boolean vectors is a more common operation than bitwise operations and so if & is going to maintain its current meaning, it might be worth tweaking the precedence to be the same as .*.

@nalimilan
Copy link
Member

I must admit that the fact that the priority order differs between |/.== and ||/== reminds me a little of the R Inferno -- something you absolutely want to avoid. Every choice is bad in some way, but inconsistent solutions are bad in all ways! ;-)

Using the same operators for so different things as bitwise and boolean operations is not ideal. Does any other language use that pattern? In Matlab and R & only does the latter, and in Numpy it only does the former.

Adding .&& and .|| might be a good and consistent solution. In that case & and | would better be eventually reserved to bitwise operations (do not offer two slightly different ways of doing the same thing).

@JeffBezanson
Copy link
Sponsor Member

What is the difference between bitwise and boolean?

The difference between & and && is that && is a control flow operator; it short-circuits. So .&& doesn't really make much sense.

@StefanKarpinski
Copy link
Sponsor Member

Yeah, .&& and .|| don't make any sense. I almost regret using | and & for bitwise or and and. Those are pretty uncommon operations and are arguably clearer if you have to write them in function notation – e.g. there are no precedence issues that way. Then we could have used | for piping without punning.

@malmaud
Copy link
Contributor

malmaud commented Dec 22, 2013

Those are fair points. Maybe something like .&? It would be consistent with most binary operators having a '.' version that is element-wise, and could have the expected precedence (later than .<).

@nalimilan
Copy link
Member

@JeffBezanson IIUC, & and | have a higher priority than && and || because it's more practical for common patterns of bitwise operations. That's why having a separate element-wise boolean operator would make sense, to provide a consistent precedence order for scalar and element-wise comparison operators.

The whole thing is quite confusing because of the multiple meanings of operators...

julia> help("&&")
Base.&&(x, y)

   Boolean and

So && is not only defined as short-circuit: it also intuitively means that you expect to work with booleans only. This gives support to the idea of having an element-wise version working only with boolean arrays. One could also have & work only for scalars (and operating bitwise), and .& for arrays.

Honestly, the issue of short-circuit really seems secondary to me; of course, when working element-wise it doesn't make sense to short-circuit. But I see that as a side effect. The fact that Matlab offers the choice between two operators (though & and | do short-circuit in if!) looks like a design mistake: when do you need that? Even C does not offer this choice. The real question is: what's the appropriate precedence of bitwise and boolean operators, and do they need to be separated because of that.

@StefanKarpinski Yeah, I was thinking too that wasting & and | for bitwise operations is not that great; but no language uses different operators. Is the alternative of using functions a good solution? Matlab and R do that, there might be a good reason.

@JeffBezanson
Copy link
Sponsor Member

&& does not have multiple meanings. The short-circuit behavior only makes
sense because of the boolean operation it does; they cannot be separated.

A function like & but accepting only boolean arrays would be silly. The
"boolean" and "bitwise" behaviors are the same; a boolean is 1 bit.
On Dec 22, 2013 12:35 PM, "Milan Bouchet-Valat" notifications@github.com
wrote:

@JeffBezanson https://github.com/JeffBezanson IIUC, & and | have a
higher priority than && and || because it's more practital for common
patterns of bitwise operations. That's why having a separate element-wise
boolean operator would make sense, to provide a consistent precedence order
for scalar and element-wise comparison operators.

The whole thing is quite confusing because of the multiple meanings of
operators...

julia> help("&&")
Base.&&(x, y)

Boolean and

So && is not only defined as short-circuit: it also intuitively means
that you expect to work with booleans only. This gives support to the idea
of having an element-wise version working only with boolean arrays. One
could also have & work only for scalars (and operating bitwise), and .&for arrays.

Honestly, the issue of short-circuit really seems secondary to me; of
course, when working element-wise it doesn't make sense to short-circuit.
But I see that as a side effect. The fact that Matlab offers the choice
between two operators (though & and | do short-circuit in if!) looks like
a design mistake: when do you need that? Even C does not offer this choice.
The real question is: what's the appropriate precedence of bitwise and
boolean operators, and do they need to be separated because of that.

@StefanKarpinski https://github.com/StefanKarpinski Yeah, I was
thinking too that wasting & and | for bitwise operations is not that
great; but no language uses different operators. Is the alternative of
using functions a good solution? Matlab and R do that, there might be a
good reason.

?
Reply to this email directly or view it on GitHubhttps://github.com//issues/5187#issuecomment-31091723
.

@nalimilan
Copy link
Member

But what about operator precedence? If they are the same, why don't & and && have the same precedence?

@JeffBezanson
Copy link
Sponsor Member

The thing is, short-circuiting is not secondary, but essential. If it were
not for that, they would be the same operator. Short-circuit also
immediately implies scalar boolean, since it has to make a single decision.

I guess I would consider changing the precedence of &. But the key is to
see && as primarily for control flow; it is not even a function, while & is
a normal generic function.
On Dec 22, 2013 1:09 PM, "Milan Bouchet-Valat" notifications@github.com
wrote:

But what about operator precedence? If they are the same, why don't & and
&& have the same precedence?


Reply to this email directly or view it on GitHubhttps://github.com//issues/5187#issuecomment-31092386
.

@mbauman
Copy link
Sponsor Member Author

mbauman commented Dec 22, 2013

Regardless of whether the precedence of & changes or a third set of boolean operators is introduced specifically for element-wise boolean arrays, I think it's very important to allow syntax such as B[A .< 1 | A .> 2] without parentheses. In a language like Julia, I think that's a much more common operation than, say, checking bitfields.

@StefanKarpinski
Copy link
Sponsor Member

+1 for what @mbauman said. I would also throw into the mix that Perl and Ruby have and and or operators that have short-circuit behavior like && and || but much lower precedence. I don't recall if that's because && and || have precedence like & and | or not, but it might be. It's quite subjective, of course, but I happen to think that and and or are much more intuitive as control flow primitive because they look like keywords, not operators.

@johnmyleswhite
Copy link
Member

I would love it if Julia had and and or as keywords that did && and ||.

@nalimilan
Copy link
Member

But currently && and || have the lowest precedence, excepting assignment operators. So in the vast majority of cases you do not need parentheses to combine several conditions (assignment in a condition is special anyway). So what would be the point of introducing and and or with the same behavior as && and || except for precedence? Looks like a source of complexity for a very limited gain.

Apparently [1], and and or where added to Perl to allow constructs like this:

open my $fh, '<', $filename or die "A horrible death!";

I'm not sure that's what you want to encourage in Julia. ;-)

1: http://stackoverflow.com/a/15193366/2413179

@StefanKarpinski
Copy link
Sponsor Member

Yeah, I already do that sort of thing all the time with ||. Guess where I picked it up?

@IainNZ
Copy link
Member

IainNZ commented Dec 23, 2013

@johnmyleswhite +1 to that

@johnmyleswhite
Copy link
Member

To give a justification for why I like and over &&: the trouble for me with & and && is that their surface similarity makes me believe that they will be more similar than they are. This was an endless problem for me in R: I would have to look up the definitions of both operators 100% of the time I used them, because they just confuse me to this day. Julia's type errors are big step up from this, but I still find the distinction between these operators non-intuitive.

@JeffBezanson
Copy link
Sponsor Member

I guess our C bias leaked through by picking && instead of and. I've been using && in C for so long that it didn't even occur to me how easy it is to confuse with &, but that should have been obvious. I don't think we can change it now.

We could lower the precedence of & and | though. While this could cause very hard-to-find bugs, I'm confident that these operators are not used that much,

@johnmyleswhite
Copy link
Member

Is there no mechanism for operators to be deprecated? If that happens over the course of months, it seems easy enough to replace && with and in existing code. Unlike a lot of breaking changes, this one seems like a simple search-and-replace will solve almost every use case without any careful thinking.

@StefanKarpinski
Copy link
Sponsor Member

Agreed. I don't see why we couldn't migrate from && and || to and and or fairly easily. Steps:

  1. Deprecate the usage of and and or as identifiers.
  2. Add and and or keywords as short-circuit operators.
  3. Deprecate && and || operators.
  4. Remove && and || operators.

We could also just have both.

@johnmyleswhite
Copy link
Member

I'd really prefer that we not have both in the long-term future. If that were the final option, I'd prefer sticking with the unloved && over redundant operators.

@JeffBezanson
Copy link
Sponsor Member

Good point that unlike most changes, this can be handled by search and
replace. But it would not fix this issue; we'd still need to change the
precedence of & or the like.

@johnmyleswhite
Copy link
Member

Very true. Let's just open another issue for debating the change of && to and.

@BobPortmann
Copy link
Contributor

+1 for lowering the precedence of & and |

@StefanKarpinski
Copy link
Sponsor Member

Precedence is often a damned if you do, damned if you don't kind of business, but I have to say that I've often wished that these operators had lower precedence.

@musm
Copy link
Contributor

musm commented Sep 27, 2016

I wonder if that could allow = and && or = and || to be on the same level and be right associative so that a = b && c parses as a = (b && c) while a && b = c parses as a && (b = c).

I really like this idea, IMO I find it really ugly to always write condition(a) && (x = 2) it would be a lot more clear as condition(a) && x = 2 .

@mbauman
Copy link
Sponsor Member Author

mbauman commented Sep 7, 2017

Could we allow && and || to participate in dot-fusion now? Especially if the @. macro applied to them, this seems like it could be a nice solution. As I noted in the first post:

A < 1 || A > 2 must be written differently if A is an array: (A .< 1) | (A .> 2)

Are there any arguments against allowing @. A < 1 || A > 2 to fuse into broadcast(x->x < 1 || x > 2, A)? Edit: I suppose we'd have to special case A .|| B to lower to broadcast((x,y)->x||y, A, B) since || isn't a real function. And .&& does parse right now, but it evaluates to a "misplaced "&" expression" error.

@JeffBezanson
Copy link
Sponsor Member

Coming back to this, it seems pretty clear to me that (a < b) & (c < d) is way more useful than a < (b & c) < d. Reopening and marking for triage.

@JeffBezanson JeffBezanson reopened this Sep 11, 2017
@JeffBezanson JeffBezanson added the status:triage This should be discussed on a triage call label Sep 11, 2017
@JeffBezanson JeffBezanson modified the milestones: 0.4.0, 1.0 Sep 14, 2017
@JeffBezanson JeffBezanson removed the status:triage This should be discussed on a triage call label Sep 14, 2017
@JeffBezanson JeffBezanson self-assigned this Nov 14, 2017
@nalimilan
Copy link
Member

For anecdotal evidence, the current precedence of & prompts some people to write things like this:

 hflights[.&(hflights[:Month] .== 1, hflights[:DayofMonth] .== 1), :]

@Sacha0
Copy link
Member

Sacha0 commented Dec 3, 2017

(Perhaps I should note that I have WIP on this front. I am prioritizing things A_mul_B though, so not certain whether I will post that work on a short timescale. Best!)

JeffBezanson added a commit that referenced this issue Dec 29, 2017
JeffBezanson added a commit that referenced this issue Dec 30, 2017
mbauman added a commit that referenced this issue Apr 12, 2018
Giving us space to allow lowering this to `.&&` and `.||` in the future. Ref #5187.
JeffBezanson pushed a commit that referenced this issue Apr 20, 2018
Giving us space to allow lowering this to `.&&` and `.||` in the future. Ref #5187.
@robsmith11
Copy link
Contributor

Has the plan for precedence change of & and | to allow for the expected behavior of B[A .< 1 | A .> 2] been changed?

I haven't seen any update in the 2.5 years since the deprecation.

mbauman added a commit that referenced this issue Feb 10, 2021
I have long wanted a proper fix for issue #5187. It was the very first Julia issue I filed.
This is a shot at such a fix. This PR:

* Enables parsing for `.&&` and `.||`.  They are parsed into `Expr(:call, :.&&, ...)` expressions at the same precedence as their respective `&&` and `||`:
    ```julia-repl
    julia> Meta.show_sexpr(:(a .&& b))
    (:call, :.&&, :a, :b)
    ```

* Unlike all other dotted operators `.op` (like `.+`), the `op`-alone part (`var"&&"`) is not an exported name from Base. As such, this effectively lowers to `broadcasted((x,y)->x && y, ...)`, but instead of using an anonymous function I've named it `Base.andand` and `Base.oror`:
    ```julia-repl
    julia> Meta.@lower a .&& b
    :($(Expr(:thunk, CodeInfo(
        @ none within `top-level scope'
    1 ─ %1 = Base.broadcasted(Base.andand, a, b)
    │   %2 = Base.materialize(%1)
    └──      return %2
    ))))
    ```

* I've used a named function to enable short-circuiting behavior _within the broadcast kernel itself_. In the case that the second argument is a part of the same fused broadcast kernel, it will only evaluate if required:
    ```julia-repl
    julia> mutable struct F5187; x; end

    julia> (f::F5187)(x) = (f.x += x)

    julia> (iseven.(1:4) .|| (F5187(0)).(ones(4)))
    4-element Vector{Real}:
        1.0
     true
        2.0
     true
    ```

* This also enables support for standalone `.&&` and `.||` as `BroadcastFunction`s, but of course they are not able to short-circuit when used as functions themselves.

Request for feedback
--------------------

* [ ] A good bikeshed could be had over the names themselves. We could actually use `var"&&"` and `var"||"` for these names — and it'd _almost_ simplify the implementation, but in order to do so we'd have to actually _export_ them from Base, too. It seems like it might just be confusing.
* [ ] Is this the implementation we want? This uses `Broadcast.flatten` to create the lazy function needed for short-circuiting the second argument. This could alternatively be done directly within the parser — perhaps by resurrecting the old 0.5 broadcast parsing behavior. Someone else would have to do that work if they wanted it.
* [ ] Do we want to support the stand-alone `.&&` and `.||` `BroadcastFunction`s if they cannot possibly short circuit?
vtjnash pushed a commit that referenced this issue Apr 9, 2021
I have long wanted a proper fix for issue #5187. It was the very first Julia issue I filed.
This is a shot at such a fix. This PR:

* Enables parsing for `.&&` and `.||`.  They are parsed into `Expr(:call, :.&&, ...)` expressions at the same precedence as their respective `&&` and `||`:
    ```julia-repl
    julia> Meta.show_sexpr(:(a .&& b))
    (:call, :.&&, :a, :b)
    ```

* Unlike all other dotted operators `.op` (like `.+`), the `op`-alone part (`var"&&"`) is not an exported name from Base. As such, this effectively lowers to `broadcasted((x,y)->x && y, ...)`, but instead of using an anonymous function I've named it `Base.andand` and `Base.oror`:
    ```julia-repl
    julia> Meta.@lower a .&& b
    :($(Expr(:thunk, CodeInfo(
        @ none within `top-level scope'
    1 ─ %1 = Base.broadcasted(Base.andand, a, b)
    │   %2 = Base.materialize(%1)
    └──      return %2
    ))))
    ```

* I've used a named function to enable short-circuiting behavior _within the broadcast kernel itself_. In the case that the second argument is a part of the same fused broadcast kernel, it will only evaluate if required:
    ```julia-repl
    julia> mutable struct F5187; x; end

    julia> (f::F5187)(x) = (f.x += x)

    julia> (iseven.(1:4) .|| (F5187(0)).(ones(4)))
    4-element Vector{Real}:
        1.0
     true
        2.0
     true
    ```

Co-authored-by: Simeon Schaub <simeondavidschaub99@gmail.com>
ElOceanografo pushed a commit to ElOceanografo/julia that referenced this issue May 4, 2021
I have long wanted a proper fix for issue JuliaLang#5187. It was the very first Julia issue I filed.
This is a shot at such a fix. This PR:

* Enables parsing for `.&&` and `.||`.  They are parsed into `Expr(:call, :.&&, ...)` expressions at the same precedence as their respective `&&` and `||`:
    ```julia-repl
    julia> Meta.show_sexpr(:(a .&& b))
    (:call, :.&&, :a, :b)
    ```

* Unlike all other dotted operators `.op` (like `.+`), the `op`-alone part (`var"&&"`) is not an exported name from Base. As such, this effectively lowers to `broadcasted((x,y)->x && y, ...)`, but instead of using an anonymous function I've named it `Base.andand` and `Base.oror`:
    ```julia-repl
    julia> Meta.@lower a .&& b
    :($(Expr(:thunk, CodeInfo(
        @ none within `top-level scope'
    1 ─ %1 = Base.broadcasted(Base.andand, a, b)
    │   %2 = Base.materialize(%1)
    └──      return %2
    ))))
    ```

* I've used a named function to enable short-circuiting behavior _within the broadcast kernel itself_. In the case that the second argument is a part of the same fused broadcast kernel, it will only evaluate if required:
    ```julia-repl
    julia> mutable struct F5187; x; end

    julia> (f::F5187)(x) = (f.x += x)

    julia> (iseven.(1:4) .|| (F5187(0)).(ones(4)))
    4-element Vector{Real}:
        1.0
     true
        2.0
     true
    ```

Co-authored-by: Simeon Schaub <simeondavidschaub99@gmail.com>
antoine-levitt pushed a commit to antoine-levitt/julia that referenced this issue May 9, 2021
I have long wanted a proper fix for issue JuliaLang#5187. It was the very first Julia issue I filed.
This is a shot at such a fix. This PR:

* Enables parsing for `.&&` and `.||`.  They are parsed into `Expr(:call, :.&&, ...)` expressions at the same precedence as their respective `&&` and `||`:
    ```julia-repl
    julia> Meta.show_sexpr(:(a .&& b))
    (:call, :.&&, :a, :b)
    ```

* Unlike all other dotted operators `.op` (like `.+`), the `op`-alone part (`var"&&"`) is not an exported name from Base. As such, this effectively lowers to `broadcasted((x,y)->x && y, ...)`, but instead of using an anonymous function I've named it `Base.andand` and `Base.oror`:
    ```julia-repl
    julia> Meta.@lower a .&& b
    :($(Expr(:thunk, CodeInfo(
        @ none within `top-level scope'
    1 ─ %1 = Base.broadcasted(Base.andand, a, b)
    │   %2 = Base.materialize(%1)
    └──      return %2
    ))))
    ```

* I've used a named function to enable short-circuiting behavior _within the broadcast kernel itself_. In the case that the second argument is a part of the same fused broadcast kernel, it will only evaluate if required:
    ```julia-repl
    julia> mutable struct F5187; x; end

    julia> (f::F5187)(x) = (f.x += x)

    julia> (iseven.(1:4) .|| (F5187(0)).(ones(4)))
    4-element Vector{Real}:
        1.0
     true
        2.0
     true
    ```

Co-authored-by: Simeon Schaub <simeondavidschaub99@gmail.com>
johanmon pushed a commit to johanmon/julia that referenced this issue Jul 5, 2021
I have long wanted a proper fix for issue JuliaLang#5187. It was the very first Julia issue I filed.
This is a shot at such a fix. This PR:

* Enables parsing for `.&&` and `.||`.  They are parsed into `Expr(:call, :.&&, ...)` expressions at the same precedence as their respective `&&` and `||`:
    ```julia-repl
    julia> Meta.show_sexpr(:(a .&& b))
    (:call, :.&&, :a, :b)
    ```

* Unlike all other dotted operators `.op` (like `.+`), the `op`-alone part (`var"&&"`) is not an exported name from Base. As such, this effectively lowers to `broadcasted((x,y)->x && y, ...)`, but instead of using an anonymous function I've named it `Base.andand` and `Base.oror`:
    ```julia-repl
    julia> Meta.@lower a .&& b
    :($(Expr(:thunk, CodeInfo(
        @ none within `top-level scope'
    1 ─ %1 = Base.broadcasted(Base.andand, a, b)
    │   %2 = Base.materialize(%1)
    └──      return %2
    ))))
    ```

* I've used a named function to enable short-circuiting behavior _within the broadcast kernel itself_. In the case that the second argument is a part of the same fused broadcast kernel, it will only evaluate if required:
    ```julia-repl
    julia> mutable struct F5187; x; end

    julia> (f::F5187)(x) = (f.x += x)

    julia> (iseven.(1:4) .|| (F5187(0)).(ones(4)))
    4-element Vector{Real}:
        1.0
     true
        2.0
     true
    ```

Co-authored-by: Simeon Schaub <simeondavidschaub99@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind:breaking This change will break code needs decision A decision on this change is needed parser Language parsing and surface syntax
Projects
None yet
Development

No branches or pull requests