Compute angle between a vector and a plane perpendicular to the image plane

[sfmig]

Is your feature request related to a problem? Please describe.
In a 2D, top view camera case, a user may want to compute the angle between a vector (e.g. the head vector) and a line that represents a plane perpendicular to the image plane (e.g., the walls of an enclosure).

In that case we would need to compute the angle between the (head) vector and the normal to the (wall) plane.

Additional context
This could be useful for boundary vector cells analyses, alongside #368

[willGraham01]

@sfmig (and @niksirbi) wanted to get some clarification here, when it comes to regions of interest.

What's described in the issue, I've drawn in the image below (on the right-hand-side) as I interpret it. I'm not sure what this is actually representing, or why it would be useful? There's also ambiguity regarding the sign of the boundary normal, amongst other things. This gets even more confusing when we start considering polygons too. If we really do want access to the normal to a line though, I can add an normal property to LineOfInterest which returns this, which means that our other functions (compute_signed_angle_2d, etc) can work out-of-the-box.

On the left of the above is what I imagined this feature would be bringing to the package. Given a forward vector and the point the forward vector originates from, compute the direction towards the closest point on the RoI boundary. Then, compute the angle between the current heading and the direction that follows the shortest heading towards the region.

So the RoI class gets a general method

class BaseRegionOfInterest

    def angle_to(
        data,
        left_keypoint,  # These two define
        right_keypoint, # the forward direction
        camera_view,    # Also needed to determine "forwards"
        position_keypoint, # Used to compute the vector towards the region of interest
        ) -> float:
        """Computes the signed angle between a forward vector and the direction of closest approach to a region of interest"""

However I think I'm missing the overall objective of what this issue originally asked for 😅 Any guidance is appreciated - feel free to use diagrams too!

[sfmig]

Hi @willGraham01 !

I think both options above are equivalent, right? Just the angle obtained with option 1 is the supplementary of the one obtained in option 2.

See diagram below (sorry it's huge!). v is the head (or forward) vector, and n the normal to the boundary plane pi. With option 1 you compute theta, with option 2, you compute 180-theta.

But I think in both cases it may get tricky because of the fact that the boundary is made up of segments and not infinite lines...

There may be another tricky aspect regarding the orientation of the normal - though if it is always closed polygons it may be alright?

Just to clarify the suggestion was to:
(1) compute the boundary segment closest to the animal (or to a keypoint in the animal? - might be tricky to choose)
(2) compute a line perpendicular to that segment that passes thru the origin of the head vector (e.g., the midpoint between the ears)
(3) compute the angle between the head vector and the normal line/vector. If that angle is 0 (or 180, depending on how we orient the normal vector), the animal is looking straight at the wall. If that angle is 180 (or 0), its back is to the wall.

I need to leave now but happy to continue discussing how we can deal with the segments issue!

[sfmig]

P.S.: In the above comment, I am assuming that the goal is to compute the orientation of the animal wrt to a plane (projected on the page as an infinite line), not orientation wrt to a specific segment (for which your options 1 and 2 wouldn't be the same) ... I think that the confusion may be there.

I don't think there is interest in computing the angle with a specific segment. An example below (the plane is in blue, the segment in red): if we compute the angle with your option 2 (which would give us alpha) that doesn't seem indicative of the orientation of the animal wrt the wall, right?

This is more clear if you imagine the animal is with its back to the wall (i.e., v is parallel to n). In that case theta would be 0, but alpha would be some "arbitrary" value, that depends on the position of the closest point in the segment.

So I think the aim here is, given a segment, compute the angle between the head/forward vector and the plane that contains that segment and is perpendicular to the paper (or image plane).

[niksirbi]

I'm frankly confused about the confusion here 🤔 .
I have to think about it a bit by sketching and looking at boundary vector cell literature. I will post here if/once I have something meaningful to contribute to your discussion.

[niksirbi]

For now just copying here Sepi's notes on boundary vectors, for easy reference:

• Allocentric boundary vector:
  • Define the coordinates of the experimental arena’s boundaries
  • Extract the time series of animal’s position (midpoint between eyes or ears)
  • For each position (each frame), calculate the distance to the arena boundaries and find the distance to the nearest boundary. In a circular arena, this is simplified as “arena’s radius - distance to centre”. For rectangular arenas, this is the shortest perpendicular distance to any of the four walls.
  • For each position (each frame), calculate the allocentric direction to the nearest boundary: In circular arenas, this is calculated as the angle between the vector from the arena centre to the animal’s position and a reference direction, i.e. the positive x axis. For rectangular arenas, this is the angle between the vector from the animal to the nearest point on the boundary and a reference direction.
• Egocentric boundary vector:
  • Define the coordinates of the experimental arena’s boundaries
  • For each frame, find the nearest point on the boundary to the animal’s head and calculate the distance to this point (same as above).
  • Calculate the egocentric direction as the angle between the animal’s head direction and the vector to the nearest boundary point.

[willGraham01]

@sfmig and I just had a in depth chat about this. I have tried to summarise it here (though please point out any inconsistencies @sfmig!):

1. In the case of a line segment, how does one define the animal's orientation with respect to the segment.

Shamelessly stealing @sfmig's diagram, which encapsulates these two cases really well:

There are two ways that seem logical (mathematically) for defining the orientation of an animal to a line segment.

• Convention $\theta$ is that the orientation is the angle given by $\theta$ in the diagram. Compute the normal to the line segment $n$ (as if one had extended it to an infinite line in the plane) and then take a suitable angle between $v$ and $n$ as the "orientation". Note that this approach is agnostic to the animal's actual position (except for determining which choice of normal to the line to use).
• Convention $\alpha$ is that the orientation is the angle given by $\alpha$ in the diagram. Compute the approach vector $a$, which is the vector from the animal's current position (this can be given by some keypoint / combination of, this detail isn't important here) to the closest point on the segment. Then the animal's orientation is signed angle between $v$ and $a$.

Convention 1 results in a constant orientation, so long as the animal remains on the same side of the line segment. Convention 2 coincides with convention 1 when the closest point is an interior point of the line, but not the endpoints. There are merits to both, and @sfmig and I actually came down on the side that $\theta$ was the interpretation intended in the wording of this issue originally. But upon seeing

Calculate the egocentric direction as the angle between the animal’s head direction and the vector to the nearest boundary point.

from your quoted block in Sepi's notes... this would imply that $\alpha$ is the convention she wants? @sfmig raised the very valid point that having two functions; compute_angle_to_nearest_pt_in_segment ($\alpha$), AND
compute_angle_to_support_plane_of_segment ($\theta$) disambiguates these cases very well, at least for line segments, and may both have applications in analyses beyond just a potential difference in convention between research groups.

2. In the case of a Polygon, how do we deal with the corner regions?

In the case of a Polygon, the closest point to the animal may belong to either 1 or 2 of its boundary segments. In the event that it only belongs to one, then the orientation is easily computed by reverting to the line-segment case for the appropriate boundary segment. Note that the $\theta$ and $\alpha$ conventions would coincide here (since $a$ will be parallel to $n$).

Difficulty arises when the closest point to the animal is a corner, between two boundary segments. Convention $\alpha$ handles this case perfectly fine, because $a$ serves as a kind of "smoothed normal" to the corner, given the animal's position.

Convention $\theta$ does not handle this case well though, it will result in a discontinuity in the computed orientation. It is possible to still determine the "closest boundary segment" in this case by looking at the angular distance of the animal to the two boundary segments in question, but when the animal crosses this "threshold" the orientation angle will jump as $n$ changes abruptly from one boundary segment to another. Sofia and I talked about using something like Slerp to potentially smooth out this discontinuity, though this might be equivalent to using approach $\alpha$.

Given what @niksirbi has quoted from Sepi's wish list:

Egocentric boundary vector:

• Define the coordinates of the experimental arena’s boundaries
• For each frame, find the nearest point on the boundary to the animal’s head and calculate the distance to this point (same as above).
• Calculate the egocentric direction as the angle between the animal’s head direction and the vector to the nearest boundary point.

This gives me the impression that convention $\alpha$ is the one we want, but Sepi's request is worded as if she is only ever considering enclosure boundaries - IE closed polygons, and animal positions inside those polygons. In this case, we would never see a difference between $\theta$ and $\alpha$ (since any point would always be "inside" the region, thus we can never be in situation 4 or 5 as per Sofia's diagram). However if the region of interest is a sub-region of the experimental arena, we need to know which convention to follow.

For the time being it isn't much effort to implement both in a draft PR. But we might want to show this to Sepi and ask for some explicit advice as to which convention she means, in the event that the region is not the entire enclosure.

[sfmig]

great summary @willGraham01 ! 🌟

Just to clarify for other readers:

Note that the θ and α conventions would coincide here

When we say coincide, we mean they are basically the same because they just differ in a constant value (θ = 180 - α).

Sepi's request is worded as if she is only ever considering enclosure boundaries - IE closed polygons, and animal positions inside those polygons. In this case, we would never see a difference between θ and α (since any point would always be "inside" the region, thus we can never be in situation 4 or 5 as per Sofia's diagram)

I think even if the region is a closed polygon, we could still have concave corners, for which the θ and α approach differ. Right? (I may be missing something tho).

I think you mention it but just to highlight it - I would say a question to clarify with Sepi is: in the diagram here, would she interpret the animal has constant orientation with respect to the wall in all 5 cases? Or would she say 4 and 5 have different orientations?

If she says definitely Yes to the first question, maybe we just implement the orientation wrt boundary function as a "compute angle to support plane of segment". If it is not clear cut or if she says both approaches are of interest, I would say we can have both compute_angle_to_nearest_pt_in_segment (α), AND compute_angle_to_support_plane_of_segment (θ).

[sfmig]

Btw there is a typo in the diagram above, the arc representing α5 is wrong - sorry!

Below the fixed version. I also added in yellow the vectors @willGraham01 calls "approach vectors" a.

Note that in the diagram, the α angle actually changes sign for 5.

[willGraham01]

I think even if the region is a closed polygon, we could still have concave corners, for which the θ and α approach differ. Right? (I may be missing something tho).

Yeah you're right - us being inside doesn't stop there from being interior concave corners 😰 I was only thinking of enclosures that would be regular polygons (in which case we wouldn't have this issue, but we of course can for irregular shapes).

[niksirbi]

Thanks both! This is an excellent summary of the problem, I understand it now!

Let's define the following vectors, for supporting this discussion.

• The "forward" vector of the animal $\vec{v}$, however we've obtained it
• The "approach" vector $\vec{\alpha}$, here defined as the vector starting from the animal's head and terminating at the closest point to the boundary (without extensions). "Closest" here means the boundary point at the smallest euclidean distance from the animal's head.
• The "perpendicular approach" vector $\vec{n*}$, starting from the animal's head going towards the closest line segment (or its infinite extension) at a right angle. This vector is aligned with the line segment's normal $\vec{n}$, i.e. is either parallel or anti-parallel to it, depending on the normal vector's orientation. I deliberately chose to define $\vec{n*}$ here, instead of just using $\vec{n}$, because $\vec{n*}$ has unambiguous orientation if we "force" it to start at the animal's head.

Essentially we can have two cases (illustrated below):

• they "easy" case, in which the "approach" vector $\vec{\alpha}$ is identical to the "perpendicular approach" vector $\vec{n*}$. In this case, the angle with respect to the boundary ("egocentric" boundary vector) is unambiguous—it's the signed angle between $\vec{v}$ and $\vec{a} = \vec{n*}$ (or its complement, that's not so important). If the lines of interest describe a convex polygon enclosing the animal, we are always in the "easy" case.
• the "hard" case, in which the "approach" vector $\vec{\alpha}$ differs from the "perpendicular approach" vector $\vec{n*}$. In this case the angle is ambiguous, because we could either take the angle between $\vec{v}$ and $\vec{a}$ or the angle between $\vec{v}$ and $\vec{n*}$.

Is this a fair summary of the question?

[willGraham01]

Is this a fair summary of the question?

I would say so.

I think it's important to note though that for a line (segment), both interpretations make sense (particularly with @sfmig's distinction into two cases / functions). The complication only comes when we consider polygons, and the "hard case" kind suddenly makes it clear we have a problem near the corners of these polygons (when the approach vector $\vec{a}$ is smooth, but $\vec{n^*}$ jumps suddenly as we "go round" the corner).

[sfmig]

Is this a fair summary of the question?

Yeah, I think that is about right! As Will says it is just missing the extension to a polygon case, but we can discuss that in person if that is unclear.

I would suggest some things from @niksirbi's bit above, just to not change or add more nomenclature unnecessarily:

• Shall we call the approach vector $\vec{a}$ and reserve $\alpha$ for the angle between the forward vector $\vec{v}$ and the approach vector $\vec{a}$?
• Instead of distinguishing between $\vec{n}$ and $\vec{n}_{*}$ and having to deal with both, Will and I were discussing defining $\vec{n}$ as the vector that is (1) perpendicular to the (infinite) line that contains the segment, and (2) pointing towards the origin point of $\vec{v}$ (i.e., in practice, pointing towards the animal). Just to not add more things, but they are "equivalent" ways as you say.
• We consider a line and a plane to be infinite (following their definition in geometry). This is in contrast to segments, which are finite and thus have end points (that is basically the crux of the thing). I think it would be clearer if we use "line" and "segment" like this, and maybe avoid "line segment" or "line extension"?
• I think in most use cases the forward vector $\vec{v}$ will be the head vector, and I think this helps for the biological interpretation. So maybe we want to use "head" vector rather than "forward" for this discussion - I use them interchangeably below.

In this case the angle is ambiguous,

A small reframing: I wouldn't say that it is ambiguous - it just depends on how we translate the biological question into a math one.

The biological question would be: What is the orientation of the animal('s head) relative to this boundary? Assuming the boundary is defined by the user as a segment, this could be translated into two possible math questions:

• What is the angle between the head vector and the approach vector?, or
• What is the angle between the head vector and the normal vector?

My current take: I think generally when thinking about orientation wrt a wall, people mean the second case. But a user may be interested in computing the angle between the head vector $\vec{v}$ and the approach vector $\vec{a}$ in some cases. For example, if the selected segment does not represent a wall, but instead a target object (e.g. a box with a reward spout?), you may want to compute how far off is any point in the target object from the centre of the animal's 2D field of view (I'm assuming the field of view is always centred at the head vector). We probably need to support both, but I think it would be clearer as separate implementations.

[sfmig]

Re-reading the egocentric/allocentric distinction, this is my current understanding:

If we are given a polygon (a list of connected segments) and the users want the..

• Allocentric orientation of the closest boundary:
  • We return: the angle between approach vector $\vec{a}$ and reference vector $\vec{x}$ (usually the positive x-axis).
• Egocentric orientation of the closest boundary
  • We return: the angle between approach vector $\vec{a}$ and head vector $\vec{v}$.

If we are given a single segment we need to clarify what the users mean / decide and explain. If they want the..

• Allocentric orientation of the segment
  • We return: the angle between approach vector $\vec{a}$ and reference vector $\vec{x}$, OR the angle between the normal vector $\vec{n}$ and the reference vector $\vec{x}$.
• Egocentric orientation of the segment
  • We return: the angle between approach vector $\vec{a}$ and head vector $\vec{v}$, OR the angle between the normal vector $\vec{n}$ and the head vector $\vec{v}$.

[niksirbi]

Shall we call the approach vector $\vec{\alpha}$ and reserve $\alpha$ for the angle between the forward vector $\vec{v}$ and the approach vector?

Agreed.

Instead of distinguishing between $\vec{n}$ and $\vec{n*}$ and having to deal with both, Will and I were discussing defining
$\vec{n}$ as the vector that is (1) perpendicular to the (infinite) line that contains the segment, and (2) pointing towards the origin point of $\vec{v}$ (i.e., in practice, pointing towards the animal). Just to not add more things, but they are "equivalent" ways as you say.

Agreed with that as well, it's nice to reduce the "normal" to one specific vector. My motivation for introducing $\vec{n*}$ was for both the "perpendicular" and the $\vec{\alpha}$ vector to point from the animal to the boundary. In your case $\vec{n}$ points from boundary to animal, while $\vec{\alpha}$ points from animal to boundary. If we end up implementing both us options (which I think we should), users may be confused about getting an acute angle in one case and an obtuse one in the other case. But these are implementation details better discussed on the PR directly. For the purposes of this issue I subscribe to your definitions for $\vec{n}$ and $\vec{\alpha}$.

We consider a line and a plane to be infinite (following their definition in geometry). This is in contrast to segments, which are finite and thus have end points (that is basically the crux of the thing). I think it would be clearer if we use "line" and "segment" like this, and maybe avoid "line segment" or "line extension"?

Agreed for this discussion, important to distinguish between infinite lines and finite segments. Though in terms of code objects we only have LineOfInterest, which may consist of one or more segments, and they are always finite.

I think in most use cases the forward vector $\vec{n}$ will be the head vector, and I think this helps for the biological interpretation. So maybe we want to use "head" vector rather than "forward" for this discussion - I use them interchangeably below.

Biologically yes, boundary vectors are defined with respect to the head position and orientation, so using "head vector" for $\vec{v}$ makes sense.

The biological question would be: What is the orientation of the animal('s head) relative to this boundary? Assuming the boundary is defined by the user as a segment, this could be translated into two possible math questions:

• What is the angle between the head vector and the approach vector?, or
• What is the angle between the head vector and the normal vector?

This is a great framing of the whole issue!

[sfmig]

From the dev meeting today:
It seems worth reviewing how would we compute the egocentric (or allocentric) angle relative to a region like the polygon shown here.

It seems like using the angle between the approach vector and the head vector would be the most intuitive here, but not sure if the wall is treated differently than the rest of the polygon.

[willGraham01]

It seems like using the angle between the approach vector and the head vector would be the most intuitive here, but not sure if the wall is treated differently than the rest of the polygon.

Worth mentioning that in our current implementation the enclosing square and the wall that's "jutting out" would have to be separate RoIs (due to constraints on the underlying shapely objects).

[sfmig]

Ok good to know!

Maybe in reality the jutting-out wall would always have a thickness so it wouldn't be a problem?

[niksirbi]

Tagging @TomGeorge1234 to pick his brain on this.

Tom, it seems like RatInABox cam simulate the responses of boundary vector cells given the arena and an agent with position and heading. This means that at some point you've probably contemplated the same issue we're discussing here. You also probably know the boundary vector literature better than any of us.

Do you have opinions on what most neuroscientists would expect in edge cases like the ones we're discussing above?
Or could you perhaps point us to relevant parts of the RatInaBox codebase?

Feel free to ignore the ping if you don't have the capacity to chime in on this right now. No pressure.

[niksirbi]

Maybe in reality the jutting-out wall would always have a thickness so it wouldn't be a problem?

Good point, if you annotating a "real" arena, that's a more likely scenario, and it reduces to the problem of non-convex polygons.

[TomGeorge1234]

BVCs in literature (possible red herring)
I'm not sure to what extent the BVC literature is useful here but if it is, I can tell you that the way BVC firing rates are calculated isn't by finding the single closest wall segment and then establishing the angle to that segment. Instead, the firing rate is an integral over all wall segments in the 360deg field-of-view. For each direction there will be a short wall segment (in the limit this segment is infinitely short so can be considered a point) and that segment contributes a small amount to the total firing rate based on the animal's angle and distance to that segment (point). In that case the normal vector to the wall segment is not a meaningful quantity nor one which is ever used. As you can imagine this is a slightly hellish calculation requiring vectorising over all directions, walls, neurons and environment positions but it has been done https://github.com/RatInABox-Lab/RatInABox/blob/5ce2a279373336fbe11ea4cd96f1912f764f36de/ratinabox/Neurons.py#L1609
You can also check equations 33 and 35 in the Riab paper https://elifesciences.org/articles/85274

Comments on your issue
I can't lie I haven't fully comprehended this whole thread but it sounds like you guys are thinking about this deeply. Some thoughts are:

• @willGraham01 points above (the $\theta$ and $\alpha$ are the same if you are inside a polygon) only holds for convex polygons, right?
• The ambiguity in which normal to use is definitely awkward. I see two ways to break the symmetry:
  • In enclosed polygons each wall segment can be directional in the sense that if you traversed the environment wall in a clockwise manner you'd always cross a wall segment in a specific direction. Use that to define the normal which doesn't depend on agents position or convexity of the environment.
  • Use the agents position as @sfmig said and define the normal as the one pointing "towards" the agent. You don't even need to establish this a priori, just limit $\theta$ and if it's >180, return its supplement.
• Not sure what you mean by "edge cases". If in reference to @willGraham01 concerns about corners where there are two equivalent closest walls - I can't tell you how to resolve that in your code I'm afraid. What I can say is my philosophy is usually to think about whether this distinction exists for the animal. It probably doesn't because it has a representation of all walls simultaneously. Thus RiaB almost entirely skirts these questions by never seeking to answer "which wall is closest?" or "which wall normal should I choose?" and instead calculating relevant quantities for all wall segment and letting them jointly affect the animals motion or BVCs firing rate etc. Not to say this is how you want to solve your problem but it was a mentality I found useful when building a "biologically plausible" package like Riab.

Closest equivalent problem in RiaB
BVCs aren't the closest problem I had to solve. More related was the way the moving agent "deflects" smoothly away from walls to as not to crash into them (one of those problems which, you'll appreciate, is much more finnicky than it sounds). Roughly the agent feels a force in a direction away from all walls scaled by how close it is to those walls. What direction do I use? I had to remind myself but it seems the way I do this is by finding the shortest vector from all wall segments to the agents position restricting to the segments (not the infinite line extension) i.e. I use what you are now calling the (negative) approach_vector. You can check my maths / code here https://github.com/RatInABox-Lab/RatInABox/blob/5ce2a279373336fbe11ea4cd96f1912f764f36de/ratinabox/utils.py#L121

It's moderately well documented, the core function docstring is :

def shortest_vectors_from_points_to_lines(positions, vectors):
    """
    Takes a list of positions and a list of vectors (line segments) and returns the pairwise  vectors of shortest distance
    FROM the vector segments TO the positions.
    Suppose we have a list of N_p positions and a list of N_v line segments (or vectors). Each position is a point like [x_p,y_p], or p_p as a vector.
    Each vector is defined by two points [[x_v_0,y_v_0],[x_v_1,y_v_1]], or [p_v_0,p_v_1]. Thus
        positions.shape = (N_p,2)
        vectors.shape = (N_v,2,2)
    Each vector defines an infinite line, parameterised by line_v = p_v_0 + l_v . (p_v_1 - p_v_0).
    We want to solve for the l_v defining the point on the line with the shortest distance to p_p. This is given by:
        l_v = dot((p_p-p_v_0),(p_v_1-p_v_0)/dot((p_v_1-p_v_0),(p_v_1-p_v_0)).
    Or, using a diferrent notation
        l_v = dot(d,s)/dot(s,s)
    where
        d = p_p-p_v_0
        s = p_v_1-p_v_0"""

what I forgot to say in the docstring but you'll see in the function is that $l_v$ is clipped to between 0 and 1 so the closest perpendicular point must be on the wall segment (see utils lines 170 and 171)....anyway, I'm sure you've got all these sorts of calculations down to a tee by now :))

[willGraham01]

@willGraham01 points above (the θ and α are the same if you are inside a polygon) only holds for convex polygons, right?

Yeah this only holds for convex polygons. So when the polygon represents the whole arena we're usually OK, but we hit differences when we have a polygon representing part of the arena (EG a nest) and the animal is outside this region but still inside the experimental area.

[niksirbi]

Had forgotten to say thank you for the very comprehensive response @TomGeorge1234!

The pointers you've provided us will come in handy well beyond the scope of this particular issue.

[sfmig]

thanks @TomGeorge1234 you raised very good points that I think will become relevant for movement sooner or later

instead calculating relevant quantities for all wall segment and letting them jointly affect the animals motion or BVCs firing rate etc

As you hint, we ended up relying on the approach vector, following a sort of "proximity estimation" approach. But it is very useful for us to be aware of more biological approaches like the one you describe for BVC - it makes sense that the integral is the biologically relevant representation of how the environment "affects" the animal.