LARMOR: Bench limits drift

[AdrianPotter]

We got this email from Rob:

From: Dalgliesh, Robert (STFC,RAL,ISIS)
Sent: 14 March 2017 22:00
To: Howells, Gareth (STFC,RAL,ISIS); Akeroyd, Freddie (STFC,RAL,ISIS); Woods, Kevin (Tessella,RAL,ISIS)
Cc: Washington, Adam (STFC,RAL,ISIS); Nilsen, Goran (STFC,RAL,ISIS); Stewart, Ross (STFC,RAL,ISIS)
Subject: Larmor bench limits

Hi,
We have been seeing some very odd problems with the Larmor rotating bench over the past couple of days.

The limit of the bench seems to not be functioning properly and keeps changing.
After determining a safe limit on Monday I set this in EPICS. However, once we tried to move back to the limit it turned out that we had to add 36deg. To this number to make it work.
Again today we moved to this limit and found that we had to add a further r2 degrees to the limit.
The bench itself seems to be moving reproducibly but this limit behaviour is both very odd and has wasted about 4 hours of beam time to date.

Gareth seems to think that it might be a similar problem to the ones seen on IMAT.

Rob

Having investigated, it doesn't seem related to #2180. I note that in the GALIL_07 logs there is the message:

[2017-03-15 12:49:30] sevr=info Poll services: MOTOR HOMED C

[2017-03-15 12:49:30] sevr=info Poll services: applying motor C raw home position 257003

[2017-03-15 12:49:30] sevr=info Poll services: applying encoder C raw home position 321254

[2017-03-15 12:49:32] Poll services: MOTOR OFF C

I wonder if the homing is causing the drift, though can offer no explanation as to why this might be occurring now and not previously.

@kjwoodsISIS has asked for more information from Rob about the context under which this problem occurred.

[kjwoodsISIS]

Rob first noticed the problem on Monday (13th March) and again on Tuesday (14th). Is there anything in the logs on these days that might point to the problem?

[AdrianPotter]

Do you know which Galil the problem is on?

[AdrianPotter]

Rotating bench is MOT:MTR0401

[AdrianPotter]

There a number of messages of the form

[2017-03-15 11:08:25] GalilAxis::beginMotion:GalilController::writeReadController:BGA:2010 COMMAND ERROR. Galil::command("BGA") got ? instead of : response. TC1 returned "22 Begin not possible due to limit switch"

[2017-03-15 11:08:25] sevr=major BGA: 22 Begin not possible due to limit switch

But that might be the Galil just ignoring a request to move beyond its limit switch. I don't see any smoking guns, but I'm not so familiar with what constitutes 'normal' Galil output.

[AdrianPotter]

The above message seems pretty common throughout the logs so is unlikely to be the culprit.

[AdrianPotter]

Autosave files indicate only the high limit (MOT:MTR0401.DHLM) has been changed since 6th March so I don't see any problems there.

[AdrianPotter]

This may need someone to go down to Larmor to examine the problem in person.

[AdrianPotter]

As mentioned by @GDH-ISIS, Rob has been known to change globals.txt which may make the problem the same as #2180, though the problem should only arise if the Galil macros were removed, not just because the file was changed. We should take a closer look at the workflow that led to the problem. We could also run a local simulation of Larmor's setup to see if the issue can be reproduced with simulated motors.

[GDH-ISIS]

In relation to limits not being applied correctly, I do not believe the limits for a Galil motor are the problem. The issue is more refined - the Galil does not know where the motor is. I have confirmed this on ALF using LabVIEW as a diagnostic. If an encoder is defined and operational, the motor must be told where it is before the move is actioned. This is already done in the LabVIEW driver. This issue is amplified much more when an offset is present on an axis, particularly with an absolute encoder (as on IMAT and ZOOM).

[kjwoodsISIS]

Bring this forward into Sprint 2017_03_09.
Whoever takes this ticket on needs to sit down with @GDH-ISIS and determine what the LabVIEW code (for Galils) is doing that IBEX is not. The solution then needs to be documented in the Wiki, so that we don't fall foul of this issue in future.
The solution will need to be deployed to all instruments - not just LARMOR and IMAT.

[FreddieAkeroyd]

Had a chat with Rob, he has been adjusting offsets to compensate for drift. The formula used for homeval means it is specified in user rather than dial coordinates, however if the direction used in the calculation does not match that used in the motor record you could get offset applied twice etc. (to test: do HOMER and HOMEF give different results?)

[FreddieAkeroyd]

Ok, dirm should be synced to the value used in the motor record, it is done so via a separate closed_loop DB record

[FreddieAkeroyd]

home5.txt

Galil 5 homing over time, I note for example

GALIL_05-20161216.log:[2016-12-16 11:48:12] sevr=info Poll services: applying motor G raw home position -762000
GALIL_05-20161216.log:[2016-12-16 11:48:12] sevr=info Poll services: applying encoder G raw home position -190500
GALIL_05-20170331.log:[2017-03-31 10:51:24] sevr=info Poll services: applying motor G raw home position -766000
GALIL_05-20170331.log:[2017-03-31 10:51:24] sevr=info Poll services: applying encoder G raw home position -191500

So a difference of 4000 motor / 1000 encoder - homeval has remained unchanged according to autosave, the offset has changed from 8.5 in December to 9.5 in march which correlates with the 1000

[GDH-ISIS]

Not sure on the calculation here - I can not quite make sense of it. Clearly I could be wrong of course, but at the risk of making myself look a fool ....

1. The Galil ioc has one home value and one homing routine (not sure about other facilities)
2. I think (1) would make the home value direction independent (if you were inverting the axis - that is different and needs a sign to be considered).
3. I would have thought homeval would include the offset. It does not make sense for it not to with a readback.
4. In calculating the encoder and motor values, there is an IF statement checking whether (homeval != 0.0000). I'm not sure I follow the reason why - the offset would still need to be considered when setting the motor and encoder values if the homeval was zero.
   I would just remove dirm and the IF statement (4) - leave the calculation in place - and try again. An additional check probably needs to be included to see whether the axis has been inverted - which I think is held in the motor record.
   Currently can not check as I think there is a problem communicating with the Galil.

[FreddieAkeroyd]

The homeval looks to be applied in user coordinates, userVAL = raw * res * DIR + OFFset with userVal being the homeval. If DIR is not the same in when the IOC defines position and when the motor record back calculates it, then I think you'd get a shift of twice the homeval in the readback? If homeval is zero it looks like this is treated as a special case, Rob has axes with both zero and non-zero homevals, he said he though all axes were drifting, but maybe that is not the case. I believe DIR is being propogated correctly though.

[FreddieAkeroyd]

Currently homing with a homeval != 0 will re-assert the homeval in user coordinates, so if homeval is 10 and Rob decides to apply an offset of .1 to make the readback 10.1, then next time he homes it will go back to 10 and he'll need to increase the offset to 0.2 to get the previous 10.1 He said he was currently setting the offset back to 0 before he homed, which would be consistent with working around this behaviour. If homeval is zero the Galil driver ignores the offset when doing DP/DE, so the user offset is then what you want the home position to look like in user space. Maybe using a direct offset approach is a better way to do home positions than using homeval? Or to make homeval in dial coordinates (i.e. minus any offset) so a user can apply an additional offset, though is this confusing that homeval isn't homeval? homeval will be 0.0 if unspecified, so i can see the reasoning for ignoring the offset in that case in the galil driver,

[FreddieAkeroyd]

Rob has confirmed this series of events

Hi Rob,

Would this describe your situation:

Home bench – it stops at a position of 30.0 say.
This isn’t the appropriate value, so you add an offset of 0.1 so it now reads 30.1
Later you home bench again – it comes back at 30.0 despite the offset of 0.1
To get 30.1 back you now need to increase the offset to 0.2
The only way to avoid an ever increasing offset is to set it to 0.0 before homing

The axes can have a “home value” (the 30.0) associated with them, it seems this is applied in user rather than dial coordinates and so will include/absorb any offset. However if the “home value” is 0 (unset) it is applied in raw/dial coordinates and so the offset is preserved and in fact the offset becomes the user readback at the home position

Regards,

Freddie

[John-Holt-Tessella]

The behaviour has now be confirmed on Demo. Use the dummy homing routines;

• Turn off Jog After Home (because it is confusing). caput %MYPVPREFIX%MOT:MTR0101_JAHV_SP 0
• Set a %MYPVPREFIX%MOT:MTR0101_HOMEVAL_SP to a value, x.
• Home
• The value set is as above, x
• Set an offset (value is now x + offset)
• Home
• Value is x but the offset is still set

Alternate behaviour:

• Set a TE:NDW1798:MOT:MTR0101_HOMEVAL_SP to 0.
• Home
• The value set is 0
• Set an offset (value is now offset)
• Home
• Value is offset

It is not clear what the correct behaviour is here. Should the home val be the value when a home is done or should it be offset (which is an EPICS thing). (NB a homeval of 0 indicates that it has never been set hence why this logic is implemented)

The options for resolving this ticket are:

1. For ISIS offset is always 0 (this allow lab view and epics to show the same values)
2. At ISIS homeval is always 0
3. Allow the users to set both homeval and offset but show a Galil OPI so they can more easily see the interplay between homeval and offset
4. Create a dhomeval (dial home value) this would be the home value without an offset
5. When a offset is added the homeval is updated to include the offset. When homeval is set the offset is set to 0.
6. Change the meaning of the current offset to be the same as DHomeVal

Discussion with users need to take place to verify what behaviour is needed.

[John-Holt-Tessella]

Impeded while @FreddieAkeroyd discusses with Galil author.

NB this is not affected by update to Galil drivers.

[FreddieAkeroyd]

Having exchanged a few emails with Mark Clift (Galil driver author), we are coming to the conclusion that dial coordinates for HOMEVAL is the best approach. This does mean that if an OFF is set then the readback at the home position will be "HOMEVAL + OFF" rather than "HOMEVAL", but it does avoid the situation we currently have of an OFF being gobbled up by the homeval logic and so creating a perceived "drift". Are we happy with this?

[John-Holt-Tessella]

OK final decision is to set HOMEVAL=0 for all cases and use the offset to mange differences between dial and user coordinates.
The acceptance criteria are:

1. All homevals are set to 0 on all instruments. This means:
   1. Home val = 0
   1. Offset needs setting
   1. Limits need adjusting
2. A new OPI linked from out motor summary page which allows users to set the offset without all the extra detail.

[FreddieAkeroyd]

This is consistent with HOMEVAL being in DIAL coordinates and a HOMEVAL of 0 currently has this behaviour with the existing driver

[John-Holt-Tessella]

see #2471 is this now done?

[FreddieAkeroyd]

I think homeval=0 solves the problem, moving to review