I cleaned it up and got it building so you can continue to test and refine. The handling of the probe probably needs to be more specific. Users need to be able to mix-and-match endstops and probes according to their designs, but we can also add options to enable specific CAN peripherals that have known protocols and hardware such as built in Z probes and sensorless XY endstops, etc.

I cleaned it up and got it building so you can continue to test and refine.

@thinkyhead Thanks, well done, you did a lot of good work there. You could have told me that I completely forgot to include CAN.h, sorry about that!

The handling of the probe probably needs to be more specific. Users need to be able to mix-and-match endstops and probes according to their designs, but we can also add options to enable specific

Agree, this needs some work, preparations are made, but it's currently only implemented for the Z-probe.
Virtual Endstops work like this:
On the head board the ENDSTOP_INTERRUPTS_FEATURE is enabled, which triggers and interrupt which sends a CAN message with the updated IO status to the host. On the host the incoming CAN message triggers, an interrupt which causes the CAN message to be reads from the CAN hardware FIFO, the read message updates CAN_io_state and "endstops.update()" is called.

Here are the masks to get the relevant bits stored in CAN_io_state:

  #define CAN_PROBE_MASK                   1 // Virtual IO bit for probe
  #define CAN_FILAMENT_MASK                2 // Virtual IO bit for filament
  #define CAN_X_ENDSTOP_MASK               4 // Virtual IO bit for X-endstop
  #define CAN_Y_ENDSTOP_MASK               8 // Virtual IO bit for Y-endstop
  #define CAN_Z_ENDSTOP_MASK              16 // Virtual IO bit for Z-endstop
  #define CAN_STRING_MESSAGE_MASK         32 // Signals the head sent a string message
  #define CAN_REQUEST_SETUP_MASK          64 // Signals the head requests setup information
  #define CAN_TMC_OT_MASK                128 // Signals the head signals a TMC Over Temp error
  #define CAN_E0_TARGET_MASK             256 // Signals E0 or E1
  #define CAN_ERROR_MASK                 512 // Signals the head encountered an error

One should be able to define which CAN virtual IO bits are relevant for the host, only these endstop results should be transferred to the host process. I'm not sure how I could transparently transfer the virtual endstop status into Marlin.
@thinkyhead Perhaps you can point me in the right direction...

CAN peripherals that have known protocols and hardware such as built in Z probes and sensorless XY endstops, etc.

I believe most Z-probes should work, they are configured on the head side. Servo commands are forwarded to the head, so BLTouch/3DTtouch and servo probes should work (I use a BLTouch).
Any inductive, capacitive or micro switch based probe will work, as they just rely on a digital pin status.
I'm note sure about MAGLEV4 probes, if they are triggered by gcode commands, then they should also work (relevant gcodes need to be forwarded).
BD_SENSOR will need some work, the head can send any data required to the host.

What CAN peripherals do you have in mind? Stepper drivers?

I have not considered sensorless XY endstops on the head board, I assumed to leave this on the host side. The board that controls X and Y should be the host.
For reference, here the pinout of the head board I use, it's a tiny 42x42mm board with only 1 stepper driver.

I would be quite interested to get the TMC2209 and ADXL345 3-axis accelerometer working.
The TMC2209 is fully supported of course, the problem is syncing the movements between the host and the head. I tried sending the extruder gcodes to the head, which works fine, but soon the extruder motion will go out of sync. I'm not sure how to handle this, perhaps I should send the motion blocks directly to the head, or some dedicated synchronization might be required.

you'll need clock synchronization between the can boards (and execute based on synchronized time). https://github.com/Duet3D/CANlib/blob/3.5-dev/doc/Duet3CAN-FDProtocol.md#time-synchronisation may be a useful read.

Just to clarify, here the current printer setup I'm running, 8 wires go to the head, 4 for the CAN bus and 4 for the extruder stepper. This is fully working already.

you'll need clock synchronization between the can boards (and execute based on synchronized time). https://github.com/Duet3D/CANlib/blob/3.5-dev/doc/Duet3CAN-FDProtocol.md#time-synchronisation may be a useful read.

Great tip, thanks. I had a look, they report that FD CAN is needed, because they want to transmit more than 8 bytes per message, which is not supported by CAN. The STM32F4xx does not support FD CAN. The STM32G0 (BTT EBB42 V1.2) supports FD CAN. Currently I don't see a real reason why time synchronization could not work over CAN because two 32bit timestamps will fit within the available 8 bytes, so a NTP style sync should work.
I will give that a try and see what numbers I find for time sync accuracy, turn-around time and drift based on Marlin's 2MHz stepper clock.

I'm testing a NTP like time sync, here's the code running on the tool head: The numbers are timestamps based on micros()

// NTP style time sync
  // t[0] = local time sync request time
  // t[1] = host time sync receive time
  // t[2] = host time sync reply time
  // t[3] = local time stamp receive time
  if (t[3] != 0) {
    t[0] += time_offset; // Adjust local time stamps with time_offset
    t[3] += time_offset; // Adjust local time stamps with time_offset
    uint32_t local_time_adjustment = (t[1] - t[0] + t[2] - t[3]) >> 1;
    time_offset += local_time_adjustment;
    uint32_t Round_trip_delay = (t[3] - t[0] - t[2] + t[1]);
    SERIAL_ECHOLNPGM("t0: ", t[0], " us");
    SERIAL_ECHOLNPGM("t1: ", t[1], " us");
    SERIAL_ECHOLNPGM("t2: ", t[2], " us");
    SERIAL_ECHOLNPGM("t3: ", t[3], " us");
    SERIAL_ECHOPGM("Local time adjustment: ", local_time_adjustment, " us");
    if (t[4]) {
      SERIAL_ECHOLNPGM(" after ",  ftostr42_52(float(t[0] - t[4]) / 1000000.0), " seconds");
      SERIAL_ECHOLNPGM("Drift: ", local_time_adjustment / (float(t[0] - t[4]) / 1000000.0), " us/s");
    }
    else
      SERIAL_EOL();

    SERIAL_ECHOLNPGM("Time offset: ", time_offset, " us");
    SERIAL_ECHOLNPGM("Round trip delay: ", Round_trip_delay, " us");
    SERIAL_ECHOLNPGM("Host response time: ", (t[2] - t[1]), " us");

    t[4] = t[0]; // Store previous time sync request time
    t[3] = 0;
  }

Here 2 logs:

t0: 150581211 us
t1: 150582363 us
t2: 150582365 us
t3: 150581446 us
Local time adjustment: 1035 us after 74.75 seconds
Drift: 13.85 us/s
Time offset: 2677282 us
Round trip delay: 233 us
Host response time: 2 us

t0: 210998246 us
t1: 210999201 us
t2: 210999209 us
t3: 210998484 us
Local time adjustment: 840 us after 60.42 seconds
Drift: 13.90 us/s
Time offset: 2678122 us
Round trip delay: 230 us
Host response time: 8 us

I'm not sure how accurate the time synchronization needs to be, anybody?
Based on these numbers I think an accurate time sync is possible, especially if the drift is taken into account actively.
I can foresee an issue.
Even if the tool head movement start time is correct, the movement finish time could be different because of the drift. The tool head could finish the movement early than the host, causing stutters, or later which could cascade to the next moves. So I guess the movement on the tool head actively needs to be adjusted (slowed down or accelerated) to counter this effect.
Any input on how to proceed is welcome.

TL;DR if you look elsewhere, RRFs time-sync protocol is precise for up to 10us.

Maybe a couple notes here:

• the round trip time of 233 us is awfully close to Klipper's maximum round trip time allowed of 250us

• RRFs CAN-FD has way lower round trip times than Klipper (mostly because it doesn't layer on top of USB). I wil ltry to find where I saw the numbers.

• A time drift of 1000us per minute is catastrophic for 3d printing. It's 0.1 seconds. In 0.1 seconds worth of deviation

  • extrusion at 30mm/sec will miss corners, and retracts and PA moves will just not happen at the right times
  • endstops and probes will report early or late, missing the mark.

Please also consider that it took RRF about 5 years to remove most, but not all limitations when using CAN-FD connected expansion boards, many of which relate to board-to-board communication delays or logic challenges.

There are some further differences between Klipper's CAN implementation and RRFs CAN-FD implementation:

• Klipper's CAN is layered on top of USB and never can be better than USB for connecting MCUs for that reason
• The Klipper host-to-mcu protocol sends step pulse trains; RRF sends complete moves
• RRF has liveness checks for CAN-FD boards and you can (and must) manage when boards disappear or appear (DangerKlipper has a facility to mark boards as non critical, such as accelerometers. Usually any of the boards disappearing makes Klipper crash in an attempt to not run uncontrolled)
• RRF allows you to assign (numeric) IDs to boards, which makes it easy to reference them for configuration. Marlin's mostly static approach to configuration probably also needs this for tunable parameters and error reporting. Trying to sort out boards by UUIDs is no fun.
• Since I am already rambling, let me add that the bandwidth of CAN will likely be sufficient for simple setups like one main board and one or two toolheads. More complex setups like tool changers or machines with a higher number of axes that need to move in coordination (not necessarily 3d printers) may put a squeeze on CAN, and CAN-FD will give you more headspace here.

Thanks a lot for the detailed reply!

TL;DR if you look elsewhere, RRFs time-sync protocol is precise for up to 10us.

OK, I will target a time-sync below 10us.

Maybe a couple notes here:
* the round trip time of 233 us is awfully close to Klipper's maximum round trip time allowed of 250us

Some background. The used CAN bus speed is 1M bit, which is the max for the STM32F4 board, the STM32G0 board supports 2M bit. A Standard ID (11 bits) CAN message with an 8 byte payload (2 x 32bit timestamp) is already 115 bits (if I counted it correctly), and thus 115us for one way of the message. I could perhaps optimize that a little bit, but not by much.
I do not understand why the round trip time is so critical (max allowed 250us, why?), I don't think there is much influence on the time sync accuracy, unless the numbers get really big.

* RRFs CAN-FD has way lower round trip times than Klipper (mostly because it doesn't layer on top of USB). I wil ltry to find where I saw the numbers.

* A time drift of 1000us per minute is catastrophic for 3d printing. It's 0.1 seconds. In 0.1 seconds worth of deviation

The boards I use have low cost crystal oscillators which have a typical accuracy of +/- 20ppm. The drift I found is about 14ppm which is the difference between 2 boards (not a bad result!). This will be the same for any firmware that is running on the boards. The software will have to actively compensate for the drift. And a suitable resync interval can be chosen to guarantee that the maximal deviation stays below 10us. I think that is possible, but more testing is needed.

  * extrusion at 30mm/sec will miss corners, and retracts and PA moves will just not happen at the right times
  * endstops and probes will report early or late, missing the mark.

I am already using this setup for probing (BLTouch), which works very reliably. The probing process is fully interrupt driven (on both sides). The CAN bus just adds a relatively constant time delay of about 100us. As long as the time delay is constant there will be no effect on the probing results. Note that depending on your hardware and Marlin setup, probing will be done with an accuracy of 1ms (1000 samples/s), so the CAN bus should not be the limiting factor.

Please also consider that it took RRF about 5 years to remove most, but not all limitations when using CAN-FD connected expansion boards, many of which relate to board-to-board communication delays or logic challenges.

There are some further differences between Klipper's CAN implementation and RRFs CAN-FD implementation:

* Klipper's CAN is layered on top of USB and never can be better than USB for connecting MCUs for that reason

* The Klipper host-to-mcu protocol sends step pulse trains; RRF sends complete moves

That is interesting, I need to read more about this. Internally Marlin has move blocks. I would expect to send these blocks to the tool head with a start timestamp. The tool head probably needs to adjust the move block to compensate for the difference in clock frequency, but the step count must not be affected.

* RRF has liveness checks for CAN-FD boards and you can (and must) manage when boards disappear or appear (DangerKlipper has a facility to mark boards as non critical, such as accelerometers. Usually any of the boards disappearing makes Klipper crash in an attempt to not run uncontrolled)

* RRF allows you to assign (numeric) IDs to boards, which makes it easy to reference them for configuration. Marlin's mostly static approach to configuration probably also needs this for tunable parameters and error reporting. Trying to sort out boards by UUIDs is no fun.

* Since I am already rambling, let me add that the bandwidth of CAN will likely be sufficient for simple setups like one main board and one or two toolheads. More complex setups like tool changers or machines with a higher number of axes that need to move in coordination (not necessarily 3d printers) may put a squeeze on CAN, and CAN-FD will give you more headspace here.

Right, there is a lot of work to do, I try to start simple, then go step by step.
I just posted the tool head side of the software, so anybody can give it a try now.

I just posted the tool head side of the software, so anybody can give it a try now.

I just want to say that I am absolutely positively excited about this and I just want to make sure CAN / CAN-FD in Marlin will be as delightful an experience as in RRF (-:

I just posted the tool head side of the CAN software, which is running on a BTT EBB42 V1.2 board (STM32G0). The board has FD CAN support at 2M bit/s, but because the host (STM32F407) only has CAN support, the FDCAN features cannot be used.

IMPORTANT NOTE:

It seems the used framework to compile Marlin for the STM32G0 does not have full support for the CAN callbacks. To solve this I needed to override a function, which means that I had to make a change to the framework. I talked to the developers about this a long time ago, but have not seen any action on it yet, perhaps it will only be done in a future version.

The framework change that is required (Windows environment in PlatformIO).
In \Users<username>.platformio\packages\framework-arduinoststm32@4.20600.231001\libraries\SrcWrapper\src\HardwareTimer.cpp
Add "__weak" in front of "void TIM16_IRQHandler(void)"
So you get: __weak void TIM16_IRQHandler(void)

Note that this part "framework-arduinoststm32@4.20600.231001" of the framework path could be different, depending on which frameworks you have installed already. An easy way to find where the framework files are located is by using the "Go to definition" function in PlatformIO (F12 or right click) on a function that is provided by the framework. For example you can right click on "HAL_FDCAN_GetTxFifoFreeLevel".

I just want to say that I am absolutely positively excited about this

Me too. Fewer wires to the toolhead will be so nice!

just a note that I think the deadline timeout of Klipper likely is 250msec not usec. Polling rate of USB is 125usec already, so reply/response hits those 250usec in an ideal stream without any processing. Sorry for mixing up the millis and the micros!

For the changes I pushed in dd5a2ff, I meant EBB42, but same difference 😄 Since CAN will be more common than the filament extruder configuration, I reordered them.

I also ran a make format-pins -j to format the MONSTER8 pins in 73bb491 which takes care of the failing "Validate Pins Files" test.

Here some new data which shows that the host and tool head time can be calculated accurately based on a few time sync cycles. Note that the earlier and shorter test periods are less accurate. Later we are within 1us even after 3 minutes.

t0: 65082290 us
t1: 65082522 us
t2: 65082532 us
t3: 65082534 us
Predicted time adjustment: 114.82 us
Local time adjustment: 115 us after 8.19 seconds
Drift: 14.04 us/s
Time offset: 2676043 us
Round_trip_delay: 234 us
Host response time: 10 us

================================

t0: 79930405 us
t1: 79930729 us
t2: 79930742 us
t3: 79930651 us
Predicted time adjustment: 208.44 us
Local time adjustment: 207 us after 14.85 seconds
Drift: 13.94 us/s
Time offset: 2676250 us
Round_trip_delay: 233 us
Host response time: 13 us

=========================

t0: 259642612 us
t1: 259645234 us
t2: 259645250 us
t3: 259642862 us
Predicted time adjustment: 2505.40 us
Local time adjustment: 2505 us after 179.71 seconds
Drift: 13.94 us/s
Time offset: 2678755 us
Round_trip_delay: 234 us
Host response time: 16 us

I would like to add that there are CAN-FD modules for RRF in stepstick or EXP1/EXP2 format that add CAN-FD via SPI. I believe @thinkyhead might have gotten one in the past, and if there is interest I could try to get you one.

Thanks for your kind offer, I guess you mean one of those MCP2518FD based boards. That is perhaps something to look into at a later time. I believe FDCAN is not required for this task, CAN has the needed speed and bandwidth. FDCAN can run faster and can send more data (64 bytes instead of 8 bytes) per message. That would make things easier of course.
I had a look at the CAN transceiver on the EBB42 board (TI SN65HVD1050), it officially only supports up to 1Mbit. Perhaps 2Mbit is still possible, I will give it a test if a have an opportunity.

just a note that I think the deadline timeout of Klipper likely is 250msec not usec. Polling rate of USB is 125usec already, so reply/response hits those 250usec in an ideal stream without any processing. Sorry for mixing up the millis and the micros!

I'm not sure if we are talking about the same thing. The data log are only about a NTP like time sync protocol I'm testing.
t0 = (client) time sync request send time
t1 = (host) time sync request receive time
t2 = (host) time sync request reply time
t3 = (client) time sync request response time
Round trip time = ((t1 - t0) + (t2 - t3)) / 2

@thisiskeithb Thanks again for the great support.

In my view there are 2 main task to get synchronized motion working:

1. Time needs to be synchronized between the host and the tool head (target better than 10us). This seems possible (see previous post).
2. The movement data needs to be transmitted from the host to the tool head.
   There are a lot of questions to be answered here. To be continued.

Some thoughts on motion synchronization. I can foresee some issues:

Even if we can accurately start a movement, the movement duration will not be the same because of the clock rates difference. We can try to adjust the movement speed, but I believe there is not enough resolution to make the required tiny adjustments.
If a tool head movement takes longer than the host movement, then the next tool head move will also start later, which can cascade to become a bigger issue. If the host is aware of this issue, then it can actively compensate for it.

Could the following approach work?

1. The host processes a gcode, for example "G1 X10 Y20 Z30 E10 F3000", which is converted into movement blocks.

2. The gcode and the start time of the first related block are transmitted to the tool head, which based on the gcode creates the same movement blocks and adds the movement start time to the first block (or perhaps all blocks).

3. If a movement block has a start time then we wait until the start time has come. If the block has no start time then the block is processed as usual. This mean that the first host movement has to be delayed a bit to allow the tool head to get ready.

I will investigate the motion blocks to get more insights.

UPDATE:

1. I had a look at the movement blocks. There is a lot going on there. The movement blocks are continuously updated when new blocks are added. A newly added block can cause an adjustment of all not yet processed blocks. When synchronizing the movement, they can start at the same time, but what if a new block is added and the block optimization process in a different phase? In my case the host is a STM32F407 running at 168MHz, while the tool head is a STM32G0 running at 64MHz, so the block processing times could be quite different. If there is enough time to process the blocks, then there is no issue, so I'm not sure if this is a problem in practice.

2. The total block processing time seems to be non-deterministic because the pulses are not created in relation to a fixed time reference, but in relation to the previous pulse. This can introduce small errors, which will add up, and become bigger if there are more pulses in a block. This implies that no exact start time for the next block can be calculated beforehand.

Any input is very welcome!

Does anybody know where I can find a detailed description of Marlin's Fixed-Time motion? Or enhance/correct my findings below.

I had a look at the source code, but don't fully understand how Marlin's Fixed-Time motion works.
It seems that 8-bit commands are used, encoding both steps and direction for X, Y, Z and E(Bit 0=E direction, Bit 1=E step etc.).
The command buffer has a size of 20.000 bytes (FTM_STEPPER_FS, 30.000 for CoreXy). The stepper clock runs as 2MHz (for STM32), but a step or direction change can only be generated every 20 (or 30 for CoreXY) stepper clocks. So basically this results in a byte stream of 100.000 bytes/s (2M / 20) which defines the steps and directions for 4 steppers (XYZE).

Sending 100.000 bytes/s will not be possible over CAN at 1Mbit/s, but if we only care about the extruder stepper, then only 25.000 bytes/s will be needed. This seems possible. Compressing the data might be possible, but with a 20.000 bytes buffer the whole process is quite time critical because there is only 0.2seconds of buffer.

I'm working on the ADXL345 3-axis accelerometer, so if anybody wants acceleration data, then please let me know.
I'm not sure what mode to set the device to, so for now I use a range of 2g (max is 16g), 10 bits of resolution(max is 13 bits), 100Hz sample rate (max is 3200Hz, which is noisy) in streaming mode(uses the 32 sample FIFO).
In this mode the LSB represents 4milli g. A value of 256 represent 1g.

Note that in streaming mode the data read is from the FIFO, which is from 32 samples ago (at 100Hz). So if the data needs to be real time as much as possible then this is not the best mode. I just followed the datasheet recommendations to get started.