Cannot use non-standard baud rates for uploading on MacOS

[MCUdude]

Hi!

It seems like I can't upload a program to a target when a non-standard baud rate is used. I'm running a version of the Optiboot bootloader (-c arduino) that runs at 250000 baud. I'm able to upload to this using a Windows computer, but not on my mac.
115200 and 230400 baud works fine.

$ /Users/hans/Downloads/SVN/avrdude/avrdude -C/Users/hans/Documents/Arduino/hardware/MightyCore/avr/avrdude.conf -v -patmega324p -carduino -P/dev/cu.usbserial-1410 -b250000 -D -Uflash:w:/var/folders/6l/ypg6qbw172v1s4vtt6g990tw0000gn/T/arduino_build_264648/Blink.ino.hex:i 

avrdude: Version 6.3-20211205
         Copyright (c) 2000-2005 Brian Dean, http://www.bdmicro.com/
         Copyright (c) 2007-2014 Joerg Wunsch

         System wide configuration file is "/Users/hans/Documents/Arduino/hardware/MightyCore/avr/avrdude.conf"
         User configuration file is "/Users/hans/.avrduderc"
         User configuration file does not exist or is not a regular file, skipping

         Using Port                    : /dev/cu.usbserial-1410
         Using Programmer              : arduino
         Overriding Baud Rate          : 250000
avrdude: serial_baud_lookup(): Using non-standard baud rate: 250000avrdude: ser_setparams(): tcsetattr() failed
avrdude: ser_open(): can't set attributes for device "/dev/cu.usbserial-1410": Invalid argument

avrdude done.  Thank you.

[dl8dtl]

I'm afraid there is not much we can do about this. This appears to be a limitation of MacOS.

I just tried with a standard FTDI adapter, and the system-supplied cu utility:

sh-3.2# cu -l /dev/cu.usbserial-FTUBISA9 -s 115200
Connected.
~.

Disconnected.
sh-3.2# 
sh-3.2# 
sh-3.2# cu -l /dev/cu.usbserial-FTUBISA9 -s 1000000
cu: Unsupported baud rate 1000000
sh-3.2# cu -l /dev/cu.usbserial-FTUBISA9 -s 230400 
Connected.
~.

Disconnected.
sh-3.2# 
sh-3.2# cu -l /dev/cu.usbserial-FTUBISA9 -s 460800
cu: Unsupported baud rate 460800

[dl8dtl]

Even though they encode the baudrates as full numbers (as opposed to bitmapping them as it used to be in historic Unices), they appear to restrict the applicable baudrates to a pre-known set of numbers, even for hardware (like the FTDI) that can support almost arbitrary baudrates.

[dl8dtl]

/Applications/Xcode.app/Contents/Developer/Platforms/AppleTVOS.platform/Developer/SDKs/AppleTVOS.sdk/usr/include/sys/termios.h has:

/*
 * Standard speeds
 */
#define B0      0
#define B50     50
#define B75     75
#define B110    110
#define B134    134
#define B150    150
#define B200    200
#define B300    300
#define B600    600
#define B1200   1200
#define B1800   1800
#define B2400   2400
#define B4800   4800
#define B9600   9600
#define B19200  19200
#define B38400  38400
#if !defined(_POSIX_C_SOURCE) || defined(_DARWIN_C_SOURCE)
#define B7200   7200
#define B14400  14400
#define B28800  28800
#define B57600  57600
#define B76800  76800
#define B115200 115200
#define B230400 230400
#define EXTA    19200
#define EXTB    38400
#endif  /* (!_POSIX_C_SOURCE || _DARWIN_C_SOURCE) */

[MCUdude]

Thanks for taking the time to find the root cause. It's a real bummer though...

[hallard]

I'm sorry for the revival, but looks like it can works because on some other tools such as Arduino IDE console I can select this baud rate and it works with no issue (verified with CH340 and FTDI)

Also works fine with awesome Serial tool for mac

Not sure how they success to do that, maybe not using termios

[dl8dtl]

Well, if you find out what they are doing, I'll happily integrate it.

Remember: it's the call to tcsetattr() above that failed when applying a non-standard baud rate. But if tcsetattr() doesn't accept it, I'm somewhat at a loss here …

[hallard]

Yeah, I know that, I already faced and played with tc for an old Serial App and this issue on mac os, but as now we can read at 250K with other, was hoping it was solved.
Not blaming anyway, you're using the correct API, but yes would be interesting to see how they are doing that with success, if I had time I would love going some investigation :-)

[MCUdude]

Just for reference, the MiniTerm (based on Pyserial) works with "odd" baud rates on mac as well.

EDIT:
The magic seems to happen in serialposix.py:

https://github.com/pyserial/pyserial/blob/master/serial/serialposix.py

[dl8dtl]

That probably means we could look what pyserial does, in order to find out how to enable them on MacOS.

[MCUdude]

That probably means we could look what pyserial does, in order to find out how to enable them on MacOS.

Excellent! It's been a while since I've used anything pyserial based on my mac, but I'm 99% sure it works with custom baud rates. I'll give it an extra try tonight just to be sure. Let me know if you need help with testing this!

[MCUdude]

Can confirm that pyserial works 250000, 500000, and 1M baud on MacOS and a CH340C USB to serial chip. It probably works with more baud rates too!

[MCUdude]

Here's an update. I got Avrdude working with custom baud rates by tweaking some code I found online.

Starting with Tiger, the IOSSIOSPEED ioctl can be used to set arbitrary baud rates other than those specified by POSIX. The driver for the underlying serial hardware ultimately determines which baud rates can be used. This ioctl sets both the input and output speed. The speed will not be reflected in the termios structure.

// ser_setparams() {
// ...

  rc = tcsetattr(fd->ifd, TCSANOW, &termios);
  if (rc < 0) {
    // Try custom baudrate (MacOS only)
    #define IOSSIOSPEED 0x80045402
    speed = baud;
    if(ioctl(fd->ifd, IOSSIOSPEED, &speed) < 0) {
      avrdude_message(MSG_INFO, "%s: ser_setparams(): tcsetattr()/ioctrl() failed\n",
            progname);
    return -errno;
    }
  }

  tcflush(fd->ifd, TCIFLUSH);
  
  return 0;
}

$ ./avrdude  -C avrdude.conf -p atmega1284p -c arduino -b 500000 -P /dev/cu.usbserial-1410 -v

avrdude: Version 6.99-20211218
         Copyright (c) Brian Dean, http://www.bdmicro.com/
         Copyright (c) Joerg Wunsch

         System wide configuration file is "avrdude.conf"
         User configuration file is "/Users/hans/.avrduderc"
         User configuration file does not exist or is not a regular file, skipping

         Using Port                    : /dev/cu.usbserial-1410
         Using Programmer              : arduino
         Overriding Baud Rate          : 500000
avrdude: serial_baud_lookup(): Using non-standard baud rate: 500000         AVR Part                      : ATmega1284P
         Chip Erase delay              : 55000 us
         PAGEL                         : PD7
         BS2                           : PA0
         RESET disposition             : dedicated
         RETRY pulse                   : SCK
         Serial program mode           : yes
         Parallel program mode         : yes
         Timeout                       : 200
         StabDelay                     : 100
         CmdexeDelay                   : 25
         SyncLoops                     : 32
         PollIndex                     : 3
         PollValue                     : 0x53
         Memory Detail                 :

                                           Block Poll               Page                       Polled
           Memory Type Alias    Mode Delay Size  Indx Paged  Size   Size #Pages MinW  MaxW   ReadBack
           ----------- -------- ---- ----- ----- ---- ------ ------ ---- ------ ----- ----- ---------
           eeprom                 65    10   128    0 no       4096    8      0  9000  9000 0xff 0xff
           flash                  65    10   256    0 yes    131072  256    512  4500  4500 0xff 0xff
           lock                    0     0     0    0 no          1    1      0  9000  9000 0x00 0x00
           lfuse                   0     0     0    0 no          1    1      0  9000  9000 0x00 0x00
           hfuse                   0     0     0    0 no          1    1      0  9000  9000 0x00 0x00
           efuse                   0     0     0    0 no          1    1      0  9000  9000 0x00 0x00
           signature               0     0     0    0 no          3    1      0     0     0 0x00 0x00
           calibration             0     0     0    0 no          1    1      0     0     0 0x00 0x00

         Programmer Type : Arduino
         Description     : Arduino
         Hardware Version: 3
         Firmware Version: 8.0

avrdude: AVR device initialized and ready to accept instructions

Reading | ################################################## | 100% 0.00s

avrdude: Device signature = 0x1e9705 (probably m1284p)

avrdude done.  Thank you.

I've pushed the fix here:
https://github.com/MCUdude/avrdude/blob/c827120513dc6ad4e20022460b49306c0a4f82b3/src/ser_posix.c#L200-L215

References:
https://github.com/pyserial/pyserial/blob/master/serial/serialposix.py
https://fpc-pascal.freepascal.narkive.com/oI4b0CM2/non-standard-baud-rates-in-os-x-iossiospeed-ioctl
https://discussions.apple.com/thread/1587991

[dl8dtl]

I'm a bit embarrassed why they introduced that hack. Nothing would have prevented them from simply accepting those non-standard baud rates in a simple tcsetattr() call as well, as an extension to those required for Posix conformance.

I'm not a friend of local hacks like #define IOSSIOSPEED 0x80045402. There ought to be a header file that correctly defines that already, and the respective code can then be #ifdefed either on that macro itself, or on a MacOS-specific compiler macro.

[MCUdude]

This is perhaps a better way. Tested and works:

    #ifndef IOSSIOSPEED
    #define IOSSIOSPEED _IOW('T', 2, speed_t)
    #endif

[MCUdude]

I also added __APPLE__:

  rc = tcsetattr(fd->ifd, TCSANOW, &termios);
  if (rc < 0) {
  // Try custom baudrate (MacOS only)
#ifdef __APPLE__
    #ifndef IOSSIOSPEED
      #define IOSSIOSPEED _IOW('T', 2, speed_t)
    #endif
    speed = baud;
    if(ioctl(fd->ifd, IOSSIOSPEED, &speed) < 0) {
      avrdude_message(MSG_INFO, "%s: ser_setparams(): tcsetattr()/ioctrl() failed\n",
            progname);
      return -errno;
    }
#else
    avrdude_message(MSG_INFO, "%s: ser_setparams(): tcsetattr() failed\n",
                progname);
          return -errno;

#endif
  }

  tcflush(fd->ifd, TCIFLUSH);
  
  return 0;
}

[MCUdude]

It looks like IOSSOISPEED is defined in

/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/Frameworks/IOKit.framework/Versions/A/Headers/serial/ioss.h

But I'm not sure if we can rely on this file being present on every MacOS machine set up to build Avrdude.

[dl8dtl]

Except that the path is even longer here (full XCode package):
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.1.sdk/System/Library/Frameworks/IOKit.framework/Versions/A/Headers/serial/ioss.h
it's there. I guess one way or another, it needs to be around anyway if you want to compile that stuff.
I would make the decision whether this needs to be called a little more sophisticated though. Rather than trying that ioctl for any error return from tcsetattr, we ought to be able to know in advance whether we are trying to configure a non-standard baud rate. In that case, just call tcsetattr without changing the baud rate first. If there's an error, handle it (and stop there). Then, if it's a non-standard baudrate, insert that IOSSIOSPEED ioctl. If that triggers an error, again, handle it.
Just chaining the ioctl for any tcsetattr error might yield very misleading errno values, hiding the actual reason.

[MCUdude]

I'm able to include ioss.h by adding the following code:

#ifdef __APPLE__
#include <IOKit/serial/ioss.h>
#endif

I would make the decision whether this needs to be called a little more sophisticated though.

Agreed. I just created a simple proof-of-concept code to help out point in the right direction. It looks like pyserial is just using a flag to indicate this.
But I'm very happy a solution is possibly on its way! As a mac user this has been one of the more annoying things with the Avrdude mac port, even though this is strictly a macOS quirk and not Avrdude's fault.

[hallard]

Awesome guys, II already faced this issue two years ago, and running an old project with custom moteino boards setup to 250K today I was unable to flash them. I flashed back all the board with standard baud rate toda) so it's awesome it's fixed.

If one could attach the compiled file fixed, I can give a try on my side (and I'm really interested)

Anyway, thank for your investigation and this prompt fix.

[dl8dtl]

#include <IOKit/serial/ioss.h>

Takk, Hans. I didn't immediately figure out how that complicated path in the filesystem is supposed to be written in sourcecode. I assumed it must be way simpler. (Alas dtruss doesn't work anymore out of the box, as the system by default runs in a pretty secure mode now.)

Give PR #898 a try (and feel free to add review comments). I could only test the ioctl is accepted by my OS but I don't have anything I could actually use at those speeds right here.

[dl8dtl]

If one could attach the compiled file fixed,

Check out the Wiki. Marius mentioned there how you can fetch the build artifacts from the CI process.

[hallard]

I'll try that, I'm trying to build with instruction with no luck, each time I need to build something with libusb and libftdi it's a mess :-)

[dl8dtl]

I'm trying to build with instruction with no luck

I think the Wiki describes the prerequisites pretty well now. It almost doesn't matter whether you obtain the prerequisites using brew (Hans uses it) or Mac ports (I use them when testing on MacOS here). Then, just use the new build.sh script in the toplevel directory.

[hallard]

Thanks for your help, but does not works either include files (I remember I needed to fix lot of stuff compiling airspy)
build.sh complained missing libftdi1.h and libusb.h and they are on my mac, I fixed changing a line in the build

changed

build_flags="${build_flags} -D CMAKE_C_FLAGS=-I/usr/local/include -D CMAKE_EXE_LINKER_FLAGS=-L/usr/local/Cellar"

by

build_flags="${build_flags} -D CMAKE_C_FLAGS=-I/usr/local/include -D CMAKE_C_FLAGS=-I/opt/homebrew/include -D CMAKE_EXE_LINKER_FLAGS=-L/opt/homebrew/Cellar"

Now it goes further but fail later,

charles@mini-m1:avrdude$ ./build.sh 
-- Configuration summary:
-- ----------------------
-- DO HAVE    libelf
-- DO HAVE    libusb
-- DO HAVE    libusb_1_0
-- DO HAVE    libhidapi
-- DO HAVE    libftdi (but prefer to use libftdi1)
-- DO HAVE    libftdi1
-- DISABLED   doc
-- DISABLED   parport
-- DISABLED   linuxgpio
-- DISABLED   linuxspi
-- ----------------------
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/charles/devt/avrdude/build_darwin
Consolidate compiler generated dependencies of target libavrdude
[  1%] Building C object src/CMakeFiles/libavrdude.dir/dfu.c.o
In file included from /Users/charles/devt/avrdude/src/dfu.c:33:
/Users/charles/devt/avrdude/src/dfu.h:32:4: error: "libusb needs either <usb.h> or <lusb0_usb.h>"
#  error "libusb needs either <usb.h> or <lusb0_usb.h>"
   ^
/Users/charles/devt/avrdude/src/dfu.h:52:3: error: unknown type name 'usb_dev_handle'
  usb_dev_handle *dev_handle;
  ^
/Users/charles/devt/avrdude/src/dfu.h:53:32: error: field has incomplete type 'struct usb_device_descriptor'
  struct usb_device_descriptor dev_desc;
                               ^
/Users/charles/devt/avrdude/src/dfu.h:53:10: note: forward declaration of 'struct usb_device_descriptor'
  struct usb_device_descriptor dev_desc;
         ^
/Users/charles/devt/avrdude/src/dfu.h:54:32: error: field has incomplete type 'struct usb_config_descriptor'
  struct usb_config_descriptor conf_desc;
                               ^
/Users/charles/devt/avrdude/src/dfu.h:54:10: note: forward declaration of 'struct usb_config_descriptor'
  struct usb_config_descriptor conf_desc;
         ^
/Users/charles/devt/avrdude/src/dfu.h:55:35: error: field has incomplete type 'struct usb_interface_descriptor'
  struct usb_interface_descriptor intf_desc;
                                  ^
/Users/charles/devt/avrdude/src/dfu.h:55:10: note: forward declaration of 'struct usb_interface_descriptor'
  struct usb_interface_descriptor intf_desc;
         ^
/Users/charles/devt/avrdude/src/dfu.h:56:34: error: field has incomplete type 'struct usb_endpoint_descriptor'
  struct usb_endpoint_descriptor endp_desc;
                                 ^
/Users/charles/devt/avrdude/src/dfu.h:56:10: note: forward declaration of 'struct usb_endpoint_descriptor'
  struct usb_endpoint_descriptor endp_desc;
         ^
/Users/charles/devt/avrdude/src/dfu.c:97:30: error: unknown type name 'usb_dev_handle'
static char * get_usb_string(usb_dev_handle * dev_handle, int index);
                             ^
/Users/charles/devt/avrdude/src/dfu.c:152:3: error: implicit declaration of function 'usb_init' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
  usb_init();
  ^
/Users/charles/devt/avrdude/src/dfu.c:153:3: error: implicit declaration of function 'usb_find_busses' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
  usb_find_busses();
  ^
/Users/charles/devt/avrdude/src/dfu.c:154:3: error: implicit declaration of function 'usb_find_devices' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
  usb_find_devices();
  ^
/Users/charles/devt/avrdude/src/dfu.c:154:3: note: did you mean 'usb_find_busses'?
/Users/charles/devt/avrdude/src/dfu.c:153:3: note: 'usb_find_busses' declared here
  usb_find_busses();
  ^
/Users/charles/devt/avrdude/src/dfu.c:191:14: error: use of undeclared identifier 'usb_busses'
  for (bus = usb_busses; !found && bus != NULL; bus = bus->next) {
             ^
/Users/charles/devt/avrdude/src/dfu.c:191:58: error: incomplete definition of type 'struct usb_bus'
  for (bus = usb_busses; !found && bus != NULL; bus = bus->next) {
                                                      ~~~^
/Users/charles/devt/avrdude/src/dfu.c:163:10: note: forward declaration of 'struct usb_bus'
  struct usb_bus *bus;
         ^
/Users/charles/devt/avrdude/src/dfu.c:192:19: error: incomplete definition of type 'struct usb_bus'
    for (dev = bus->devices; !found && dev != NULL; dev = dev->next) {
               ~~~^
/Users/charles/devt/avrdude/src/dfu.c:163:10: note: forward declaration of 'struct usb_bus'
  struct usb_bus *bus;
         ^
/Users/charles/devt/avrdude/src/dfu.c:192:62: error: incomplete definition of type 'struct usb_device'
    for (dev = bus->devices; !found && dev != NULL; dev = dev->next) {
                                                          ~~~^
/Users/charles/devt/avrdude/src/dfu.c:161:10: note: forward declaration of 'struct usb_device'
  struct usb_device *found = NULL;
         ^
/Users/charles/devt/avrdude/src/dfu.c:193:46: error: incomplete definition of type 'struct usb_bus'
      if (dfu->bus_name != NULL && strcmp(bus->dirname, dfu->bus_name))
                                          ~~~^
/Users/charles/devt/avrdude/src/dfu.c:163:10: note: forward declaration of 'struct usb_bus'
  struct usb_bus *bus;
         ^
/Users/charles/devt/avrdude/src/dfu.c:196:23: error: incomplete definition of type 'struct usb_device'
        if (strcmp(dev->filename, dfu->dev_name))
                   ~~~^
/Users/charles/devt/avrdude/src/dfu.c:161:10: note: forward declaration of 'struct usb_device'
  struct usb_device *found = NULL;
         ^
/Users/charles/devt/avrdude/src/dfu.c:198:28: error: incomplete definition of type 'struct usb_device'
      } else if (vid != dev->descriptor.idVendor)
                        ~~~^
/Users/charles/devt/avrdude/src/dfu.c:161:10: note: forward declaration of 'struct usb_device'
  struct usb_device *found = NULL;
         ^
/Users/charles/devt/avrdude/src/dfu.c:200:38: error: incomplete definition of type 'struct usb_device'
      else if (pid != 0 && pid != dev->descriptor.idProduct)
                                  ~~~^
/Users/charles/devt/avrdude/src/dfu.c:161:10: note: forward declaration of 'struct usb_device'
  struct usb_device *found = NULL;
         ^
/Users/charles/devt/avrdude/src/dfu.c:218:36: error: incomplete definition of type 'struct usb_device'
                    progname, found->descriptor.idVendor, found->descriptor.idProduct,
                              ~~~~~^
/Users/charles/devt/avrdude/src/dfu.c:161:10: note: forward declaration of 'struct usb_device'
  struct usb_device *found = NULL;
         ^
fatal error: too many errors emitted, stopping now [-ferror-limit=]
20 errors generated.
make[2]: *** [src/CMakeFiles/libavrdude.dir/dfu.c.o] Error 1
make[1]: *** [src/CMakeFiles/libavrdude.dir/all] Error 2
make: *** [all] Error 2
Build failed.

I need to focus on other urgent task so I'm leaving here, may be give a try tomorrow on my office's Mac

Thanks for your time

[MCUdude]

Fixed in #898