Add Apple M1 silicon to Nightly-Builds

[hallard]

Hi there,

Tried back from #771, I tried today to build for my Mac M1 and and can't get the compile process going correctly to end on M1 silicon (I tried to investigate but each time I pass a new step, new errors fire until I'm stuck on last error that I can't fix)

Anyway, since I just wanted to try with this non standard baud rate setting on mac OS I gave a try with nightly builds
https://github.com/avrdudes/avrdude/wiki/Getting-Nightly-Builds-for-AVRDUDE
But there is no arm64 architecture (Apple M1) available. As rosetta can run executables with x86_64 architecture it can't mix it with arm64 lib (which are by default native on my MAC), firing this message which makes sense

charles@m1-office:avrdude-macos-x86_64$ ./avrdude 
dyld[68388]: Library not loaded: /usr/local/opt/hidapi/lib/libhidapi.0.dylib
  Referenced from: /Users/charles/Downloads/avrdude-macos-x86_64/avrdude
  Reason: tried: '/usr/local/opt/hidapi/lib/libhidapi.0.dylib' (no such file), '/usr/local/lib/libhidapi.0.dylib' (mach-o file, but is an incompatible architecture (have 'arm64', need 'x86_64')), '/usr/lib/libhidapi.0.dylib' (no such file)
Abort trap: 6

I succeded to run downloading x86_64 libraries for libhidapi and libftdi and put them into /usr/local/lib (I'm lucky the native arm64 lib are not located into this folder so it should not conflict)

Process is not so straighforward, fetch correct architecture

brew fetch --force --bottle-tag=x86_64_monterey hidapi libftdi

untargz them where you want then create symlink (or copy files)

sudo ln -s /Users/charles/Downloads/avrdude-macos-x86_64/hidapi/0.11.2/lib/libhidapi.0.dylib /usr/local/lib/
sudo ln -s /Users/charles/Downloads/avrdude-macos-x86_64/libftdi/1.5_2/lib/libftdi1.2.dylib /usr/local/lib/

And finally it works

charles@m1-office:avrdude-macos-x86_64$ ./avrdude 
Usage: avrdude [options]
Options:
  -p <partno>                Required. Specify AVR device.
  -b <baudrate>              Override RS-232 baud rate.
  -B <bitclock>              Specify JTAG/STK500v2 bit clock period (us).
  -C <config-file>           Specify location of configuration file.
  -c <programmer>            Specify programmer type.
  -D                         Disable auto erase for flash memory
  -i <delay>                 ISP Clock Delay [in microseconds]
  -P <port>                  Specify connection port.
  -F                         Override invalid signature check.
  -e                         Perform a chip erase.
  -O                         Perform RC oscillator calibration (see AVR053). 
  -U <memtype>:r|w|v:<filename>[:format]
                             Memory operation specification.
                             Multiple -U options are allowed, each request
                             is performed in the order specified.
  -n                         Do not write anything to the device.
  -V                         Do not verify.
  -t                         Enter terminal mode.
  -E <exitspec>[,<exitspec>] List programmer exit specifications.
  -x <extended_param>        Pass <extended_param> to programmer.
  -v                         Verbose output. -v -v for more.
  -q                         Quell progress output. -q -q for less.
  -l logfile                 Use logfile rather than stderr for diagnostics.
  -?                         Display this usage.

avrdude version 6.99-20220410 (60960ba), URL: <https://github.com/avrdudes/avrdude>

But would be much easier if we could have native build for Apple M1 silicon :-)

[dl8dtl]

I'd love to have native M1 builds, but what's the status of the required backend libraries? Are they available for M1 CPUs?

[mariusgreuel]

Github Actions do not support Apple M1 builds, yet. I have subscribed to this issue and plan to add this to the avrdude builds as soon it is available:
actions/runner-images#2187

@hallard What was the problem with the native build? I do not have a M1 Mac, but I would assume it to be not much different from the x86 builds.

[hallard]

@dl8dtl what do you mean by backend libraries are you talking about libusb and hidapi?

Yes, and libftdi as well. Cool if they're there as native builds already. I don't think an entire native M1 build will cause any much problems then. Back some years ago, I even occasionally ran native Solaris builds on UltraSPARC CPUs (big endian), but my reference machine doesn't boot anymore.

[mcuee]

libusb, libftdi and hidapi all have no issues with Apple M1. I have the Mac Mini M1 (2020) and I use homebrew.

[dl8dtl]

Macports works as well, including the existing build.sh script.

[hallard]

@mcuee I use homebrew and also these lib with other build project with no issue on M1, but I'm unable to compile avrdude using wiki instruction (manually or using build.sh) can you confirm a git clone and build.sh run out of the box for you?

[dl8dtl]

but I'm unable to compile avrdude using wiki instruction

Can you elaborate on that? Error messages, what does "unable to compile" actually mean?

I didn't have any problems whatsoever.

[hallard]

@dl8dtl sure it's described on the original #771 (at the end)

[hallard]

And the first issue it can't compile is that build.sh assume homebrew located in /usr/local and on M1 (and latest homebrew version) it's in /opt/homebrew/.
But once I fixed that one, it still does not compile so this is why I'm asking if someone suceeded from scratch to build on M1 with homebrew, just to be sure it's not on my side (tried on 2 MAC mini M1)

[dl8dtl]

Even with the current codebase?

Strange.

Yes, I can confirm I could run it straight out of a current Git version, using build.sh. Prerequisites came from Macports in my case, but Homebrew ought to work as well.

I tried to run the resulting binary on a number of programmers (AtmelICE, USBasp, AVRISPmkII, FT245 board), and only rolled out the v7.0 release after this all went well.

[dl8dtl]

And the first issue it can't compile is that build.sh assume homebrew located in /usr/local and on M1 (and latest homebrew version) it's in /opt/homebrew/

Then please provide a patch for that. Shouldn't be all that hard. build.sh already contains the logic to probe for different locations, just extend it.

Sorry, I don't use Homebrew myself.

[hallard]

Yeah I'm sure binary is working, but I'm not able to build nor get it on nighty builds so if you have one and can attach it on I would love to try it :-)

[hallard]

Then please provide a patch for that. Shouldn't be all that hard. build.sh already contains the logic to probe for different locations, just extend it.

Would like to but for now I can't get compilation process to works on M1 with homebrew, so patch would be useless

[mcuee]

@mcuee I use homebrew and also these lib with other build project with no issue on M1, but I'm unable to compile avrdude using wiki instruction (manually or using build.sh) can you confirm a git clone and build.sh run out of the box for you?

I am following the instruction here using CMake (just change /usr/local to /opt/homebrew)
https://github.com/avrdudes/avrdude/wiki/Building-AVRDUDE-for-macOS#build-instructions

Build log:

mcuee@mcuees-Mac-mini avrdude % cmake -D CMAKE_C_FLAGS=-I/opt/homebrew/include -D CMAKE_EXE_LINKER_FLAGS=-L/opt/homebrew/Cellar -D CMAKE_BUILD_TYPE=RelWithDebInfo -B build_macos
-- The C compiler identification is AppleClang 13.1.6.13160021
-- The CXX compiler identification is AppleClang 13.1.6.13160021
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /Library/Developer/CommandLineTools/usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /Library/Developer/CommandLineTools/usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found Git: /opt/homebrew/bin/git (found version "2.36.0") 
-- Found FLEX: /usr/bin/flex (found version "2.6.4") 
-- Found BISON: /usr/bin/bison (found version "2.3")
-- Looking for libelf.h
-- Looking for libelf.h - not found
-- Looking for libelf/libelf.h
-- Looking for libelf/libelf.h - found
-- Looking for usb.h
-- Looking for usb.h - found
-- Looking for lusb0_usb.h
-- Looking for lusb0_usb.h - not found
-- Looking for libusb.h
-- Looking for libusb.h - not found
-- Looking for libusb-1.0/libusb.h
-- Looking for libusb-1.0/libusb.h - found
-- Looking for hidapi/hidapi.h
-- Looking for hidapi/hidapi.h - found
-- Looking for ftdi_tcioflush
-- Looking for ftdi_tcioflush - found
-- Configuration summary:
-- ----------------------
-- DO HAVE    libelf
-- DO HAVE    libusb
-- DO HAVE    libusb_1_0
-- DO HAVE    libhidapi
-- DO HAVE    libftdi (but prefer to use libftdi1)
-- DO HAVE    libftdi1
-- DISABLED   doc
-- DISABLED   parport
-- DISABLED   linuxgpio
-- DISABLED   linuxspi
-- ----------------------
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/mcuee/build/avr/avrdude/build_macos

mcuee@mcuees-Mac-mini avrdude % cmake --build build_macos
[  1%] [FLEX][Parser] Building scanner with flex 2.6.4
[  3%] [BISON][Parser] Building parser with bison 2.3
[  4%] Building C object src/CMakeFiles/libavrdude.dir/arduino.c.o
[  6%] Building C object src/CMakeFiles/libavrdude.dir/avr.c.o
[  8%] Building C object src/CMakeFiles/libavrdude.dir/avr910.c.o
[  9%] Building C object src/CMakeFiles/libavrdude.dir/avrftdi.c.o
[ 11%] Building C object src/CMakeFiles/libavrdude.dir/avrftdi_tpi.c.o
[ 13%] Building C object src/CMakeFiles/libavrdude.dir/avrpart.c.o
[ 14%] Building C object src/CMakeFiles/libavrdude.dir/bitbang.c.o
[ 16%] Building C object src/CMakeFiles/libavrdude.dir/buspirate.c.o
[ 18%] Building C object src/CMakeFiles/libavrdude.dir/butterfly.c.o
[ 19%] Building C object src/CMakeFiles/libavrdude.dir/config.c.o
[ 21%] Building C object src/CMakeFiles/libavrdude.dir/confwin.c.o
[ 22%] Building C object src/CMakeFiles/libavrdude.dir/crc16.c.o
[ 24%] Building C object src/CMakeFiles/libavrdude.dir/dfu.c.o
[ 26%] Building C object src/CMakeFiles/libavrdude.dir/fileio.c.o
[ 27%] Building C object src/CMakeFiles/libavrdude.dir/flip1.c.o
[ 29%] Building C object src/CMakeFiles/libavrdude.dir/flip2.c.o
[ 31%] Building C object src/CMakeFiles/libavrdude.dir/ft245r.c.o
[ 32%] Building C object src/CMakeFiles/libavrdude.dir/jtagmkI.c.o
[ 34%] Building C object src/CMakeFiles/libavrdude.dir/jtagmkII.c.o
/Users/mcuee/build/avr/avrdude/src/jtagmkII.c:491:18: warning: variable 'checksum' set but not used [-Wunused-but-set-variable]
  unsigned short checksum = 0;
                 ^
1 warning generated.
[ 36%] Building C object src/CMakeFiles/libavrdude.dir/jtag3.c.o
[ 37%] Building C object src/CMakeFiles/libavrdude.dir/linuxgpio.c.o
[ 39%] Building C object src/CMakeFiles/libavrdude.dir/linuxspi.c.o
[ 40%] Building C object src/CMakeFiles/libavrdude.dir/lists.c.o
[ 42%] Building C object src/CMakeFiles/libavrdude.dir/micronucleus.c.o
[ 44%] Building C object src/CMakeFiles/libavrdude.dir/par.c.o
[ 45%] Building C object src/CMakeFiles/libavrdude.dir/pgm.c.o
[ 47%] Building C object src/CMakeFiles/libavrdude.dir/pgm_type.c.o
[ 49%] Building C object src/CMakeFiles/libavrdude.dir/pickit2.c.o
[ 50%] Building C object src/CMakeFiles/libavrdude.dir/pindefs.c.o
[ 52%] Building C object src/CMakeFiles/libavrdude.dir/ppi.c.o
[ 54%] Building C object src/CMakeFiles/libavrdude.dir/ppiwin.c.o
[ 55%] Building C object src/CMakeFiles/libavrdude.dir/serbb_posix.c.o
[ 57%] Building C object src/CMakeFiles/libavrdude.dir/serbb_win32.c.o
[ 59%] Building C object src/CMakeFiles/libavrdude.dir/ser_avrdoper.c.o
[ 60%] Building C object src/CMakeFiles/libavrdude.dir/ser_posix.c.o
[ 62%] Building C object src/CMakeFiles/libavrdude.dir/ser_win32.c.o
[ 63%] Building C object src/CMakeFiles/libavrdude.dir/serialupdi.c.o
[ 65%] Building C object src/CMakeFiles/libavrdude.dir/stk500.c.o
[ 67%] Building C object src/CMakeFiles/libavrdude.dir/stk500v2.c.o
[ 68%] Building C object src/CMakeFiles/libavrdude.dir/stk500generic.c.o
[ 70%] Building C object src/CMakeFiles/libavrdude.dir/teensy.c.o
[ 72%] Building C object src/CMakeFiles/libavrdude.dir/updi_link.c.o
[ 73%] Building C object src/CMakeFiles/libavrdude.dir/updi_nvm.c.o
[ 75%] Building C object src/CMakeFiles/libavrdude.dir/updi_readwrite.c.o
[ 77%] Building C object src/CMakeFiles/libavrdude.dir/updi_state.c.o
[ 78%] Building C object src/CMakeFiles/libavrdude.dir/usbasp.c.o
[ 80%] Building C object src/CMakeFiles/libavrdude.dir/usb_hidapi.c.o
[ 81%] Building C object src/CMakeFiles/libavrdude.dir/usb_libusb.c.o
[ 83%] Building C object src/CMakeFiles/libavrdude.dir/usbtiny.c.o
[ 85%] Building C object src/CMakeFiles/libavrdude.dir/update.c.o
[ 86%] Building C object src/CMakeFiles/libavrdude.dir/wiring.c.o
[ 88%] Building C object src/CMakeFiles/libavrdude.dir/xbee.c.o
[ 90%] Building C object src/CMakeFiles/libavrdude.dir/__/lexer.c.o
[ 91%] Building C object src/CMakeFiles/libavrdude.dir/__/config_gram.c.o
[ 93%] Linking C static library libavrdude.a
[ 93%] Built target libavrdude
[ 95%] Building C object src/CMakeFiles/avrdude.dir/main.c.o
[ 96%] Building C object src/CMakeFiles/avrdude.dir/term.c.o
[ 98%] Building C object src/CMakeFiles/avrdude.dir/whereami.c.o
[100%] Linking C executable avrdude
[100%] Built target avrdude

[mcuee]

As for build.sh, it will fail as it does not detect Homebrew M1 properly.

   darwin)
	# determine whether we are running using Mac Ports
	# if not, assume Mac Brew
	if [ -f /opt/local/bin/port ]
	then
	    build_flags="${build_flags} -D CMAKE_C_FLAGS=-I/opt/local/include -D CMAKE_EXE_LINKER_FLAGS=-L/opt/local/lib"
	else
	    build_flags="${build_flags} -D CMAKE_C_FLAGS=-I/usr/local/include -D CMAKE_EXE_LINKER_FLAGS=-L/usr/local/Cellar"
	fi
	;;

[hallard]

interesting that I also modified script (like I already done before) also works now :-)

    darwin)
        # determine whether we are running using Mac Ports
        # if not, assume Mac Brew
        if [ -f /opt/local/bin/port ]
        then
            build_flags="${build_flags} -D CMAKE_C_FLAGS=-I/opt/local/include -D CMAKE_EXE_LINKER_FLAGS=-L/opt/local/lib"
        else
            # Apple M1 (may be new version of homebrew also)
            if [ -d /opt/homebrew ]  
            then
                build_flags="${build_flags} -D CMAKE_C_FLAGS=-I/usr/homebrew/include -D CMAKE_EXE_LINKER_FLAGS=-L/usr/homebrew/Cellar"
            else
                build_flags="${build_flags} -D CMAKE_C_FLAGS=-I/usr/local/include -D CMAKE_EXE_LINKER_FLAGS=-L/usr/local/Cellar"
            fi
        fi
        ;;

Thanks for your support.

[mcuee]

There is a minor issue here.

-- DO HAVE    libftdi (but prefer to use libftdi1)
-- DO HAVE    libftdi1

There is no libftdi here at all, rather the detection is a bit off. It should probably detect libftdi1 first. If libftdi1 exists then should skip libftdi-0.1 detection.

[mcuee]

interesting that I also modified script (like I already done before) also works now :-)

    darwin)
        # determine whether we are running using Mac Ports
        # if not, assume Mac Brew
        if [ -f /opt/local/bin/port ]
        then
            build_flags="${build_flags} -D CMAKE_C_FLAGS=-I/opt/local/include -D CMAKE_EXE_LINKER_FLAGS=-L/opt/local/lib"
        else
            # Apple M1 (may be new version of homebrew also)
            if [ -d /opt/homebrew ]  
            then
                build_flags="${build_flags} -D CMAKE_C_FLAGS=-I/usr/homebrew/include -D CMAKE_EXE_LINKER_FLAGS=-L/usr/homebrew/Cellar"
            else
                build_flags="${build_flags} -D CMAKE_C_FLAGS=-I/usr/local/include -D CMAKE_EXE_LINKER_FLAGS=-L/usr/local/Cellar"
            fi
        fi
        ;;

Thanks for your support.

Nice patch. This change of build.sh should probably be merged.

[hallard]

Nice patch. This change of build.sh should probably be merged.

thanks done with #950

[mcuee]

BTW, not so sure if someone can create a pull request to Homebrew to add --HEAD option. In that case, Homebrew users can easily install the git head version.
https://github.com/Homebrew/homebrew-core/blob/master/Formula/avrdude.rb

Interestingly they have a patch for the release tarball.
https://raw.githubusercontent.com/Homebrew/formula-patches/03cf8088210822aa2c1ab544ed58ea04c897d9c4/libtool/configure-big_sur.diff

[mcuee]

#950 has been merged. Hopefully github action for Apple ARM64 (M1 or later) can be added.

[mcuee]

BTW, not so sure if someone can create a pull request to Homebrew to add --HEAD option. In that case, Homebrew users can easily install the git head version.
https://github.com/Homebrew/homebrew-core/blob/master/Formula/avrdude.rb

Homebrew has updated the formula for avrdude and now they use libusb-1.0 and libftdi1 API. Somehow they dropped libusb-compat-0.1 which I think is incorrect.
Homebrew/homebrew-core@1adc277

Example: it will not support usbtiny.

mcuee@mcuees-Mac-mini bin % brew info avrdude
avrdude: stable 7.0 (bottled), HEAD
Atmel AVR MCU programmer
https://www.nongnu.org/avrdude/
/opt/homebrew/Cellar/avrdude/7.0_1 (9 files, 2.3MB) *
  Poured from bottle on 2022-06-18 at 09:07:46
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/avrdude.rb
License: GPL-2.0-or-later
==> Dependencies
Build: libelf ✔
Required: hidapi ✔, libftdi ✔, libusb ✔
==> Options
--HEAD
	Install HEAD version
==> Analytics
install: 3,389 (30 days), 8,720 (90 days), 34,016 (365 days)
install-on-request: 1,488 (30 days), 3,914 (90 days), 15,446 (365 days)
build-error: 2 (30 days)

mcuee@mcuees-Mac-mini bin % avrdude -p m2560 -c usbtiny -vv 

avrdude: Version 7.0
         Copyright (c) Brian Dean, http://www.bdmicro.com/
         Copyright (c) Joerg Wunsch

         System wide configuration file is "/opt/homebrew/Cellar/avrdude/7.0_1/bin/../etc/avrdude.conf"
         User configuration file is "/Users/mcuee/.avrduderc"
         User configuration file does not exist or is not a regular file, skipping

         Using Port                    : usb
         Using Programmer              : usbtiny
avrdude: error: no usb support. Please compile again with libusb installed.
avrdude: opening programmer "usbtiny" on port "usb" failed

avrdude done.  Thank you.

[mcuee]

Just FYI, homebrew formula has been fixed.
Homebrew/homebrew-core#103964

[mcuee]

The github action macOS Apple Silicon infrasture mentioned earlier is still not ready.
actions/runner-images#2187

[mcuee]

For those who are using Homebrew and wants to use the latest, you can use the following command one time.

brew install avrdude (install 7.0 release in case you need it)
brew unlink avrdude 
brew install avrdude --HEAD

After that, you can use the following command to be always on the latest.

brew reinstall avrdude

Example:

mcuee@mcuees-Mac-mini avrdude_sr % brew reinstall avrdude
==> Cloning https://github.com/avrdudes/avrdude.git
Updating /Users/mcuee/Library/Caches/Homebrew/avrdude--git
From https://github.com/avrdudes/avrdude
   c81f52f..affe4cb  main       -> origin/main
==> Checking out branch main
Already on 'main'
Your branch is behind 'origin/main' by 73 commits, and can be fast-forwarded.
  (use "git pull" to update your local branch)
HEAD is now at affe4cb Merge pull request #1034 from stefanrueger/config_file
==> Reinstalling avrdude 
==> cmake -S . -B build/shared -DCMAKE_INSTALL_SYSCONFDIR=/opt/homebrew/etc -DBUILD_SHARED_LIBS=ON -DCMAKE_INSTALL_RPATH=@loader_path/../lib -DCMAKE_SHA
==> cmake --build build/shared
==> cmake --install build/shared
==> cmake -S . -B build/static -DCMAKE_INSTALL_SYSCONFDIR=/opt/homebrew/etc
==> cmake --build build/static
🍺  /opt/homebrew/Cellar/avrdude/HEAD-affe4cb_2: 13 files, 1.5MB, built in 7 seconds
==> Running `brew cleanup avrdude`...
Disable this behaviour by setting HOMEBREW_NO_INSTALL_CLEANUP.
Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).


mcuee@mcuees-Mac-mini avrdude_sr % avrdude -v

avrdude: Version 7.0-20220719 (affe4cb)
         Copyright (c) Brian Dean, http://www.bdmicro.com/
         Copyright (c) Joerg Wunsch

         System wide configuration file is "/opt/homebrew/etc/avrdude.conf"
         User configuration file is "/Users/mcuee/.avrduderc"
         User configuration file does not exist or is not a regular file, skipping


avrdude: no programmer has been specified on the command line or the config file
         Specify a programmer using the -c option and try again

[mcuee]

Reference:

• GitHub Actions: Apple Silicon (M1) powered macOS runners (Public Beta) github/roadmap#528

[mcuee]

github has (almost) sorted out the issue: still public beta and not yet reached GA (general availability) status.
https://github.blog/2023-10-02-introducing-the-new-apple-silicon-powered-m1-macos-larger-runner-for-github-actions/

GA ticket

• GitHub Actions: macOS Larger Runners (GA) github/roadmap#819

[mcuee]

I tried to use my avrdude repo to test out the public beta but it does not seem to work.

[mcuee]

Close this one and convert it into a discussion.

[mcuee]

Latest update:
https://github.com/actions/runner-images?tab=readme-ov-file#available-images