Monitor port autodetection based on VID:PID #3349

thomastech · 2020-01-21T04:49:39Z

Configuration

Operating system:
WIN10 Pro/64

PlatformIO Version
Platformio IDE 1.10.0 installed on VSCode
Platform is espressif32 V1.11.1
lolin_d32_pro board
USB is assigned to COM4

PIO Home=>Devices:
 COM1 	Communications Port (COM1)	ACPI\PNP0501\0
 COM2 	High-Speed USB Serial Port (COM2)	QUADPORT\QUAD_SERIAL_INTERFACE\7&29C3DB4D&0&0000
 COM3 	High-Speed USB Serial Port (COM3)	QUADPORT\QUAD_SERIAL_INTERFACE\7&29C3DB4D&0&0001
 COM4 	USB-SERIAL CH340 (COM4)	USB VID:PID=1A86:7523 SER=5 LOCATION=1-9
 COM5 	USB Serial Port (COM5)	USB VID:PID=0403:6001 SER=PXF3PKCPA

Description of problem

Serial port Auto detection chooses the wrong USB COMx port. Success can occur too, but it depends on the USB serial devices that are currently installed.

I originally thought that only the Monitor port was affected. But after some experimentation (noted further below) I found that Upload can also choose the wrong COMx port.

Initially I found that the Upload Port auto-detect was working correctly (ESP32 Upload flash worked normally). But Serial Monitor would not auto-detect. Instead it lists the available ports and waits for me to enter the COMx port name. As expected, typing "COM4" (or "4") launches the serial monitor.

This is the list I see in the terminal window when Serial Monitor is launched from the IDE:

--- Available ports:
---  1: COM1                 'Communications Port (COM1)'
---  2: COM2                 'High-Speed USB Serial Port (COM2)'
---  3: COM3                 'High-Speed USB Serial Port (COM3)'
---  4: COM4                 'USB-SERIAL CH340 (COM4)'
---  5: COM5                 'USB Serial Port (COM5)'
--- Enter port index or full name:

Steps to Reproduce

No. 1
I tried specifying the COM port with a wildcard in platformio.ini. Like this: monitor_port = COM*

The good news is that this automatically launches the monitor terminal after the flash Upload, but it chooses the wrong port for the serial monitor. For example, the ESP32 board is on COM4, but the monitor_port wildcard is picking COM5.

No. 2
I found that specifying the monitor_port COM4 value in platformio.ini works correctly. Like this: monitor_port = COM4
After a flash Upload this automatically opens the serial monitor for my serial debugging. But specifying a fixed COMx is problematic since port usage can vary with the other project members.

No. 3
I removed the COM5 USB adapter and everything works perfectly; Both upload and serial monitor are correctly auto-detected. I wondered if it could it be a bad USB device.

So I installed a randomly chosen FT232RL USB adapter. It was assigned COM10. I rebooted the PC and got back into the Platformio IDE. As before, when I launched the Serial Monitor it did not auto-detect. The available COM ports list appeared, like this:

--- Available ports:
--- 1: COM1 'Communications Port (COM1)'
--- 2: COM2 'High-Speed USB Serial Port (COM2)'
--- 3: COM3 'High-Speed USB Serial Port (COM3)'
--- 4: COM4 'USB-SERIAL CH340 (COM4)'
--- 5: COM10 'USB Serial Port (COM10)'
--- Enter port index or full name:

Choosing COM4 enables the ESP32’s serial monitor. So at this point the behavior follows what I saw when the COM5 device was present.

But now when I do a flash upload it auto-detects COM10! I removed the COM10 USB device and flash Upload correctly auto-detects COM4.

No. 4
The Lolin D32 Pro uses a CH340 that has VID=1A86 and PID=7523. As an experiment, I edited the build section of the d32_pro board.json file and added the USB ID's as follows:

    "hwids": [
      [
        "0x1A86",
        "0x7523"
      ]
    ],
    "ldscript": "esp32_out.ld",

This did not change anything, all observed symptoms remained the same.

No. 5
As an alternate method to what was described in No. 4 (see above), I also tried using scripting to override the board configuration. In platformio.ini I added this line to the [env:lolin_d32_pro] section:
extra_scripts = pre:custom_hwids.py
I also added the custom_hwids.py file to the project's main folder. The file contained this code:

Import("env")

board_config = env.BoardConfig()

board_config.update("build.hwids", [
   "0x1A86", #VIDS
   "0x7523" #PIDS
])

Results were the same as No. 4, all observed symptoms remained the same.

Expected Results

The intended action is that Upload and Serial Monitor should auto-detect the COMx.
{For convenience, a properly working auto-detection would avoid keyboard entry to choose the port. And my use case involves an open source project that is intended to be shared; So requiring special user edits to the platformio.ini to workaround the issue should be avoided.}

The text was updated successfully, but these errors were encountered:

ivankravets · 2020-01-22T19:18:25Z

I don't see any problems with PlatformIO Core. Please provide an output from

pio device list

thomastech · 2020-01-22T21:36:38Z

@ivankravets: Thanks for the helping hand.

My ESP32 is on COM4. Here is the device list, including the COM5 and COM10 USB serial devices used in the testing scenarios described earlier.

COM1 | Communications Port (COM1) | ACPI\PNP0501\0
COM10 | USB Serial Port (COM10) | USB VID:PID=0403:6001 SER=A50285BIA
COM2 | High-Speed USB Serial Port (COM2) | QUADPORT\QUAD_SERIAL_INTERFACE\7&29C3DB4D&0&0000
COM3 | High-Speed USB Serial Port (COM3) | QUADPORT\QUAD_SERIAL_INTERFACE\7&29C3DB4D&0&0001
COM4 | USB-SERIAL CH340 (COM4) | USB VID:PID=1A86:7523 SER=5 LOCATION=1-9
COM5 | USB Serial Port (COM5) | USB VID:PID=0403:6001 SER=PXF3PKCPA

Partial good news:
I found the reason for the failure with upload port auto-detection. That is to say, upload port would incorrectly choose COM10 even when I used the build.hwids script that explicitly called out the required VIDS & PIDS.

The upload port bug is in the pioupload.py file at the _look_for_serial_port() function. Instead of using the hwids string array vars, it incorrectly calls out the char array vars. So build.hwid port match detection always fails.

Here is the code snippet that shows the corrected function. The old code line has been commented out so you can see the code revision more distinctly.

See File: .platformio\penv\Lib\site-packages\platformio\builder\tools\pioupload.py

            for hwid in board_hwids:
                #hwid_str = ("%s:%s" % (hwid[0], hwid[1])).replace("0x", "")
                hwid_str = ("%s:%s" % (board_hwids[0], board_hwids[1])).replace("0x", "")
                if hwid_str in item["hwid"]:
                    print(env.subst("Found build.hwids match at %s" % (port)))
                    return port

This patch also includes a user message that reports when a build.hwids port match has occurred, which is a useful confirmation.

Although this patch fixes upload port, it doesn't help solve the monitor port's auto-detection issue. It would be great to get some help with that.

Thomas

ivankravets · 2020-01-22T21:38:55Z

#3349 (comment)

thomastech · 2020-01-22T22:03:27Z

#3349 (comment)

PS C:\users\tom\.platformio\penv\Scripts> pio device list
COM5
----
Hardware ID: USB VID:PID=0403:6001 SER=PXF3PKCPA
Description: USB Serial Port (COM5)

COM1
----
Hardware ID: ACPI\PNP0501\0
Description: Communications Port (COM1)

COM4
----
Hardware ID: USB VID:PID=1A86:7523 SER=5 LOCATION=1-9
Description: USB-SERIAL CH340 (COM4)

COM10
-----
Hardware ID: USB VID:PID=0403:6001 SER=A50285BIA
Description: USB Serial Port (COM10)

COM2
----
Hardware ID: QUADPORT\QUAD_SERIAL_INTERFACE\7&29C3DB4D&0&0000
Description: High-Speed USB Serial Port (COM2)

COM3
----
Hardware ID: QUADPORT\QUAD_SERIAL_INTERFACE\7&29C3DB4D&0&0001
Description: High-Speed USB Serial Port (COM3)

ivankravets · 2020-01-23T22:04:21Z

PlatformIO depends on VID:PID which declared in board manifest. You board has generic PID/VID. Million of other boards use the same UART chip. In this case, we recommend specifing uploader port manually.

thomastech · 2020-01-23T22:24:42Z

@ivankravets. Thanks for the help. I dug into the platformio code this morning and saw that the "VID:PID" string matching was the method used to choose the monitor port. I learned that if only one of these type of COM ports exists then it is chosen, else the user has to manually choose it. That is exactly what I am seeing, so monitor port it is working as designed.

I was under the impression that specifying the build.hwids would be used in all COMx port selection. But now I know only applies to the Upload port.

But as I reported earlier, the hwids matching is broken in pioupload.py. As-is, the build.hwids directives are ignored due to incorrect vars (hwid[ ] should be board_hwids[ ]).

Please see this revision:
See File: .platformio\penv\Lib\site-packages\platformio\builder\tools\pioupload.py

            for hwid in board_hwids:
                #hwid_str = ("%s:%s" % (hwid[0], hwid[1])).replace("0x", "")
                hwid_str = ("%s:%s" % (board_hwids[0], board_hwids[1])).replace("0x", "")
                if hwid_str in item["hwid"]:
                    print(env.subst("Found build.hwids match at %s" % (port)))
                    return port

Before the fix hwid_str = "0x"
After the fix hwid_str = "1A86:7523"

Thomas

ivankravets · 2020-01-23T22:27:18Z

board_config.update("build.hwids", [
   "0x1A86", #VIDS
   "0x7523" #PIDS
])

This should be array of array

Fixed code

board_config.update("build.hwids", [
   ["0x1A86", #VIDS
   "0x7523" #PIDS
]])

thomastech · 2020-01-23T22:47:38Z

@ivankravets, thank you! That solves the problem. My apologies for any unnecessary work I created.

BTW, I have a proposal for "auto-detecting" monitor port; A new platformio.ini directive like this:
monitor_port = upload_port
or
monitor_port = build.hwids

Thomas

ivankravets · 2020-01-24T12:47:32Z

See docs https://docs.platformio.org/en/latest/projectconf/advanced_scripting.html#override-board-configuration

thomastech · 2020-01-24T16:30:54Z

Thanks.

From what I saw in the core code the hwids's manifest override is for the Upload, Debug, and Test ports. So I think it would be a good idea to add a note to the Override Board Configuration text that states it does not apply to the Monitor port.

Thomas

ivankravets · 2020-01-24T16:45:51Z

Thanks! I renamed this issue and reopened. We will implement in 4.1.1.

thomastech · 2020-01-24T17:20:49Z

Nice. Adding Monitor port to the hwids override is fantastic. Looking forward to V4.1.1.

Thomas

thomastech · 2020-02-07T22:40:19Z

@ivankravets : Thanks for adding the monitor port auto-detect, I appreciate it.

If you desire any user feedback then I'm available to test the new feature. To do it I'll need advice on how to temporarily install the platformio-core's develop version (will revert back after the validation test). I'm running VS Code Platformio IDE on a WIN10/64 machine.

Thomas

ivankravets · 2020-02-08T14:11:35Z

All kudos goes to @valeros, he resolved this issue 😊

Thanks for the willingness to help with testing. You can easily switch to PlatformIO Core dev-version running the next command:

pio upgrade --dev

thomastech · 2020-02-08T18:00:45Z

All kudos goes to @valeros, he resolved this issue
Thanks @valeros!

I've switched to the dev version and tested the new Serial Monitor hwids detect feature. Unfortunately Serial Monitor port detection is still not working for me (same behavior as before).

I've rebooted and re-verified the dev version using terminal:

> pio --version
PlatformIO, version 4.2.0rc1

My platformio.ini has this:
extra_scripts = pre:custom_hwids.py

custom_hwids.py is in the project directory (same as platformio.ini). This is the file contents:

Import("env")

board_config = env.BoardConfig()

board_config.update("build.hwids", [
   ["0x1A86", #VIDS
   "0x7523" #PIDS
]])

My Lolin D32 Pro board is on COM4. The device list is as follows:

pio device list
COM5
----
Hardware ID: USB VID:PID=0403:6001 SER=PXF3PKCPA
Description: USB Serial Port (COM5)

COM1
----
Hardware ID: ACPI\PNP0501\0
Description: Communications Port (COM1)

COM4
----
Hardware ID: USB VID:PID=1A86:7523 SER=5 LOCATION=1-9
Description: USB-SERIAL CH340 (COM4)

COM2
----
Hardware ID: QUADPORT\QUAD_SERIAL_INTERFACE\7&29C3DB4D&0&0000
Description: High-Speed USB Serial Port (COM2)

COM3
----
Hardware ID: QUADPORT\QUAD_SERIAL_INTERFACE\7&29C3DB4D&0&0001
Description: High-Speed USB Serial Port (COM3)

With the above USB Com devices the Upload port is correctly chosen. I noted that the port is incorrect without the custom_hwids.py, so the file appears to be OK). But the Serial Monitor is still requiring me to manually choose the Com port.

I suspect I'm doing something wrong. So advice is welcome.

Thomas

ivankravets · 2020-02-08T18:09:49Z

I see the problem and I'm not sure that it could be fixed. The extra scripting is a part of a build process. The device monitor works in another way. You have 2 options to resolve this issue without extra script (yes, you can remove it):

Edit board JSON directly in ~/.platformio/platforms/espressif23/boards/your_board.json. See example https://github.com/platformio/platform-atmelavr/blob/develop/boards/uno.json#L6
Clone this your_board.json to project-dir/boards/your_board.json and edit here. Please do not forget to add "platform": "espressif32", to new JSON to instruct PlatformIO to override only for ESP32 dev-platform.

thomastech · 2020-02-08T18:23:42Z

Oh, that's not the outcome I was hoping for.

Early on I tried Choice # 1. But abandoned it because it requires custom edits outside the project folder. The goal is to keep it all in the project.

Choice # 2 seems reasonable since it allows the workaround to be included in the project folder. I'll be able to try it out in a couple days and will report back on my success.

Thanks again for your advice.

Thomas

thomastech · 2020-02-08T22:07:55Z

@ivankravets,
I implemented choice # 2. Unfortunately the Serial Monitor still requires me to manually choose the serial port.

Edit/Update: Ignore that comment, It works! The updates in the 4.2.0rx2 dev core are successfully using the custom board's json hwids for Serial Monitor. So this issue is indeed solved with the dev release.

It would be helpful to add a note to the Override Board Configuration text about the Serial Monitor limitation.
https://docs.platformio.org/en/latest/projectconf/advanced_scripting.html#override-board-configuration
That is to say, the platformio.ini override applies to the Upload, Debug, and Test ports, but not the Monitor port. Of course Serial Monitor can use either solution you proposed earlier as a workaround.

Thomas

…or command // Issue #3349

ivankravets · 2020-02-12T19:08:39Z

Thanks! Docs is updated b870480

thomastech · 2020-02-12T19:27:44Z

@ivankravets, thank you for all the hands-on advice.
@valeros, thank you for coding the new feature.

Looking forward to the official release. In the meantime I'll continue to use the Dev release.

Thomas

pfeerick · 2020-02-13T01:35:52Z

@thomastech You don't have to wait long... it's already been released! ;) https://community.platformio.org/t/platformio-core-4-2/11980

thomastech · 2020-02-13T03:10:26Z

Thanks for the heads-up!

Thomas

thomastech mentioned this issue Jan 21, 2020

[Moved] Serial Monitor Port Auto-Detection Issues platformio/platform-espressif32#273

Closed

ivankravets added the help wanted label Jan 22, 2020

ivankravets closed this as completed Jan 23, 2020

ivankravets added a commit that referenced this issue Jan 24, 2020

Add notice about VID:PID pairs // Resolve #3349

8fce660

ivankravets changed the title ~~Serial Port Auto-Detection Issues~~ Monitor port autodetection based on VID:PID Jan 24, 2020

ivankravets added enhancement and removed help wanted labels Jan 24, 2020

ivankravets added this to the 4.1.1 milestone Jan 24, 2020

ivankravets reopened this Jan 24, 2020

thomastech mentioned this issue Jan 24, 2020

Refactor thomastech/Sparky#5

Merged

ivankravets self-assigned this Jan 24, 2020

ivankravets assigned valeros and unassigned ivankravets Feb 5, 2020

ivankravets closed this as completed in 7716fe1 Feb 5, 2020

ivankravets mentioned this issue Feb 12, 2020

platformio 4.2.0 Homebrew/homebrew-core#50058

Closed

5 tasks

ivankravets added a commit that referenced this issue Feb 12, 2020

Add warning that overriding board data will not work for device monit…

b870480

…or command // Issue #3349

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Monitor port autodetection based on VID:PID #3349

Monitor port autodetection based on VID:PID #3349

thomastech commented Jan 21, 2020 •

edited

Loading

ivankravets commented Jan 22, 2020

thomastech commented Jan 22, 2020

ivankravets commented Jan 22, 2020

thomastech commented Jan 22, 2020

ivankravets commented Jan 23, 2020

thomastech commented Jan 23, 2020 •

edited

Loading

ivankravets commented Jan 23, 2020

thomastech commented Jan 23, 2020 •

edited

Loading

ivankravets commented Jan 24, 2020

thomastech commented Jan 24, 2020

ivankravets commented Jan 24, 2020

thomastech commented Jan 24, 2020

thomastech commented Feb 7, 2020

ivankravets commented Feb 8, 2020

thomastech commented Feb 8, 2020 •

edited

Loading

ivankravets commented Feb 8, 2020

thomastech commented Feb 8, 2020 •

edited

Loading

thomastech commented Feb 8, 2020 •

edited

Loading

ivankravets commented Feb 12, 2020

thomastech commented Feb 12, 2020 •

edited

Loading

pfeerick commented Feb 13, 2020

thomastech commented Feb 13, 2020

Monitor port autodetection based on VID:PID #3349

Monitor port autodetection based on VID:PID #3349

Comments

thomastech commented Jan 21, 2020 • edited Loading

Configuration

Description of problem

Steps to Reproduce

Expected Results

ivankravets commented Jan 22, 2020

thomastech commented Jan 22, 2020

ivankravets commented Jan 22, 2020

thomastech commented Jan 22, 2020

ivankravets commented Jan 23, 2020

thomastech commented Jan 23, 2020 • edited Loading

ivankravets commented Jan 23, 2020

thomastech commented Jan 23, 2020 • edited Loading

ivankravets commented Jan 24, 2020

thomastech commented Jan 24, 2020

ivankravets commented Jan 24, 2020

thomastech commented Jan 24, 2020

thomastech commented Feb 7, 2020

ivankravets commented Feb 8, 2020

thomastech commented Feb 8, 2020 • edited Loading

ivankravets commented Feb 8, 2020

thomastech commented Feb 8, 2020 • edited Loading

thomastech commented Feb 8, 2020 • edited Loading

ivankravets commented Feb 12, 2020

thomastech commented Feb 12, 2020 • edited Loading

pfeerick commented Feb 13, 2020

thomastech commented Feb 13, 2020

thomastech commented Jan 21, 2020 •

edited

Loading

thomastech commented Jan 23, 2020 •

edited

Loading

thomastech commented Jan 23, 2020 •

edited

Loading

thomastech commented Feb 8, 2020 •

edited

Loading

thomastech commented Feb 8, 2020 •

edited

Loading

thomastech commented Feb 8, 2020 •

edited

Loading

thomastech commented Feb 12, 2020 •

edited

Loading