[all] Serialport.list Information Wrong/Missing

[McClellandLaboratories]

• SerialPort Version: 4.07

• NodeJS Version: 7.1

• Operating System and Hardware Platform: Windows 10 and Ubuntu 16.04

• Have you checked the right version of the api docs?: Yes

• Are you having trouble installing and you checked the Installation Special Cases docs? No

• Are you using Electron and have you checked the Electron Docs?: No

Summary of Problem

FTDI device serial numbers are missing and manufacturers are incorrect on Windows 10.

Steps and Code to Reproduce the Issue

Using Windows 10, plug in a FTDI USB device. Next, assign it a unique USB device descriptor using FT_Prog. After programming the FTDI device with the new USB device descriptors, run serialport-list -f jsonline, resulting in output similar to the following:

{"comName":"COM23","manufacturer":"FTDI","pnpId":"FTDIBUS\\VID_0403+PID_6001+A51MAMMEA\\0000","vendorId":"0403","productId":"6001"}

Using Ubuntu 16.04, plug in the same FTDI devices. Run serialport-list -f jsonline, resulting in output similar to the following:

{"comName":"/dev/ttyUSB0","manufacturer":"MLI","serialNumber":"MLI_scale_A51MAMME","pnpId":"usb-MLI_scale_A51MAMME-if00-port0","vendorId":"0x0403","productId":"0x6001"}

Why are the manufacturers listed differently between the two operating systems. Why doesn't windows see the serialNumber? Is there any way to fix this.

[reconbot]

It's been a long time since I've fully explored the apis involved but you can find the code responsible for returning the information here; https://github.com/EmergingTechnologyAdvisors/node-serialport/blob/master/src/serialport_win.cpp#L525-L573

Long story short, we ask the system for the info with SetupDiGetDeviceRegistryProperty and return it to the list function. For manufacturer we use SPDRP_MFG which is defined as

The function retrieves a REG_SZ string that contains the name of the device manufacturer.

So it's probably down to the drivers your using to load the FTDI device. Maybe FTDI has newer ones or will take a bug report?

For the serial number, I don't see that as a thing we can read from the pnp subsystem but I might be overlooking it. I do see we read some values out of the registry with RegQueryValueEx so maybe there's a way we can accomplish it with that approach.

Happy to add it if you can figure it out. Also happy to be shown references to how other serial port libraries (eg pyserial) if they include the missing info, so we can copy them.

[McClellandLaboratories]

Ok so i searched through the windows registry after taking a look at :
https://github.com/EmergingTechnologyAdvisors/node-serialport/blob/master/src/serialport_win.cpp#L525-L573

On Windows 10 the USB registry files are located under: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Enum\USB

This is what I see on windows:

Note that I have highlighted one of the folders A51MAMME. This name of the folder is the serial number for this device.

If I go to the Device Parameters folder I see the following:

Note the SymbolicName REG_SZ :
??\USB#VID_0403&PID_6001#A51MAMME#{a5dcbf10-6530-11d2-901f-00c04fb951ed}

It contains the VID (0403), PID (6001), and the serialnumber (A51MAMME). So it looks like the serial number is available in the registry :) So maybe we can retrieve it with RegQueryValueEx?

[McClellandLaboratories]

So I installed pyserial on windows and ran their equivalent command line list and got the following:

COM23
desc: USB Serial Port (COM23)
hwid: USB VID:PID=0403:6001 SER=A51MAMMEA

So I dug into their code and it looks like they are retrieving the information from the registry using regular expressions, shown in the highlighted lines in the link below:

https://github.com/pyserial/pyserial/blob/master/serial/tools/list_ports_windows.py#L251-L257

[reconbot]

I'm confused. We have quite a few results.

• linux node-serialport: Atlas_pH_DO1KHTT8 (from this command udevadm info)
• registry A51MAMME
• pyserial A51MAMMEA

Is A51MAMME the expected value in all cases? What are we getting on linux?

[McClellandLaboratories]

Ok sorry for the confusion, I switched devices midway through, which was definitely confusing. I have gone back through my posts and fixed them so that the sole device discussed is the one with the serial number A51MAMME. A51MAMME is the expected value in all cases.

linux node-serialport: MLI_scale_A51MAMME
(Note: This is actually the manufacturer + _ + manufacturer description + _ + serial number)
So this isn't exactly the "serial number", but at least it contains the serial number.

registry: A51MAMME

pyserial: A51MAMMEA
(Note there is an extra "A" at the end of the serial number, the same error occurs for other FTDI devices. They all seem to have an extra "A" appended to the end of the serial number. )

So I didn't notice this before, but it looks like windows node-serialport is picking up the serial number in the pnpID. For reference here is the windows node-serialport -f -jsonline printout for the A51MAMME device:

{"comName":"COM23","manufacturer":"FTDI","pnpId":"FTDIBUS\VID_0403+PID_6001+A51MAMMEA\0000","vendorId":"0403","productId":"6001"}

Note the pnpID contains "A51MAMMEA", I am not sure why we are getting an extra "A" character at the end.

It would probably be easy enough to extract the serialNumber from windows registry or pnpID, and then return a "serialNumber" entry in the serialport-list json object. If we wanted the linux "serialNumber" and windows "serialNumber" entries to match, it would probably be best to remove the extra information stored in the linux entry (aka manufacturer, which already exists as a separate entry, and the manufacturer description (perhaps a new entry could be added for this information?)).

[reconbot]

I only have an arduino uno to test with

linux

{
  "comName":"/dev/ttyACM0",
  "pnpId":"usb-Arduino__www.arduino.cc__0043_752303138333518011C1-if00",
  "productId":"0x0043",
  "serialNumber":"Arduino__www.arduino.cc__0043_752303138333518011C1",
  "manufacturer":"Arduino__www.arduino.cc_",
  "vendorId":"0x2341"
}

osx

{
  "comName":"/dev/tty.usbmodem1431",
  "manufacturer":"Arduino (www.arduino.cc)",
  "serialNumber":"752303138333518011C1",
  "locationId":"0x14300000",
  "vendorId":"0x2341",
  "productId":"0x0043"
}

windows

{
  "comName":"COM3",
  "manufacturer":"Microsoft",
  "pnpId":"USB\\VID_2341&PID_0043\\752303138333518011C1",
  "locationId":"Port_#0003.Hub_#0001",
  "vendorId":"2341",
  "productId":"0043"
}

Ideal

{
  "manufacturer":"Arduino (www.arduino.cc)",
  "serialNumber":"752303138333518011C1",  
  "vendorId":"2341",
  "productId":"0043"
}

The locationId and pnpId are not consistant, I don't know what they should be.

osx

{
  "locationId":"14300000",
  "pnpId": undefined
}

windows

{
  "locationId":"Port_#0003.Hub_#0001",
  "pnpId":"USB\\VID_2341&PID_0043\\752303138333518011C1",
}

linux

{
  "locationId": undefined,
  "pnpId":"usb-Arduino__www.arduino.cc__0043_752303138333518011C1-if00"
}

Does the ideal look correct to you? Do you have thoughts on the locationId and pnpId?

[thiago-sylvain]

To fuel the discussion, I am also using a FTDI chip, and facing this problem.

Here is the value of the serialport object on windows:

comName: "COM29"
locationId: undefined
manufacturer: "FTDI"
pnpId: "FTDIBUS\VID_0403+PID_6015+DO008I0CA\0000"
productId: "6015"
serialNumber: undefined
vendorId: "0403"

Here is the equivalent on Linux:

comName: "/dev/ttyUSB0"
manufacturer: "FTDI"
pnpId: "usb-FTDI_FT230X_Basic_UART_DO008I0C-if00-port0"
productId: "0x6015"
serialNumber: "FTDI_FT230X_Basic_UART_DO008I0C"
vendorId: "0x0403"

The expected serial number here for me is DO008I0C.
Yes, Windows sometimes append the 'A' at the end, sometimes not, it was already the case when I was using pyserial so I coped with by using a regexp.

Pyserial seems to face the same difficulties pyserial/pyserial#61.
Libraries based on D2XX driver like https://www.npmjs.com/package/ftdi seems to work ok.

But today I faced a new behavior, while I was testing on Windows, some ports identified this way:

comName: "COM29"
locationId: undefined
manufacturer: "FTDI"
pnpId: "FTDIBUS\VID_0403+PID_6015+7&C0956A8&0&1\0000"
productId: "6015"
serialNumber: undefined
vendorId: "0403"

So here the regexp is pretty useless.
After rebooting windows, the enumeration is done correctly again.
My two cents is that the problem is related to the implementation of a low level windows driver but I don't have the skill to dig into it, so except if someone here has it, I think I am about to drop the idea of relying on the serial number to identify my devices.

[McClellandLaboratories]

@reconbot The ideal looks correct to me. It appears that windows and linux have unique pnpID formats (Ref. https://www.netiq.com/documentation/platespin-migrate-12-2/migrate-user/data/t40xx337skwt.html). I am not sure what the locationID.

@thiago-sylvain what version of Windows are you using?

[thiago-sylvain]

Windows 10, x64.

[reconbot]

I spent a few hours on this today. It looks like on windows usb connected serialports the serial number is in the last bit of the pnp id.

@thiago-sylvain that means your device drivers are reporting a serial number of 0000

[frank-dspeed]

@reconbot the A\0000 could be some kind of internal address it looks like that maybe windows uses some kind of internal addresses was facing that with cctalk devices also

[reconbot]

Fixed serial numbers on linux and usb connected windows devices. Manufacturers are reported by windows drivers, but seem to be equal on osx and linux.

Since I now pull serialnumbers from the pnpid, I think maybe it will still be different A51MAMMEA\0000. That last slash doesn't exist among the serial devices I have. (Arduino's and FTDI devices)

@McClellandLaboratories Can you run a test of master with your device?

[McClellandLaboratories]

OK, so I tested four different devices:

Arduino Mega
(https://www.amazon.com/keyestudio-development-board-compatible-arduino/dp/B016JWNYBE)

Atlas Scientific EZO-RGB Probe
https://www.atlas-scientific.com/product_pages/probes/ezo-rgb.html
Programmed (using FTProg) USB string descriptors to:
Manufacturer: Atlas
Product Description: RGB

Gearmo FTDI2-LED USB RS-232 Serial Adapter
(https://www.amazon.com/gp/product/B01DT6K8G2/ref=oh_aui_detailpage_o07_s00?ie=UTF8&psc=1)
Programmed (using FTProg) USB string descriptors to:
Manufacturer: hamilton
Product Description: MVP4

Arducam Nano V3.0 (Arduino Nano with FTDI)
(https://www.amazon.com/Arducam-Atmega328p-Controller-Development-Compatible/dp/B01983R7PK/ref=sr_1_1?ie=UTF8&qid=1501520623&sr=8-1&keywords=ftdi+nano)
Programmed (using FTProg) USB string descriptors to:
Manufacturer: MLI
Product Description: scale

Linux

Arduino Mega
{"manufacturer":"Arduino (www.arduino.cc)","serialNumber":"85531303630351C081D2","pnpId":"usb-Arduino__www.arduino.cc__0042_85531303630351C081D2-if00","vendorId":"2341","productId":"0042","comName":"/dev/ttyACM0"}

This one is perfect, all of the information is reporting to the correct fields.

Atlas Scientific EZO-RGB Sensor
{"manufacturer":"Atlas","serialNumber":"DJ1XJE67","pnpId":"usb-Atlas_RGB_DJ1XJE67-if00-port0","vendorId":"0403","productId":"RGB","comName":"/dev/ttyUSB1"}

The only thing wrong here is the productId, which should be 6015. RGB is the product description.

Gearmo FTDI2-LED USB RS-232 Serial Adapter
{"manufacturer":"hamilton","serialNumber":"AL1WHZWF","pnpId":"usb-hamilton_MVP4_AL1WHZWF-if00-port0","vendorId":"0403","productId":"MVP4","comName":"/dev/ttyUSB0"}

The only thing wrong here is the productId, which should be 6001. MVP4 is the product description.

Arducam Nano V3.0 (Arduino Nano with FTDI)
{"manufacturer":"MLI","serialNumber":"A51MAMME","pnpId":"usb-MLI_scale_A51MAMME-if00-port0","vendorId":"0403","productId":"scale","comName":"/dev/ttyUSB2"}

The only thing wrong here is the productId, which should be 6001. scale is the product description.

Windows 10

Arduino Mega
{"comName":"COM8","manufacturer":"Arduino LLC (www.arduino.cc)","serialNumber":"85531303630351C081D2","pnpId":"USB\\\\VID_2341&PID_0042\\\\85531303630351C081D2","locationId":"Port_#0002.Hub_#0003","vendorId":"2341","productId":"0042"}

This one is perfect, all of the information is reporting to the correct fields.

Atlas Scientific EZO-RGB Sensor
{"comName":"COM5","manufacturer":"FTDI","pnpId":"FTDIBUS\\VID_0403+PID_6015+DJ1XJE67A\\0000","vendorId":"0403","productId":"6015"}

No serialNumber field is reported. The serial number for this device is DJ1XJE67.

Gearmo FTDI2-LED USB RS-232 Serial Adapter
{"comName":"COM5","manufacturer":"FTDI","pnpId":"FTDIBUS\\VID_0403+PID_6001+AL1WHZWFA\\0000","vendorId":"0403","productId":"6001"}

No serialNumber field is reported. The serial number for this device is AL1WHZWF.

Arducam Nano V3.0 (Arduino Nano with FTDI)
{"comName":"COM6","manufacturer":"FTDI","pnpId":"FTDIBUS\\VID_0403+PID_6001+A51MAMMEA\\0000","vendorId":"0403","productId":"6001"}

No serialNumber field is reported. The serial number for this device is A51MAMME.

Results Summary

The Arduino Mega data is reported correctly for Linux and Windows.

The FTDI devices had some issues. In Linux, the productID was reporting the product description instead of the PID numbers. In Windows, no serialNumber field was reported. However, the serialNumber is present in the pnpID.

Notice the difference in formatting when comparing the Windows 10 pnpIDs:

Arduino Mega
"pnpId":"USB\\VID_2341&PID_0042\\85531303630351C081D2"

Atlas Scientific EZO-RGB Sensor
"pnpId":"FTDIBUS\\VID_0403+PID_6015+DJ1XJE67A\\0000"

Gearmo FTDI2-LED USB RS-232 Serial Adapter
"pnpId":"FTDIBUS\\VID_0403+PID_6001+AL1WHZWFA\\0000",

Arducam Nano V3.0 (Arduino Nano with FTDI)
"pnpId":"FTDIBUS\\VID_0403+PID_6001+A51MAMMEA\\0000"

The Arduino Mega is not a FTDI device, which is shown first by the 'USB\\'. The VID and PID are seperated by an '&', and the serial number is separated from the serialNumber by '\\'.

In comparison, the FTDI devices are reported with a 'FTDIBUS\\'. The VID, PID, and serialNumber are seperated by '+', an 'A' is tacked on to the end of the serial number and \\0000 is added at the end.

Maybe the FTDI device information on Windows would be easiest to extract from the pnpID using a regular expression that can catch both results from normal USB devices (like the Arduino Mega) and FTDI devices.

[thiago-sylvain]

node-serialport v5.0.0 and FTDI 230X

Linux

comName: "/dev/ttyUSB0"
locationId: undefined
manufacturer: "FTDI"
pnpId: "usb-FTDI_FT230X_Basic_UART_DO004ZB7-if00-port0"
productId: "FT230X Basic UART"
serialNumber: "DO004ZB7"
vendorId: "0403"

In the productId I was rather expecting something like 0x6015.

Windows 10

comName: "COM13"
locationId: undefined
manufacturer: "FTDI"
pnpId: "FTDIBUS\VID_0403+PID_6015+DO004ZB7A\0000"
productId: "6015"
serialNumber: undefined
vendorId: "0403"

So nothing new since beta8. Concerning the bug I reported above, still no hint on how to reproduce.

[reconbot]

The productId on linux issue has been fixed in #1279

The windows serialnumber... I wish I could easily test this... I guess we'll have to break it out into a testable function.

Collecting all your test data;

FTDI Device
FTDIBUS\VID_0403+PID_6015+DO004ZB7A\0000 -> DO004ZB7

Arduino Mega
USB\\VID_2341&PID_0042\\85531303630351C081D2 -> 85531303630351C081D2

Atlas Scientific EZO-RGB Sensor
FTDIBUS\\VID_0403+PID_6015+DJ1XJE67A\\0000 -> DJ1XJE67

Gearmo FTDI2-LED USB RS-232 Serial Adapter
FTDIBUS\\VID_0403+PID_6001+AL1WHZWFA\\0000 -> AL1WHZWF

Arducam Nano V3.0 (Arduino Nano with FTDI)
FTDIBUS\\VID_0403+PID_6001+A51MAMMEA\\0000 -> A51MAMME

[reconbot]

Well FTDIBUS was pretty easy to sus out, the latest master is available for you perusal. If you have any other kind of devices now's the time to break them out. =)

[McClellandLaboratories]

Awesome job on the update! I have tried all of the devices I have, and everything looks great. Both Linux and Windows report the same "serialNumber".

One question somewhat unrelated to this issue, but why does the serialport-list in Linux show a bunch of blank com ports (as shown below):

{"manufacturer":"Arduino (www.arduino.cc)","serialNumber":"9563533373035191F0D2","pnpId":"usb-Arduino__www.arduino.cc__0042_9563533373035191F0D2-if00","vendorId":"2341","productId":"0042","comName":"/dev/ttyACM0"}
{"vendorId":"8086","productId":"a127","comName":"/dev/ttyS4"}
{"comName":"/dev/ttyS0"}
{"comName":"/dev/ttyS1"}
{"comName":"/dev/ttyS10"}
{"comName":"/dev/ttyS11"}
{"comName":"/dev/ttyS12"}
{"comName":"/dev/ttyS13"}
{"comName":"/dev/ttyS14"}
{"comName":"/dev/ttyS15"}
{"comName":"/dev/ttyS16"}
{"comName":"/dev/ttyS17"}
{"comName":"/dev/ttyS18"}
{"comName":"/dev/ttyS19"}
{"comName":"/dev/ttyS2"}
{"comName":"/dev/ttyS20"}
{"comName":"/dev/ttyS21"}
{"comName":"/dev/ttyS22"}
{"comName":"/dev/ttyS23"}
{"comName":"/dev/ttyS24"}
{"comName":"/dev/ttyS25"}
{"comName":"/dev/ttyS26"}
{"comName":"/dev/ttyS27"}
{"comName":"/dev/ttyS28"}
{"comName":"/dev/ttyS29"}
{"comName":"/dev/ttyS3"}
{"comName":"/dev/ttyS30"}
{"comName":"/dev/ttyS31"}
{"comName":"/dev/ttyS5"}
{"comName":"/dev/ttyS6"}
{"comName":"/dev/ttyS7"}
{"comName":"/dev/ttyS8"}
{"comName":"/dev/ttyS9"}

[reconbot]

Yay!

Because linux doesn't differentiate between tty's and serialports in general. Do you know a way?

[McClellandLaboratories]

I wonder if filtering just based on the presence of a VID or product PID would work. Note in the above list the:

{"vendorId":"8086","productId":"a127","comName":"/dev/ttyS4"} represents a USB hub
The remaining tty's are empty, and do not contain a PID or VID.

I suppose another option would be to add a flag to serialport-list like -vidReq.

Command:
serialport-list -f jsonline -vidReq

Output:
{"manufacturer":"Arduino (www.arduino.cc)","serialNumber":"9563533373035191F0D2","pnpId":"usb-Arduino__www.arduino.cc__0042_9563533373035191F0D2-if00","vendorId":"2341","productId":"0042","comName":"/dev/ttyACM0"}
{"vendorId":"8086","productId":"a127","comName":"/dev/ttyS4"}

Anyways these are just thoughts.

[reconbot]

I'm not sure how to confirm this will always be the case but I'm up for filtering it in our api. I wonder what python or tcl does...