Improper duplicate field exception?

[peci1]

I must be doing something wrong, or would such a bug go unnoticed from the beginning of rosmsg?

I have a custom message which works with all the CLI tools and C++ (rosbag play, roscpp pubsub, rosmsg show). I can even load it in python using from darpa_msgs.msg import LocalizationHypotheses. But I can't get it to work with rosbag python module.

$ python
>>> import rosbag
>>> b = rosbag.Bag("robot_data_0_TEAMBASE.bag", "r")
>>> msg = next(b.read_messages(topics=["/robot_data/X1/confirmed_hypotheses_batch"]))[1]
WARNING: For type [darpa_msgs/LocalizationHypotheses] stored md5sum [7da59d174d4c697d0571b3a6fc9fed73] has invalid message definition."
>>> import genpy
>>> import rosbag.bag
>>> old = rosbag.bag._get_message_type
>>> def tmp(info):
...   genpy.dynamic.generate_dynamic(info.datatype, info.msg_def)[info.datatype]
...   return old(info)
...
>>> rosbag.bag._get_message_type = tmp
>>> msg = next(b.read_messages(topics=["/robot_data/X1/confirmed_hypotheses_batch"]))[1]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/ros/melodic/lib/python2.7/dist-packages/rosbag/bag.py", line 2699, in read_messages
    yield self.seek_and_read_message_data_record((entry.chunk_pos, entry.offset), raw, return_connection_header)
  File "/opt/ros/melodic/lib/python2.7/dist-packages/rosbag/bag.py", line 2862, in seek_and_read_message_data_record
    msg_type = _get_message_type(connection_info)
  File "<stdin>", line 2, in tmp
  File "/opt/ros/melodic/lib/python2.7/dist-packages/genpy/dynamic.py", line 138, in generate_dynamic
    specs = {core_type: genmsg.msg_loader.load_msg_from_string(msg_context, core_msg, core_type)}
  File "/opt/ros/melodic/lib/python2.7/dist-packages/genmsg/msg_loader.py", line 269, in load_msg_from_string
    spec = MsgSpec(types, names, constants, text, full_name, package_name)
  File "/opt/ros/melodic/lib/python2.7/dist-packages/genmsg/msgs.py", line 251, in __init__
    raise InvalidMsgSpec("Duplicate field names in message: %s"%names)
genmsg.base.InvalidMsgSpec: Duplicate field names in message: ['header', 'seq', 'stamp', 'frame_id', 'hypotheses', 'id', 'active', 'merged_to', 'position', 'x', 'y', 'z', 'covariance', 'objectness', 'confirmed', 'class_probabilities', 'n_detections', 'last_detection_stamp', 'image', 'header', 'seq', 'stamp', 'frame_id', 'format', 'data']
>>>

And the message is:

$ rosmsg show darpa_msgs/LocalizationHypotheses
std_msgs/Header header
  uint32 seq
  time stamp
  string frame_id
darpa_msgs/LocalizationHypothesis[] hypotheses
  int32 id
  bool active
  int32 merged_to
  geometry_msgs/Point position
    float64 x
    float64 y
    float64 z
  float64[9] covariance
  float64 objectness
  bool confirmed
  float32[] class_probabilities
  int32 n_detections
  time last_detection_stamp
  sensor_msgs/CompressedImage image
    std_msgs/Header header
      uint32 seq
      time stamp
      string frame_id
    string format
    uint8[] data

Is it possible that all the names that are being checked are not "namespaced" by their relative path in the message definition? I.e., putting two Header fields in different parts of the message is treated invalid?

The exception is raised here

genmsg/src/genmsg/msgs.py

Lines 250 to 251 in 7d8b6ce

	if len(set(names)) != len(names):
	raise InvalidMsgSpec("Duplicate field names in message: %s"%names)

And the names reported by the exception contains the "non-namespaced" field names: ['header', 'seq', 'stamp', 'frame_id', 'hypotheses', 'id', 'active', 'merged_to', 'position', 'x', 'y', 'z', 'covariance', 'objectness', 'confirmed', 'class_probabilities', 'n_detections', 'last_detection_stamp', 'image', 'header', 'seq', 'stamp', 'frame_id', 'format', 'data']

[peci1]

So it seems that rosbag expects another format of the message definition - the one with ===-divided sections specifying each subtype separately, instead of this "all-in" message definition I showed in the issue description.

I just can't figure out how did this kind of definition get into our bags. I checked the bag directly via a debugger and the definition in the connection info is the "all-in" type. When I tried to recreate this situation artificially without running the whole pipeline, I tried roscpp publishers, rospy publishers, topic_tools relay, and feeding all of them to rosbag record (which is how we record the bags), but all of these produce the ===-sectioned definitions. The rosbag record normally runs in a container that doesn't know this message definition, and I even simulated that by calling record from a workspace which doesn't source the message definition package, but it still recorded the ===-separated definition (probably provided by publisher). Any ideas how could the "all-in" definitions sneak into the bags? We only use rospy and roscpp.

[peci1]

Even rosbag check is confused:

$ rosbag check /media/data/bags/subt_virtual/robot_data_0_TEAMBASE.bag
WARNING: For type [darpa_msgs/LocalizationHypotheses] stored md5sum [7da59d174d4c697d0571b3a6fc9fed73] has invalid message definition."
The following migrations need to occur:
 * From: darpa_msgs/LocalizationHypotheses [d41d8cd98f00b204e9800998ecf8427e]
   To:   darpa_msgs/LocalizationHypotheses [7da59d174d4c697d0571b3a6fc9fed73]
    1 rules missing:
     * From: darpa_msgs/LocalizationHypotheses [d41d8cd98f00b204e9800998ecf8427e]
       To:   darpa_msgs/LocalizationHypotheses [7da59d174d4c697d0571b3a6fc9fed73]

But:

$ rosbag info /media/data/bags/subt_virtual/robot_data_0_TEAMBASE.bag
path:        /media/data/bags/subt_virtual/robot_data_0_TEAMBASE.bag
version:     2.0
duration:    1hr 0:23s (3623s)
start:       Jan 01 1970 01:00:48.84 (48.84)
end:         Jan 01 1970 02:01:11.85 (3671.85)
size:        223.4 MB
messages:    1083074
compression: none [281/281 chunks]
types:       darpa_msgs/LocalizationHypotheses [7da59d174d4c697d0571b3a6fc9fed73]

and

$ rosmsg md5 darpa_msgs/LocalizationHypotheses
7da59d174d4c697d0571b3a6fc9fed73

So rosbag check "invented" md5 d41d8cd98f00b204e9800998ecf8427e that doesn't correspond to anything in the bag or in the system.