[WIP] j1939 updates and upgrades #335

ebroecker · 2019-03-25T19:34:11Z

Add functionality to find a frame in matrix by given (j1939-)pgn

add Test data
create png extraction tests
move pgn functionality to ArbitrationId

codecov-io · 2019-03-25T19:53:17Z

Codecov Report

Merging #335 into development will increase coverage by 0.79%.
The diff coverage is 68.55%.

@@               Coverage Diff               @@
##           development     #335      +/-   ##
===============================================
+ Coverage        40.49%   41.29%   +0.79%     
===============================================
  Files               37       39       +2     
  Lines             7008     7168     +160     
  Branches          1649     1683      +34     
===============================================
+ Hits              2838     2960     +122     
- Misses            3968     3990      +22     
- Partials           202      218      +16

Impacted Files	Coverage Δ
src/canmatrix/cli/convert.py	`0% <ø> (ø)`	⬆️
src/canmatrix/__init__.py	`100% <ø> (ø)`	⬆️
src/canmatrix/join.py	`0% <0%> (ø)`	⬆️
src/canmatrix/formats/arxml.py	`11.73% <0%> (ø)`	⬆️
src/canmatrix/tests/test_dbc.py	`98.88% <100%> (+0.04%)`	⬆️
src/canmatrix/tests/test_j1939_decoder.py	`100% <100%> (ø)`
src/canmatrix/tests/test_canmatrix.py	`97.31% <100%> (+0.06%)`	⬆️
src/canmatrix/j1939_decoder.py	`53.96% <53.96%> (ø)`
src/canmatrix/canmatrix.py	`77.12% <60.65%> (-0.87%)`	⬇️
src/canmatrix/formats/dbc.py	`54.15% <66.66%> (+1.75%)`	⬆️
... and 4 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5ca617e...a00f7d8. Read the comment docs.

ebroecker · 2019-03-27T21:14:03Z

anyone out there to review the changes of CanId?

J1939 PGN and Destiation calculation seemed incorrect to me.

Funth0mas · 2019-03-28T14:34:39Z

I don't know LKWs but quickly found an online converter which looks like the whole is a bit more complicated. No idea.

Funth0mas · 2019-03-28T14:50:12Z

src/canmatrix/canmatrix.py

@@ -1843,7 +1861,12 @@ def __init__(self, arbitration_id):  # type: (ArbitrationId) -> None
        if arbitration_id.extended:
            self.source = arbitration_id.id & int('0xFF', 16)


maybe if you're tochuing the code anyway, also refactor the two crazy int('0xFF', 16) to simply 0xFF

Funth0mas · 2019-03-28T14:54:11Z

src/canmatrix/canmatrix.py

+        :rtype: Frame or None
+        """
+
+        if pgn <= 0xEFFF:


Instead of handle the pgn internals (for me "magic") here, I'd think about to defining some equality check to the CanId class itself (to prevent copy paste such if in another places).

Funth0mas · 2019-03-28T14:55:31Z

src/canmatrix/tests/test_canmatrix.py

@@ -731,14 +731,14 @@ def test_define_for_float():
 def test_canid_parse_values():
    can_id = canmatrix.canmatrix.CanId(canmatrix.ArbitrationId(id=0x01ABCD02, extended=True))
    assert can_id.source == 0x02
-    assert can_id.destination == 0x01


Of coure I wrote the tests to match the current code, not to match the J1939 specification :-(

ebroecker · 2019-03-28T15:47:19Z

I merged the CanID-Code a long time ago. I had no idea about J1939.

I also think, there is more wrong with it and it should be completely rewritten...
I'll try to do like spec. But I don't have any example data here...

Funth0mas · 2019-03-28T16:19:54Z

The core code of the online javascript converter, which works with bits-in-string, not integers, is

idAsBinary = string_representing_bites_in_the_can_id
var sa = idAsBinary.slice(-8);  // byte
var ps = idAsBinary.slice(-16, -8);  // byte
var pf = idAsBinary.slice(-24, -16);  // byte
var edp = idAsBinary.slice(-26, -25); // single bit
var priority = idAsBinary.slice(-29, -26); //3 bits
if(int(pf) >= 240){
 pgn=pf+ps;  // note this is not aritmhmetic but string addition
}else{
 pgn=pf+'00000000';  //kind of zero the lower byte
}

It think the >= 240 (>= 0xF0) is the main magic. If the converter works correctly, obviously.

Funth0mas

We shall (later) add one test to cover the if self.pf >= 240 condition but otherwise I see no more problems.

MatinF · 2019-03-29T09:03:08Z

Hi all,

We're very interested in this functionality and I'd be happy to provide some input on J1939 - and answer any questions I can. We work with numerous J1939 end users across the globe. J1939 is special since it is a common standard across most heavy duty vehicles.

We've recently done a more detailed version of our online J1939 CAN ID to PGN converter, which I think can be useful in getting a detailed understanding of the inner workings in going from a J1939 29 bit CAN ID to the J1939 PGN. Also, this J1939 intro may also be helpful.

The key to understand about converting J1939 data vs. other CAN data is that the matching of IDs should be based only on the PGN, not the full 29 bit CAN ID. Therefore, J1939 has to be handled as a special case in the code, where the PGN is extracted from the full CAN ID - both from the raw data being converted, as well as the J1939 DBC file itself.

The J1939 PGN of a 29 bit CAN ID starts at bit 9, with length 18 (indexed from 1) - see also the below illustration:

Example:
29 bit CAN ID (hex): CF00400
18 bit J1939 PGN (hex): F004

Here, the PGN F004 equals 61444 in DEC, which corresponds in the J1939 standard to Electronic Engine Controller 1. For a detailed breakdown of how to extract the PGN, I suggest checking out the converter I linked as it breaks down the CAN ID into sub-components - and reconstructs these into the PGN.

For the sake of illustration, I've added below an example of converting 2 different J1939 CAN IDs to illustrate that they both result in the same J1939 PGN, despite having different source addresses.

Further, I've added the corresponding J1939 DBC entry (both incl/excl. the 3 bit "extended ID flag") a third composition - yet still results in the same PGN. This allows the two different CAN IDs to trigger the same J1939 conversion when using adequate conversion methodology.

Below I've attached a sample MDF4 (and CSV) log file containing one repeated CAN frame with CAN ID 0CF00400 and the same data bytes F4 DE DE 30 28 FF F0 FF. Using standard DBC conversion, this file will not yield the right result as there is no match in the DBC ID specifically 0CF00400 (the DBC has 29 bit CAN ID CF004FE). However, using PGN matching you'll be able to convert the data and find that the scaled value of EngSpeed is 1541 rpm:

J1939_MDF4_DBC_SAMPLE.zip

J1939 is one of the most used protocols globally in our experience - so adding the support for this as a special case would be worthwhile I believe. We're specifically interested in having this implemented as we hope to use J1939 conversion via ASAMMDF, which depends on canmatrix.

Happy to answer any questions on this topic!

Best,
Martin

ebroecker · 2019-03-29T09:37:30Z

@MatinF
Hi Martin,

thanks for these lots of information.
~~I learned now that there is also a J1939 for 18-Bit CANIDs. ...~~

I'll add a checkbox-list to top of this PR with todos for J1939-PGN support

In branch ´j1939_decode´ I started with a j1939 decoder.
Currently only supports decoding BAM-Messages. But I also want to decode well-known SPN.
I have already found some Information about SPN decoding, but I'd be happy to get some correct example data for BAM, CMDT and several well known SPN .

Funth0mas · 2019-03-29T10:08:41Z

I thing the 18 bits is misunderstanding. He meant that PGN is 18 bits length (and classic can is 11 bits only). @MatinF , the j1939 works only with extended IDs, right?

ebroecker · 2019-03-29T10:10:23Z

@Funth0mas
you are right, I need more coffee now!

ebroecker · 2019-03-29T11:01:15Z

@Funth0mas
what do you think about dismissing the CanId class and add j1939-functionality to ArbitrationId-Class?
Just a thought...

Funth0mas · 2019-03-29T12:17:43Z

That might be a good idea if we only need to parse the id no pgn etc. What will be PNG value for a standard CAN id? None? Exception?

MatinF · 2019-03-29T12:29:41Z

Hi guys,

To clarify, what I mean is not that there will be an 18 bit CAN ID observed in J1939 raw data. The J1939 raw data will always have the extended 29 bit CAN IDs. You can see an example of raw J1939 data from a truck below:

Timestamp;Type;ID;Data
09T120102191;1;c00000b;fcffff7dffffffff
09T120102192;1;cef27fd;20fffae1ff00ffff
09T120102192;1;cffcafd;c0fffffffffff800
09T120102193;1;c000003;ecffffffffffffff
09T120102194;1;cf00203;cc00000000b812ff
09T120102195;1;18fe4a03;fffcffffffffffff
09T120102196;1;c010305;ccfffaffff204e0a

For each of the above IDs, you would need to extract the PGN (i.e. as per the method outlined in the illustration I parsed and in the converter Google Sheet). To verify that your CAN-ID to PGN extraction method leads to the right results, you can use our converter.

My suggestion would be that you maintain the current methodology for DBC conversion for both regular/extended CAN IDs as the default. Ideally, the user can then somewhere specify that he wants to convert J1939 data - as a special case.

When the user selects this method, the code will look for PGNs in the raw data - and compare this with the PGNs in the loaded DBC file.

Hope clear,
Martin

Funth0mas · 2019-03-29T13:38:27Z

Martin, the "conversion" in canmatrix terminology is rather the catalog conversion between dbc, arxml, json etc. There are no raw data from CAN.
But what you're talking about is rather data representation, the Frame.decode() etc, right? FAIK there is no batch can-trace representer in this project, which would be provided by dbc (catalog) and asc (can trace) data providing parsed messages from the whole can trace.

You can programatically load the dbc to a CanMatrix object, and than for every message from your CAN trace convert the CANId to PGN (what is this ticket about) and than call matrix.frame_by_pgn(pgn).decode(my_raw_data) or something like that.

remove CanId class

Funth0mas · 2019-04-02T05:19:11Z

src/canmatrix/canmatrix.py

+
+        :rtype: tuple"""
+
+        if not self.extended:


this can be omitted, exception would be raised by self.j1939_destination when creating the return tuple.

src/canmatrix/canmatrix.py

Funth0mas · 2019-04-02T05:21:54Z

src/canmatrix/canmatrix.py

+        return _pgn
+
+    @property
+    def j1939_tuples(self):  # type: () -> typing.Tuple[int, int, int]


Consider to rename to j1939_tuple as it is single tuple carrying three values.

src/canmatrix/canmatrix.py

Co-Authored-By: ebroecker <eduard.broecker@gmail.com>

Funth0mas · 2019-04-02T11:03:21Z

You need to remove the CanId (I added recently) from the canmatrix/init.py to fix the build. Sorry :-)

work on hints from Funth0mas

j1939-dbc is not validated in any way, so its just a compilation of informations I found in the net make dbc more robust

MatinF · 2019-04-04T09:32:30Z

Thanks for the clarification on the conversion Funth0mas. My use case is "indirect" via the asammdf module, which has a dependency on canmatrix. Hence I'm hoping that these updates will pave the way for PGN based conversion of *.mf4 files in that project, as that would be really useful to a lot of people :-)

ebroecker · 2019-04-04T11:25:57Z

@MatinF
Do you have some example J1939-Can-Trace with J1939-transportprotocol in?

MatinF · 2019-04-04T11:47:56Z

Hi ebroecker,

Below is a MDF file (MDF 4.11) and 2x demo J1939 DBCs for PGN 61444 (EEC1).

The MDF data is simulated J1939 data in the buslogging format. It includes a number of CAN IDs (all with constant data byte frames), incl. the extended CAN ID CF004FE (which equals PGN 61444).

I've included two DBC demo files, both for PGN 61444. One DBC has an "exact" 29 bit match vs. the MDF file, i.e. it uses CAN ID CF004FE. The other has CAN ID CF00400. Both DBC files should be able to convert the MDF file when correctly utilizing the J1939 PGN match method - but only the CF004FE DBC will work with the regular DBC matching methodology.

Hope that is clear, else let me know. Happy to provide further input on this.

DBC_MDF.zip

ebroecker · 2019-04-06T20:27:39Z

@MatinF
I tried your mdf, I found only EEC1, CCVS and PGN 64587 (arbitration_id 0x18fc4bfe)

MatinF · 2019-04-06T20:47:39Z

@ebroecker
The MDF and the DBC in my zip is a "sample" with simulated data, so it should only return a limited set of data - so I'd assume this to be the expected behavior. I tried to supply a limited example to make it clear whether it would convert with both DBC files or not.

Let me know if further clarification is required, I'd be happy to provide input.

…d decodes them depending on PGN dbc read j1939 frame format attribute

MatinF · 2019-04-07T08:42:24Z

Also, one quick note: A J1939 DBC may sometimes contain meta info indicating that it's a J1939 DBC (rather than some regular 29 bit CAN DBC). However, in our experience it's not always the case that such a meta info exists.

I'd therefore assume that the implementation would require that the user explicitly chooses to use J1939 decoding - rather than an automatic attempt to detect that a J1939 DBC is utilized.

Hope clear,
Martin

ebroecker · 2019-04-08T11:59:12Z

@MatinF
For now CanMatrix.decode takes the j1939 meta info into account and tries to decode a given frame by finding it by arbitration_id. If no arbitration_id matches, but j1939 database is detected it trys to match by pgn (only if frame is marked as j1939-frame by metadata).

Additionally I add canmatrix.j1939_decoder.j1939_decoder for explicit decode j1939 data.
This module additionally comes along with it's own j1939.dbc which should decode all well known j1939-frames (whereas the j1939.dbc is for sure not complete or correct for every pgn).
This own j1939.dbc is only used, if no match was found with given canmatrix.

Also I wanted this decoder to get arround with BAM and J1939-TP messages.
For BAM I have some little test, but for TP I don't have any sample data.

For an example usage of canmatrix.j1939_decoder.j1939_decoder look at src/canmatrix/tests/test_j1939_decoder.py

N.B: this is everything only in this PR - not merged to development yet.

MatinF · 2019-04-13T18:43:25Z

Hi ebroecker,

Can you provide an example of the DBC meta data you use to distinguish J1939 frames from regular extended CAN ID frames? That would help us understand how to ensure that this is done consistently when using this functionality.

Thanks,
Martin

ebroecker · 2019-04-13T19:39:45Z

Hi @MatinF ,

the j1939.dbc in this pr (which is merged already) contains the information.

canmatrix/src/canmatrix/j1939.dbc

Line 3119 in a00f7d8

    
           BA_DEF_ BO_ "VFrameFormat" ENUM "StandardCAN","ExtendedCAN","StandardCAN_FD","ExtendedCAN_FD","J1939PG";

canmatrix/src/canmatrix/j1939.dbc

Line 3126 in a00f7d8

BA_ "VFrameFormat" BO_ 2147483648 4;

At last this means, that there is an Attribute VFrameFormat with the value J1939PG.

at last the a frame in canmatrix gets the attribute is_j1939, this is done here:

canmatrix/src/canmatrix/formats/dbc.py

Lines 945 to 946 in a00f7d8

    
           if "J1939PG" in frame.attributes.get("VFrameFormat", ""): 
        
               frame.is_j1939 = True

add "frame_by_pgn" to canmatrix

137aa19

ebroecker changed the title ~~[WIP] add "frame_by_pgn" to canmatrix~~ add "frame_by_pgn" to canmatrix Mar 27, 2019

change destination evaluation of j1939 arbitration ids

58388f8

ebroecker changed the title ~~add "frame_by_pgn" to canmatrix~~ [WIP] j1939 updates and upgrades Mar 27, 2019

Merge branch 'development' into frame_by_pgn

f9f86f0

Funth0mas reviewed Mar 28, 2019

View reviewed changes

reengineer j1939/PGN/...

fe3f5aa

Funth0mas approved these changes Mar 29, 2019

View reviewed changes

ebroecker and others added 3 commits March 29, 2019 22:48

Merge branch 'development' into frame_by_pgn

de5d545

add further pgn tests

b864a8a

move j1939-properties to ArbitrationId class

c500cf3

remove CanId class

Funth0mas reviewed Apr 2, 2019

View reviewed changes

Funth0mas and others added 4 commits April 2, 2019 08:55

Update src/canmatrix/canmatrix.py

7d861cc

Co-Authored-By: ebroecker <eduard.broecker@gmail.com>

Update src/canmatrix/canmatrix.py

21ab564

Co-Authored-By: ebroecker <eduard.broecker@gmail.com>

Merge branch 'development' into frame_by_pgn

f33f4f4

Merge branch 'development' into frame_by_pgn

49caa82

ebroecker and others added 4 commits April 2, 2019 19:50

Merge branch 'development' into frame_by_pgn

da86320

merge development

3775780

work on hints from Funth0mas

add some j1939 decoding example (even with j1939-dbc)

8b60e88

j1939-dbc is not validated in any way, so its just a compilation of informations I found in the net make dbc more robust

fix join module

50c58e3

ebroecker and others added 6 commits April 4, 2019 22:14

improve j1939 support

3503cc0

improve j1939 decoder

f209672

fix py2 tests

1b7451a

add j1939 default dbc

4699aef

Merge branch 'development' into frame_by_pgn

12169fe

make matrix optional for j1939-decoding

b143d14

add warnings for not yet implemented j1939 transport protocols

8bfbd65

ebroecker added 3 commits April 6, 2019 22:50

merge changes from #331

fdfdff2

add j1939 frame format to j1939 dbc

2172026

CanMatrix.decode now automatically takes j1939 frames into account an…

f4bac22

…d decodes them depending on PGN dbc read j1939 frame format attribute

add test for j1939 frame format in dbc

fc3513c

Merge branch 'development' into frame_by_pgn

a00f7d8

ebroecker merged commit 6afc8e6 into development Apr 12, 2019

ebroecker mentioned this pull request Apr 12, 2019

I-Signals with initial value of type float crash conversion and logger levels need improvement #331

Closed

ebroecker deleted the frame_by_pgn branch May 6, 2019 13:23

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] j1939 updates and upgrades #335

[WIP] j1939 updates and upgrades #335

ebroecker commented Mar 25, 2019 •

edited

Loading

codecov-io commented Mar 25, 2019 •

edited

Loading

ebroecker commented Mar 27, 2019

Funth0mas commented Mar 28, 2019

Funth0mas Mar 28, 2019

Funth0mas Mar 28, 2019

Funth0mas Mar 28, 2019

ebroecker commented Mar 28, 2019

Funth0mas commented Mar 28, 2019

Funth0mas left a comment

MatinF commented Mar 29, 2019 •

edited

Loading

ebroecker commented Mar 29, 2019 •

edited

Loading

Funth0mas commented Mar 29, 2019

ebroecker commented Mar 29, 2019

ebroecker commented Mar 29, 2019

Funth0mas commented Mar 29, 2019

MatinF commented Mar 29, 2019

Funth0mas commented Mar 29, 2019

Funth0mas Apr 2, 2019

Funth0mas Apr 2, 2019

Funth0mas commented Apr 2, 2019

MatinF commented Apr 4, 2019

ebroecker commented Apr 4, 2019

MatinF commented Apr 4, 2019

ebroecker commented Apr 6, 2019

MatinF commented Apr 6, 2019

MatinF commented Apr 7, 2019

ebroecker commented Apr 8, 2019

MatinF commented Apr 13, 2019

ebroecker commented Apr 13, 2019

		@@ -1843,7 +1861,12 @@ def __init__(self, arbitration_id): # type: (ArbitrationId) -> None
		if arbitration_id.extended:
		self.source = arbitration_id.id & int('0xFF', 16)

[WIP] j1939 updates and upgrades #335

[WIP] j1939 updates and upgrades #335

Conversation

ebroecker commented Mar 25, 2019 • edited Loading

codecov-io commented Mar 25, 2019 • edited Loading

Codecov Report

ebroecker commented Mar 27, 2019

Funth0mas commented Mar 28, 2019

Funth0mas Mar 28, 2019

Choose a reason for hiding this comment

Funth0mas Mar 28, 2019

Choose a reason for hiding this comment

Funth0mas Mar 28, 2019

Choose a reason for hiding this comment

ebroecker commented Mar 28, 2019

Funth0mas commented Mar 28, 2019

Funth0mas left a comment

Choose a reason for hiding this comment

MatinF commented Mar 29, 2019 • edited Loading

ebroecker commented Mar 29, 2019 • edited Loading

Funth0mas commented Mar 29, 2019

ebroecker commented Mar 29, 2019

ebroecker commented Mar 29, 2019

Funth0mas commented Mar 29, 2019

MatinF commented Mar 29, 2019

Funth0mas commented Mar 29, 2019

Funth0mas Apr 2, 2019

Choose a reason for hiding this comment

Funth0mas Apr 2, 2019

Choose a reason for hiding this comment

Funth0mas commented Apr 2, 2019

MatinF commented Apr 4, 2019

ebroecker commented Apr 4, 2019

MatinF commented Apr 4, 2019

ebroecker commented Apr 6, 2019

MatinF commented Apr 6, 2019

MatinF commented Apr 7, 2019

ebroecker commented Apr 8, 2019

MatinF commented Apr 13, 2019

ebroecker commented Apr 13, 2019

ebroecker commented Mar 25, 2019 •

edited

Loading

codecov-io commented Mar 25, 2019 •

edited

Loading

MatinF commented Mar 29, 2019 •

edited

Loading

ebroecker commented Mar 29, 2019 •

edited

Loading