)
Author: Wolfgang Buescher ( MKT Systemtechnik, www.mkt-sys.de,
)
Date: 2016-02-28 (ISO8601, YYYY-MM-DD)
The CAN log files described in this article contain everything you need to decode all all logged signals. For this reason, the file header contains a small database with signal definitions in a "textual" form. The CAN logger actually writes messages (not signals) to the memory card. A single data record in the logfile consists of 16 bytes of information, including...
A complete and up-to-date desciption of the CAN logger's primary file format may be obtained from MKT Systemtechnik. Please ask the software development engineer Mr Buescher for the "CAN-logger file format specification" .
If you are a "C" programmer and want to write your own tool to process logfiles, look into the header file with the definition of data types (logtypes.txt, original name : logtypes.h). This heavily commented file contains structure- and union definitions for the binary data section in a logfile. Studying this file carefully should answer 90 % of all questions about the structure of the logfiles. The rest will be explained later in this document.
Please note that writing your own logfile-converter is not an easy task, due to the large number of data types, CPU byte orders, and logger options ;-)
Besides the actual logged data, a logfile contains a lot of info about how the data was logged, and how the data in the file can be decoded into human-readable form.
The first few hundred bytes in every logfile are text lines (separated with
carriage return + new line characters). It contains most of the logger
configuration data. The reason to have this in the header: The author
of the CAN logger wanted to have all informations required to decode the
binary "data section" in the file itself. So there is no need to have the
original logger configuration file (*.clc) to decode a logger data file
(*.cld).
Example for a text header section (without the trailing end-marker)..
[Common]
Encryption=0
PwdHint1=""
Password1=""
[CanLoggerConfig]
InfoString="Logger Konfig fuer KM_221 Nr.2"
DbFileBus1="C:\CanLoggerTest\Bus1.dbc"
DbFileBus2="C:\CanLoggerTest\Bus2.dbc"
ReceiveAll=0
ReceiveAppl=0
FifoMode=0
ReopenToAppend=0
FileSizeLimit=1000
MsgRateLimit=1
TimestampFreq=39065.5
InitTrigger=1
Pretrig10ms=2000
Posttrig10ms=2000
TrigOnDigIn=0
TrigOnRxId=0x00000000
TrigOptions=0x00000000
Msg00: id=0x00000123,mn="Msg123",nn="TestNode1"
Sig00: id=0x00000123,sn="Temp_01",ty=U,un="mm",fa=1,of=-20,mi=-20,ma=50,sb=32,nb=8,bo=I
The most important line you need to parse from the text header section is
the
TimestampFreq value.
It contains the frequency of the high-resolution 32- and 24-bit timestamps
in the binary data section. You will need it to
convert the timestamp values into seconds: time_in_seconds = timestamp_value
/ TimestampFreq .
The "Msg" and "Sig"-definitions are required to decode the signal values later. There is one entry per text line, the parameters are marked with two-letter tokens, which are:
If your conversion utility has access to the CAN database files which were
used to configure the CAN logger, you may skip most sections in the text
header (except for the TimestampFreq token).
To skip the text header, read all lines until you reach the
[EndOfHeader]
section. After this, a few dummy bytes are inserted to fill the file up to the next 512-byte boundry. This was required to optimize the disk access for the following "binary" parts of the logfile (the microcontroller in the CAN-logger writes the file sector-by-sector; a disk sector contains 512 bytes).
The binary header follws after the text header. It is defined as a C-structure named "T_CanLog_BinaryHeader512". It is defined in the file LOGTYPES.H (or logtypes.txt), which you will find in the same directory as this document.
The structure begins with a string containing "BinaryLogfileHeader", which can be used to check if your program really found this header. All entries in the section commented as "Debugging stuff" in the header file are subject to change.
Note: The T_CanLog_BinaryHeader512 structure contains some
"reserved" entries, which may be used in future versions of the CAN logger
firmware. In fact, you can skip (ignore) the binary header part completely.
But the Logger firmware needs it to re-open an existing file quickly.
The rest of the logfile (after the "header" parts) is filled with binary
data blocks, which may be very large (several megabytes). Most important
to understand the structure of this part:
All entries in this sections are squeezed into 16-byte-blocks. Most of these
blocks (let's call them entries) will contain CAN MESSAGES, but there may
be other data too - despite the fact that we are still talking about a CAN
LOGGER here, not a SIGNAL LOGGER !
Look into at the structure named T_CanLog_MsgData in logtypes.txt (or logtypes.h if you have) .
This data type defines the basic structure of the 16-byte-entries in the
binary "data" section of a logfile. The most significant bits in the first
byte in this structure (bTypeFlags) defines the type of the structure. The
bits in the first byte are:
Bit 7,6 : Union Type Code.
CANLOG_UNION_TYPE_MSG or CANLOG_UNION_TYPE_HDR.
Bit 5,4 : Subtype, usage and meaning depends on the union type
code.
Bit 3..0 : Count of received data bytes (= data length code, 4 bit!)
The next three bytes (if the entry contains a CAN MESSAGE) are used as a
24-bit timestamp. Note: INTEL-BYTE-ORDER = least significant byte FIRST !
...
BYTE bTimeL; // timestamp, low
byte (sometimes 1st of 3-letter-code)
BYTE bTimeM; // timestamp, medium byte (sometimes
2nd of 3-letter-code)
BYTE bTimeH; // timestamp, high byte
(sometimes 3rd of 3-letter-code)
// The remaining 12 bytes in a 16-byte-struct are UNION-TYPE-specific
components,
// here : a CAN MESSAGE ...
BLONG blBusAndId; // 4 byte: BusNr + 29BitFlag + 11..29-bit
Message-ID .
// Note: a BLONG is a
byte-adressable 32-bit integer value.
BYTE bData[8]; // 8 byte raw message data
The CAN-message identifier is combined with the bus-number ("CAN channel")
and the standard/extended flag. The following "C"-defines may be useful to
split the information in the "BusAndId"-value:
#define C_CAN_29_BIT_ID_FLAG 0x20000000L /* Maske fuer 29-Bit-Flag */
#define C_CAN_ID_MASK_BUS2 0x40000000L /* Bitmaske fuer "Messages von Bus 2" */
#define C_CAN_ID_MASK_BUS3 0x80000000L /* Bitmaske fuer "Messages von Bus 3/4 (Reserve) */
#define C_CAN_ALL_ID_BITS 0x1FFFFFFFL /* Maske fuer "alle ID-Bits" */
If neither C_CAN_ID_MASK_BUS2 nor C_CAN_ID_MASK_BUS3 are set in the 32-bit
"ID"-value, the message has been received from the first bus (first CAN channel).
What makes the analysis of a CAN logfile a bit more complicated than you may expect are the timestamps. Why ? For efficient logging, the file is written sector-wise, which means the size of the entries must be an integer fraction of 512. Here, for CAN frames with 8 data bytes, the only reasonable struct size is 16 byte. For reasons explained in the previous chapter, only 3 bytes remain for the timestamp in a logged CAN-message entry . In other words, 24 bits for the timestamp ... too few ... if the timestamp is incremented every 25.6 microseconds(*) , the 24-bit value will overflow every 429 seconds ! So the 24-bit timestamp overflow must be handled by the logfile converter, like this:
if the new 24-bit timestamp value is less than the previous
one,
(*) assuming the timestamp generator runs at 40 MHz divided by 1024, which is 39065.5 Hz, or 1 / 25.598 microseconds. But beware, the timer frequency is hardware dependent ! It may be completely different if the CPU clock frequency is not 40 MHz, because the timer's prescaler can only divide by powers of two. For this reason, you must read the timer frequency from the logfile's text header .
The logger (firmware) also places absolute timestamps in the logged data stream, which contains 32(!)-bit timestamps along with the current date (year, month, day, hour, minute, second). You can easily see such entries in the binary part of a logfile by looking at the first three characters (in each 16-byte entry). The "union type code" in the first byte of an absolute timestamp-entry is 0x40 (=CANLOG_UNION_TYPE_HDR), so the next three bytes (which would be time 24-bit timestamp in a CAN message) contain the ASCII-characters "tim".
Example: CAN logfile opened with a HEX file viewer:
In this example, a timestamp entry can be seen at file position 0002400 (hex). The first 4 bytes are union type (here: 0x40 = "not a CAN message"), followed by three characters (0x74 0x69 0x6D = "tim"). The next 4 bytes are the 32(!)-bit timestamp, here: 0x000FD945 (little-endian format, marked red). The remaining 8 bytes contain the calendar date and -time as described below. At file positions 0002410, 0002420 and 0002430, three logged CAN-messages are shown, each containing a 24-bit timestamp only (also little-endian byte order, marked red in the screenshot).
Note: The Calendar-date and -time is not frequently updated ! It must be seen in conjunction with the (32-bit) timestamp value ! For example, the "calendar" time may contain 09:00:00 (logging started at 09:00:00), while the timestamp-value (converted to seconds) may be 3662.0 in a logged CAN message -> the time of reception was 10:01:02 then ! The logger firmware will update the calendar-date and time only to avoid overflow of the 32-bit timestamp, or after the logger was shut down and restarted with the option "continue old logfile".
If you need to combine the absolute time with the high-resolution timestamp for some reasons, first convert the calendar-time and date into UNIX-format ("seconds passed since 1970-01-01, 00:00:00) as a double precision floating point value, then combine(!) the 32-bit timestamp value from the "tim"-entry with the 24-bit timestamp value from the a CAN-message. When reading a "tim"-entry from the logfile, keep a copy of it in memory, and use the most significant byte to expand the 24-bit timestamp in a CAN-message to 32 bits, before converting that value into seconds, milliseconds, microseconds or whatever.
Sounds too tricky ? Indeed. If you seriously plan to write your own logfile converter, and need an "absolute, high-resolution time channel", ask the author for a snippet from his own logfile converter (written in C / C++ ).
/****************************************************************************/
int CanLog_ExprResultsToBulkObject(
T_CanLog_ExprResult *pResults, int iHowMany, // [in] pResults[ 0 .. iHowMany-1 ]
DWORD dwTimestamp, // [in] timestamp of the 'origin' of the latest entry in pResults[]
BOOL fFlushAll, // [in] TRUE=flush *all* valid results, FALSE=flush only *modified* values
BYTE *pbOutput, BYTE *pbEndstop ) // [out] : later copied to CanLog_LargeBuffer.logger[ CanLog_dwLargeBufIndexIn++ ]
// Places the *values* of logged display variables, script variables, etc
// in a 'bulk' object (variable-sized block in the recorded file).
// Returns the number of bytes(!) actually placed in *pbOutput .
//
// Since 2014-01-29, those logged 'calculated expressions'
// may actually be SCRIPT VARIABLES declared with attribute 'logged:' .
// For efficient disk space usage, a new data type "BULK"
// ( logtypes.h :: CANLOG_UNION_TYPE_BULK ),
// which may be much larger than the old 16-bytes-per-entry objects,
// is used to embed those 'expressions' and SCRIPT VARIABLES
// in the logged stream.
// Also (at least for numeric data) the values are only emitted
// to the stream when 'modified' (since the last written sample).
{
int i,j,n,nBytesToEncodeBlocksize,nEmittedValues;
long nBytes;
BYTE *pbOutputStart = pbOutput;
T_CanLog_ExprResult * pCanLogResult;
// Place all logged expressions and script variables for the current "sampling point"
// inside a single 'BULK' object. That bulk object only has a single timestamp for all values,
// so there is less overhead than the older method (with fixed-size 16-byte entries) .
// A 'BULK' object now contains ONE OR MORE 16-byte entries in the CAN LOGFILE BUFFER !
// Run through all entries in CanLog_sExprResult[0..iHowMany-1], to count
// THE TOTAL NUMBER OF BYTES required to assemble this 'BULK' block :
for( i=nBytes=nEmittedValues=0; i<iHowMany; ++i ) // 1st loop through CanLog_sExprResult[i] :
{ // detect the total size in bytes, and the number of modified values (there may be none at all)
pCanLogResult = &pResults[i];
if( (pCanLogResult->fModified || fFlushAll) && (CanLog_Config.Expr[i].sz40Expression[0]!=0) )
{ // What's the current data type of *this* variable (data types may change at runtime, at least in the script language)
switch( pCanLogResult->iCanLogDataType ) // CANLOG_UNION_DATA_TYPE_.., defined in C:\cproj\UPT_xx\logtypes.h
{ case CANLOG_UNION_DATA_TYPE_UNKNOWN: // (0x00) here used to mark INVALID entries (don't log yet)
default : // ... or for UNSUPPORTED data types
break;
case CANLOG_UNION_DATA_TYPE_BOOLEAN: // (0x01, "CANopen slang", originally a SINGLE BIT VARIABLE)
case CANLOG_UNION_DATA_TYPE_INTEGER8: // (0x02, CANopen slang...)
case CANLOG_UNION_DATA_TYPE_UNSIGNED8: // (0x05, CANopen slang for a BYTE )
++nEmittedValues;
nBytes += ( 1/*for the table index*/ + 1/*for the data type*/ + 1/* for the netto data */);
break;
case CANLOG_UNION_DATA_TYPE_INTEGER16: // (0x03)
case CANLOG_UNION_DATA_TYPE_UNSIGNED16: // (0x06)
++nEmittedValues;
nBytes += ( 1/*for the table index*/ + 1/*for the data type*/ + 2/* for the netto data */);
break;
case CANLOG_UNION_DATA_TYPE_INTEGER32: // (0x04)
case CANLOG_UNION_DATA_TYPE_UNSIGNED32: // (0x07)
case CANLOG_UNION_DATA_TYPE_REAL32: // (0x08, CANopen slang for a 'single precision floating point')
++nEmittedValues;
nBytes += ( 1/*for the table index*/ + 1/*for the data type*/ + 4/* for the netto data */);
break;
case CANLOG_UNION_DATA_TYPE_VISIBLE_STRING: // (0x09, CANopen slang for a simple TEXT STRING with variable length)
// In this case ("strings"), the string itself is NOT contained in CanLog_sExprResult[i],
// because the union in that array is restricted to FOUR BYTES :
// If CanLog_sExprResult[i].iCanLogDataType == CANLOG_UNION_DATA_TYPE_VISIBLE_STRING,
// then j := CanLog_sExprResult[i]u.i32 is an index into a STRING TABLE,
// and CanLog_sz255LoggedStrings[j] is the actual STRING VALUE .
j = pCanLogResult->u.i32; // HERE: j = index into CanLog_sz255LoggedStrings[j] !
if( j>=0 && j<CANLOG_MAX_LOGGED_STRINGS && CanLog_iLoggedStringUsedByExprIndex[j]==i )
{ n = strlen( CanLog_sz255LoggedStrings[j] );
}
else
{ n = 0;
}
++nEmittedValues;
nBytes += ( 1/*for the table index*/ + 1/*for the data type*/ + n/* for the characters */ + 1/*for the trailing zero*/ );
break;
case CANLOG_UNION_DATA_TYPE_OCTET_STRING: // (0x0A, CANopen slang for an 'array of bytes')
// Not supported yet. Could be used (one fine day) to log 'binary objects' from the script ?
break;
} // end switch( pCanLogResult->iCanLogDataType )
}
} // end for < determine the NUMBER OF BYTES required for the 'BULK' data block >
if( nEmittedValues > 0 )
{ // Arrived here: there's at least ONE modified value, and now we know
// nBytes = the number of 'netto' bytes for the BULK data block.
// Additional overhead for the 'BULK' block:
// - ONE byte for the '1st byte [=LSByte] in every T_CanLog_LoggingUnion (bTypeFlags)',
// here: with bits 7+6 set to CANLOG_UNION_TYPE_BULK;
// - plus four bytes for the TIMESTAMP (common for all channels in the sampling point)
nBytes += 5;
// - plus ZERO, TWO, or FOUR additional bytes to encode the length of the BULK block;
// - plus additional padding bytes (up to 15) to 'round up' the BULK block to N*16 bytes.
nBytesToEncodeBlocksize = 0;
if( nBytes > (240-15) ) // for up to 240 bytes, the block size can be encoded in the lower bits of the 1st byte..
{ nBytesToEncodeBlocksize = 2; // up to 65535 bytes, TWO bytes are sufficient to encode the block size
if( nBytes > (65535-15) )
{ nBytesToEncodeBlocksize = 4; // whow, SUCH a large block... use FOUR BYTES to encode it's size !
}
}
nBytes += nBytesToEncodeBlocksize;
nBytes = (nBytes+15) & 0x7FFFFFF0; // *round up* to the next multiple of 16 bytes for the 'BULK' block..
if( nBytes < 32 ) // because the 4-bit numbers 0000 and 0001 are used to indicated 'two or four byte LENGTH'..
{ nBytes = 32; // .. the minimum 'brutto' size of a BULK block is 32 !
}
// Assemble a 'BULK' structure for the CAN logger's output file,
// based on the specification in C:\cproj\UPT_xx\logtypes.h :
// *ppbDest is a BYTE POINTER to the first byte in a 16-byte T_CanLog_LoggingUnion .
// The 'BULK' object (=large quantity of data) will actually span over multiple of those 16-byte units !
*pbOutput = CANLOG_UNION_TYPE_BULK | CANLOG_UNION_TYPE_BULK_T_EXPR; // fill out the 'BULK' header byte...
switch( nBytesToEncodeBlocksize )
{ case 2: // bulk block's length encoded as a TWO-BYTE entity (after the header byte)
*pbOutput |= CANLOG_UNION_TYPE_BULK_SIZE2;
++pbOutput;
CanLog_AppendInt16( &pbOutput, nBytes );
break;
case 4: // bulk block's length encoded as a FOUR-BYTE entity (after the header byte)
*pbOutput |= CANLOG_UNION_TYPE_BULK_SIZE4;
++pbOutput;
CanLog_AppendInt32( &pbOutput, nBytes );
break;
default: // bulk block's length encoded in the lower 4 bits of the header byte :
i = (nBytes+15) / 16; // the result (N*16 bytes) will be <= 15 !
if( i<2 ) i=2; // size codes 0 and 1 must be avoided here !
if( i>15) i=15; // size code > 15 should never happen, but for safety..
*pbOutput |= i;
++pbOutput;
break;
} // end switch( nBytesToEncodeBlocksize )
CanLog_AppendInt32( &pbOutput, dwTimestamp );
for( i=0; (i<iHowMany) && ((pbOutput+6)<pbEndstop); ++i ) // 2nd loop : assemble a 'BULK' block
{ pCanLogResult = &CanLog_sExprResult[i];
if( (pCanLogResult->fModified || fFlushAll)
&& ((pbOutput+3)<pbEndstop)
&& (CanLog_Config.Expr[i].sz40Expression[0]!=0) )
{ if( (pCanLogResult->iCanLogDataType > CANLOG_UNION_DATA_TYPE_UNKNOWN)
&& (pCanLogResult->iCanLogDataType <= CANLOG_UNION_DATA_TYPE_VISIBLE_STRING) )
{ // Ok, the value has been modified, and the type is supported here: Append it to the bulk block
*pbOutput++ = (BYTE)i; // append the table index (to identify this value later)
*pbOutput++ = (BYTE)pCanLogResult->iCanLogDataType; // append the data type (it may CHANGE for script-variables during runtime !)
// Append THIS value, with 1..4, and sometimes even more bytes :
switch( pCanLogResult->iCanLogDataType )
{ case CANLOG_UNION_DATA_TYPE_UNKNOWN: // (0x00) here used to mark INVALID entries (don't log yet)
default : // ... or for UNSUPPORTED data types
break;
case CANLOG_UNION_DATA_TYPE_BOOLEAN: // (0x01, "CANopen slang", originally a SINGLE BIT VARIABLE)
case CANLOG_UNION_DATA_TYPE_INTEGER8: // (0x02, CANopen slang...)
case CANLOG_UNION_DATA_TYPE_UNSIGNED8: // (0x05, CANopen slang for a BYTE )
*pbOutput++ = (BYTE)(pCanLogResult->u.i32 & 0x00FF);
break;
case CANLOG_UNION_DATA_TYPE_INTEGER16: // (0x03)
case CANLOG_UNION_DATA_TYPE_UNSIGNED16: // (0x06)
CanLog_AppendInt16( &pbOutput, pCanLogResult->u.i32 );
break;
case CANLOG_UNION_DATA_TYPE_INTEGER32: // (0x04)
case CANLOG_UNION_DATA_TYPE_UNSIGNED32: // (0x07)
case CANLOG_UNION_DATA_TYPE_REAL32: // (0x08, CANopen slang for a 'single precision floating point')
CanLog_AppendInt32( &pbOutput, pCanLogResult->u.i32 ); // also works to 'transport' 32-bit FLOAT !
break;
case CANLOG_UNION_DATA_TYPE_VISIBLE_STRING: // (0x09, CANopen slang for a simple TEXT STRING with variable length)
// In this case ("strings"), the string itself is NOT contained in CanLog_sExprResult[i],
// because the union in that array is restricted to FOUR BYTES .
j = pCanLogResult->u.i32; // HERE: j = index into CanLog_sz255LoggedStrings[j] !
if( j>=0 && j<CANLOG_MAX_LOGGED_STRINGS && CanLog_iLoggedStringUsedByExprIndex[j]==i )
{
CanLog_AppendString( &pbOutput/*destination*/, pbEndstop, CanLog_sz255LoggedStrings[j], 255 );
}
else
{ *pbOutput++ = 0; // emit only trailing zero as limiter for an empty 'C' string
}
break;
case CANLOG_UNION_DATA_TYPE_OCTET_STRING: // (0x0A, CANopen slang for an 'array of bytes')
// Not supported yet. Could be used (one fine day) to log 'binary objects' from the script ?
break;
} // end switch( pCanLogResult->iCanLogDataType )
} // end if < data type supported for the export in a 'BULK' block of logged expressions >
pCanLogResult->fModified = FALSE; // no need to write this entry again (until its value is modified)
} // end if( pCanLogResult->fModified )
} // end for < assemble a 'BULK' data block with all logged 'expressions' and SCRIPT VARIABLE>
// If there are unused bytes remaining (between the end of the 'BULK' block
// and the next 16-byte boundary, here: limited by "pbEndstop"), fill those bytes with FF's (!):
n = pbOutput - pbOutputStart; // -> number of bytes 'produced' in the loop above
while( ((n&15)!=0) && (pbOutput <= pbEndstop) )
{ *pbOutput++ = 0xFF; // fill the rest of the 'BULK' block with FFs (not zeroes!)
++n;
}
} // end if( nEmittedValues > 0 )
return pbOutput - pbOutputStart; // > Returns the number of bytes(!) actually placed in *pbOutput
} // end CanLog_ExprResultsToBulkObject()
Note: Instead of writing your own logfile converter, how about using
one of the common text-based CAN-Logfile formats ("ASCII") ? The ASCII files
are less prone to "change", which may happen to the internal format. Latest
example: Implementation of the 512-byte BINARY logfile header which is described
in logtypes.txt
only.