Data Logging

Datalogger - more updates

Posted: 
Tuesday, July 25, 2017 - 15:00

It's been about six weeks since I posted the last update to the site, so thought I better post something...

Since the last post I've been working on implementing a modular system, with the idea that the basic logger can handle control inputs from USB, Bluetooth LE, a simple stop/go button interface, or a touch panel. The "Pass Through" interface shown in the last post is intended as a way of allowing applications running on mobile devices to communicate with the ECU without the need to micro-manage the low level protocol. You send a Service ID, Resource ID and any other necessary data for the request and get back a positive response plus data, or negative response if the request fails.

The second module is a configurable logger. And this is what I've been working on since start of July. The logger is currently set up with definitions for the 18 or so diagnostic requests that I think you'd realistically want to log.
Requests are set by using something like..

AT M LC  09 0D 1B

LC indicates a "Log Configure" command, and the items to log are specified using the Request ID. In this example 0x09 = RPM, 0x0D = Road Speed, and 0x1B = Throttle Voltages. The command input is not case or space sensitive.

Starting and stopping logging is controlled by AT commands.

AT M LS  
AT M LE  

These are the "Log Start" and "Log End" commands respectively.

By default the data from the logger is in ECU format with no transformations. Using the LR command

AT M LR  

toggles between "Raw" and "Convert" mode which converts Kelvin to Celsius and shifts decimal places where appropriate.

The next thing I need to tackle is the logging to SD Card and the button control interface.

Td5 Datalogger updates

Posted: 
Sunday, June 11, 2017 - 08:00

I've come to something of a Catch 22 where I need detailed logging to properly document and verify the waste gate modulator, and the code that handles torque reduction requests from the auto gear box before I can post details of the maps. On the other hand, if I work on the datalogger nothing much happens on the website.

The combination of changes to the Microchip tool chain which caused issues with my old code, a failed hard drive which lead to the discovery that the Microchip peripheral library installer does not work on current versions of macOS, and the need for a rethink and rewrite has meant the datalogger has been out of action since late December.

When I was playing around with the VAGCOM adapter and Python earlier in the year I had ported//rewritten part of my C code and found that using a publish and subscribe framework solved a few of the issues I'd run into. As a result of that little "ah ha" moment and the hassle of dealing with the Microchip libraries I've been working on writing and unit testing peripheral drivers and a publish & subscribe framework in C.

Over the past few weeks I've been progressively bolting together the code that manages the communications with the Td5 ECU.

The way I've decided to approach this is to look at the ISO14230 docs and build a module that handles the startCommuncations, sendData and stopCommunications commands as the base of the logger/interface. That section is now working quite well and at this stage only requires some additional error handling to deal with cases where the ECU response times out mid-message.

From that base I've been working on adding a command interface and USB interface. The idea is to provide a basic set of ELM-like AT commands to configure the settings from default. The current defaults for "pass through" operation are set to connect to a Td5 ECU in manufacturer specific diagnostic mode - Testbook diagnostics in other words.

In practice this means you can hook up the interface and send requests without worrying about the underlying communications protocols. This is similar to how the ELM327 operates.

This is a short demo of the pass through mode, operating over a serial usb connection. The delay between hitting return on the "21 01" command and the response includes the time taken to connect and authenticate to the ECU.

Nanocom One log converter

Posted: 
Sunday, May 14, 2017 - 08:00

Updated: I decided to revisit the Nanocom One log converter code today with view to incorporating it into a larger project. When I made a couple of changes to tidy things up, I discovered that I'd managed to wallpaper over a small problem in the Nanocom One log format, and also missed the bleeding obvious in regard to the record delimiter. The post is getting a rewrite to correct.

While the Nanocom One is viewed as a bit of a dinosaur there is still a body of opinion that it is far more reliable for .map uploads than it's replacement - the Evo. And there are still a lot of them in circulation, and that means there a quite a few logs in the binary .fu1 format.

The usual procedure is to open up the old graphic viewer PC application and use that to check files or to export as a.csv file and then use a spreadsheet app to look at the data. Either way it's not especially flexible and a bit of a PITA.

Anyway last night I sat down with a copy of Synalyze It! (Hexinator on Windows/Linux) and worked out the structure of the .fu1 log files.
Synalyze It! uses "grammar" files to apply structures a binary file which is a fairly nice way of reverse engineering file formats.

This shows how a single fuelling record looks with the fu1 grammar applied:

       Fuel Record [0] (0)
        {
            RecordDelimiter: CR LF: 0D 0A  [0]
            FuelData [0] (2)
            {
                DiagRequest_0x09 [0] (2)
                {
                    RPM: 754 [0]
                }
                DiagRequest_0x0D [0] (4)
                {
                    RoadSpeed: 1 [0]
                }
                DiagRequest_0x1A [0] (5)
                {
                    ECT: 3523 [0]
                    ECT mV: 772 [0]
                    IAT: 3311 [0]
                    IAT mV: 1309 [0]
                    AAT: 3200 [0]
                    AAT mV: 1681 [0]
                    FT: 3325 [0]
                    FT mV: 1260 [0]
                }
                DiagRequest_0x1B [0] (21)
                {
                    TPS1 mV: 571 [0]
                    TPS2 mV: 4460 [0]
                    TPS3 mV: 4677 [0]
                    Throttle %: 0 [0]
                    TPS Supply mV: 5021 [0]
                }
                DiagRequest_0x1C [0] (31)
                {
                    MAP: 10143 [0]
                    MAP raw: 10158 [0]
                    MAF: 575 [0]
                    MAF mV: 1977 [0]
                }
                DiagRequest_0x21 [0] (39)
                {
                    idleErrorRpm: -3 [0]
                }
                DiagRequest_0x40 [0] (41)
                {
                    Cyl 1: 3 [0]
                    Cyl 2: -2 [0]
                    Cyl 3: -1 [0]
                    Cyl 4: -2 [0]
                    Cyl 5: 2 [0]
                }
                DiagRequest_0x23 [0] (51)
                {
                    AAP: 9971 [0]
                    AAP raw: 10013 [0]
                }
                DiagRequest_0x37 [0] (55)
                {
                    EGR Inlet: 0 [0]
                }
                0x38 [0] (57)
                {
                    WGM Duty: 0 [0]
                }
                0x38 [0] (59)
                {
                    WGM Duty: 0 [0]
                }
                0x10 [0] (61)
                {
                    BAT mV: 13807 [0]
                    BAT_MSB_only: 35  [0]
                }
            }
        }

The format is actually pretty simple. It consists of a CRLF ('0x0D0A') pair delimiting each "observation" and then the block of data which makes up the observation. The final byte of the file looks to be a mod256 checksum.

The hex number appended to each DiagRequest_0x refers to a predefined diagnostic request used by the ECU to look up engine parameters. These are common to MSB, and EU2 and EU3 NNN, so if you request 0x10 you'll always get back 4 bytes of data containing corrected and uncorrected Battery Voltage. The problem I discovered when revising the script is that the Nanocom One logs only 3 of the 4 bytes of data for the Battery Voltage. This means the corrected voltage is OK, but only the most significant byte for the uncorrected voltage is available making it useless. I've corrected the script so it ignores the corrupt value.

Note also that until the most recent update (v1.32) the Nanocom was making duplicate requests for the WGM Duty Ratio, and using one read for the EGR Output. This update fixed the duplication in fueling mode, but it remained in Instrument mode when I last looked.

What is interesting is that the full data for each request is saved in the file, so that the Throttle % and AAT reading is present in the data. As can be see from the decoded structure the file also includes sensor voltages for all temperature sensors, and the MAF.

I found that there is also a small error in the Nanocom One's AAP value - there are two values returned - one is the raw reading, and the other is the reading corrected for variation in sensor supply voltage. The corrected value is used by the ECU for fuelling and boost limiting purposes, however the Nanocom (One) reports the raw value.

Converter

I've put together a small python script which is attached.

It's is a bit less basic than previous version, and has only been tested on macOS 10.13 with Python 2 and 3. You'll need to remove the txt extension.

The script is run from command line, with the path to the file to be converted as an argument.

python fuconvert.py file.fu1  

The converted file is saved as filename.csv in the current working directory. The .csv contains the full data from the log saved in the same order the requests appear in the .fu1 file. This is different to the Nanocom CSV order. The converter alters temperatures from Kelvin to Celsius, and adjusts decimal points to match the the Nanocom, with the exception of voltages which are left as millivolts.

Note that the AAT and TPS3 voltages will read 5000mV if the input is not connected.

Desktop diagnostics...

Posted: 
Tuesday, January 24, 2017 - 08:00

Garmen has posted up a Python script which at the moment allows the user to read out the "Nanocom Fuelling" parameters. github

The project uses a cheap serial programming interface based on the CP2102 USB to TTL converter plus some external circuitry to interface with comms levels. The trick is that the CP2012 needs to be reconfigured to suit the "non-standard" baud rates, and to allow sending a 0x00 byte with a period of exactly 25ms. It's a fairly neat trick.

Anyway Garmen's work prompted me to pull out a "dumb" OBD-II interface that I'd picked up a couple of years ago and thrown in the cupboard. Most OBD-II interfaces are based on the ELM-327 chip which basically does all the grunt work of managing the diagnostic connection and allows users to simply send OBD commands over a serial connection and read back the response.

A "dumb" interface is a USB to serial converter chip (usually a variant or clone of the FTDI FT232) and hardware to interface with the diagnostic port line levels. These are usually sold as something like "VAG-COM KKL USB Interface" - just google it... These require that the user application to handle all of the communications protocol details.

The interface I have has a cloned FT232BL. The complete lack of markings on the chip is the give away. But as the part cost $14.00AUS including postage it's hard to feel ripped off...

The nice thing about the FTDI chips is that they have a "Bit Bang" mode. This allows you to write a byte to the chip and it will hold that state until you write another byte. It's perfect for doing things like sending the 25ms wake-up message or bit-banging the 5 Baud initialisation sequence.

There are a couple of Python modules that allow access to these FTDI specific features. I'm using pyftdi.

As a rough example of how you do it...

#!/opt/local/bin/python3

from pyftdi.ftdi import Ftdi
import time

HIGH = bytearray([0x01])
LOW = bytearray([0x00])
message = bytearray([0x81,0x13,0xF7,0x81])


uart = Ftdi()

uart.open(0x403, 0x6001)
uart.set_baudrate(10400)
uart.set_line_property(8, 1, 'N')

# Enable Bitbang Mode
uart.set_bitmode(0x01, 0x01)

# Set TX line high
uart.write_data(HIGH)


# Short sleeps in the loop seem to improve accuracy...
# Using time.sleep(.025) for the full delay is very jittery.
# time.perf_counter() can be used instead time.monotonic()
# requires Python 3.3 or later

start = time.monotonic()
while (time.monotonic() <= start + 0.35):
    time.sleep(0.01)

# Set TX line low
uart.write_data(LOW)

start = time.monotonic()
while (time.monotonic() <= start + 0.0245):
   time.sleep(0.00025)

# Set TX line high
uart.write_data(HIGH)

start = time.monotonic()
while (time.monotonic() <= start + 0.0245):
     time.sleep(0.00025)

# Disable Bitbang Mode
uart.set_bitmode(0x00, 0x00)

# Send serial message
uart.write_data(message)

Which should give you something like:

Wakeup Sequence

The ISO14230 spec states the total time for the wakeup time (Twup) should be 50ms +/- 1ms. This code gives Twup of 50.67ms, so is within spec.
As reference the Nanocom sets the line low for 26.54ms and then idles high for 118.9ms giving a Twup = 145.44ms. The Td5 ECU is pretty forgiving.

Td5 KeyGen now on github

Posted: 
Tuesday, January 10, 2017 - 15:45

As a copy of the Seed-Key.txt database has recently turned up in a post on the AULRO forum I've decided there isn't much point in keeping my keygen code under wraps.

Td5 Keygen Github repo

I started doing some testing against the Seed-Key database and came up with some anomalies. As a result I've double checked the code against the ECU assembly code, and I also wrote ended up writing a version in MATLAB to verify what I had was ok. When I ran the MATLAB version against the Seed list I discovered that there are about 308 questionable keys in the database. These are mainly blocks of either 0x2020 or 0xF781 keys which are clearly fillers. This is an example...

SEED_KEY.txt:

F0DD 2020
F0DE F781
F0DF F781
F0E0 F781
F0E1 F781
F0E2 F781
F0E3 F781
F0E4 F781
F0E5 2020

KEY_GEN:

F0DD 7D51
F0DE F9A1
F0DF FCD1
F0E0 2607
F0E1 9303
F0E2 2A0F
F0E3 9506
F0E4 321E
F0E5 990E

There are also two additional values which I believe are wrong.

The Keygen generated values match the remaining 65228 "good" keys.

I'm quietly confident the code is OK, but someone has a 0.47% error rate.

Datalogger updates

Posted: 
Sunday, April 24, 2016 - 14:45

After a week or two distracted by the OLED graphic display - which turned out to be damaged in the end - I've refocused on the core logger component.

I'm now using a simple LED which changes state every time a new line is written to the log buffer. Currently this occurs at slightly less than 0.5 second intervals so it's very obvious when logging stops.

The "refocusing" has meant I've found and squashed a couple of bugs that were causing intermittent issues with the log files. I've also discovered that the road speed data from the ECU is intermittently transmitted with an incorrect checksum. Until I found this issue I had assumed that I was logging too fast. Now that I've added handling for incorrect checksums the diagnostic connection will keep running after "ignition off" until the ECU shuts down.

With a bunch of bugs fixed the logger had a short road test this afternoon. In previous tests the connection would drop after 3 minutes maximum. This time around I was able to take the D2 for a 15 minute drive including a short run on a nearby freeway without out a hitch. I did have a moment of fuel cut as I accelerated on to the freeway and the logs show clearly the MAF sensor voltage climbing above the 4950mV upper limit, and the MAF reading dropping to 0.

A second run to work, to pick up a few things gave another opportunity to test the stability of the logger. The round trip was just under 30 minutes and I found I was able turn the motor off, switch back to accessory position, grab the things I needed from the office, then restart the engine and drive home without dropping the connection. The Nanocom is lucky to do the trip one way without throwing in the towel.

This is a quick comparison of the "Airmass per cycle" figures the ECU calculates from the MAF and MAP done using the MegaLog Viewer scatter plot function.

Live data, at last.

Posted: 
Wednesday, April 6, 2016 - 19:45

The logging to sd card is proving to be a bit of a case of one step forward, two steps back.

Most of the the difficulty has resulted from the decision to make the logging "user" configurable.

Rather than the fixed set of logged parameters that Td5 owners are familiar with from the Nanocom, one of the goals is to allow granular parameter logging . Not all data is available as a single request, but for those that are it's possible to log at more than 10 times per second giving very detailed snapshot of sensor performance.

The screen grab below shows some of the data captured on a short 4 minute run around the block. The top graph is Fuel Temp in C and FT sensor voltage. The lower graph is Engine Coolant Temp and ECT sensor voltage. These are being sampled at slightly more than 4 times per second vs the Nanocom sample time of once every 1.25 seconds.

The log viewer app I'm using is MegaLogViewer HD . The big plus of this over the free Nanocom log viewers is that it's very flexible and allows user defined calculations based on the log data.

Td5 Datalogger Reboot

Posted: 
Monday, March 28, 2016 - 18:00

After a long hiatus I've been working on getting the Td5 Data Logger (aka Raijin Project) up and running.

The low level connection management code has been rewritten so it's a lot more robust. One of the big improvements is that it now uses the 5ms minimum inter byte timing specified by ISO14230 - in practice it's actually between 4.25 and 4.75ms. My theory is this timing allows the ECU more time to handle core processes so should help prevent dropped communications sessions under load. It's still missing error handling and recovery but otherwise working well.

The point to point wired board I've been using for development was a bit impractical for in-car testing, so I've had a piggy back board made up which should make it slightly easier to work on. The board is designed to plug on to a Fubarino SD development board, and has a ISO9141 interface chip, k-type thermocouple interface, 5V voltage regulator running off the OBDII port supply, a 5 way tact switch, and a connector for a small 0.96" OLED. The whole thing is about 25mm by 75mm.

I got notification today that the test boards have finally been shipped, a week later than the original estimated shipping date. I'll be interested to see how they turn out as this was the first time I've used Eagle to design a board and the first time I've had one of my own designs fabricated.

Td5 Datalogger

Posted: 
Tuesday, July 15, 2014 - 17:30

While the Nanocom is a pretty useful diagnostic tool it leaves a lot to be desired in terms of datalogging, so I've been messing around with the idea of building a dedicated datalogger for the Td5.

The time I've put into reserve engieering the ECU code has been very useful in this regard. I have been able to identify the routines used to generate and verify the security keys - done using linear feedback shift registers - which are required to gain access to many of the diagnostic functions of the Td5 ECU. 

At this point I've been able to sucessfully negotiate a connection with the ECU and request "live" data. On my test ECU running on the bench I'm able to request a single parameter every 20ms. Unfortunately this seems to take a bit of a hit when the ECU is actually hooked up to a running engine, so this appears to best possible under ideal conditions.  Nonetheless it appears that it should be possible to log full engine data at around 3-5 times per second, which is substainally better than the Nanocom's once every 1.1 seconds. By prioritising requests so that slow changing data is requested less frequently it should be possible to log critical parameters 10 times per second or faster.

I've also verified that it is possible to request specific data from the ECU providing you where in memory the data is stored. This opens up the possibility of logging the values returned for ECU map lookups.  As an example you could log throttle amount, aircharge and the result of the smoke map lookup.

It's still early days yet and I'm currently working to make the code I have already written more robust.

The current wish list also includes an  interface for a EGT probe, and ability to connect of other sensors that output 0-5V.

I've attached a screen capture showing a capture of "Raijin" ( the shinto god of storms and thunder) sending a key response and the ECU responding with a security access OK message.  Top line is the transmit line and the bottom is the recive line. The ISO interface chip (ST-Micro L9637) I'm using echoes the transmitted data back on the rx line hence the duplication of the TX packets.

 

Security access granted response from ECU