Telemetry Database

On one of my previous for-work projects, I was involved in flight software for an instrument bound for low-earth orbit. I am Inordinately Proud of that, because the instrument is still working, in year eight of its two-year mission. The instrument has never crashed. However, that's not what I am talking about today.

One of the things that project used was a command/telemetry database. In the spirit of One Authoritative Source, that database was used to generate PDF documentation for humans (which was then printed out and put into a big three-ring binder for us to take notes on) but also for the ground system to create command packets and interpret telemetry packets. Further, the database was used to generate some of the flight software code involved in interpreting commands and generating telemetry. Mostly in the flight software it was used to make structs, which the flight software used to parse commands and form telemetry. The code had to use the right struct names, but the actual definition of the struct came from the cmd/tlm db.

I have wanted to implement this in Yukari for a long time now, and I have finally done it. The TelemetryDatabase.ods file is the authoritative source for all packet creation and interpretation. This table contains one row for each packet field, grouped into contiguous blocks describing each packet, and eleven columns:

1. Apid - Application Process ID, the CCSDS term for "packet ID". Each different kind of packet has a different apid, and all packets with the same apid have the same structure.
2. shortName - Name of the packet, must be a valid C++ identifier
3. wrapRobot - set to 'y' if we want to automatically generate code to start and end the packet. This is typically set to 'n' when the header is generated in one routine, most commonly in main.cpp, and the body is generated somewhere else, like a sensor routine.
4. extract - set to 'y' if you want to extract this kind of packet with extractPacket
5. hasTC - C++ expression, which when evaluated in the embedded code in the right context, produces the timestamp to use on this packet. If the packet doesn't require a timestamp, this field is blank
6. extractor - One of "csv", "source", "dump". Used to control how the extractPacket ground support program creates extracted packet tables.
7. source - C++ expression, which when evaluated in the embedded code in the right context, produces the value for this field
8. field - field name to the outside world
9. type - C++ field type. If the field is an array (char[] is most typical) include the array size right after the 
10. unit - Unit of this field, typically some SI unit for floating-point values, or DN or TC ticks for integer values
11. description in English

Columns 1 through 6 only need to be set for the first field in a packet.  A blank apid is treated as having the same apid (and therefore being in the same packet) as the previous field. Fields in a packet must be contiguous, apids must be unique for packets, and order of fields counts, but order of apids doesn't.

This table is converted from ODS format to CSV using ssconvert, a program which comes with Gnumeric. It is then run through a perl script tlmDb.pl, which generates the following things:

• A bunch of include files of the form write_packet_$shortName.inc which contain C++ code to write the packets in the embedded code.
• extract_head.inc  - This file contains all the packets as C++ structs. This file is used in the head of extractPacket.cpp, which is why it is called extract_head.inc
• extract_body.inc - This file contains a series of blocks of code. Each one starts an appropriate dump file if necessary, reverses each field from (CCSDS-mandated) big-endian to little-endian as necessary, then dumps the packet in the dump file. This is where the extractor field above comes into play. CSV dumps the packet into a CSV file, with all fields printed as ASCII decimal. "Dump" is used for packets with a variable-length payload, like source images and NMEA sentences. These packets have their last field just concatenated to the dump file.

The table itself is useful as a human-readable reference for the packets. Since it is the One Authoritative Source, any change here is immediately reflected in the code. 

This is worked into the makefiles, particularly into the makefile for the packet library. Since the output of tlmDb.pl depends on the table, we treat extract_head.inc as the primary output (it will always be present). We make it depend on TelemetryDatabase.csv , which in turn depends on TelemetryDatabase.ods (we keep the telemetry database in spreadsheet form to take advantage of formulas). All C++ files which create packets then use #include "write_packet_$shortName.inc" in the proper context, and depend on extract_head.inc even if that file isn't used directly.

I spend some time scouring the Internet for a make-friendly program which could convert an ODS spreadsheet to CSV. I finally found ssconvert after trying to get soffice --convert-to to work without success. I also tried the program unoconv, but this appears to be a wrapper around soffice.