A better C++ than C++?

tl;dr -- Is Rust the language that C++ promised but failed to deliver? It's still unclear.

Programmers are adults. We don't need our hands held, and we certainly don't need to be told "Don't do that!" The only kind of advice that is needed is "I see what you are *trying* to do. This might be a better way to do it..."

Case in point: reading data from external sensors. I am still working on the rollercoasterometer (now for 13 glorious years!) and am doing my usual fighting with C++ about getting formatted data out of a byte buffer. Back in the bad old days, we would cast the address of the buffer as a pointer to a struct, and read the data out directly. Then the compiler writers, with their fancy-smancy alignment and such, said to not do that, because there is no guarantee that the structure will line up with what you think. The compiler is free to put any amount of padding between fields, order the fields however it likes, put invisible fields like vptr, etc. 

That by itself isn't bad. The bad part is when I ask how to do what I want, and the answer comes back: "Don't". There is no portable way to guarantee that any field in a struct lands anywhere. This way we can write C++ targeting the Apollo Guidance Computer, with its 15-bit words, no such concept as "byte", ones-complement arithmetic with +0 and -0, etc. It doesn't matter that basically every machine in the last 40 years has been twos complement, 8-bit bytes, and word size of a power of 2 bytes. Basically the only disagreement is endian-ness. 

But, C++ won't let me take advantage of the fact that both machines in a transaction have the same native word format. No, I have to individually extract each byte, shift and OR it myself, etc, to get the data out. If I am lucky, the compiler will see that I am translating from English to English, and optimize it out.

Game designers learned long ago to learn from their users. If all the Minecraft users are building farms, then support the building of farms. If the farm depends on a glitch, consider formalizing the glitch and making it an official feature. Don't just shower them with whatever they are farming for "free", but don't take away their ability to farm either. For instance, Mojang has considered in several instances changing the mechanics of how villagers and iron golems work to discourage farming iron. They got quite a bit of pushback from the community, and have therefore backed off. They don't "support" iron farming, but they haven't removed it either.

The C++ committee on the other hand seems to be driven by two factors:

1. Backward compatibility indefinitely into the past
2. Ability of compiler writers to game benchmarks

C++ has a lot of good ideas (some might say too many) but it is a language which has evolved for decades while at the same time hasn't been able to shed old, bad ideas or old, bad implementations of good ideas. 

C++ for whatever reason also has a burning hatred for the preprocessor. The preprocessor and compiler are married, but they are both trapped in a loveless marriage, and the simmering hatred has boiled over to the point where the official C++ FAQ considers macros to be "evil". Now the preprocessor does have some minuses, mainly in type checking. So, we were given constants and templates. We were told that templates would basically eliminate the need for macros. However, when we tried to use them like that, we found that the promise has not quite been kept. Templates do some but not all of the compile-time processing that we want. Constexpr functions are helping, but aren't there yet. 

My use case is that I want to make a self-documenting logger. The rollercoasterometer reads a bunch of sensors, formats the data into packets, then writes the packets to a file on the SD card. Since the data from the sensors doesn't naturally come in packets, and doesn't naturally have a timestamp, the main firmware timestamps the data and formats it into packets. Since I write the firmware, I also have to write the code on the host which interprets the packets. I came up with a [[clever idea]] to have the program emit a series of packet-documentation packets as it starts up. One way to do it is to have the program create a documentation packet the first time it emits each packet, and I have done this. It looks something like this:

void start_packet(apid, packet_desc_str) {

  if apid is not yet documented:

     write a documentation packet for this packet, using the packet_desc_str

  start the packet, perhaps to a backup buffer if we are documenting the packet

}

void fill<type>(data_value, field_desc_str) {

  if apid is not yet documented:

    write a documentation packer for this field, using the field_desc_str

  write the field to the packet, perhaps to the backup buffer

}

void finish(apid) {

  if the apid is being documented:

    take note that we have documented it and don't do it next time

  finish the packet and write it, perhaps from the backup buffer

}

Note that we need to have two buffers. It would be far better if we did something like this:

void start_packet(apid,packet_desc_str) {

  compile_time_code {

    create a packet describing this packet in the packet description block. This block will be a read-only blob as far as the runtime code is concerned

  }

  start the packet, no need for backup buffer

}

void fill<type>(data_value,field_desc_str) {

  compile_time_code {

    Add a packet describing this field to the packet description block

  }

  add the field to the packet

}

void finish() {

  finish the packet and write it

}

void setup() {

   open sd card

   write packet description block

   set up sensors etc

}

void loop() {

   read sensor

   start_packet(0x1234,"sensor packet");

   fillu16(value_from_sensor,"value from sensor")

   ...

   finish_packet()

}

Preprocessor macros might be able to do it, but the preprocessor is Evil. Templates might be able to do it, but it might require template metaprogramming, which is actually evil. It seems like there isn't a way to do it in C++, certainly not a clean way. Therefore we are forced to either use the official preprocessor, write our own preprocessor (which has its own headaches), or do it at runtime.

The language which might be a better C++ than C++ is Rust. This is a statically typed language which is designed to be compiled into good machine code (the reference compiler uses LLVM as its back end) but with some features added and some taken away. I'm not sure if I like mutability and ownership yet, and haven't gotten used to the concept of borrowing yet, but it does look like there is support for forcing a struct to land on certain bytes, and it does look like (with procedural macros) there might be enough compiler support for compile-time computation.

To test this out, I am going to work on three projects:

1. A conversion of kwantrace to Rust, to experiment with plain application-domain programming.
2. A packet parser for reading rollercoasterometer logs
3. Firmware for the rollercoasterometer

The last one is probably not going to be ready in time for my next expedition to a roller coaster.

Check PCLK measurement

Check if the user code measures PCLK properly. If it doesn't, then the baud rate calculation will be wrong. Since one of the symptoms that has been seen is that the RX light on the FT232 doodad flickers, but no characters appear in putty, it is possible that the baud rate isn't what we think it is.

Consider also calculating the baud rate registers and stuffing them manually. If this works, then it's the PCLK stuff that is broken.

Cortex M4 FPU

For a while I was having trouble getting my part to print any FPU calculations. Finally it occurred to me that maybe the FPU has to be turned on, and that the ISP wasn't doing it since it didn't use it.

It turns out that you DO need to turn on the FPU:

4.6.6 Enabling the FPU
The FPU is disabled from reset. You must enable it before you can use any floating-point instructions. Example 4-1shows an example code sequence for enabling the FPU in both privileged and user modes. The processor must be in privileged mode to read from and write to the CPACR.

Example 4-1 Enabling the FPU

; CPACR is located at address 0xE000ED88
LDR.W R0, =0xE000ED88
; Read CPACR
LDR R1, [R0]
; Set bits 20-23 to enable CP10 and CP11 coprocessors
ORR R1, R1, #(0xF << 20)
; Write back the modified value to the CPACR
STR R1, [R0]; wait for store to complete
DSB
;reset pipeline now the FPU is enabled
ISB

In effect, the FPU is counted as coprocessor 10 and 11. Cortex-M doesn't fully support the concept of coprocessors, but it does in this context. We allow full unpriveleged access to coprocessors 10 and 11.

I don't know about waiting for the store to complete and resetting the pipeline. I just put some C++ code to do this long before the FPU is used, and let a bunch of other work instructions flush the pipeline.

Getting on board the LPC4078

A few things make the LPC4078 dramatically different from the LPC2148 that I am used to:

1. This is a Cortex-M4, with a much different interrupt and reset vector table. Instead of a set of ldr pc,[pc,#24] instructions followed by addresses 24 bytes later, we have just a table of addresses. The first one is the value to put into the stack pointer upon reset (so there is no required stack setup code) and the second one is the value to put in PC on reset. Subsequent values include exception and interrupt handler addresses.
2. As with the 2148, there is a bootstrap program. On reset, the bootstrap vector table is mapped to 0x00000000, rather than the more obvious solution of having the reset value of the vector table address register point at the bootstrap.

One of the ugly things about my old LPC2148 is that much of it was Not Invented Here. Since this is a hobby project, I can use as much of my time to do whatever I feel like, including reinvent wheels! So, I get to figure out how to start up a Cortex-M4 from scratch.

I'm not there yet.

I haven't gotten my code to run directly yet, so there is still a problem with my vector table. But, I can get my code to run with the help of the ISP, after some experimentation.

The ISP maps a total of 512 (0x200) bytes of memory as its vector table. This is room for 128 vectors, while the actual number of vectors it uses is only 7. I can tell because the reset vector points at what would be in slot 7. In any case, this code is mapped to address 0 at startup. When I was using the ISP to launch my code, I originally had only exactly enough memory reserved to hold my table, and my code started immediately after, at address 0xE4 as it happened. Well, that code was covered up by the bootstrap. After rearranging my program so that a full 512 bytes was allocated for the table, things worked much better.

Also, the ISP uses an autobaud feature when it sets up the serial port. You send a question mark, and it times the bits on that to set up thebaud rate registers. However, the ISP uses a feature I don't, and that is the fractional baud rate register. This is able to tune the baud rate at a relatively fine grain to between 1x and 2x the rate called for by the coarse baud rate registers. When the ISP is used to kick off my code, my code sets the coarse baud rate, but didn't touch the fractional baud rate register, which was left at about 1.5 . Therefore the part was programmed to talk at the wrong rate by 50%, out of tolerance for the serial port. The FT232 could tell that the part was talking, but couldn't understand any of it.

With those two things taken care of, I can now start my code with the ISP, from the very beginning of my code. Next step is to see what I am doing wrong such that my code doesn't start itself.

LPC4078 operational

I am continuing my project of doing robot stuff with a buddget of zero, plus the stuff on my bench. Well, the stuff on my bench includes a Loginator2368 purple board, and a box of LPC4078 Cortex-M4 microcontrollers. Those controllers happen to be pin-compatible with the LPC2368 the board was designed for. In this case that means that all the power pins have the same jobs and voltage levels (even if they have different names and perhaps different internal connections), all the special pins like reset are in the same place, and all the GPIO pins have the same numbers, and where the 2368 and 4078 have the same peripherals, they have compatible pin assignments.

One thing that the 4078 has that the 2148 doesn't is internal pullup resistors. I can take advantage of these to reduce the part count in several places.

Therefore I got out my soldering iron and finally attached the board and LPC4078. I was trying to figure out what was the minimum amount of components I could get away with, since I can't find my solder paste stencil for this board, and would have to manually solder everything. I decided to skip the power section, so no external battery or regulator. I skipped the USB section, the LEDs, and the voltage reference. I also skipped the crystal, since the part has an internal RC oscillator which is good enough for now. I even skipped all of the bypass caps.

The only part that it looked like I absolutely needed was the reset button, since my first read on the datasheet didn't show the reset pin as having a pullup. It turns out that it does, so, I didn't even need that. I did end up stealing a push button from another board and using it as a reset switch.

I did the normal SMD IC soldering thing. I carefully lined up the part on the board, then used a normal soldering iron and normal solder to glob all the pins on each side down, and to each other. I then used wick to clean up all the bridges. This naturally leaves the connections to the board intact. I then soldered on the reset button, more as a convenience than anything. (Next time, leave a reset terminal on the edge!) Also solder a shield 6-pin top-and-bottom connector onto the power and serial connections.

Then, hook it up to power. No smoke, the chip isn't hot, it passes the smoke test. But, I've seen this board not work before, with the LPC2368 that it was designed for. So, the real test is running the ISP.

?
Synchronized
Synchronized
OK
12000
OK 

It works!!! I had to guess at the frequency it wants, since I don't have an oscillator attached at all. The internal RC oscillator runs at 12MHz, so I put that in.

Now to massively restructure the code. A Cortex-M4 is quite a different beast than an ARM7TDMI. It has an FPU and built-in VIC instead of the VIC as a peripheral. It also doesn't use 32-bit ARM instructions, but (mostly) 16-bit Thumb2 instructions. The 4078 is also a much different part than the 2148, with a different set of registers in different places. But, probably 90% of the code can be in common, if we carefully separate the code into the part which is different and the part which is common. That's the next big adventure. Throw one switch in the Makefile and compile for a 4078 instead of 2148.

Practical CRTP (or lack thereof)

I have recently performed an experiment where I took the Loginator code and removed all dynamic polymorphism in place of static polymorphism. The motivation was the fact that since the sensors were all hard-wired to the board, all references to them could be figured at compile-time, and static polymorphism plus the optimizer and inliner could make "better" code. I also had an irrational fear of the vtables being corrupted.

Having gone through this exercise, I think I can say that CRTP is not the right way to do static polymorphism. It works, but it is ugly and infectious. If a class uses an object which is defined with CRTP, that class itself must also be made to use CRTP. I don't know if the machine code produced is faster, but I do know that it is significantly larger.

The motivation for changing back is the hardware description block which I am adding to the Loginator code, in hopes of being able to unify the Logomatic, Loginator, Rocketometer, Rollercoasterometer, and Pocketometer. My idea is to be able to load the same binary image into any LPC2148 based hardware I have or make, and have the software read the hardware description block to see what hardware it is attached to. This is all done at runtime, so we need dynamic polymorphism again.

CRTP isn't all bad. It works fine for instance in the dump class, where you can use it to make specializations of dump which are resolved at compile time. The problem arises when you want to pass a CRTP object as a parameter. At that point, the reference or pointer to the object has to be CRTP also, which frequently causes the whole class to need to be CRTP.

The idea then is that the general-purpose drivers are built into the code, and constructed at runtime from the hardware description block.

Gem of the Once-In-A-While: Ragel FSM generator

Yukari 3 ran a hand-built state machine to parse NMEA sentences coming from the GPS. It worked, but it was hard to maintain. I went looking for a tool that was a better solution. I was looking for something like a C++ template library that implements a state machine by abusing the template processor. What I found was Ragel.

This a nice, simple, clean program for generating finite state machines and weaving them into your programs. It does so in a way which I vastly prefer over Lex. It's main drawback is that since it is not a template library, it does require an external program to handle. Well, so do several other parts of my system. I use Perl without embarassment, so I will use Ragel similarly.

Ragel is structured very similarly to Yacc, in that it creates a program which executes user-chosen actions when a certain sentence form is encountered. However, it only handles part 1 languages, IE regular languages.

Ragel takes as input a source file written primarily in C/C++ (other languages are supported, but I don't use them) but which is marked with certain Ragel blocks. These blocks are used to define the machine and actions. The actions are written in the main language (C/C++ as I use it) and Ragel builds code which executes the state machine on an input stream and executes your actions as they come up.

It demonstrates a couple of interesting ideas, one of which is that you can just make up a new language and mix it into your existing source code.

Certain of the Ragel commands indicate where the state machine's tables should be written, where the state machine code should go, etc. Interfacing to the scanner is pretty simple.

A couple of examples below the fold:

Worse is Better

The thesis "Worse is Better" states that Simplicity of implementation is the overriding design goal of quality software, to the expense of everything else, even Correctness. I don't know if I agree with it in all cases, and I'm not even sure the author agrees with it in all cases. Be that as it may, I am applying this principle to Project Yukari. The main result of this is that if it is simpler (by which I mean takes less time to implement) then do it that way. If I run into trouble using some advanced language or hardware feature, I will see if I can get around it.

Will it be elegant? No.
Will it be extensible? No.
Will it be an example of how to code, a work of art? No.
Will it work? That is the goal above all else. If the robot navigates the course in three weeks, that is Mission Accomplished, nothing else matters.

• The hardware has a great USB interface. I know that the part is capable of simultaneously acting as a Mass Storage Class and a serial port. However, I haven't learned how to code it. I can't use the Sparkfun bootloader since it doesn't work with SDHC cards. I can't use my upgraded Bootloader_SDHC since it is too slow. Therefore I don't use the USB at all, except for power. I load the software with the LPC2148 monitor. This means that since the PWM is using serial port 1, the GPS is forced to use the same port 0 as the bootloader.
• The GPS Rx line doesn't seem to be working on the part that I have. I know it has worked in the past, but I can't get it to work now. As a result, I will let the part speak its native NMEA 4800 baud, 1Hz update.
• The Rocketometer used a timer-interrupt driven task to read the sensors at a precise given cadence. Yukari will instead read the sensors at whatever rate it can, record the timestamp at which it did read, then go with that. 
• I am having trouble with reading the serial port, and I suspect it has to do with the interrupt-driven nature. Somehow I am not acknowledging an interrupt, and as a consequence no new interrupts are being generated and no new data is being processed. I will make a new NoIntSerial class, as a replacement for HardwareSerial, which will have blocking write and will use the Rx FIFO as its one and only buffer. This will be fine, as long as the main loop is called sufficiently often. In this case the low bitrate from the first item above works to our advantage. There are only 480 characters per second, about one every 2ms. If we run the main loop at least once every 32ms, we won't drop any serial data. This is only 30Hz.
• There will only be a major loop, with no interrupts. In order, in the main loop:

1. The inertial sensors are read
2. An inertial packet is written to the card
3. Every 10th time, the magnetic sensors are read, and a magnetic packet is written to the card
4. The serial port is checked, and while there is a byte present, the port is read and parsed as NMEA. If there is a complete sentence, update the 2D state and GPS heading.
5. The heading Kalman filter is run based on the inertial sensors and GPS heading, if new. This is the navigation portion of Navigate, Guide, Control.
6. The waypoints are consulted and updated as necessary, and the heading to the correct waypoint is caluclated. This is the Guide portion of Navigate, Guide, Control.
7. The difference between the navigation heading and guidance heading is used to calculate the steering. This is the control part of Navigate, Guide, Control.
8. The control value is written to the steering PWM.

Bugs crossing Boundaries

I am getting the Rocketometer ready for delivery. The hardware appears to be fine, but the software had one particularly weird bug -- the name of the FAT volume on the SD card was changing. This meant that something was poking into the root directory block where it did not belong, which meant that it could be anything.

As it turned out, it was the packet buffer becoming full. Now how in the world did that corrupt the file system buffers? As it turned out, the file system by design wasn't using buffers, in order to save memory.

An SD card is addressed a sector at a time. Data can only be read and written in full sectors. So, if you want to change a single entry in the root directory or file allocation table, you have to read the whole sector into memory, change the bits you want, then write the whole sector back. Now the root directory and file allocation table only needs to be changed when the file is written, and in this case there is already a buffer which has just been used, so I foolishly decided to have the code use this.

In the mean time, this same buffer was part of the circular buffer accumulating packets. If the buffer ever filled up, new bytes would be rejected, and the head pointer would not move. If you start to write a packet to an already full buffer, nothing gets written, and when you finish, the packet length seems to be zero. Now the fun begins.

Upon finishing a CCSDS packet, the code counts how many bytes have been written, subtracts 7 because that's according to the protocol, then goes back and changes the buffer. In the current case, since no data had been written, it actually changed marked data.

Now we combine this with the fact that the interrupt task was filling the buffer. Here then is the final sequence:

1. The buffer is full. No more bytes can be written.
2. The main loop detects this, and starts to drain the buffer and write it to the SD card. It writes the packet, then reads the root directory sector, changes it, then writes it back.
3. During the write, an interrupt fires.
4. The interrupt task writes another packet. The bytes are dropped on the floor.
5. The interrupt task finishes the packet. The calculated length is zero, since no new bytes were written
6. The length to be written is -7, 0xFFF9
7. The length is written to the buffer, but because of timing, it writes it over the top of the root directory sector in memory. The root directory is now corrupted.
8. The interrupt task finishes
9. The SD writing code writes the corrupted root directory sector back to the card.
10. The card is now named RKT\FF\F9 instead of RKTO3.

It would be pretty easy for this same process to shoot holes anywhere in the filesystem metadata. This bug is hazardous!

So, to fix the bug, we do the following:

1. Give the file system its own scratch buffer, and never let it do scratch work with the user buffer. This doesn't actually fix anything, but it does make the bug, and any similar undiscovered bug, less dangerous.
2. Handle the full buffer case better. When the buffer is full, it throws away the new incoming bytes. Further, it backs the head pointer to the mid pointer, in effect throwing away the fractional packet accumulated before the buffer became full. Even further, since the buffer is now no longer full, to prevent getting the last half of a packet, we keep a flag that says that the buffer was full, and don't accept any more bytes until the flag is cleared by a drain operation.
3. The interrupt task checks if the buffer is already full. If it is, we don't bother to read the sensors, a time-consuming chore which will just discard the result and take time away from the SD writing routines.

This clears a major bug, perhaps one causing all the observed halts. A bug which punches random holes in the metadata can do anything. It only triggered recently when I cranked the data recording rate to 200Hz, which appears to be just beyond the equipment capability. Previous rocket flights must have run at lower rate and not triggered the bug, or not triggered it as hard.

So how did I find it? Lots of print statements and blinklocks. First, I had the SD driver print out the buffer it was going to write. Then I had the direntry driver (the thing which actually handles the root directory) print out its scratch buffer after it had been read, as it changed each byte in turn, and before it wrote it out. From that I found out that the buffer was changing in between lines. That meant that an interrupt task was doing it. Further puzzling out and seeing 0xFFF9, then counting on my fingers to see what number that is in signed decimal, revealed that it was -7, a key number in calculating CCSDS packet sizes. However, it wasn't writing bytes 4 and 5, where it would be if the packet started at byte 0 in the buffer. Instead it wrote one byte before that, implying that the phantom packet started at -1, a key number in the circular buffer pointers. Once the head pointer is one less than the tail pointer, the buffer is full.

For a while I thought that the code was starting a new packet before it finished an old one, which would result in weird numbers. So, I put in some code which blinklocked the device if a packet had not been finished before the next one was started. That never hit. So, I put in some code which blinklocked when the circular buffer filled up, and that finally hit and proved it all. This interacted weirdly with the overlapping packet checker, so I had to fix a bug there. 

Now the only remaining known weakness is what we do if the first cluster of the root directory is full. We can make a cluster as big as 64kiB, we can handle multiple sectors in a cluster in the root directory, and we can fit 2ki entries into a cluster that big. But, what if we fill it? The file search code will need to know how to go to the next cluster, and the file creation code will need to know how to allocate a new cluster. Or, the format process can write 10000 files then delete them all.

The flight version of the code will not blinklock, but reset, when blinklock() is called. Between that and gathering enough runtime, I can gain some confidence that the device won't lock up.

I set the circular buffer size back to two sectors, down from the 6 I had given it. Next version will put the circular buffer into USB ram to save space for the stack, and allow me to have up to 16 sectors. USB memory doesn't work for some reason... maybe the USB system needs to be on for it? Then we are exchanging memory for power consumption. I know that the system is dropping packets now, more so than it was at 6 sectors.

Yet another Rocketometer

The Arduino Leonardo can effectively be thought of as an Arduino on a chip, except that an Arduino pretty much already is one chip. What makes a Leonardo special is that its core is an ATMega32U4, which has a USB port. The chip can be programmed as normal over ISP, and once it is programmed with the appropriate Arduino bootloader, it can be programmed over the USB port. The Arduino library for Leonardo also supports using the USB as a CDC serial port, and has much other USB support.

The chip only runs at 8MHz. It can run at 16MHz at 5V, but I am planning on running it in the Rocketometer, which only has 3.3V available, which limits the chip to the aforementioned 8MHz.

Now how do I think I can get away with this? Simple: Even with the LPC2148 running at 60MHz, it was spending most of its time waiting for the bits to crawl across the I2C and SPI buses. And how much can we optimize a busy wait loop?

I bought a Leonardo to practice with, and a Sparkfun Pro Micro to practice some more, especially programming the bootloader. I will not be using the 6-pin ISP header on the Rocketometer, but I got the great idea to use the MicroSD sniffer to get at the SPI signals through the Rocketometer MicroSD slot, and I can just hold down the reset button, or solder one wire to the contact of the Reset switch - which reminds me, I should put the solder bridge on the top side of the reset switch. In that case, I can use the solder bridge itself as a temporary terminal, solder a wire to it long enough to program, connect the rest of the SPI signals to the microSD sniffer, then program it with an ArduinoISP.

While I was at it, I also designed a Precision clock around this new chip. It doesn't need a hardware UART to program or talk to the host computer, so no FT232 and no multiplexer is needed. The chip has more pins, so this design doesn't use literally every single possible pin (those two ADC-only pins don't count, since they are not possible to use in this project).

DirectTask and nested interrupts

One problem with the old rocketometer code is that the sensor readout and SD card writing code were in the same thread, meaning that when the SD card took its endless milliseconds to write the data, the sensors were not being read, leaving an irregular gap in the record. My brilliant idea was to read the sensors in a task at interrupt priority, effectively creating another thread. First effort was with the task manager I described below, which was a dismal failure.

For whatever reason, and perhaps the same reason (see below), the task was not able to read the sensors. I came up with a much simpler task manager with which I am getting incredible accuracy.

I call it the DirectTask manager. Its concept: Rather than using one match channel of the timer and a heap priority queue, we just use all three available match channels (there are four, but the zeroth channel resets the timer every 1 minute). This limits the flexibility enormously, but I only need two tasks. I set up one task to reschedule itself on a regular basis (5ms in my first test) and I use the other task to read the BMP sensor.

However, the sensor readout runs an I2C read, which itself is interrupt driven. The code does not currently support nested interrupts, which means all interrupts are delayed until the current one returns. The I2C state machine was interrupt drive, and its interrupts were getting delayed, including the one which makes the state machine stop waiting forever, so the state machine waited forever.

So, we put in an option to I2C to run without interrupts, instead using a busy loop to check the I2C interrupt status bit, then calling the same state machine driver. We weren't getting anything by being interrupt driven anyway, since we had to wait for the read to finish.

With that, the DirectTask manager worked fine. Maybe the heap task manager would have worked fine too, but this one is simpler.

It takes 641 microseconds to read the sensors. We could probably easily bump the read rate to 500Hz, and maybe to 1000Hz, but this doesn't count anything but reading the MPU and HighAcc sensors.

Gem of the Week - the Parallax Propeller

I write this before ever using one, so consider this a review of the concept rather than the implementation. The Parallax Propeller is a microcontroller with a couple of interesting features, and perhaps more interestingly, a couple of intentionally missing features.

First off, it has no peripherals other than 32 GPIO pins.

Secondly, it has no such concept as interrupt.

Thirdly, it has eight independent processors, called "cogs", each with its own memory. Each one can run on its own resources without interfering with any other processor. Each cog is a 32-bit processor, and gets access to 2kiB of ram, shared between code and data.

Fourth, it has a set of resources that all processors can access, called a "hub". This basically consists of more memory and a round-robin memory controller which each cog can access in turn. The hub has 32kiB of ROM with the bootloader, Spin interpreter, and a couple of tables, and 32kiB of RAM.

The missing features are what make the controller interesting. Want an I2C? Write a program for a cog which can bit-bang it. Same for SPI, UART serial, etc. Presumably it could bit-bang low-speed USB, but high-speed would be difficult due to limited processor speed.

Further -- want an interrupt? Too bad. Instead, assign a cog to sleep-wait for the appropriate signal.

Programming such a beast is clearly a different problem than programming an ARM of any flavor. ARMs are all about peripherals, registers, interrupts, etc. Propeller is about bit-banging. Effectively you can use a cog as a soft-peripheral to do basically any digital process.

As I mentioned above, the processor comes with a Spin interpreter. Spin is a custom language for programming Propeller, which gets compiled into byte code and interpreted with the Spin virtual machine. Of course you give up performance, but they used an interpreter in the Apollo Guidance Computer. There, it was for memory saving - a single interpretive instruction could take the place of many instructions in AGC machine language. They gave up time to gain space. Spin could have similar benefits, but it seems like the main purpose is providing a language which natively handles the very different concepts needed to handle a processor as weird as the Propeller.

A Propeller program then consists of a bunch of Spin and machine language routines, all stored in hub RAM and copied from some external source whenever the chip resets. Aside from self-modification or using a cog to bit-bang a memory bus, this is all the space you get. This is a rather tight restriction, in fact less memory than in the AGC. But, they fit a full-blown Kalman filter into that.

Of course I have already designed it into a Rocketometer. It's what I do. One of the great things about the GPIO and bit-banging style of the Propeller is that I can put the SPI bus on the pins which are closest to their targets outside the chip. This makes routing the board MUCH easier.

So I will say for now that Propeller shows a lot of promise. The design is a gem. We will see about the implementation.

File system driver

My C++ification of the Loginator code continues. As noted below, I have the startup code now in full C++ (with 18 lines of inline asm), and I have overthrown the tyranny of main() (that sounds familiar, have I written on this topic before?). I have taken Roland Riegel's sd_raw driver and heavily modified and simplified it. Basically I made it a pure block device. I have dropped all the buffering. You can open an SD card (SDHC fully supported), read a block, write a block, and get the card info.

I looked into extending c++ifying the partition and fat32 driver, but it looked too complicated and messy. One of the things I am dead set against is dynamic memory allocation in an embedded process. What if it fails? When that happens, the software crashes (it wouldn't ask for memory if it didn't desperately need it) and when that happens, it is a good possibility that the device it is flying crashes too.

So, I get to write a fat32 driver myself. Once again, only whole blocks at a time. And to start with, only that which the USB bootloader and Logomatic need: read a file, write a file, delete a file. Also to start with, we fully support FAT32, but do not support long filenames.

One area where I am going to get myself in trouble is writing the file. Sometimes when you write a file, you have to change the file allocation table. When you do so, you need to read the sector containing the change, make the change, then write the new sector. This is all easy, but you need a buffer to do it. Also, you will need to read the table to find the next cluster. What buffer do you use? I know that the LPC2148 is not really memory-limited, but it still seems a waste to set aside a whole block buffer for this.

I started by writing a partition driver. You pass it an open and started SD object and a partition number, and it reads the partition table to get the info for that partition. From then on, you use the partition object to read and write blocks.

A review of the ADXL345 and BMA180

First off, the ADXL345 is exactlty what it claims to be. It's my own fault for not reading the data sheet, or rather reading it but not comprehending the information in it.

So, my first experience with digital accelerometers was with the Bosch BMA180. The part had 14 bits of precision and 6 scale settings, from 1g to 16g. When you switched to the lower scales, you got more precision, as expected.

I finally got all 11 DoFs working on the 11DoF board. The first SPI part I broughs up was the ADXL345, so I learned about its SPI protocol that way. For one thing, those of you used to I2C, SPI is different! For instance, on both the ADXL345 and the gyro on the 11DoF, the L3G4200D, the register addresses are six bits. This is fine, since that is a big enough space. You send this address as the first word of any transfer in either direction from these devices. However, both devices use 8 bit words in the protocol. The other two bits control the direction of transfer for the other words, and whether the device is to expect more than one word -- that is, whether it should increment the register address pointer for the next words.

For instance, say you want to read all the measurement registers in the ADXL345. The register map says that this is registers 0x32 through 0x37. Since SPI is a full-duplex protocol, every time you send a byte, you receive a byte. So, send 0x32, ignore what you receive, then send six more bytes (doesn't matter what, I used 0) to read 6 bytes that you care about. Right? Almost, but not close enough. I did this first with the ID register, and it worked, sometimes. But, when I actually tried to read the data, I got back the same word six times. What gives? First, bit 7 of the address is read/#write. You have to set it to read the register, otherwise it interprets the MOSI data as data to be written to the registers. The data registers are read-only, so the part ignored me. Next, bit 6 is the multibyte flag. Set this bit if you are going to read/write multiple registers in one transaction (one continuous #cs assert). Doing this will cause the part to increment the address pointer each time it sends or receives 8 bytes. Since I set neither of these, my part became very confused, since I was telling it to write to a single read-only register six consecutive times.

tl;dr - Tell it you are using address 0xF2, then read 6 more times to get the 6 data registers.

This is an unnamed protocol layered on top of SPI, which by itself knows nothing of registers. It happens that the gyro uses the same protocol, so turning it on was a simple matter of verifying that the protocol was the same, and copypasting the code.

Now for the review. The ADXL345 has a programmable range, with choices ±2, 4, 8, and 16g. We will need 16g for the rocketometer. It has a readout precision of 13 bits, almost equal to the 14 bits the BMA180 gives. Now for the bad part. The readout precision is only 12 bits for 8g, 11 bits for 4g, and 10 bits for 2g. It is as if the part was always running at 16g, but reporting saturation if it was out of its current range.

I might as well use an analog part if I am only going to get 10 bits. I had always known this, but only realized the significance when I finally got it up and running, and only got about 250DN out of the part in the 1g field. So, the ADXL gets 2 of 5 stars, not recommended on the Chizumatic scale.

The BMA180 is what I used before, in flight. It has a programmable range, with choices ±1, 1.5, 2, 3, 4, 8, and 16g. It produces 14 bits of precision, and this 14 bits is constant across all ranges, so if you use the 1.0g range, you actually have zoomed in, and get better resolution than when you are at 16g. I can't speak to its accuracy, so it gets 4 of 5 stars, not recommended. Why not? The part was discontinued without a suggested replacement as of today, and is no longer available on Digikey. In fact, I am hoarding three of them still in their cut tape, purchased from Sparkfun today at great expense, not even on a breakout board. None of the other BMA accelerometers are as good, and none of the Analog Device accelerometers are as good either.

8 DoFs so far...

I have gotten the ADXL345 accelerometer, HMC5883L compass, and BMP180 pressure/temperature sensors working with the Loginator and 11DoF board. This means that all SPI and I2C signals work on both boards. Last thing to do is to get the STMicro L3G4200D gyroscope working.

I just saw a board basically identical to the 11DoF (same sensors except a BMP085). The problem is that it is all done up in I2C.

Starting up an ARM processor with 90% C code

Now why would you want to do a darn fool thing like that? Startup.S works fine, why mess with it?

One word: Understanding.

Let's face it: Assembly is hard to read. Especially ARM assembly, with its special registers, shift-operands, and treating memory (load/store) fundamentally different from registers (mov). You can't get more than a few lines through an assembly listing without having to crack the ARM ARM.

Besides, I have a philosophy to use a consistent language throughout a program to the extent possible. So, for the embedded stuff, it's C++ when you can, C when you have to, and asm only when you really have to.

So what stands in the way? Mostly it's the fact that the toolchain makes it difficult to put things exactly where you want:

1. The interrupt vector table. This is what keeps the code from being 99% C instead of 90%. On an x86 processor, the interrupt vector table is purely a list of addresses. Upon interruption, the processor looks up the correct vector, and loads it straight into the instruction pointer, causing the processor to branch to the handler. In many other processors, the table is actually code. When an interrupt happens, the processor jumps to the correct slot in the table and executes it. Usually this is a jump to where the handler really is. In ARM, there isn't enough space for a long jump in a single instruction, so each vector says to load the program counter with another value from memory, usually in a table located right after the true interrupt table. In any case, C just isn't a good language for building the table. What we do then is use inline asm. We make a function called vectorg which is purely inline asm. The first half is instructions, specifically the long jump instructions with embedded pointers to the second half, which is the address table. This is populated with symbols, so the linker can patch it up.
2. Putting things where we want. The old Turbo Pascal had a mechanism for assigning a variable to a particular point in memory. In asm, this is easy: just define a symbol with a hard-coded address. This just isn't possible in C. So, we need cooperation from the linker. We need to specify the linker script. In particular, we need to say that a particular named ELF section is to be linked right at the beginning of flash, and then make sure that the table is in fact in that section, at the beginning of it. The easiest way to do that is to turn on -ffunction-sections when compiling, then link .text.vectorg at the beginning. The source code and the linker script have to agree on this.
3. Registers - This is the other 1%. To set up the program, the reset handler has to set up the stacks, move the initialized data from flash to RAM, zero out the uninitialized data, and call all the constructors for global objects. But, in order to set up the stacks, the code has to be able to set the CPSR register, so it can flip through modes and set each mode's stack register. It also has to be able to write directly to the stack register itself.

A new weapon in the war against Bugs and Photino Birds

Keil μVision 4 Debugger/Simulator. This is a small part of the larger (450MB, really guys?) μVision suite, but the only part I care about. With it I was able to debug my Task code, which is interesting given that I did not write or compile my code in μVision. Get the program and install it (it installs fine under Wine on Linux) and then bring up any example project, compile it, and get into debug mode. Then, right-click the disassembly window and select "Load Hex or Object File...". Now select the master .elf file that was compiled with GnuARM. It will load, and the simulated processor will be reset, so it starts at location 0 (of your code, apparently it skips the ISP).

And now for a word (or thousand) about tasks...

More on SDHC

So here's the problem. There are in fact two SD card drivers in the Logomatic firmware. Hey, by the engineering method, anything that works is good, and the Logomatic code works, but it's not great code. There are entire sections that are included but not used, like the USB driver in the main logging firmware. Now we have two pieces of code to do the same thing -- access the SD card. There is Roland Riegel's code used by rootdir.c and used by both the bootloader and main code to read and write the card, and there is the code used by the USB driver, which is entirely separate and entirely unready to do SDHC.

So, I am going to update the USB driver to use Roland's code. This just means mapping what the code is doing now to the various parts of sd_raw.c . Piece of cake, probably. We'll see.

Anyway, once this is done, I'll post a message about it on the Sparkfun forums, direct them here, and get literally some readers on this blog, rather than just myself like I get now.

No bragging until I can put my code where my mouth is.

By the way: I am doing all of this development on the Loginator 1.1, so the SD card slot and USB port work. Yay for me!

Source Code Embedding

I am quite proud of this, and it took me several iterations to get right.

Once upon a time I wanted to know about how the In-Application Programmer (IAP) worked. This is a bit of code loaded onto the LPC214x by its manufacturer. I wanted to see what special thing it was doing that my program couldn't. So, I wrote a little doodad into Logomatic Kwan which caused it to go through flash memory and dump its content into the normal log file. This would include the Sparkfun bootloader at the bottom of memory, my own code starting at 0x10000, and the IAP up at 0x7D000. I used $PKWNH packets and simple hex encoding.

I didn't end up doing any serious poking around with the IAP (it still is a mystery) but I was interested in dumping the firmware itself as part of the metadata for any particular run. The problem is that it takes a megabyte and at least a minute of run time before the dump is completed and a normal log run commences. So, I put a switch into the config file to turn it on or off.

Also, the actual bits of the program are interesting but hard to interpret. Better would be the original source code. So, I tweaked the make file to tack onto the end of FW.SFE a listing (ls -l) of the code used to build it, then all the files concatenated together. Now all the source code is there, but pretty hard to get to, being encoded in hex and quite difficult to parse out. Plus, it was in danger of not fitting into memory.

So, the next idea is to use that same file listing, but instead of doing a simple concatenate, concatenating a gzipped tar file. This is what Tar exists for, and it does a better job than anything I could roll on my own. Mostly it's better because a tar/untar utility already exists in Linux, and I would have to write one for my roll-my-own solution. Plus, the compression saved something like 80% of the space.

Then I (mis)remembered something we did with AIM/CIPS flight software. We wanted to check if the software was ever corrupted by things like radiation hits, so it was told to take a CRC of a series of blocks of memory which should have been constant, and transmit the CRC down. The misremembering is that originally I thought that it actually transmitted down the whole firmware one packet at a time. Since they are much better at NASA at configuration control than I am at home, they always know what software should be on the spacecraft, so they don't need to send the firmware down. But I do. So, I just had it do the old firmware dump routine, but one packet at a time, once every time a GPS update comes in. (Update: I finally re-remembered. We did have a memory dump command which would act pretty much as described. We said dump this memory bank from this address to that address, and it sent down multiple packets, one per second, with that data until it was finished. We could even dump RAM, so we could dump a chunk of memory that was changing as we dumped it. We could even dump the FPGA registers.)

Finally, I wrote a perl script which extracted the flash dump from a logging run, translated it from hex to binary, and then wrote out four files: Bootloader, User firmware, User firmware source tar file, and IAP.

Unfortunately, at 5 packets per second, the firmware dump now takes 54 minutes to complete.

Linker scripts

Linkers are pretty smart, but they need some help to tell them what to do. That is where the linker script comes in. Mine originally descended from the Sparkfun Logomatic firmware (like my whole program) where it is called main_memory_block.ld , but it turned out to have a flaw.

I remember back in the 90s when I was first learning a real language, Turbo Pascal. Don't laugh. One of the features of its compiler/linker was what it called "smart linking" where it would only include code and/or data in the final executable if it had a chance to be used. It eliminated dead code.

GNU ld by default doesn't do this. The linker is a lot simpler and dumber than we think. For instance, the objects that it links are composed of sections and labels. It uses the labels to do the linking, but the sections are trackless expanses of bytes. They might be code, they might be data, they might be unlabeled static functions or read-only data. The linker doesn't know, it can't know.

In order to do a semblance of smart linking, GCC has a feature called -ffunction-sections and -fdata-sections. This tells GCC that when it is building an ELF object, it should put each function into its own section. The code goes into .text.function_name, while zeroed data goes into .bss.variable_name and initialized data goes into .data.variable_name.

The complementary feature in ld is called --gc-sections. The linker script tells the linker where to start, where the true program entry point is. All labels used in the section where the entry point is, are live. All labels used in the sections where those live labels are are live also, and so on and so on. Any section which has no live labels is dead, and the linker doesn't put the dead sections into the final executable.

With the much smaller sections provided by -ffunction-sections, the .text section is no longer a trackless waste of bytes. It's probably empty, in fact. All the functions live in their own sections, so the linker can know what is what, and can remove dead code. These complementary features are the GCC implementation of smart linking.

However, when the linker is done garbage collecting the dead code, the linker script might tell it to bundle together all the sections whose names match a pattern, together into one section in the output. No one is going to try to link the output into some other program, so this is ok. The Sparkfun linker script bundled together all the .text.whatever sections into a single .text section in the output, and all the .data.whatever, but not the .bss.whatever. This is important, because the linker creates a label at the beginning and end of the .bss section, and a block of code in Startup.S, the true entry point, fills memory between the labels with zeros. With all these unbundled .bss sections, the final .bss was very small and did not include all the variables, so some of the variables I expected to be zeroed, were not. This is a Bad Thing. Among other things, it meant that my circular buffer of 1024 bytes had its head pointer at byte 1734787890. As the wise man Strong Bad once said, "That is not a small number. That is a big number!"

It turns out this does not have anything to do with C++. I turned on -ffunction-sections to try and reduce the bloat from the system libraries needed in C++, but if I had turned it on in C, the same thing would have happened.

The fix:

Open up main_memory_block.ld . Inside is a section like this:


/* .bss section which is used for uninitialized (zeroed) data */
.bss (NOLOAD) :
{
__bss_start = . ;
__bss_start__ = . ;
*(.bss)
*(.gnu.linkonce.b*)
*(COMMON)
. = ALIGN(4);
}
> RAM

. = ALIGN(4);
__bss_end__ = . ;
PROVIDE (__bss_end = .);


This says among other things, that in the output file, the .bss section (which has a new label __bss_start at its beginning) includes all .bss sections from all the input object files. It also creates a new label at the end of the section called naturally enough, __bss_end . The startup code zeroes out all the RAM between these two labels.

The problem is that *(.bss) only includes the .bss sections from each object, not the .bss.variable stuff. So, change the


*(.bss)


line to:


*(.bss .bss.*)


This says to bundle in not only all the .bss sections from all the inputs, but also all the sections which start with .bss. in their names. Now with them all bundled, the bss zero-er will do the right thing, and the program will work.