Sunday, June 1, 2014

Remind me why I am doing this again?

By the third day his eyes ached unbearably and his spectacles needed wiping every few minutes. It was like struggling with some crushing physical task, something which one had the right to refuse and which one was nevertheless neurotically anxious to accomplish.
-- George Orwell, 1984

Here we are, three weeks until the contest. I have no motivation to do this, but at the same time no desire to call it off. As I hate making decisions by default, that means I have to do it. Since I am unlikely to win the contest, nothing tangible is gained by finishing. Last time, I was tremendously motivated, to the point of using every waking hour, even to the point of taking vacation time to work on the robot. This time I can barely motivate myself to take a weekend to work on it. I just can't get into it.

Yesterday I went to the course. As the newspaper says, "'Tis a privilege to live in Colorado". One benefit is that with 20 minutes of driving and $6.25 to get in the gate, I can visit the course whenever I want. I did so yesterday, with the robot brain on a breadboard. I had the GPS on and locked, connected through the Arduino Nano used as a pure passthrough for its FT232 chip. This was piped through the SiRFDemo program running on Natsumi, and recorded.

Yesterday, however, they were setting up for the Boulder Triathlon. I guess I take a certain amount of inspiration from the runners in that race. Thousands of people join up, most of them with no thought of winning, just finishing. Similarly, I do not plan on winning. I just want to do this so that I can say that I finished something. I am not doing this again by myself. Next time, I plan on being part of a team, preferably as a sponsor/adviser.

In any case, here is the plan. I am going to do this by waypoint navigation. The robot will continuously estimate its 2D position and heading (speed doesn't matter). The position estimate will come straight from the GPS. The heading will be a Kalman filtered result from the GPS heading (to remove gyro biasing) and the gyro (for precision and response time). Perhaps GPS position will work in as well, if I can figure out how.

The hardest part is deciding what to do when we are close to the waypoint. I can easily imagine the waypoint getting inside of the robot's turning circle and the thing chasing its tail forever in vain.

Imagine we have three consecutive waypoints, one at the start line (point 0), one at the first turn (point 1), and one at the second turn (point 2). Also imagine we have just left the start line and are heading towards the first turn. How do we know when to turn? Also, what course do we steer heading towards the point? I think we want to set an imaginary steer-to point (point 1') about 20 meters beyond the actual waypoint 1, on the line from point 0 through point 1. We steer towards 1', while keeping track of the dot product between the vector from point 1 to point 0 and the vector from point 1 to the robot. When this dot-product becomes negative, we have crossed a line perpendicular to the line from 0 to 1. At this time, we set the target point to point 2', 20m beyond waypoint 2 on the line from 1 to 2. We then change all the indexes and steer towards point 2'.

The 20m is arbitrary, but intended to be larger than the GPS error.

Friday, May 16, 2014

June 1 Test

When I last entered the AVC in earnest, I remember there being an April 1 test, where you had to demonstrate the vehicle moving and steering under its own power and control. As it happened, I burned out my helicopter and shifted to the car design after this date, but I got a waiver when I demonstrated the car a mere 1 week before the contest. And we all know how that wound up.
April 1 test from last time, completed on April 17


So, I was expecting something like this, this time around. I was watching the Sparkfun web page, expecting to see something there. My phone automatically checks my gmail, so I was watching that. As it turns out, their email got stuck in a spam trap, and I didn't see it until today. So without further ado, I present the June 1 test, completed on February 22, 2014.
This test shows the steering servo connected to an Arduino Nano, powered by the BEC from the ESC, just like the real controller will be used.
Maya wants to help. Some tests are more successful than others...

This test shows the Arduino controlling both the steering and throttle.

Steering is the easy part. It is half of the Control function in the Guidance, Navigation, and Control triad. All the mechanical parts work and if the robot knew where to go, it could control itself and get itself there. The latter is the hard part. I anticipate using a Kalman filter with inputs from the GPS, magnetic compass, and inertial rotation rate sensor. I also am going to add a bumper, so in case it hits anything, it can back up and try again. This is to get around the barrels. I am not currently planning on using the line for line-following robots.

The parts breakdown is about as follows:

Robot chassis - RC car. Includes drive motor, steering servo, battery, ESC, and radio receiver (still carried in robot but disconnected). about $150. Maybe $100 if you count only the parts in the chassis that I actually use, not the receiver, transmitter, or battery charger.
Controller - Kwan Systems Loginator (Logomatic clone) - about $60
Inertial Sensors - Kwan Systems 11DoF - about $40
GPS - GP2106 - about $50
Miscelaneous wires and breadboards - scattered on my workbench.

After last time's debacle, I decided to not go crazy with parts this time. All parts except the chassis are from other projects. I have all these parts already. This is all the hardware I anticipate needing. I use lots of other parts for bench support, such as Arduinos, FTDI chips, oscilliscopes, logic analyzers, computers, etc. None of these are actually part of the robot.

As such, I believe it is a small enough budget that it might fit in the micro class.

As noted in the parts list above, I have chosen to go with the more powerful LPC2148, in the form of the Kwan Systems Loginator. You can read about the design of the Loginator and 11DoF further down this blog. I have repeated all these tests with that controller, but I didn't record them, and they would just look the same as these tests anyway. Video from those tests when it does something more advanced.










Tuesday, April 8, 2014

Music Structure and Program Structure

I am a programmer by nature. I learned on my father's knee when I was single-digits old. I passed him in skill when I was a teenager. But, I am not only a programmer. "Specialization is for insects."

One of the other things that I am is an amateur musician. I may not have ever been a good, or even an average, tuba player, but I had fun doing it. I had fun marching, and I had fun watching football and basketball games. I realized early on that I was there not really so much as a musician, but as the carrier of a big metal flag. I was there for visual impact as much as anything. I even achieved a certain amount of fame as the "spinning tuba guy" in the early 2000's. One of my proudest moments was when I was featured in the opening montage of SportsCenter for one whole second.

As I said, I was never very good, but to be even a below-average tuba player, you must acquire certain skills. You have to know how to read music, and I did. I could convert note positions into fingerings and play approximately the right note. I could count or clap out rhythms, and with some practice, I could play the music well enough to fit in with the rest of the band.

One of the things I noticed being both a programmer and musician is that there are some similarities between program flow and music flow. In both cases, the most common flow is from one line or measure to the next, just sequential. Programming has loops, music has repeat signs. Programming has if/then, music has first and second endings. Music is somewhat limited, in that it is deterministic. It doesn't have to deal with input. Therefore, in music, some things have to go together, like multiple endings and repeat signs. Otherwise, the second ending would never be played, and would effectively get optimized out.

I have often wondered if this mapping could be made more complete. There are certain concepts we use in programming that tend not to get used in music, but maybe could, like subroutines. That got me wondering if there were concepts in music that could be mapped to programming, but aren't, and I finally came up with one today: the coda.

In music, you will see markings such as D.S. al Coda, indicating that the flow jumps from here, back (never forward) to a special marking, and then continues from there to another mark coda. This mark is ignored the first time through, but the second time, indicates a jump forward (never back) to the marked coda section.

It first occurred to me today that this is similar to exception handling. When a handled exception is thrown, the flow jumps to the handler. In a sense, this is like taking the coda branch in music.

You could use this in normal flow, throwing an exception when you want to make an early exit from a loop or function. Most functions and loops should have one entry point (enforced by the language, except for Fortran), and one exit point. However, sometimes it is convenient to do an early return from a routine, an early break from a loop body, or an early re-loop in a loop body. Most languages I use support these things with the return, break, and continue statements, respectively. However, there is a good reason for the 'only one exit' rule. Often the routine needs to do some cleanup on exit, closing files, calculating final results from accumulated variables, etc. If you do an early exit, you have to make sure that the cleanup is done appropriately each time you have an exit. If you want to change the cleanup, you have to do that in multiple places. The alternative is that rather than having multiple cleanup-and-exit blocks, you have something like goto end, and then at the end label, you do the cleanup.

However, gotos are to be avoided, for good reason. While the code will work just fine, source code is for communication with humans, not machines. If it were otherwise, we would write code once, keep the binary code, and delete the source. In this case, the humans we are communicating with are most likely our future selves.

Since source code is for humans, semantics matter more than they do if it were just for machines. Machine language doesn't normally have many advanced control structures, just jumps. The code is effectively strewn with goto statements, and the more optimized the code is, the worse this tends to get. Even disassembled code from a modern compiler is hard to reconcile with its source code. Optimizations make everything dramatically out-of-order.

Source code on the other hand, has structured statements rather than a spaghetti nest of goto statements, because they mean something to us. To a machine, a for loop means set up an induction variable, do something to it at the end of each loop, check the loop condition, and go back and do it all again as necessary. To a human, it means run the loop a predetermined number of times. This is why we have the rule to not monkey with the induction variable, as it disrupts the loop semantics. The human reading it will think that this block will run a predetermined number of times, but monkeying with the induction variable disrupts this expectation. In a sense, this is why break and continue are frowned upon, because they also disrupt the expectation.

Random goto statements are just that, random. They have no semantics. They can be used to do anything, so they mean nothing. All structured programming constructs can be done with goto (in fact they have to be in machine language) but not all goto constructs can be represented by structured programming. Sure, there is a proof that you can do it, but that's just because structured programming and goto statements are both Turing complete. I was once dealing with a program, SDP4, which was originally written in Fortran with no structured programming constructs, but pure goto spaghetti. The code arrived to me translated into (Turbo) Pascal, and I translated it further into Java. The previous programmer did a pretty good job of translating most of the gotos into structured constructs, but there is one part where he left in the gotos because the original code was so tangled he couldn't figure it out. He had the luxury of leaving in the goto statements in that case, because (Turbo) Pascal supports it, but I in Java did not. I ended up using a case statement inside a while loop and depending on fall-through. It was technically a structured construct, but just emulated the original goto nest. Vallado points to a solution that involved jettisoning that entire block of code and re-writing it from the original math.

So, while you can write a goto end, it doesn't carry the appropriate semantics. What I want is a statement that means to coda and coda. This carries the semantic that we will shortly be exiting the routine, but that there is a certain amount of cleanup to do. It would be added to the arsenal along with the early return, break, and contiune. One way to implement it in Java is to put the coda in a finally block of a try/catch/finally statement. Then when you want an early cleanup, you throw an exception. However, this violates the semantics of an exception, which is supposed to only be used for an error condition. I once wrote a program which used exceptions in this sense, asked the guys on TheDailyWTF forum what is a better way to do it, and basically got laughed off the forum.

Therefore, I think that in order to capture this semantic, a new statement is needed. Coda is a fine word for it for me, since I am a musician, but maybe there is a better term. Until this construct is added to our arsenal of structured constructs, we are stuck with goto coda, which is better than nothing, because it does capture the appropriate semantics.

Wednesday, April 2, 2014

Regulator

Once again this is hooked up as indicated in the MIC5319 datasheet. I finally am using the switch right, no more useless machine (and useless transistor) for me. As it turns out, that transistor was effectively built-in to the regulator, that's what the EN input is there for. This circuit supplies a nice 3.3V output on VCC to everything else. Previous versions had a current sensor, but this one doesn't, as I haven't gotten one of those to work yet.

It isn't visible in this image, but the bottom rail is GND.

Charging Circuit

This one is a pretty straightforward hookup of the MCP73831 charging circuit, with one addition: D302 is there so that the charging circuit does not have to treat the whole device as a battery to be charged. If USB is connected, VIN will get the full 5V (minus the voltage drop of a Schottky diode). If not, the battery is used (again minus the voltage drop). Finally, both diodes need to be there so that the battery doesn't feed back to the charger input and try to charge itself. The programming resistor tells the resistor to charge the battery at a maximum of 100mA. This is 1C for a 100mAh battery, the smallest one I have and the one that flies with the Rocketometer.

The status light uses a resistor from the 1.5k pack in the USB connection.

Other versions of this circuit have had a voltage divider between VLIPO and GND so as to allow the MCU to measure the battery voltage. I haven't used it in a while, and it does draw some current, so it is gone from this circuit.

Doing USB right

AN11392 - Guidelines for full-speed USB on NXP's LPC microcontrollers (19 Feb 2014). This finally answers all my questions about what all the USB parts are. Based on it, let's review the Loginator USB/charge/power supply section.

We are now using a Micro USB connector with through-hole posts for better mechanical security and easier alignment during soldering. Micro USB takes up less board space and is compatible with the cords used for Android phone charging.

The first thing the app note says is that there must be a 33 ohm resistor on each of D+ and D-. This, plus the internal resistance of the LPC itself, add up to the 90 ohm total impedance required. It implies that there is 12 ohms on each of the pins inside the LPC. This is what I have been carrying all along from the Logomatic design.

Secondly, USB_Softconnect is required if the device is self-powered, or if it takes longer than a certain amount of time to boot up. Since my device can be self-powered and might never boot up and connect, I intend to use this as intended. However, I still like a PMOS rather than PNP transistor. The 1.5k resistor required for softconnect to work is also well-sized for the LED, so I use an array.

The signal lines have capacitors to ground, for exactly the purpose I deduced - shorting high-frequency signals to ground. The app note says that they are not strictly necessary but that it has been reported to improve certain noise issues. I have always built circuits with these included, so I shall continue. These are one of the rare cases where it makes sense to use a capacitor array.

Next, we have R011 and R015, which I completely whiffed on. In my defense, I am not the only one. My design came from the Logomatic, which came from the microbuilder.eu LPC2148 Reference Design which appears to be a mistranscription of the Keil MCB2140 schematic. Even then, the Keil board does not seem to be what was intended.

The idea is that P0.23, USB_ON, is 5V tolerant if and only if the rest of the circuit is powered. So, if it is not possible for the MCU to be off while the USB voltage is present, then you can just plug VBUS into P0.23. However, if the MCU is not powered, the voltage on that pin is supposed to be limited to less than 3.6V. The app note recommends a voltage divider, with 24k on the high side and 39k on the low side, resulting in about 3.09V on the input.

The way both the microbuilder circuit and Logomatic circuits are arranged, that isn't a voltage divider at all, and the pin eats a full 5V. Since the MCU can be turned off (the power switch disables the 3.3V VCC line), this is technically out of spec.

R011 should be connected to the right pin of R015B, and should be closer to 20k. This divider will draw three times as much current as the recommended value, but that is still less than 1mA.

There is supposed to be between 1 and 10 uF between VBUS and ground, visible through regulators and other parts. The real spec is that the inrush current should be limited, but I don't intend to ever submit my device to USB compliance testing, so as long as it works for me, it's fine. There is 4.7uF on the input to the voltage regulator, so I do not inculde any intentional capacitance in this section.

Monday, March 10, 2014

Rocketometer Flight Data Published

This documents the data produced by the Rocketometer during NASA Sounding Rocket Flight 36.290, 2013 Oct 21.

I was getting sick of looking at this data, unable to fully process it but holding it jealously. This has to change. The data wants to be free. Get it at https://drive.google.com/#folders/0B_jwoDhgAF3cNXo0OG5MS2RoM3c

Thursday, February 20, 2014

Once More Unto the Breach...

It's official, St Kwan's has re-entered the robot business. Project Yukari Mk2 will be racing on June 21, 2014.

Monday, October 21, 2013

IT WORKED!!!

I turned the Rocketometer over to the rocket guys back in July, and had no contact with it until after the flight today. It had been exposed to every environment it was going to see except for vacuum, but I still had no confidence that it would work.

Well, it did.

The first time I stuck the SD card into my computer after the flight, it said "what card?"

 That was disappointing.

 It took a few minutes for the computer to recognize the card, but when it did, I saw that it had recorded 419 MiB in 66 files. One of the last was also the longest, so I ran it through my quick extraction program, then through IDL, and saw the characteristic acceleration pulse during first and second stage.

The first thing I did after that was press the Victory button on my phone:

 No one else in the lab either heard that or got it, so I had to shout, "It Worked!!!".

Now I have to analyze the data...



Minimum mission success achieved!

At about 12:01:12MDT today, the Rocketometer achieved minimum mission success by riding above 100km and therefore reaching space.

It will be some time yet before I can recover the device to see that it worked, which will represent full mission success.

Saturday, October 12, 2013

The Curiously Recurring Template Pattern

Go look up on Wikipedia what it is. I am going to talk about how I am having to use it.

I was doing fine with the Rocketometer analysis code in C++, using the NewMat library to handle matrices, with a Quaternion layer on top that I wrote myself. After five days of doing battle with various things, I finally got something that worked, but I was curious if this was the "best" way to do it. The C++ Standard Template Library didn't have anything directly related to matrices. The Boost library had a section called uBLAS, but the documentation for it kind of de-recommended itself. It suggested several alternatives, and the one that looked best is called Eigen.

Eigen is interesting in that it is entirely header files, containing almost all of its code in C++ templates. Templates are cool, mostly because when they are instantiated, the compiler gets to see the code in the context that it is used, and gets to inline and optimize it there. Specifically, Eigen contains a dynamic-sized matrix, but also is a template for fixed-sized vectors and matrices. I want to use these as much as possible because all vector sizes used in Rocketometer data analysis are known at compile-time, so the compiler can unroll loops and so on to best optimize the code.

However, templates do not mix with virtual methods, so I had to figure out how to make that work, since I used virtual methods to implement the physics model.  I had code that looks like this with NewMat:

class SimModel {
protected:
  /** Physics function. Calculates the derivative of the state with respect to time
  virtual ColumnVector fd_only(double t, const ColumnVector& x)=0;
  /** Measurement function. Calculates the measurement from the given state and time   */
  virtual ColumnVector g_only (double t, const ColumnVector& x, int k)=0;
  /** Physics function with process noise. Uses fd virtual function to calculate physics, then adds process noise.*/
  ColumnVector fd(double t, const ColumnVector& x,        const ColumnVector* v);
public:
  /** Measurement function with measurement noise. Uses g virtual function to calculate measurement, then adds measurement noise.   */
  ColumnVector g (double t, const ColumnVector& x, int k, const ColumnVector* w) {ColumnVector result=g_only (t,x,k);if(w)result+=*w;return result;};
};
 
But I wanted to adapt that to use Eigen, specifically with the fixed-length vectors, since the size of the state vector is determined by the problem and known at compile time. That means that ColumnVector has to go, to be replaced by Matrix<double,n,1> where n is a template parameter determining the size of the state vector. But what about the measurement? The purpose of the k parameter to g_only is to tell which of several kinds of measurements to use. For instance, in the Rocketometer problem, we have a measurement vector coming from the inertial and magnetic sensors, treated as a single 9-element vector. We also have measurements coming from radar or equivalent, treated as a 3-element vector. So, we need a template function g_only, which generates either a 9-element vector or a 3-element vector. You can't do that and have it be virtual, too. Basically, virtual functions are a runtime binding issue, while templates are a compile-time binding. So, I can't have a virtual g_only function, callable by the base class g function.

Enter the Curiously Repeating Template Pattern (CRTP). As it happens, this is something that I read about just a few days ago, just reading up on the C++ language in general. For us, the pattern goes something like this:

template<int n, class Derived> class SimModel {
public:
  template<int m> Matrix<double,m,1> g (double t, const Matrix<double,n,1>& x, int k, const Matrix<double,m,1>* w) {
    Matrix<double,m,1> result=static_cast<Derived*>(this)->template g_only<m>(t,x,k);
    if(w)result+=*w;
    return result;
  };
};
 
Note that g_only isn't even defined in this class template, only used. In fact, one of the weaknesses of CRTP is that it implies definitions without expressing them, so it is hard to document. Also note that extra template keyword there after the arrow operator. See here for details.

The derived class then looks like this:


template<int n> class RocketometerModel: public SimModel<n,class RocketometerModel<n>> {
  template<int m> Matrix<double,m,1> g_only(double t, const Matrix<double,n,1>& x, int k);
};
 So what happens is that the compiler
  1. Parses the template for SimModel, but doesn't compile it, because it's a template, not actual code yet. Therefore it doesn't matter that g_only is undefined yet.
  2. Parses the template for RocketometerModel, and again doesn't compile it.
  3. Parses the main code, compiling as it goes along until it hits RocketometerModel. It instantiates and compiles RocketometerModel, in the process instantiating and compiling SimModel.
  4. When SimModel is being instantiated and compiled, it has a call to RocketometerModel g_only, but this is ok, since that is available already, since step 2 has already happened.
Now the derived class might not be a template, it might be an actual class. In this case, the base class is instantiated and compiled with the derived class. In either case, everything is available before it is used, even though the code might look otherwise.

Now this part I will write in bold, so that Google can see it. The other curiously repeating template pattern is having to use the word .template when using (not defining) a template member function. This solves the error invalid operands of types ‘<unresolved overloaded function type>’ and ‘int’ to binary ‘operator<’ .

It appears that there is a weakness in the C++ operator precedence table, such that in certain cases it can't distinguish the opening angle bracket of a template parameter set from a normal compare-for-less-than. In order to disambiguate, you throw in the word template after the dot (it will also work after a pointer arrow -> if you are using one of those). I don't understand it completely myself, but Eigen uses this itself, which is how I found out about how it works in the first place.

I am gradually coming to the conclusion that Java did it right, with generics which are real compiled code. Java generics are enabled by the "everything is an object" model in which all objects descend from a common class. I am also beginning to think that Java did it right with operator overloading. Operator overloading, even when fully appropriate, like defining * to mean matrix or quaternion multiplication, is fun to use but a nightmare to implement. And, if it is implemented wrong, it might be left to the user to find out when he tries to do something the implementer did not foresee.

All in all, I give Eigen 4/5, especially recommended for new projects, but not for converting old projects. The biggest advantage is speed. What took IDL over an hour, took Java and C++ with NewMat about 4 minutes, but takes Eigen only 20 seconds. Also, templated matrix and vector sizes are nice, because they resolve matrix size mismatches at compile-time. Finally, zero-based component indexing is what I expect, and the reason I don't suggest converting old projects from NewMat. Also be aware that the Eigen quaternion library uses the convention \(\vec{v}_{to}=\mathbf{p}\vec{v}_{from}\mathbf{p}'\), which is fine and internally consistent, but not consistent with the math I had seen for quaternion derivatives. As a consequence, my code is liberally festooned with q.conjugate() and in some places q.conjugate().conjugate(). It's almost a case of two wrongs making a right, but not quite.

Thursday, October 10, 2013

Calibration

It's kinda weird, but it turns out that C++ is the best language for processing Rocketometer data. There is a cool library NewMat which creates appropriate operator overloads to do matrices in C++, and I have extended it to include quaternions. C++ was doing in minutes what it was taking IDL hours. However, it took me 5 days to translate from IDL to C++, so I had better process a lot of data to ever get that time back.

The time that the Rocketometer spends in zero gravity may be the most valuable calibration time I ever get. Zero acceleration, zero rotation, and I expect a wide range of temperatures.

Remember that the goal of this is to get something into space, but that goal requires no further effort on my part to achieve. The secondary goal is to be able to calibrate the data and report something useful to Tom. The long-term goal though is to measure the track of Space Mountain.

So I am putting something in Space to get it ready for a roller coaster.

That made me laugh.

Saturday, August 3, 2013

Code A113

I was thinking about this at work, explaining it to one of the students I work with. He had never seen Wall-E, so I explained it in a generic and non-spoilery way. Since no one but me reads this anyway, I am going to write it the spoilery way here. You have been warned.

In that story, the systems on Axiom are programmed to look for habitability on Earth, and when it is proven, the ship is to return to Earth. It's an interesting but very expensive backup for just a normal recall message.

So, the pseudocode looks something like this:

while in_space do begin
  if(check_earth_habitability()) {
    return_to_earth();
  }
  sleep(1 year);
end


The function check_earth_habitability() is incredibly expensive to evaluate, as it involves sending a spacecraft with several EVE probes all the way back to Earth. It's like us making an implementation of go_to_saturn().

Directive A113 instructed the autopilot to not return to Earth under any conditions. It could have been implemented by cancelling the entire loop (including never sending EVE back to Earth), but instead the implementation looks like this:


while in_space do begin
  if(check_earth_habitability()) {
    //return_to_earth();
  }
  sleep(1 year);
end


A normal person would ask "Why does Auto send Eve back to Earth year after year if it has been ordered to not return?" I ask a similar question "Why did they do the code that way?" The thing is, I have done things similar to this: Evaluate an incredibly expensive function, just to ignore almost all of its output.

An example from work is sim_sample. A lot of times I only care about a small part of the result of sim_sample, or I have already calculated the result and could just run viusalize_1c manually. I don't because it is easier to run sim_sample and wait for it to finish than it is to do it efficiently.

Wednesday, July 31, 2013

Rocketometer has been integrated

The rocketometer is now an integral part of the rocket payload
Attached to the CCD heater card

In the control section card cage



And here we have a short video demonstrating the size of the Rocketometer:

Wednesday, June 26, 2013

Unattended running

The Rocketometer has been running unattended now for several hours. For instance, last I checked it was just over 7 hours, and had generated 77 files of 4MiB each. This represents a data rate of about 12kiB/second. At this rate, the device will fill up in about 350 hours, or 14 straight days.

So, I asked about which direction would be vertical when the Rocketometer is installed in the rocket. It is vertical when accelerometer reads most strongly in the  +Z direction. Coincidentally, this means that the interface is on the bottom, and the lights will be hidden.

I put in a piece of code which detects when the Z axis is stronger than the others. Mindful of the Genesis reentry failure, I put in code such that if the board is either +Z or -Z, it will decide that it is vertical. I can't test that Z is correct, but I can observe the board in place in the control section and visually confirm that Z is correct.

Now this piece of code will just reduce the rate of readout from once every 3 ms to once every 30ms, still quite quick, but only 1/10 the data rate. Even if it is wrong, we will still get some data.

I will need a log of when the CCD board is powered in order to match up non-flight measurements with anything, but the flight should be either the last or second-to-last record on the board.

For the unattended run, the system will be set to chunk its data into 1GiB pieces, and use only the last digit of the 4-digit number for chunks in a run, reserving 3 digits for runs. I do not expect the Rocketometer to be power-cycled more than 1000 times.

Monday, June 24, 2013

Bugs crossing Boundaries

I am getting the Rocketometer ready for delivery. The hardware appears to be fine, but the software had one particularly weird bug -- the name of the FAT volume on the SD card was changing. This meant that something was poking into the root directory block where it did not belong, which meant that it could be anything.

As it turned out, it was the packet buffer becoming full. Now how in the world did that corrupt the file system buffers? As it turned out, the file system by design wasn't using buffers, in order to save memory.

An SD card is addressed a sector at a time. Data can only be read and written in full sectors. So, if you want to change a single entry in the root directory or file allocation table, you have to read the whole sector into memory, change the bits you want, then write the whole sector back. Now the root directory and file allocation table only needs to be changed when the file is written, and in this case there is already a buffer which has just been used, so I foolishly decided to have the code use this.

In the mean time, this same buffer was part of the circular buffer accumulating packets. If the buffer ever filled up, new bytes would be rejected, and the head pointer would not move. If you start to write a packet to an already full buffer, nothing gets written, and when you finish, the packet length seems to be zero. Now the fun begins.

Upon finishing a CCSDS packet, the code counts how many bytes have been written, subtracts 7 because that's according to the protocol, then goes back and changes the buffer. In the current case, since no data had been written, it actually changed marked data.

Now we combine this with the fact that the interrupt task was filling the buffer. Here then is the final sequence:

  1. The buffer is full. No more bytes can be written.
  2. The main loop detects this, and starts to drain the buffer and write it to the SD card. It writes the packet, then reads the root directory sector, changes it, then writes it back.
  3. During the write, an interrupt fires.
  4. The interrupt task writes another packet. The bytes are dropped on the floor.
  5. The interrupt task finishes the packet. The calculated length is zero, since no new bytes were written
  6. The length to be written is -7, 0xFFF9
  7. The length is written to the buffer, but because of timing, it writes it over the top of the root directory sector in memory. The root directory is now corrupted.
  8. The interrupt task finishes
  9. The SD writing code writes the corrupted root directory sector back to the card.
  10. The card is now named RKT\FF\F9 instead of RKTO3.
It would be pretty easy for this same process to shoot holes anywhere in the filesystem metadata. This bug is hazardous!

So, to fix the bug, we do the following:

  1. Give the file system its own scratch buffer, and never let it do scratch work with the user buffer. This doesn't actually fix anything, but it does make the bug, and any similar undiscovered bug, less dangerous.
  2. Handle the full buffer case better. When the buffer is full, it throws away the new incoming bytes. Further, it backs the head pointer to the mid pointer, in effect throwing away the fractional packet accumulated before the buffer became full. Even further, since the buffer is now no longer full, to prevent getting the last half of a packet, we keep a flag that says that the buffer was full, and don't accept any more bytes until the flag is cleared by a drain operation.
  3. The interrupt task checks if the buffer is already full. If it is, we don't bother to read the sensors, a time-consuming chore which will just discard the result and take time away from the SD writing routines.
This clears a major bug, perhaps one causing all the observed halts. A bug which punches random holes in the metadata can do anything. It only triggered recently when I cranked the data recording rate to 200Hz, which appears to be just beyond the equipment capability. Previous rocket flights must have run at lower rate and not triggered the bug, or not triggered it as hard.

So how did I find it? Lots of print statements and blinklocks. First, I had the SD driver print out the buffer it was going to write. Then I had the direntry driver (the thing which actually handles the root directory) print out its scratch buffer after it had been read, as it changed each byte in turn, and before it wrote it out. From that I found out that the buffer was changing in between lines. That meant that an interrupt task was doing it. Further puzzling out and seeing 0xFFF9, then counting on my fingers to see what number that is in signed decimal, revealed that it was -7, a key number in calculating CCSDS packet sizes. However, it wasn't writing bytes 4 and 5, where it would be if the packet started at byte 0 in the buffer. Instead it wrote one byte before that, implying that the phantom packet started at -1, a key number in the circular buffer pointers. Once the head pointer is one less than the tail pointer, the buffer is full.

For a while I thought that the code was starting a new packet before it finished an old one, which would result in weird numbers. So, I put in some code which blinklocked the device if a packet had not been finished before the next one was started. That never hit. So, I put in some code which blinklocked when the circular buffer filled up, and that finally hit and proved it all. This interacted weirdly with the overlapping packet checker, so I had to fix a bug there. 

Now the only remaining known weakness is what we do if the first cluster of the root directory is full. We can make a cluster as big as 64kiB, we can handle multiple sectors in a cluster in the root directory, and we can fit 2ki entries into a cluster that big. But, what if we fill it? The file search code will need to know how to go to the next cluster, and the file creation code will need to know how to allocate a new cluster. Or, the format process can write 10000 files then delete them all.

The flight version of the code will not blinklock, but reset, when blinklock() is called. Between that and gathering enough runtime, I can gain some confidence that the device won't lock up.

I set the circular buffer size back to two sectors, down from the 6 I had given it. Next version will put the circular buffer into USB ram to save space for the stack, and allow me to have up to 16 sectors. USB memory doesn't work for some reason... maybe the USB system needs to be on for it? Then we are exchanging memory for power consumption. I know that the system is dropping packets now, more so than it was at 6 sectors.

Tuesday, June 18, 2013

Not another Rocketometer

I got an Arduino Leonardo and a Sparkfun Pro Micro as breadboard models of the Arduino-based Rocketometer. Based on my experience with those, I conclude that the ATMega32U4 and the current bootloader are Not Ready for Prime Time, and Not Ready for Spaceflight. Further, I would have had to write my own USB Mass Storage driver anyway.


Wednesday, May 29, 2013

Yet another Rocketometer

The Arduino Leonardo can effectively be thought of as an Arduino on a chip, except that an Arduino pretty much already is one chip. What makes a Leonardo special is that its core is an ATMega32U4, which has a USB port. The chip can be programmed as normal over ISP, and once it is programmed with the appropriate Arduino bootloader, it can be programmed over the USB port. The Arduino library for Leonardo also supports using the USB as a CDC serial port, and has much other USB support.

The chip only runs at 8MHz. It can run at 16MHz at 5V, but I am planning on running it in the Rocketometer, which only has 3.3V available, which limits the chip to the aforementioned 8MHz.

Now how do I think I can get away with this? Simple: Even with the LPC2148 running at 60MHz, it was spending most of its time waiting for the bits to crawl across the I2C and SPI buses. And how much can we optimize a busy wait loop?

I bought a Leonardo to practice with, and a Sparkfun Pro Micro to practice some more, especially programming the bootloader. I will not be using the 6-pin ISP header on the Rocketometer, but I got the great idea to use the MicroSD sniffer to get at the SPI signals through the Rocketometer MicroSD slot, and I can just hold down the reset button, or solder one wire to the contact of the Reset switch - which reminds me, I should put the solder bridge on the top side of the reset switch. In that case, I can use the solder bridge itself as a temporary terminal, solder a wire to it long enough to program, connect the rest of the SPI signals to the microSD sniffer, then program it with an ArduinoISP.

While I was at it, I also designed a Precision clock around this new chip. It doesn't need a hardware UART to program or talk to the host computer, so no FT232 and no multiplexer is needed. The chip has more pins, so this design doesn't use literally every single possible pin (those two ADC-only pins don't count, since they are not possible to use in this project).

Monday, May 13, 2013

NTRS is back!

I am happy to say that I was wrong about NTRS. Plus, the EDL kernel has been released (at least is present on NAIF).

I know what we're doing tonight.