Shipometer progess

 We are on Shipometer 23.04 now. Shipometer 23.03 is a failure, for perhaps multiple reasons.

First, the shipometer is a HAT for a Raspberry Pi that can carry a ZED-F9R GPS+IMU, a BME280 pressure/temperature/humidity sensor, and an ICM20948 9DoF sensor. Even though the GPS+IMU has an IMU built into it, I want the ICM20948 because I can control it at a lower level, and because it has a magnetometer in it as well. 

Finally, it carries an LPC210x as a precision timer. The ZED-F9R generates a pulse-per-second (PPS) which is accurately timed to have its rising edge right at the top of each second, with sub-microsecond accuracy. This PPS is routed to and captured by the Pi, but is also routed to the timer capture input of the LPC210x. This microprocessor is an old ARM7TDMI, but has a hardware 32-bit timer capable at running at 60MHz, with multiple usable capture inputs. The LPC210x is programmed to count at 60MHz, reset after 3.6 billion cycles, capture several inputs, and output the exact timer count of each pulse to its serial port, which is wired to the Pi UART1. This way, the time of each PPS and sensor data-ready signal can be recorded with no latency due to things like interrupts and real-time software.

Today's Episode of "It's my responsibility, but..."

"...The datasheet diagrams didn't have the pins labeled."

This time, it is the encoder board.

Here, it isn't obvious which pad in the footprint goes with which pad on the part.

It doesn't matter anyway, because a closer examination of the footprint and the Digikey list of optical sensors reveals that I bought the wrong part. The footprint on the board is marked QRE1113, but it is actually for an Omron EE-SY193. D'oh!

Well, back to the fab again, for both boards. I'll use the QRE1113 parts that I have to test if the parts even fit in the hole. If not, I'll have to use the EE-SY193.

All parts of this project are my responsibility. It can't be otherwise, since there isn't anyone I report to or who reports to me. My partner is a special case.

Dagnabbit!

That beautiful purple board I received yesterday won't work. If it had been stuffed and plugged in, it would have immediately shorted out anything plugged into it.

One of the features on the board is a super-wide (for a 6-mil PCB) strap between the two adjacent 5V pins coming from the Pi. When I had a close look at it, I saw a thin little gold crescent around part of the hole. It took a little bit for it to dawn on me that this was the ground plane, which was visible through the mask, plated in gold, and not protected from shorting with the other contacts:

 The effect is everywhere on the board. I saw it first on the strap in the upper-left, but this magnified version shows it on the motor power section and connection to the Arduino. Note the crescents around the top of D3~ and RST, and the slivers of ground plane visible through the mask around the isolator footprint.

I don't think that there is anything that can be done to fix it. I also don't think that the encoder board is affected, so at least I can still use that. I'll try to stuff it, but I will check continuity closely. If any of the solder bridges the 6mil gap, then the short will exist.

I'm pretty sure the problem comes from settings in the ground plane. Since OSHPark can make 6mil boards, I take full advantage of the feature. Unfortunately, this interacts with a bad default in Kicad. Fortunately, that is easy to change, but it would have been nice to know fifteen dollars and two week ago. The money isn't a big deal, it's the time.

Kicad was even trying to tell me that something bad was about to happen. Not from the DRC (although that would have been nice) but in the images. Here is the old, bad design:

The purple rings around the pads are the soldermask gap. You can see a slight red tinge around the edge where the mask gap overlaps the ground plane.

The OpenGL preview shows it too, perhaps in an easier-to-understand form.

To fix it, use the Dimensions/Pads Mask Clearance menu option

Change the Solder mask clearance value from its default... (Yes, I design my boards in US units. You got a problem with that?)

Change it to zero.

The borders of all the pads are now black, indicating the solder mask gap doesn't span the space between the pad and the ground plane

 This option has settings in several places: The global setting I talk about above, the zone setting (which I haven't found), the footprint setting, and the individual pad setting. The later settings in the list have priority over the earlier ones, but the earlier ones are used as defaults.

In general with modern fab processes, you should set this value to zero. This will describe a hole in the solder mask the exact size of the pad. The fab can and will edit this to match their process, so don't worry about getting it too small.

Thanks to a YouTube video by My 2uF for help finding this setting.

Hooray for distributed backups!

BACK UP ALL THE THINGS!

I found a backup of my database from April 21 of this year, which was only a couple of days before the failure. That should have all of my wikis and gallery data, and therefore represents the second-most important data I have. The most important is the code, and that is backed up by means of git. I know for sure that there is a valid git repository on one of my portable USB disks

Yet Another Episode in the Annals of Data Stewardship

Having learned my lesson from before, I did not set up my filesystem as one big raid0. I did a btrfs raid5 instead. When one of the disks finally did give out, it wasn't with the click of death I heard before, but with read errors. The btrfs degraded, and by mounting read-only in recovery mode, I was able to use the two good disks in order to get my data.

Or so I thought.

A word on the issue I was having. I was seeing "stale file handle" warnings, of the type you see when you are in a folder that is NFS mounted, after you lose connection. But, this wasn't an NFS system. I rebooted the system and it wouldn't come up, because the btrfs refused to mount. After manually mounting in degraded mode, many of the disk accesses reported errors in dmesg, about the generation of certain metadata being off of the expected value, often by hundreds or thousands of generations.

First, I decided that I had lost confidence in btrfs -- if it wasn't going to keep working in the presence of a disk failure, what was the point? I spent the next several days scraping data off of the btrfs and putting it wherever I could find a place for it - on the USB disks I have, on other computers, on the system disk, etc. I then replaced the bad disk and formatted them all as zfs - now possible since Ubuntu 16.04 includes a native zfs driver.

Finally, I started copying data back onto the zfs. All appeared to go well, until I tried to bring up the wiki. The LocalSettings.php file was completely blank - it had the expected value, but all bytes in the file were 0x01 . Hrm.

Turns out a lot of files were like this. Files I care about, like the database, the git repositories, etc. It seems like the newer the file is, the more likely it is to be damaged like this.

No problem, I've got backups. A raid5 is not a backup, so I had the most important data copied off onto several other systems.

Or so I thought.

My backup script runs on a cron every night, and had backed up the bad data and spread it all around over the good data.

Oops.

It isn't a total loss. I have all my code in a git repository on the big USB disk. I have an old backup (from December, I think) of all the data I considered important. I did lose a lot of video :( but I don't think I lost anything from Florida 5.

So I think.

Check PCLK measurement

Check if the user code measures PCLK properly. If it doesn't, then the baud rate calculation will be wrong. Since one of the symptoms that has been seen is that the RX light on the FT232 doodad flickers, but no characters appear in putty, it is possible that the baud rate isn't what we think it is.

Consider also calculating the baud rate registers and stuffing them manually. If this works, then it's the PCLK stuff that is broken.

The Secret to Success

1. Pick something you like doing.
2. Do it and do it and do it until you don't like doing it any more. This will always happen at some point.
3. Keep doing it.

Following these steps don't guarantee success, but failing to follow them guarantees failure.

Splitting a git repository

I keep all my hobby code in a big git repository. The problem is that it is too big. I have something like 28GB in it, of which the vast majority is data, by which I mean results of data-collection processes, such as photos, rocketometer data, etc. It also includes data acquired from other sources, such as spice kernels, shape models, image maps, etc. It is making everything slow.

So, the solution is to split the repository. We split it into code, data, and docs.

Now, how do we split a repository? One option is to use the data we have to build new repositories. We go through the existing one commit-by-commit, generate a diff, filter that diff so that only the right files go into the new repository, then make a new commit in the new repository, making sure to keep the commit message, user information, and timestamp.

That seems hard. Also, it seems like someone should have already done that. So, I did some research to see if anyone else has done this. Mostly I am looking for people who are permanently removing files from a repository. My idea is to make three copies of the existing repository, then remove the files that don't belong using the methods described on the Internet.

In the course of my research, I found a program called BFG. This program does a full remove of a large set of files, in a way that is much quicker than git filter-branch. In order to do this, we need to remove the files from the head of the repository, so we do things like this:

1. Make three copies of the original workspace with git clone --mirror . These are going to be the three new repositories, so call them code.git, data.git, and docs.git
2. For each repository, check it out with git clone (no --mirror). This working copy is temporary. We will use the code repository as typical.
3. Because of the way BFG works, we have to rename any folders which are called Data or Docs. BFG doesn't remove just /Data, but any folder called Data. As it turned out, there were some such files. They needed to be either renamed (to data and docs, note lower case), deleted, or moved.
4. Remove the files from the working copy with git rm -r Data Docs .
5. Move all the files and folders from code down one level, since the old code folder will be the new root: git mv code/* .; git rm code
6. Commit and push the removal and move
7. Now go back to the new mirrored repository code.git and run java -jar ~/bfg-1.12.3.jar --delete-folders Data and java -jar ~/bfg-1.12.3.jar --delete-folders Docs . This process runs very quickly.
8. At this point, the repository is cleaned, you can't see any evidence of the files, but there are garbage objects which need to be removed to actually make the repository take less space. To do this, run git reflog expire --expire=now --all --verbose and git gc --prune=now --aggressive . Git garbage collection takes a long time and an immense amount of memory, more than the 8GiB my file server actually had. I had to run the gc over the network on my game rig, which has 32GiB.
9. Now the repository is shrunk. We went from 28GB to 3GB for the code repository.

AVC 2015 Wrapup

OR: Will there be a Yukari 4?

Multipath

This thing might actually work. This morning the biggest problem I was having with it was the fact that the GPS was going crazy. I think it is multipath, and I think putting a big plane of copper below the antenna will work.

In the mean time, I have calibrated all the gyro axes, which needed a 15% scaling in order to read accurately. I trust the gyro compass far more than the GPS at this point, so I will tune the configuration parameters such that the GPS heading is ignored.

Heading Reference

The hardest thing is getting an accurate heading reference. There are at least 4 possible sources:

1. Magnetic compass - We are carrying one, but it is close to the motor. I'll have to check some of the data to see if the motors influence it a lot.
2. Gyroscope - Subject to random walk drift and needs an accurate starting heading
3. GPS heading - The GPS calculates a heading itself and reports it in RMC
4. GPS delta-position - Use trigonometry and the position 1 second ago

 A proper solution would use a Kalman filter. I haven't gotten that to work on the robot yet. So, I am using primarily 2, with 3 to initialize and 4 to correct when in motion. It goes like this:

• We initialize the gyro using "Average-G". This is the average of the last 50 gyro readings on each axis. This average is subtracted from the in-motion readings to produce a zero-biased rotation rate
• We use the first RMC sentence to get our initial heading
• Once in motion, we use the gyro as the primary heading reference. We set up an inertial frame parallel to the initial orientation of the robot, and an initial identity quaternion. We use normal quaternion integration to integrate the gyro readings and get a new gyro-based orientation. We use that orientation to transform the vector pointing forward (the nose vector) from the body to the reference frame. The gyro heading is then the angle between the initial nose vector (same as the body nose vector) and the current nose vector. We use the initial heading to calculate the offset between the gyro heading and true north, and add this offset to the gyro heading to get the instantaneous heading relative to true north
• Whenever a large maneuver is executed, IE a large yaw rate is sensed, a countdown is set to 400 readings (4 seconds). When this countdown expires, we use method 4 to get the heading if the speed of the vehicle is high enough. We use this heading to calculate a new heading offset.

Improvements are as follows:

• We know the initial heading from the first two waypoints. Provided the robot is accurately aimed toward the waypoint, it will be plenty good enough.
• We are going to add a green button. Before the button is pushed, the robot will continually be recording the gyro readings, but will remember the last 50. Each reading, we will pop one reading out of the queue, and average it with the running total. The weighting will be 1 point for the new reading and 49 for the old average. In effect this takes the average of all the measurements from robot reset, but each old measurement has an exponentially decaying weight. We include some code to wait for 49 measurements before using a weighting of 49. We remember 50 measurements because the very act of pushing the green button will cause the robot to shake and will ruin the current measurements.

With these improvements,  I expect the initial heading to be much better and I expect to be able to run almost entirely on gyro readings.

(Real life) Failure

Time to test the robot in the real world. Because it is faster than I am, and because I don't trust it, and because I am testing it on the open road, I am testing it by carrying it, walking with it, and turning myself and the robot as its steering wheels turn. Here is the first result: The green line is the intended course, and the blue track is the measured track (it actually started almost exactly on the start of the green line, and never travelled towards the west).

Notice that it didn't turn. Analysis of the guidance packets suggests that the robot was only going at one tenth the rate we expected. As a consequence, when it reached the waypoint in reality, it only thought it had reached one tenth of the way to the waypoint. What happened? It seems to be a problem related to parsing the GPS data. The simulator gave out data with 7 digits after the decimal place, while the real GPS uses 4. In both cases, the intent was to force the length to 5 digits. It turned out that the code was forcing the part after the decimal point to 4 digits, if the number needed to be stretched, which is the case for real data and not for simulated data. The effect was that the fractional minute part was only 1/10 of what we expected. As it happens, the test course does not cross a whole-minute grid line, so the gigantic leap when it thinks it is going from 11.09999 to 12.00000 doesn't happen. The robot only thinks it is going 1/10 as far as it really is going.

Furthermore, this test inspired a new simulator/reality hybrid, called playback. The playback program is quite similar in structure to the simhost program from before. The difference is that it reads a recording from a real robot run rather than simulating from scratch. Whenever a gyro packet is encountered in the recording, a simulated gyro reading with the same DN and timestamp is created. Whenever an NMEA packet is found, this is stuffed into the virtual GPS serial port. In both cases the simulated timestamp is set to the time indicated in the packet.

I was able to put the actual robot code under the microscope with gdb and the playback seen above, and by stepping it there, found the problem with the NMEA parsing. I am not able to use playback to completely test the fix, but I can check if the robot tried to turn at the waypoint.

Results:
The simulator was adjusted to only give 4 digits after the decimal and was re-run against the actual AVC course, to see that this still works, and it does.

The donuts are "victory donuts" done after the robot crosses the finish line. Victory donuts are a bug, but they are after the robot crosses the finish line.

The playback also changes waypoints at about the right time, if I actually have it using the right course. Too many times I ran the playback with the AVC course rather than the local test course.

Cleaning up the code

"All the most important mistakes on a project are made on the first day"
--Old design adage

"There is no teacher but the enemy. No one but the enemy will tell you what the enemy is going to do. No one but the enemy will ever teach you how to destroy and conquer. Only the enemy shows you where you are weak. Only the enemy tells you where he is strong. And the only rules of the game are what you can do to him and what you can stop him from doing to you."
-- Mazer Rackham

"Reinhardt regarded the mysteries of the Universe not as indifferent questions of physics or chemistry, but as implacable, malicious foes. They were to be assaulted with science, vanquished at any cost, forced to yield their treasure house of knowledge."
-- from The Black Hole by Alan Dean Foster

In any interesting problem, you don't know enough about the problem to begin with to even form a plan for solving it. You do the best you can, and in the process of solving the problem, you learn what you should have been doing from the beginning. If time and resources weren't an obstacle, you could go back and solve the problem right from the beginning. But, usually for any interesting problem, time and resources are an obstacle, and you are stuck with what you started with. At each point in solving the problem, you only change what came before to the absolute minimum degree necessary.

As a result, sometimes legacy design choices make the solution quite a bit more complicated than a clean solution would be.

Now I have time and resources -- a whole year of time. It's time to clean house.

AVC 2014 Event Report

Or: Give Up! It's time for you to throw in the towel, capitulate and raise the white flag...

I made it farther than I did last time. I made it to the starting line. I turned on the robot, and watched it run straight as an arrow into the fence. It looked like it didn't even try to make the turn.

After that first run, I looked at the wiring and misread it. Since the LPC2148 is short on pins, all of the PWM outputs are multiplexed with other functions, functions it turned out I needed. PWM 1 and 3 are also UART 0, PWM 4 and 6 are also UART 1, and PWM 2 and 5 are also part of SPI 0. As it turns out, I needed all of those things, so I had to pick one to discard. I would have liked the GPS on UART 1 so that I could program the device over UART 0, but since that was the one thing that was possible to do without, I had to give it up. I had the GPS wired to UART 0, but only when it was in robot mode. When it was in bootloader mode, UART0 had to be connected to the host (through the FTDI cable). When I glanced at the board after the run, as I said, I misread it. I thought that the wiring was set up for bootloader. This would have made it impossible to do waypoint navigation. It would have resulted in the robot running straight as an arrow forever (or until it hit the fence).

As it turns out, it did try to make the turn. Like I said, I had misread the wiring and the GPS was fine. I could see from the logs that at about the right place it signalled to turn -100 (out of the full servo range of -128 to 127). It wanted to turn -600, but of course it was limited. I was let down by the steering mechanism again, but this time it wasn't a twitchy servo, it was my own circuit.

I was worried about running a solderless breadboard as the base of the robot controller, so I got some matching solder protoboards, but ended up never using them, mostly because I didn't want to commit the Loginator to this project. I didn't want to solder things down, so I ended up with two short breadboards stuck side by side. One supported the Loginator and FTDI socket, while the other supported the GPS interface and the opto-isolators for the servo control. The opto-isolator was the weakest part, partly from the sheer number of wires in such a small space, partly because I had to switch the channels at the last minute.

During the test run before the race, the thing was running long and seemed to want to turn only after I had picked it up. It might have already had this fault at this point. I have the logs and can analyze at this point. Soldering down the opto-isolator and its wires probably would have worked.

In any case, I was looking for an excuse to give up. It is one thing to surrender before you have to, and another to realize that there really is an insurmountable obstacle. During the test run on Friday, I had manually driven the thing around the track and flipped it on the finish line. The board popped out, but was apparently undamaged. I was almost hoping that the board was damaged beyond repair so that I could honestly say that I was beaten, rather than had given up.

I had convinced myself on Saturday that the steering problem was in the former  class. I had convinced myself that I needed to solder the whole thing to a board to get it to work. Thinking back on it now, if I had disassembled and reassembled it, it very well might have worked. I had the solder protoboard with me, and I could have borrowed a soldering iron. I could have done it there at the race.

This means I gave up too soon. I haven't finished. I will have to go back next year. Joseph is interested, so maybe I can use him as the motivation I need. Maybe he can help with Yukari, maybe he will help with another 'bot. Yukari is so close, I can feel it.

I did meet some nice people. Team Bloomberg was a dad with his family who had driven out from Wisconsin. Team Deep Space was an RC car with a robot controller, pretty similar to mine, except he went with odometry and a 4WD chassis. However, it had a cool hat, a flying saucer.

All-Nighter

If I was a week ago where I am now, or even a day ago, I would be in great shape.

As it is, I gave the robot its first free run, about 6 feet down the driveway. I also walked it down the street course, holding it up, checking that the wheels turned and roughly following its steering. It found the waypoint, turned around, and headed right off the other end. I may need to have a catcher's mitt for this one.

Expectations:

1. I'd like to win
2. I'd settle for finishing
3. I'll get the one-corner program.

I am going with two boxes, one computer bag, one live and one dead robot and one robot hat. The boxes are filled with a whole bunch of random stuff, not even in the hopes that I need it, but in the hopes that someone else is helped out by my being there. I have programmed the robot with my laptop, so I know I am at least ready for the nominal case.

Capacitance Loss

I didn't learn this one myself the hard way, but that's only luck. I very well could have.

I use 100nF (why doesn't the nanofarad get any respect?) capacitors all over the place on my boards, mostly as "filter" or "bypass" caps. They sit next to each Vcc pin of each digital IC, and the way I have heard it, they act like a little bitty power supply right next to where you need it. If the part needs a bit more power than average, it will suck charge off that capacitor before it sucks current off the Vcc rail. The part is protected from its own variations in power, and the rail (and therefore the other parts) are protected from this part.

I also use much larger caps as specified for regulators and other such power supply and sensor analog devices. The sensors want a much smoother power supply, and the regulators and such use them for feedback. These tend to be 1μF or even larger. Some parts call for as much as 10μF. The USB spec says no more than 10μF equivalent on a device across Vcc to ground, since too large of a value will draw a large inrush current as those capacitors are initially charged.

I tend to use ceramic caps for all of these, because I am obsessed with board space. Besides, a ceramic cap is pretty close to ideal. No polarization to worry about, etc.

Our good friend over on the EEVBlog demonstrates an issue which does bite ceramic caps -- capacitance change with applied voltage. I had heard of this before, but effectively ignored it as something I couldn't do anything about. Ignore the accent and presentation quirks, I am sure I would sound even worse.

He takes a 10μF 6.3V 0805 ceramic capacitor and demonstrates how to measure it with an oscilloscope and an RC circuit. With a 0-5V square wave, he demonstrates that the cap has its rated 10μF. But, when he changes the signal to 5V-6V, the capacitance drops to less than 5μF.

At first I breathed a sigh of relief when I saw the first demo. No capacitance loss even with a 5V signal. But then I thought about how the second demo applies to my circuits. A bypass cap is run with a large DC bias -- the Vcc voltage. This is directly relevant to my interests.

I guess all I can say is that the designs I am following call for a certain size capacitor. When these were tested by their original manufacturer, they either took these effects into account, or didn't, and just put in a larger cap than they needed when the circuit didn't originally work. If the design turns out to call for only 3μF, they specify a 10μF cap knowing that it will still have at least 3μF under the prevailing conditions.

More details at www.murataamericas.com/murata/murata.nsf/promo_dcbias.pdf .

Code A113

I was thinking about this at work, explaining it to one of the students I work with. He had never seen Wall-E, so I explained it in a generic and non-spoilery way. Since no one but me reads this anyway, I am going to write it the spoilery way here. You have been warned.

In that story, the systems on Axiom are programmed to look for habitability on Earth, and when it is proven, the ship is to return to Earth. It's an interesting but very expensive backup for just a normal recall message.

So, the pseudocode looks something like this:

while in_space do begin
  if(check_earth_habitability()) {
    return_to_earth();
  }
  sleep(1 year);
end

The function check_earth_habitability() is incredibly expensive to evaluate, as it involves sending a spacecraft with several EVE probes all the way back to Earth. It's like us making an implementation of go_to_saturn().

Directive A113 instructed the autopilot to not return to Earth under any conditions. It could have been implemented by cancelling the entire loop (including never sending EVE back to Earth), but instead the implementation looks like this:


while in_space do begin
  if(check_earth_habitability()) {
    //return_to_earth();
  }
  sleep(1 year);
end

A normal person would ask "Why does Auto send Eve back to Earth year after year if it has been ordered to not return?" I ask a similar question "Why did they do the code that way?" The thing is, I have done things similar to this: Evaluate an incredibly expensive function, just to ignore almost all of its output.

An example from work is sim_sample. A lot of times I only care about a small part of the result of sim_sample, or I have already calculated the result and could just run viusalize_1c manually. I don't because it is easier to run sim_sample and wait for it to finish than it is to do it efficiently.

Bugs crossing Boundaries

I am getting the Rocketometer ready for delivery. The hardware appears to be fine, but the software had one particularly weird bug -- the name of the FAT volume on the SD card was changing. This meant that something was poking into the root directory block where it did not belong, which meant that it could be anything.

As it turned out, it was the packet buffer becoming full. Now how in the world did that corrupt the file system buffers? As it turned out, the file system by design wasn't using buffers, in order to save memory.

An SD card is addressed a sector at a time. Data can only be read and written in full sectors. So, if you want to change a single entry in the root directory or file allocation table, you have to read the whole sector into memory, change the bits you want, then write the whole sector back. Now the root directory and file allocation table only needs to be changed when the file is written, and in this case there is already a buffer which has just been used, so I foolishly decided to have the code use this.

In the mean time, this same buffer was part of the circular buffer accumulating packets. If the buffer ever filled up, new bytes would be rejected, and the head pointer would not move. If you start to write a packet to an already full buffer, nothing gets written, and when you finish, the packet length seems to be zero. Now the fun begins.

Upon finishing a CCSDS packet, the code counts how many bytes have been written, subtracts 7 because that's according to the protocol, then goes back and changes the buffer. In the current case, since no data had been written, it actually changed marked data.

Now we combine this with the fact that the interrupt task was filling the buffer. Here then is the final sequence:

1. The buffer is full. No more bytes can be written.
2. The main loop detects this, and starts to drain the buffer and write it to the SD card. It writes the packet, then reads the root directory sector, changes it, then writes it back.
3. During the write, an interrupt fires.
4. The interrupt task writes another packet. The bytes are dropped on the floor.
5. The interrupt task finishes the packet. The calculated length is zero, since no new bytes were written
6. The length to be written is -7, 0xFFF9
7. The length is written to the buffer, but because of timing, it writes it over the top of the root directory sector in memory. The root directory is now corrupted.
8. The interrupt task finishes
9. The SD writing code writes the corrupted root directory sector back to the card.
10. The card is now named RKT\FF\F9 instead of RKTO3.

It would be pretty easy for this same process to shoot holes anywhere in the filesystem metadata. This bug is hazardous!

So, to fix the bug, we do the following:

1. Give the file system its own scratch buffer, and never let it do scratch work with the user buffer. This doesn't actually fix anything, but it does make the bug, and any similar undiscovered bug, less dangerous.
2. Handle the full buffer case better. When the buffer is full, it throws away the new incoming bytes. Further, it backs the head pointer to the mid pointer, in effect throwing away the fractional packet accumulated before the buffer became full. Even further, since the buffer is now no longer full, to prevent getting the last half of a packet, we keep a flag that says that the buffer was full, and don't accept any more bytes until the flag is cleared by a drain operation.
3. The interrupt task checks if the buffer is already full. If it is, we don't bother to read the sensors, a time-consuming chore which will just discard the result and take time away from the SD writing routines.

This clears a major bug, perhaps one causing all the observed halts. A bug which punches random holes in the metadata can do anything. It only triggered recently when I cranked the data recording rate to 200Hz, which appears to be just beyond the equipment capability. Previous rocket flights must have run at lower rate and not triggered the bug, or not triggered it as hard.

So how did I find it? Lots of print statements and blinklocks. First, I had the SD driver print out the buffer it was going to write. Then I had the direntry driver (the thing which actually handles the root directory) print out its scratch buffer after it had been read, as it changed each byte in turn, and before it wrote it out. From that I found out that the buffer was changing in between lines. That meant that an interrupt task was doing it. Further puzzling out and seeing 0xFFF9, then counting on my fingers to see what number that is in signed decimal, revealed that it was -7, a key number in calculating CCSDS packet sizes. However, it wasn't writing bytes 4 and 5, where it would be if the packet started at byte 0 in the buffer. Instead it wrote one byte before that, implying that the phantom packet started at -1, a key number in the circular buffer pointers. Once the head pointer is one less than the tail pointer, the buffer is full.

For a while I thought that the code was starting a new packet before it finished an old one, which would result in weird numbers. So, I put in some code which blinklocked the device if a packet had not been finished before the next one was started. That never hit. So, I put in some code which blinklocked when the circular buffer filled up, and that finally hit and proved it all. This interacted weirdly with the overlapping packet checker, so I had to fix a bug there. 

Now the only remaining known weakness is what we do if the first cluster of the root directory is full. We can make a cluster as big as 64kiB, we can handle multiple sectors in a cluster in the root directory, and we can fit 2ki entries into a cluster that big. But, what if we fill it? The file search code will need to know how to go to the next cluster, and the file creation code will need to know how to allocate a new cluster. Or, the format process can write 10000 files then delete them all.

The flight version of the code will not blinklock, but reset, when blinklock() is called. Between that and gathering enough runtime, I can gain some confidence that the device won't lock up.

I set the circular buffer size back to two sectors, down from the 6 I had given it. Next version will put the circular buffer into USB ram to save space for the stack, and allow me to have up to 16 sectors. USB memory doesn't work for some reason... maybe the USB system needs to be on for it? Then we are exchanging memory for power consumption. I know that the system is dropping packets now, more so than it was at 6 sectors.

They took down NTRS

Back when NASA was testing the X-43, they said something that has stuck in my mind ever since. The X-43 is a scramjet test vehicle. The thing is boosted on a Pegasus to near mach 10, then the scramjet is fired for 10 seconds. Afterwards, the vehicle is not recovered. "The only thing we get back from this mission is data."

When you think about it that way, all the robotic space missions are like that. We just get back data. No artifacts, no samples, just bits. We then assemble those bits back into knowledge back here on the ground. When we talk about the $800M SDO mission, we are spending the $800M to get those bits, so they better be valuable. Furthermore, I paid for those bits so I should have access to them.

The NASA Technical Report Server is where the knowledge collected from those bits go. It is now closed.

They claim that it is due to an ITAR review, and that it will be reopened when that is complete. There are over a million papers on NTRS. That is never going to get the kind of review they are implying.

So: For all those mission that only send back data, the only evidence that we have that they even happened is now gone. We spent $XB a year on NASA, and what we paid for is now locked up. This is a bad precedent. Furthermore, the NASA library is now untrustworthy. We build libraries to hold knowledge, and if they close at random without notice, they fail to serve their purpose.

Similarly, I now don't think that the kernels for reconstructed entry of MSL will ever be released.

This is not a good way to start the day.

A review of the ADXL345 and BMA180

First off, the ADXL345 is exactlty what it claims to be. It's my own fault for not reading the data sheet, or rather reading it but not comprehending the information in it.

So, my first experience with digital accelerometers was with the Bosch BMA180. The part had 14 bits of precision and 6 scale settings, from 1g to 16g. When you switched to the lower scales, you got more precision, as expected.

I finally got all 11 DoFs working on the 11DoF board. The first SPI part I broughs up was the ADXL345, so I learned about its SPI protocol that way. For one thing, those of you used to I2C, SPI is different! For instance, on both the ADXL345 and the gyro on the 11DoF, the L3G4200D, the register addresses are six bits. This is fine, since that is a big enough space. You send this address as the first word of any transfer in either direction from these devices. However, both devices use 8 bit words in the protocol. The other two bits control the direction of transfer for the other words, and whether the device is to expect more than one word -- that is, whether it should increment the register address pointer for the next words.

For instance, say you want to read all the measurement registers in the ADXL345. The register map says that this is registers 0x32 through 0x37. Since SPI is a full-duplex protocol, every time you send a byte, you receive a byte. So, send 0x32, ignore what you receive, then send six more bytes (doesn't matter what, I used 0) to read 6 bytes that you care about. Right? Almost, but not close enough. I did this first with the ID register, and it worked, sometimes. But, when I actually tried to read the data, I got back the same word six times. What gives? First, bit 7 of the address is read/#write. You have to set it to read the register, otherwise it interprets the MOSI data as data to be written to the registers. The data registers are read-only, so the part ignored me. Next, bit 6 is the multibyte flag. Set this bit if you are going to read/write multiple registers in one transaction (one continuous #cs assert). Doing this will cause the part to increment the address pointer each time it sends or receives 8 bytes. Since I set neither of these, my part became very confused, since I was telling it to write to a single read-only register six consecutive times.

tl;dr - Tell it you are using address 0xF2, then read 6 more times to get the 6 data registers.

This is an unnamed protocol layered on top of SPI, which by itself knows nothing of registers. It happens that the gyro uses the same protocol, so turning it on was a simple matter of verifying that the protocol was the same, and copypasting the code.

Now for the review. The ADXL345 has a programmable range, with choices ±2, 4, 8, and 16g. We will need 16g for the rocketometer. It has a readout precision of 13 bits, almost equal to the 14 bits the BMA180 gives. Now for the bad part. The readout precision is only 12 bits for 8g, 11 bits for 4g, and 10 bits for 2g. It is as if the part was always running at 16g, but reporting saturation if it was out of its current range.

I might as well use an analog part if I am only going to get 10 bits. I had always known this, but only realized the significance when I finally got it up and running, and only got about 250DN out of the part in the 1g field. So, the ADXL gets 2 of 5 stars, not recommended on the Chizumatic scale.

The BMA180 is what I used before, in flight. It has a programmable range, with choices ±1, 1.5, 2, 3, 4, 8, and 16g. It produces 14 bits of precision, and this 14 bits is constant across all ranges, so if you use the 1.0g range, you actually have zoomed in, and get better resolution than when you are at 16g. I can't speak to its accuracy, so it gets 4 of 5 stars, not recommended. Why not? The part was discontinued without a suggested replacement as of today, and is no longer available on Digikey. In fact, I am hoarding three of them still in their cut tape, purchased from Sparkfun today at great expense, not even on a breakout board. None of the other BMA accelerometers are as good, and none of the Analog Device accelerometers are as good either.