Saturday, June 27, 2015

Splitting a git repository

I keep all my hobby code in a big git repository. The problem is that it is too big. I have something like 28GB in it, of which the vast majority is data, by which I mean results of data-collection processes, such as photos, rocketometer data, etc. It also includes data acquired from other sources, such as spice kernels, shape models, image maps, etc. It is making everything slow.

So, the solution is to split the repository. We split it into code, data, and docs.

Now, how do we split a repository? One option is to use the data we have to build new repositories. We go through the existing one commit-by-commit, generate a diff, filter that diff so that only the right files go into the new repository, then make a new commit in the new repository, making sure to keep the commit message, user information, and timestamp.

That seems hard. Also, it seems like someone should have already done that. So, I did some research to see if anyone else has done this. Mostly I am looking for people who are permanently removing files from a repository. My idea is to make three copies of the existing repository, then remove the files that don't belong using the methods described on the Internet.

In the course of my research, I found a program called BFG. This program does a full remove of a large set of files, in a way that is much quicker than git filter-branch. In order to do this, we need to remove the files from the head of the repository, so we do things like this:

  1. Make three copies of the original workspace with git clone --mirror . These are going to be the three new repositories, so call them code.git, data.git, and docs.git
  2. For each repository, check it out with git clone (no --mirror). This working copy is temporary. We will use the code repository as typical.
  3. Because of the way BFG works, we have to rename any folders which are called Data or Docs. BFG doesn't remove just /Data, but any folder called Data. As it turned out, there were some such files. They needed to be either renamed (to data and docs, note lower case), deleted, or moved.
  4. Remove the files from the working copy with git rm -r Data Docs .
  5. Move all the files and folders from code down one level, since the old code folder will be the new root: git mv code/* .; git rm code
  6. Commit and push the removal and move
  7. Now go back to the new mirrored repository code.git and run java -jar ~/bfg-1.12.3.jar --delete-folders Data and java -jar ~/bfg-1.12.3.jar --delete-folders Docs . This process runs very quickly.
  8. At this point, the repository is cleaned, you can't see any evidence of the files, but there are garbage objects which need to be removed to actually make the repository take less space. To do this, run git reflog expire --expire=now --all --verbose and git gc --prune=now --aggressive . Git garbage collection takes a long time and an immense amount of memory, more than the 8GiB my file server actually had. I had to run the gc over the network on my game rig, which has 32GiB.
  9. Now the repository is shrunk. We went from 28GB to 3GB for the code repository.

Friday, June 26, 2015

More on sensor lag

I have done further analysis of the practice runs, and also hooked up the PPS and recorded some data while fixed.

From the PPS data, the robot takes an average of 0.54s from the time that the PPS fires to the time that it finishes receiving and parsing the RMC sentence. This is true every 4 seconds out of 5, but every fifth second, the GPS sends a burst of GSA and GSV sentences, which apparently delays the RMC to an agonizing 0.96s.

During practice run 3, the log file YUKARI02.SDS was recorded when the robot ran into the barrel. During its run on the second leg, it reached a peak speed of 6.57kt, or over 354cm'/s. It had a sustained speed on the second leg of over 300cm'/s . This is around 3m/s.

So, the sentence is typically processed 0.54s or 150cm' delayed, but up to 0.96 or over 300cm' delayed. Plus, the fixes are uniformly randomly distributed relative to the actual time of waypoint passage, with the delay equally likely to be any time between 0 and 1 second. This is another 0-300cm' of delay, meaning that the robot will detect and react to the waypoint between 1.5m and 6m beyond. I pulled the waypoint back 6.6m when I did the short waypoint test.

Tuesday, June 23, 2015

Yukari 4 and nowcasting

I have been convinced to do Yukari 4, by of all people, my lovely wife Kellie. I have a theory about what is wrong, and how to fix it with one wire and a software change.

Hey mom! I'm on the Internet!

I just noticed this picture on the Sparkfun AVC2015 site:

That's from AVC 2014, and that is Yukari II, breadboard and everything. Those are my toes on the right.

Friday, June 19, 2015


This thing might actually work. This morning the biggest problem I was having with it was the fact that the GPS was going crazy. I think it is multipath, and I think putting a big plane of copper below the antenna will work.

In the mean time, I have calibrated all the gyro axes, which needed a 15% scaling in order to read accurately. I trust the gyro compass far more than the GPS at this point, so I will tune the configuration parameters such that the GPS heading is ignored.

Wednesday, June 17, 2015

Operations Manual

The instructuions for the robot in its current state, which might actually work.

  1. The robot has two batteries, one for running the motors and one for running the controller. Make sure that both are charged. Use the motor battery charger to charge the motor battery. The battery may get hot during a charge, so its instructions say to charge it outside the vehicle. The controller may be charged either with the 5V/3.3V FTDI cable plugged into its jack, or with a USB MiniB plugged into the jack on the Loginator. Only do one of these, and don't try to charge with a 3.3V FTDI module.
  2. Make sure that the SD cards are in place. The Loginator card is the 16GB red/gray Sandisk card. It is a good idea to format it (backing up old data first, of course). Make sure that it has the proper CONFIG.TXT for the course. 'make format' will do both of these things. The camera card can be formatted using the menu of the camera.
  3. Set the switches as follows: SW1 (GPS TX->Loginator RX) is ON, SW2 (FTDI TX->Loginator RX) is OFF, SW3 (Boostrap mode) is ON.
  4. Make sure the GPS is in lock. You should see the red PPS light on the GPS interface blink once per second.
  5. Turn on the Loginator. You will see the light blink green or cyan, as the GPS and bootloader try and fail to talk to each other.
  6. Switch SW3 to OFF. This will boot the robot into the main firmware the next time that the controller is reset
  7. Turn on the motor power by sliding the switch forward. The green light on the ESC should blink but then stay off. If the steering is off-center, manually turn the wheels to center.
  8. Take the robot to the starting line. Approach the line from behind, and walk for several seconds in the direction the robot needs to go for the first leg. This sets the GPS heading appropriately.
  9. Set the robot on the line.
  10. Turn on the camera and start recording.
  11. At the starting signal, push the RESET button. This will boot the robot and it will take off within 2-4 seconds.
  12. When the robot comes to a halt, either due to finishing or to hitting an obstacle, turn the motor power off by sliding the switch backwards.
  13. Turn off the Loginator.
  14. Stop recording on the camera and turn it off.
  15. Remove the card, plug it into a computer, and copy the data from the card on to the host computer
  16. Use extractPacket and a spreadsheet to analyze the data. Use the NMEA files produced to drop into Google Earth for map visualization.
  17. Pull the MicroSD card from the camera and copy the results to the host.
The intended startup is slightly different:

  1. Make sure both batteries are charged as above.
  2. Make sure the SD cards are in place as above.
  3. Set the switches to SW1 ON, SW2 OFF, SW3 OFF (changed from above). We will bring the robot up into the main firmware, because it won't go until we push a button.
  4. Make sure the GPS is in lock as above.
  5. Turn on the Loginator as above. The red light for the LED should come on also, which means that the light will blink red/white. This is not an error condition - Those are indicated by a red or blue light blinking out a blink code at one second on, one second off. The robot is now recording GPS and inertial measurements, and actively sending zero throttle and steering commands.
  6. SW3 will already be off.
  7. Turn the motor on as above. It should center the steering automatically, and the drive wheels should not spin. The green light on the ESC should be steady on.
  8. Take the robot to the starting line by any route. The robot will no longer use GPS heading to initialize
  9. Set the robot so that it is pointed as accurately as possible towards the first waypoint. The robot assumes that its initial heading is towards the first waypoint, and uses that as its initial reference.
  10. Turn on the camera and start recording as above. Do this at least 5 seconds before the start of the race. Any jiggling is in principle remembered forever and incorporated int Average G, but in practice is more than 99% forgotten after 2.5 seconds.
  11. At the starting signal, push the STOP button on the Loginator or the green button. Do this quickly, since if the robot is jiggling before half a second before you push the button, the robot will incorporate that jiggling into its average G. We don't want that.
  12. When the robot stops, turn the motor off as above.
  13. Turn off the Loginator as above.
  14. Stop camera recording as above.
  15. Read the log as above.
  16. Use the analysis tools as above.
  17. Copy the camera video as above.
To burn a new program into the robot

  1. Set SW1 to OFF and SW2 to ON. Don't let both SW1 and SW2 be on at the same time.
  2. Set SW3 to ON.
  3. Plug in the FTDI cord to its jack, paying attention to the colors. The black wire is closest to the corner. Make sure there is no USB cord plugged into the Loginator USB jack.
  4. Reset the robot by pushing RESET on the Loginator. The robot should come up in bootloader mode.
  5. Use 'make program' to program the robot.

Tuesday, June 16, 2015

Heading Reference

The hardest thing is getting an accurate heading reference. There are at least 4 possible sources:

  1. Magnetic compass - We are carrying one, but it is close to the motor. I'll have to check some of the data to see if the motors influence it a lot.
  2. Gyroscope - Subject to random walk drift and needs an accurate starting heading
  3. GPS heading - The GPS calculates a heading itself and reports it in RMC
  4. GPS delta-position - Use trigonometry and the position 1 second ago
 A proper solution would use a Kalman filter. I haven't gotten that to work on the robot yet. So, I am using primarily 2, with 3 to initialize and 4 to correct when in motion. It goes like this:

  • We initialize the gyro using "Average-G". This is the average of the last 50 gyro readings on each axis. This average is subtracted from the in-motion readings to produce a zero-biased rotation rate
  • We use the first RMC sentence to get our initial heading
  • Once in motion, we use the gyro as the primary heading reference. We set up an inertial frame parallel to the initial orientation of the robot, and an initial identity quaternion. We use normal quaternion integration to integrate the gyro readings and get a new gyro-based orientation. We use that orientation to transform the vector pointing forward (the nose vector) from the body to the reference frame. The gyro heading is then the angle between the initial nose vector (same as the body nose vector) and the current nose vector. We use the initial heading to calculate the offset between the gyro heading and true north, and add this offset to the gyro heading to get the instantaneous heading relative to true north
  • Whenever a large maneuver is executed, IE a large yaw rate is sensed, a countdown is set to 400 readings (4 seconds). When this countdown expires, we use method 4 to get the heading if the speed of the vehicle is high enough. We use this heading to calculate a new heading offset.
Improvements are as follows:

  • We know the initial heading from the first two waypoints. Provided the robot is accurately aimed toward the waypoint, it will be plenty good enough.
  • We are going to add a green button. Before the button is pushed, the robot will continually be recording the gyro readings, but will remember the last 50. Each reading, we will pop one reading out of the queue, and average it with the running total. The weighting will be 1 point for the new reading and 49 for the old average. In effect this takes the average of all the measurements from robot reset, but each old measurement has an exponentially decaying weight. We include some code to wait for 49 measurements before using a weighting of 49. We remember 50 measurements because the very act of pushing the green button will cause the robot to shake and will ruin the current measurements.
With these improvements,  I expect the initial heading to be much better and I expect to be able to run almost entirely on gyro readings.