Totality 2024

 We almost didn't even make this trip. We had seen it before, and the eclipse was scheduled for a very inconvenient time for us.

The original plan was to drive to Oklahoma and stay with family. Their place in Oklahoma isn't in the path, but was close. So the plan was to drive from there to whatever part of the path that had the best combination of closeness and good weather. We had penciled in Dallas, since that was the closest spot on the path that would have good historical weather. The history showed that further south was more likely to be cloud-free.

As it happens, the weather was very different from the average. New England was clear, while Texas was cloudy with danger of severe storms. If we had gone with this plan, we probably would have headed towards Arkansas.

For various reasons, we didn't do that plan. For a while, we had decided to skip it altogether. However, another family event with different family came up. They live in Indianapolis, and the other event was the Friday before the eclipse on Monday. So our plan changed once again to driving to Indiana, then judging the weather there. We were willing to drive wherever we needed, but Indianapolis is already in the path so we were hoping for good weather there.

Three weeks ago, the longest-range forecast was excellent for Indiana, so we decided to go there. Ten days ago, the forecast was about the worst imaginable, with a frontal weather system almost perfectly along the path, cloudy from coast to coast. Since then, the forecast improved, with Indianapolis having a cloud cover of between 6% and 50% depending on the model and time out.

As late as yesterday I was still contemplating driving to Maine, but we couldn't have taken all the family. We decided to stay here and take our chances. If it was cloudy, we would just see the clouds get dark.

The weather report on Monday morning wasn't great. The front had passed overnight, but there were still lots of cirrus clouds, and I was worried.

As the eclipse started, the sky was still full of cirrus clouds. It wasn't affecting visibility of the partial phase. As usual, the eclipse was only barely noticeable until it got over 90%. 

The "party" as such was set up in the back yard, with a table and a bunch of lawn chairs. The rest of the party played cards while I did my usual anti-social thing. 

We made a pinhole camera with a cardboard box, foil, paper, and a glue stick. It worked fine, projecting an image maybe a couple of millimeters in diameter but clearly with a bite out of it. We also had a full complement of eclipse glasses and one solar binoculars. 

With the binoculars, I could see one sunspot near the center of the sun, maybe at 9:00. 

The soundtrack for the partial phase is "There's a little black spot on the Sun today"

The most interesting thing i remember from the partial phase of the 2017 eclipse was the crescent shadows cast by the trees. There were no trees in the back yard. The family had a number of kids, and I shouted out if anyone wanted to come out front with me to look for something interesting. There were several trees out front, so I did see the crescent shadows, but it also meant I had my wife and all of the family's kids out front with me during totality.

The memorable thing this time was that totality looked like a hole in the sky, like a portal. "There's a hole in the sky, through which things can fly." The eclipse itself was silent of course, but the soundtrack in my head was the sound of the "whale probe" from Star Trek 4. 

Also there was a clear red spark at about 6:30. This turned out to be a prominence. I only remember seeing it after a minute or so, but everyone around me saw it too.

The moon was the same color as the sky, and the corona was white. During the 2017 eclipse, there were several long corona streamers. I didn't see anything like that this time -- the corona was a relatively thin band, with a definite sharp inner edge and a fade-out on the outer edge, but still pretty thin. The sketch above is about what I saw, and a couple of my party agree it looked like that.

The sky in general was dark like twilight. I don't remember exactly how dim it got last time, but it might have been dimmer this time.

Totality was long enough for me to get a video of my party, a couple of telephoto images through my camera, and still experience the event fully with my Mk1 eyeballs.

I did see the diamond ring at third contact, for about 2 seconds before glasses on. I could still see the corona, but along with an overwhelmingly bright patch of sun. The corona was still visible on the opposite side.

Gravity Simulator #1,000,006

I've written a lot of gravity simulators. This one is the first time I've built one into a game engine. First, I created a Node2D and a Sprite2D with a texture so it's visible. I then attached the following code to the sprite node:

extends Sprite2D


var v:Vector2=Vector2(100,0)
var center:Vector2=Vector2(500,200)
var mu=3e6


# Called when the node enters the scene tree for the first time.
func _ready():
	var vec=Vector2(500,0)
	global_position=vec


# Called every frame. 'delta' is the elapsed time since the previous frame.
func _process(delta):
	var r:Vector2=(global_position-center)
	var a:Vector2=-mu*r/r.length()**3
	v+=a*delta
	global_position+=v*delta

The _process() function is a very simple Euler integrator. The acceleration is the standard test-particle gravity equation of motion, where the center is considered to be infinitely heavier than the test particle, such that the gravity of the test particle doesn't affect the motion of the center. 
The next obvious things to do:

• Use Runge-Kutta instead of Euler
• Gravitate towards another sprite instead of an invisible point
• Give both points finite mass 
• Gravitate towards any number of masses, to make an N-body simulator.

It looks like I might be right about how to do physics in Godot, but it still doesn't feel quite right. Godot includes ragdoll physics and joints. In order to integrate custom physics, I would have to be able to generate custom forces and moments, instead of directly doing the numerical integration myself. I expect something more like:

# Called every frame. 'delta' is the elapsed time since the previous frame.
func _process(delta):
    var r:Vector2=(global_position-center)
    var F:Vector2=-mu*global_mass*r/r.length()**3
    add_force(F)
    var M=... #calculate moment on the object
    add_moment(M)

This way if the engine does a higher-order integrator or has its own forces, this slides in nicely.

func _process(delta)

 I wish that GodotScript was just Python. It isn't, for a reason, but I don't know the reason yet. It could be:

1. GodotScript came first, or before Python was widely known.
2. Python is too hard to integrate. In this case, they considered *creating an entire new language interpreter* easier than incorporating Python
3. True Python implies that the entire marketplace of Python libraries can be used, and that may have been too difficult to achieve.
4. Python doesn't match the authors' internal model for how to do scripting
5. Godot was started as a personal project, and a scripting engine was one of the "fun" things that they wanted to do with it.

In any case, I have reached the part of the tutorial where scripting is introduced. The template for a script for a node includes the following interesting code:

extends Node2D

# Called when the node enters the scene tree for the first time.

func _ready():

pass # Replace with function body.

# Called every frame. 'delta' is the elapsed time since the previous frame.

func _process(delta):

pass

The interesting one is _process(). I can imagine a mode of using the game engine where I just use it as a graphics engine -- every frame, this function is executed by the graphics engine, where it does all of my custom physics etc, changes the position properties of a bunch of nodes, then returns and lets the graphics part of the engine do its thing. 

Seriously, this function can implement a Runge-Kutta numerical integrator from the ground up -- it could use the node properties as part of its state, can keep global or static variables for the rest, and implement any law of motion I can think of. I'm not smarter than Newton or Euler and I intend to use their laws, but I don't *have* to.

If this was all Godot did for me, it might be just right. I'm still exploring, seeing what else it can do for me (I'm on part 25 of 47 of the Intro to Godot course). I strongly suspect that the engine does have a physics component though, since I have seen "gravity" as a property of some of the nodes.

Fallout from the Unity Disaster

 I have always been a low-level kind of programmer. I have a good intuitive sense of How Things Work, but I tend to build my own mental abstractions on top of these and I often have trouble learning other peoples' abstractions.

For instance, one of my unfinished projects is making math explainer videos. I knew about Manim from 3blue1brown, but he uses a completely different mental model of how animations work. It's tough to argue with success -- he has hundreds of videos and millions of subscribers, while I have zero and zero. Plus, I could never get the dancing equations to work as well in PictureBox as he does with TransformMatchingTex().

Similarly, I have always wanted to get into 3D simulator development -- not necessarily a game, more like a virtual environment where I can program virtual robots. One of my other long-term unfinished goals is to animate the launch of various historical spacecraft, much like Jim Blinn animated the flybys of Voyager. I want to be able to write a physically realistic rocket guidance program similar to what a Titan would have actually used, and use that to create a physically accurate, properly scaled animation of the launch. The legend goes that the second Voyager launch (Voyager 1, don't ask) only had about two seconds of fuel left in the upper stage when it finished the boost. How much did the first launch (Voyager 2) have? Is two seconds a lot? This can't really be answered without proper scale.

Anyway, one of the things I have been avoiding as someone else's abstraction, is game engine design and usage. I always thought myself capable of writing my own, even though I kept bouncing off of OpenGL and *its* abstractions. I have written simulator engines any number of times, basically just fancy numerical integrators. I have never learned anyone's game engine -- I have never explored anyone else's abstractions.

In the last couple of weeks, the Unity game engine has claimed authority to alter its relation with its developers. 

Lets just say that the development community hasn't taken it well.

One aspect of the fallout of this announcement is a mad scramble to other engines. It might be too late for programs which are years along and too intertwined with Unity, but many many people are looking at alternatives, and one of the ones that came up is Godot (https://godotengine.org/). This one is free and open source, so it can't help but stay free.

Even though it is free, I did spend about $26 on an online course for it (https://www.humblebundle.com/software/everything-you-need-to-know-about-godot-4-encore-software). The biggest thing I am looking for is how well their abstractions match up with my own. This will be interesting, as I'm not even sure what I expect a game engine to do for me. How does the physics part work? Can I write my own physics? Is there a numerical integrator underlying things? Are translational and rotational kinematics already implemented, and I just have to write my own dynamics? The engine might do too little, and require me to implement my own numerical integrator, or it might do to much, and implement things in such a way that I can't do a rocket.

Velocicoaster

 On Friday, May 26, I rode the Velocicoaster in Universal Florida. I almost didn't for a couple of reasons, but I am glad I did. I'm not quite as sure I would go on it again, but I did offer to do so with my niece once she heard that I went on it.

Shipometer progess

 We are on Shipometer 23.04 now. Shipometer 23.03 is a failure, for perhaps multiple reasons.

First, the shipometer is a HAT for a Raspberry Pi that can carry a ZED-F9R GPS+IMU, a BME280 pressure/temperature/humidity sensor, and an ICM20948 9DoF sensor. Even though the GPS+IMU has an IMU built into it, I want the ICM20948 because I can control it at a lower level, and because it has a magnetometer in it as well. 

Finally, it carries an LPC210x as a precision timer. The ZED-F9R generates a pulse-per-second (PPS) which is accurately timed to have its rising edge right at the top of each second, with sub-microsecond accuracy. This PPS is routed to and captured by the Pi, but is also routed to the timer capture input of the LPC210x. This microprocessor is an old ARM7TDMI, but has a hardware 32-bit timer capable at running at 60MHz, with multiple usable capture inputs. The LPC210x is programmed to count at 60MHz, reset after 3.6 billion cycles, capture several inputs, and output the exact timer count of each pulse to its serial port, which is wired to the Pi UART1. This way, the time of each PPS and sensor data-ready signal can be recorded with no latency due to things like interrupts and real-time software.

Jewel of the whenever: Matrix Multiplication

Eigenchris (I kinda wish I thought of that first) has a series of videos on tensors. I can't give my opinion on the series yet, because I haven't seen them all yet. However, he does something cool that I have never seen before as a mnemonic for matrix multiplication (start at 4:32):

 

We arrange the multiplication as follows. The final product goes in the bottom right. To the left of that, we put the left matrix, and *above* that we put the right matrix. Each cell is then the dot product of the row vector it is on and the column vector it is on

\[\begin{matrix}  & \begin{bmatrix}    . & w_0 & . & . \\    . & w_1 & . & . \\    . & w_2 & . & . \\\end{bmatrix}\\ \begin{bmatrix} v_0 & v_1 & v_2 \\ . & . & \end{bmatrix}& \begin{bmatrix} . & \vec{v}\cdot\vec{w} & . & . \\ . & . & . & . \end{bmatrix}\end{matrix}\]

The One Picture that explains Phase Locked Loops

 A Phase Locked Loop has always been mysterious to me until now. The following three pictures explain it all, and the third one is where the light goes on. Here are the first two, to build dramatic tension and also to do the best job I have ever seen of explaining the block diagram of a PLL. It's from Shawn Hymel's series on FPGA programming.

First diagram:

The PLL consists of three sections:

• The phase detector produces a signal based on whether the reference and fed back signals are in phase. I'm not sure of the details, but it might be something as simple as comparing both signals to zero (returning 1 if positive and 0 if negative) then XORing those comparisons. If the signals are in phase, they will always be on the same side of zero, and the phase detection output will be constant. If they are out of phase, sometimes they will be on opposite sides and the phase detection will not be constant.
• The low-pass filter takes the phase detection signal, treats it as a PWM, and converts it to analog just by running it throug a resistor-capacitor (RC) circuit. The output is then some analog signal that is a function of the average of the phase detection signal.
• The voltage-controlled oscilator (VCO) then takes that signal as an error signal. I'm sure it does some fancy PID magic, which finds just the right output signal to keep the input error signal at zero. It feeds this to the oscillator which then runs at the commanded frequency.
• The output is fed back to the phase detector to produce a proper closed-loop control system.

In this diagram, the output of the VCO is significantly out of phase with the reference, *because* it is not the right frequency. It's impossible for two signals of different frequency to stay in phase.

Second diagram:

In this diagram, the output phase has locked. The error signal from the phase detector and LPF is zero, and the controller in the VCO knows that whatever settings it is using now are correct, and keeps them there. (Note that in this diagram, the clock is an analog sine wave. Just pretend it's a digital square wave.)

Third diagram, and critical part:

This shows a clock divider in the feedback part. Digital clock dividers are relatively easy to implement, requiring a counter. To divide by N, make a counter big enough to count to N. Each input clock, increment the counter, but when the counter is about to reach N, reset it instead. If N is even, then it's pretty easy to set up some logic so that whenever the counter is in the first half of its run, a low signal is output, and vice versa. Odd is a little bit trickier, but still doable.

Multipliers on the other hand are difficult, and in fact are why we need all this fancy PLL stuff to begin with. With a PLL and a *divider* in the feedback path, we can implement a *multiplier*.

If you put a divider on the input reference signal as well, you can get frequency multiplication by any rational factor.

Shipometer

 Having given up on #SoME2, it's time to move on to the next project. Next week I will be going on a cruise. On the cruise, and also on the plane, I wish to record GPS signals. The pocketometer has all the right sensors for that, but unfortunately is not sufficiently reliable. The Raspberry Pi has already proven itself capable of recording GPS from one of the ZED-F9R breakout boards. Now the question is if it can record the sensors on I2C, and the time pulse.

I want to see if I can do this with just the parts that I already have on my desk.

I have a belt bag big enough to hold all the sensors, the Pi, and a 20Ah USB battery pack. That would be a lot less suspicious than stuffing stuff in my pocket.

It would also be great if the Pi could act like a wifi hotspot and serve SSH through it. That way I could look at it on the phone while (literally) in flight.

The last thing that would be awesome is timer capture on the GPIO, of at least the PPS and maybe others, like the interrupt lines from the sensors. If it can't, maybe we could get a program on the Teensy that would do the timer capture and output on UART or as an I2C slave.

#SoME2 post-mortem

 I did not get a video out on time for SoME2. Even for just the descoped "good part" video, I couldn't get it done in time this morning.

Deadline pressure

I ended up going with the Kalman gain video. I will use this blog as "making of" documentation.

• I am going to use PictureBox much more than Manim -- I have already forgotten how to use Manim. I think I will only need it for dancing equations, and I don't plan on using those much. MatPlotLib knows how to use TeX, so it can make nice-looking math, but can't make them dance as well.
• One video, two videos, N videos? I have about 9 minutes of narration so far, and have just covered up to measurement space. It might take 20 minutes to cover the stuff I want just for linear Kalman filtering. KalmanGain implies that the most important or interesting stuff is calculating the gain. That by itself might only take a minute, but all of the pre-requisites might take even more than 20 minutes. For now, I am planning on one long video, covering transformation of uncertainty from state to measurement space and back, and the hand-waving justification for why there even is an optimal gain matrix to transform innovations back into state vector adjustments.
• Pure video, or text plus video? This interacts with above. The hard thing about videos, is that the maker of the video always has to leave stuff on the cutting room floor. This leads to a contradiction: If I include everything I know, the video will be too long and boring, while if I don't, the pedants in the audience will use this as an opportunity to show how much smarter they are than me. Text plus video would allow me to put the most important bits in a video, which could be run as a continuous playlist. The text part would then include the videos, and footnotes with all the pedantry included.
• Narration. I hate the sound of my recorded voice. Plus, I don't have a good audio setup yet. Therefore, I am using Amazon Polly to generate narration from a script. I am having neural British Amy read it. This is the first text-to-speech I have heard that I would say is good enough to be plausible as a human-read narration. I would say it's 80% of the way there. I just wish I could adjust the emphasis on some words. American Joanna is also good enough, but as an American, I at least subconsciously buy into the "intelligent British accent" stereotype. I know that I write slightly different for Amy than I would for myself, and a lot different than I do on this blog.

Which brings us to deadline pressure. SoME2 was announced in the middle of June, but I didn't start on it in earnest until this past Monday (August 11). If I had all the time in the world, I would make a video covering time updating, nonlinear models, and maybe various filters beyond Extended Kalman. It might take an hour, but I would break it up into roughly 20-minute chapters. With the deadline pressure, I am not going to be able to carry out this whole program. I might be limited to just the gain matrix.

I am *not* going to do things like publish one chapter before the #SoME2 deadline and extend a playlist afterwards. I am committing to publishing a complete thought, or nothing at all.

Why is #SoME2 even important? For one thing, I don't expect to win. This might be a winning concept in the right hands, but I am learning that my hands are not those. I am also learning why 3b1b only publishes as infrequently as he does. That's the non-reason, so the reason is: I like having large numbers in my YouTube channel. My last-years late #SoME1 got 104,000 views, even though it didn't make the official #SoME1 playlist.

Also, without deadline pressure, I may never actually make this. One of the issues with my projects is that whenever I am working on one, I wish I was working on another. I'm sure that's a common character flaw among humans in general, but I don't know what it's called or how to fight it.

Decisions, Decisions

Summer of Math Exposition #2 was finally announced. Deadline is August 15, 2022, which fits in quite well with my summer plans -- it's all in between the Florida trip and the other Florida trip.

I could do the next logical step from Exponential beats All. This one was supposed to be a quick explainer for my real video idea, why a rocket doesn't need a heat shield.

Or, I could do a visualization of the Kalman filter. There are of course lots of interesting visualizations to do, but the one I am interested in is showing off the Kalman gain, as the gain matrix which is a linear unbiased estimator, and demonstrating that picking other values for the gain will necessarily produce larger covariance.

Eventually I plan on doing both, but I am leaning towards the "why doesn't a rocket need a heat shield" video. It's more physics and less math, but "math" is given an especially wide breadth in Some2.

A better C++ than C++?

tl;dr -- Is Rust the language that C++ promised but failed to deliver? It's still unclear.

Programmers are adults. We don't need our hands held, and we certainly don't need to be told "Don't do that!" The only kind of advice that is needed is "I see what you are *trying* to do. This might be a better way to do it..."

Case in point: reading data from external sensors. I am still working on the rollercoasterometer (now for 13 glorious years!) and am doing my usual fighting with C++ about getting formatted data out of a byte buffer. Back in the bad old days, we would cast the address of the buffer as a pointer to a struct, and read the data out directly. Then the compiler writers, with their fancy-smancy alignment and such, said to not do that, because there is no guarantee that the structure will line up with what you think. The compiler is free to put any amount of padding between fields, order the fields however it likes, put invisible fields like vptr, etc. 

That by itself isn't bad. The bad part is when I ask how to do what I want, and the answer comes back: "Don't". There is no portable way to guarantee that any field in a struct lands anywhere. This way we can write C++ targeting the Apollo Guidance Computer, with its 15-bit words, no such concept as "byte", ones-complement arithmetic with +0 and -0, etc. It doesn't matter that basically every machine in the last 40 years has been twos complement, 8-bit bytes, and word size of a power of 2 bytes. Basically the only disagreement is endian-ness. 

But, C++ won't let me take advantage of the fact that both machines in a transaction have the same native word format. No, I have to individually extract each byte, shift and OR it myself, etc, to get the data out. If I am lucky, the compiler will see that I am translating from English to English, and optimize it out.

Game designers learned long ago to learn from their users. If all the Minecraft users are building farms, then support the building of farms. If the farm depends on a glitch, consider formalizing the glitch and making it an official feature. Don't just shower them with whatever they are farming for "free", but don't take away their ability to farm either. For instance, Mojang has considered in several instances changing the mechanics of how villagers and iron golems work to discourage farming iron. They got quite a bit of pushback from the community, and have therefore backed off. They don't "support" iron farming, but they haven't removed it either.

The C++ committee on the other hand seems to be driven by two factors:

1. Backward compatibility indefinitely into the past
2. Ability of compiler writers to game benchmarks

C++ has a lot of good ideas (some might say too many) but it is a language which has evolved for decades while at the same time hasn't been able to shed old, bad ideas or old, bad implementations of good ideas. 

C++ for whatever reason also has a burning hatred for the preprocessor. The preprocessor and compiler are married, but they are both trapped in a loveless marriage, and the simmering hatred has boiled over to the point where the official C++ FAQ considers macros to be "evil". Now the preprocessor does have some minuses, mainly in type checking. So, we were given constants and templates. We were told that templates would basically eliminate the need for macros. However, when we tried to use them like that, we found that the promise has not quite been kept. Templates do some but not all of the compile-time processing that we want. Constexpr functions are helping, but aren't there yet. 

My use case is that I want to make a self-documenting logger. The rollercoasterometer reads a bunch of sensors, formats the data into packets, then writes the packets to a file on the SD card. Since the data from the sensors doesn't naturally come in packets, and doesn't naturally have a timestamp, the main firmware timestamps the data and formats it into packets. Since I write the firmware, I also have to write the code on the host which interprets the packets. I came up with a [[clever idea]] to have the program emit a series of packet-documentation packets as it starts up. One way to do it is to have the program create a documentation packet the first time it emits each packet, and I have done this. It looks something like this:

void start_packet(apid, packet_desc_str) {

  if apid is not yet documented:

     write a documentation packet for this packet, using the packet_desc_str

  start the packet, perhaps to a backup buffer if we are documenting the packet

}

void fill<type>(data_value, field_desc_str) {

  if apid is not yet documented:

    write a documentation packer for this field, using the field_desc_str

  write the field to the packet, perhaps to the backup buffer

}

void finish(apid) {

  if the apid is being documented:

    take note that we have documented it and don't do it next time

  finish the packet and write it, perhaps from the backup buffer

}

Note that we need to have two buffers. It would be far better if we did something like this:

void start_packet(apid,packet_desc_str) {

  compile_time_code {

    create a packet describing this packet in the packet description block. This block will be a read-only blob as far as the runtime code is concerned

  }

  start the packet, no need for backup buffer

}

void fill<type>(data_value,field_desc_str) {

  compile_time_code {

    Add a packet describing this field to the packet description block

  }

  add the field to the packet

}

void finish() {

  finish the packet and write it

}

void setup() {

   open sd card

   write packet description block

   set up sensors etc

}

void loop() {

   read sensor

   start_packet(0x1234,"sensor packet");

   fillu16(value_from_sensor,"value from sensor")

   ...

   finish_packet()

}

Preprocessor macros might be able to do it, but the preprocessor is Evil. Templates might be able to do it, but it might require template metaprogramming, which is actually evil. It seems like there isn't a way to do it in C++, certainly not a clean way. Therefore we are forced to either use the official preprocessor, write our own preprocessor (which has its own headaches), or do it at runtime.

The language which might be a better C++ than C++ is Rust. This is a statically typed language which is designed to be compiled into good machine code (the reference compiler uses LLVM as its back end) but with some features added and some taken away. I'm not sure if I like mutability and ownership yet, and haven't gotten used to the concept of borrowing yet, but it does look like there is support for forcing a struct to land on certain bytes, and it does look like (with procedural macros) there might be enough compiler support for compile-time computation.

To test this out, I am going to work on three projects:

1. A conversion of kwantrace to Rust, to experiment with plain application-domain programming.
2. A packet parser for reading rollercoasterometer logs
3. Firmware for the rollercoasterometer

The last one is probably not going to be ready in time for my next expedition to a roller coaster.

Babbage's Dream

 I just got a pair of 512GB MicroSD cards. Of course, any such card should be tested, since these are about the easiest things in the world to counterfeit and it can be done in a way which normally wouldn't be detected for a while.

For instance, imagine you wanted to fake a 512GB card. If it was inert plastic, it would immediately fail and the user would dispute the transaction. So instead, you make a card with less capacity (say 32GB) and reprogram it to do the following:

• Report 512GB of capacity
• Whenever doing a read or write, take the block address given and just mod it with the actual capacity. 

So if you write data to block 0, then to the block at 32GB, the later block would overwrite the first block. This is less than ideal for a storage device... It would work fine until you wrote 33GB on it, which might take a while.

Having said this, there are very many MicroSD cards which are counterfeit in this manner. It behooves us then to test each card immediately.

You can't just test it by writing zeros, as the card may already have zeros on it. You can't just write a fixed pattern (say 0xAA) because a smart enough counterfeit (and note that MicroSD cards have a full-blown microcontroller with its own firmware) would notice this and read back the pattern. You can't just test a small amount of it, because the card (or host) may have a cache. 

So, I have put together a program which writes the output of a cryptographically-secure random number generator to stdout. This can be directed to a file on the SD card to be tested. The chosen CSPRNG is the Keccak-1600 sponge function. We absorb any arbitrary string as the seed, then keep squeezing it forever, or until the device fills up and throws an error message. 

There is no way to beat this, since a CSPRNG by its nature is unpredictable -- there is no detectable pattern unless you know the arbitrary string used as the seed. The internal microcontroller probably could calculate it, since Keccak is a well-known standard alogrithm, but it doesn't know the seed. It's not reasonable for the counterfeiters to guess that I am going to be checking the chip this way, and to try to guess the seed (hint -- it's close to a substring of the title of this blog).

There is no way to derive the seed from the output of the CSPRNG -- that's one of the properties that makes the PRNG cryptographically secure. The stream might have a period of 2^1600 bits, but I haven't seen a proof or even any good reason to believe that Keccak is full-period when used this way. Even if it is much less, it is almost certainly much greater than the 2^42 bits it takes to fill this device. So there is no way to store or cache this stream without actually having a functional amount of storage equal to the advertised amount.

In my case, I couldn't find the exact routine I wanted, so I wrote my own based on the XKCP package (since it isn't wise to implement a cryptographic primitive yourself).

#include "KeccakSponge.h"

#include "stdio.h"

#define r 576

#define c (1600-r)

int main(int argc, const unsigned char** argv) {

  KeccakWidth1600_SpongeInstance s;

  KeccakWidth1600_SpongeInitialize(&s,r,c);

  char buf[r/8];

  KeccakWidth1600_SpongeAbsorb(&s,argv[1],strlen(argv[1]));

  if(argc==2) {

    for(;;) {

      KeccakWidth1600_SpongeSqueeze(&s,buf,sizeof(buf));

      fwrite(buf,1,sizeof(buf),stdout);

    }

  } else {

    FILE* inf=fopen(argv[2],"rb");

    char inbuf[r/8];

    size_t last_match=0;

    while(!feof(inf)) {

      KeccakWidth1600_SpongeSqueeze(&s,buf,sizeof(buf));

      size_t incount=fread(inbuf,1,sizeof(inbuf),inf);

      for(size_t i=0;i<incount;i++) {

        if(buf[i]!=inbuf[i]) {

          fclose(inf);

          printf("Different at byte %ld\n",last_match);

          return 1;

        }

        last_match++;

        if ((last_match%(1024*1024*32))==0) {

          printf(".");

          if((last_match%(1024*1024*1024))==0) {

    printf("%ld\n",last_match/(1024*1024*1024));

  }

          fflush(stdout);

        }

      }

    }

    fclose(inf);

    printf("All bytes up to %ld matched",last_match);

    return 0;

  }

}

This program takes one or two strings as command-line arguments. The first is the seed. If there is only one argument, it uses this seed as mentioned above to absorb, then squeezes the sponge an unlimited amount of times, limited only by the device filling up. It writes to stdout, so the way I use it is to pipe it through pv to see how fast it is going, and then pipe it to a file on the device under test. If the card is genuine, then this test is non-destructive. It also doesn't disturb the original exFAT filesystem formatting that the card came with -- I have heard that there is a non-obvious optimum way to format these cards, and that they come formatted optimally.

If there are two arguments, the first is still the seed, and the second is a file to check to see if it matches. It does this in the simplest way possible, by absorbing the seed, then going into a loop: squeezing once, reading the same amount from the file, and byte-for-byte checking the blocks. It prints a period every 32MiB and a number every 1024MiB. It prints the byte offset of the first non-match (and then exits) or prints the total number of bytes it checked.

As it happens, my chip worked properly. This means the system produces a stream of over 4 TRILLION bits, then perfectly reproduces those 4 TRILLION bits. That's amazing when you think about it, and is Babbage's dream. The legend goes that after being disillusioned by the number of errors in an almanac, he remarked that he wishes that the tables could be generated by steam power. One of his colleagues said "that is possible", which statement changed the course of Babbage's life. He originally wanted to use the difference engine and analytical engine to literally crank out mathematical and almanac tables, with no human intervention between the initial conditions and the printed page. The machine was intended to perform the calculation, then with the results, automatically make a plate to be used in a printing press. Many years later, the first and simplest such difference engine was finally constructed, and printed the first several integer multiples of the circle constant pi. It made a mistake in the last entry. 

Dusting off some old code

 In preparation for the Mars Science laboratory landing, I made a video using the best pre-EDL data available, including a simulated EDL kernel set, MOLA topography, and HiRISE imaging. I'm pretty proud of it:

Palpatine was Dead

Palpatine was dead: to begin with. There is no doubt whatever about that. His flailing body was flung down the open reactor shaft. Crackling with dark side power, he fell faster and faster until he hit a bridge far down the shaft, and burst like a bomb. Anakin and Luke felt the concussion even hundreds of meters away. And if that wasn't enough, he was killed twice, once by his former apprentice, and once by the terrorists who destroyed the planet-sized station that the pieces of his body were at rest upon. Anakin saw it, Luke saw it, we in the audience saw it.

There is no doubt that Palpatine was dead. This must be distinctly understood, or nothing wonderful can come of the story I am going to relate....

What makes a story?

"I wonder what they'll be like?" he mused. "Will they be nothing but wonderful engineers, with no art or philosophy?" --From Rescue Party by Arthur C. Clarke

Specialization is for insects  --Attributed to Robert A. Heinlein

I might think of myself as an engineer (and by no means wonderful) but sometimes I think about art and philosophy as well. Today over lunch I was thinking about what makes a great story -- what makes a story entertaining to me? For instance, why do I like Star Wars? Is it the characters? Is it the message?

For me, the most entertaining part of the original trilogy was the Battle of Endor (which the Empire totally should have won, by the way. Ewoks?). Specifically the best part of that was the run to the Death Star core. Why? Great visuals. We got to see the intricate detail of the inside of a massive, complicated object. The interior of the Death Star is itself a work of art.

Similarly, the boarding and launch of USS Enterprise in Star Trek: The Motion(less) Picture is the best part of that film by far. The new Star Trek had its moments, but they were too fast and filmed in shaky-vision such that we never got a clean look at all of the great models. From the clips I have seen of Star Trek: Beyond, it looks like Starbase Yorktown was done right. It might be way too large and expensive to be practical, but it does look awesome.

So, why aren't all movies just spectacular visual effects?

Firstly, visual effects are not cheap. It is far cheaper per minute to put a bunch of actors on a sound stage and just record a play, compared to special effects.

Second, if we do, we end up with such works as Sonic Vision and The Mind's Eye. These are works of art in themselves, but there is still something missing. I don't think even I could watch Sonic Vision for two straight hours. I finally think I know what it is. It's consistency. In a well-constructed story, all the pieces just fit together. As long as it maintains consistency, the larger the story, the better. In such a story, you can understand what is going on. You can make predictions, and evaluate the actions of the characters. Did it make sense for Han Solo to do that? Did it make sense for Admiral Holdo to do that? Harry Potter seems consistent, and it maintained that through seven books.

A good consistent story, then it seems, must be well-planned from the start. A good story universe, must have a solid scaffold of ideas, and all new ideas added to it must remain consistent. The best ideas might expand the scaffold, but in a way that makes it stronger and able to hold even more ideas. The core of any good story universe is one good story.

So: Why did Admiral Holdo do that? It was a visually stunning 10 seconds, but how does it work as for consistency? In order for it to make sense, she must have had some idea that it could work -- not necessarily a sure thing, but at least a reasonable chance to be worth trading her life for. If hyperspace ramming works, then why isn't it always used?

Also, the Holdo Maneuver didn't even get a mission kill -- Supremacy was not destroyed, only damaged. It was not stopped, only slowed. It still was able to launch an invasion of Crait. She aimed for a wing, rather than the core.

Here are the facts as shown in The Last Jedi:

• Raddus was almost out of fuel -- it had enough for one jump, and no further fuel to travel through normal space.
• Admiral Holdo ordered the abandonment of Raddus with her alone staying aboard. 
• She turned the ship to face Supremacy.
• The crew of Supremacy thought that Raddus was either trying to escape or was trying to block/stall to give the rest of the fleet a chance to escape.
• Just before Raddus jumped to hyperspace, the crew of Supremacy realized what Raddus was trying to do, and started to take action to counter, but they no longer had enough time to prevent it.
• Raddus jumped to hyperspace with its trajectory through the right wing of Supremacy. It isn't clear from the footage whether Raddus actually entered hyperspace, or just hit Supremacy at high speed in normal space. In any case, the right wing of Supremacy was sheared off, and at least four trailing ships were destroyed by debris from the collision.

Other facts seen in other parts of canon:

• During Rogue One, Devastator jumps into Scarif orbit just as the rebel fleet is trying to flee. At least one GR75 (transport-class) ship hits Devastator's hull and is destroyed, and its pieces are just brushed off.
• In one of the Legends comic books, a squadron of three star destroyers  drops out of hyperspace right on top of Executor. All three ships are smashed to atoms, and while Executor has to stop what it is doing, its shields brush the collision aside, such that the paint isn't even scratched.

We can enumerate all the possibilities, and dismiss each of them. The Holdo Maneuver is inconsistent with what has come before.

• Admiral Holdo is such a great military leader that this idea is original to her. In the twenty thousand year history of the galaxy, no one else has had this idea. We dismiss this because even a cursory study of either galactic or Earth history would have revealed many examples of ramming as a reasonable tactic. I have seen Youtube videos by one of the numberless online jabberers stating exactly this, before The Last Jedi was released, as an alternate way to destroy the Death Star, use an X-Wing in kamikaze mode.
• The shields of Supremacy should have been able to protect it from the collision, as seen in other hyperspace-shield interactions. Admiral Holdo should have known this and not even have attempted the ram.
• You can't just jump one X-wing to hyperspace and expect to destroy the Death Star -- a single X-wing isn't big enough. Anything bigger is supposedly too expensive. However, if just mass will do the trick, there is enough plain mass in the form of such things as asteroids to do cheaply.
• There is something special about Raddus which makes it uniquely qualified to ram. If this is the case, Raddus is just a machine, and any machine can be duplicated.
• The only theory which is not immediately dismissable is that there is something special about Supremacy, perhaps related to the hyperspace tracker. Admiral Holdo could not have successfully rammed any other ship. Even so, Holdo would have had to know how the hyperspace tracker worked, at least well enough to know that it created this vulnerability. Besides, one of the Incredible Cross Sections books contains text to the effect that hyperspace tracking was merely an algorithm, and the hyperspace tracker on Supremacy was merely a large computer facility.

The point is that in any conceivable scenario, whatever Admiral Holdo did at that moment could have been, and therefore should have been, weaponized long before. Either it was, and all battles in the galaxy would have been different, or it wasn't, because it is impossible.

C++ Cleverness

Yukari was in some ways an amazing piece of code. It didn't actually drive the robot all that well, but of all the things in it, I am most proud of it's self-documentation. Each run of Yukari produced a file which recorded three things:

1. Data packets showing what the robot was doing and what it was thinking
2. An image of its code and any other files I thought needed attaching
3. A description of how to parse the record file, partly in English, partly machine readable. With this description, anyone who had the file could in principle write a piece of code to parse the file.

I am redoing this code for the Loginator. While it is easy to just use the same logic that I used on Yukari, that code was inefficient. It did the following:

• The packet start function took a pointer to a string describing the packet, and each kind of fill function took a pointer to a string describing the field. This was nice because the documentation for each field in the code is right next to the actual code for it.
• The start function and each fill function called the writeDoc() function, which took care of documentaion. After that was finished, it wrote the field.
• The writeDoc() function kept track of apids which have already been documented. If this apid has already been documented, writeDoc() returns immediately. Otherwise it write a field description packet.
• In order to make this work, the actual packet data had to be stashed somewhere. If the packet was in the process of being documented, writeDoc() for a packet start set up pointers such that the packet being written went to this stash buffer, and writeDoc() got to create actual packets in the proper buffer.

What I have in mind is much more clever. It will see the compiler generate the core of the documentation packets at compile-time. These will then just be in ROM, which we have bucketloads of. I think this will involve template meta-programming and constexpr functions. 

Each call to start and fill will be immediately followed by a template class instantiation. This class template will take as parameters the apid, the string description, and perhaps some other stuff (units, conversion, etc). The class will declare a couple of static constant member fields, which will result in them getting stuck in the .rodata section, or maybe a special section. Once it is someplace in the read-only image, the startup code will write out all the documentation in the same way that 

The interface will look like this:

start(apid_blah,TTC(0)); template class packet_doc<apid_blah,0,"This packet records exactly how blah things are">;

fillu16(blah); template class packet_doc<apid_blah,t_u16,"This field records the blah level">;

It's not quite as clean as the old way, but it is purely a run-time thing. To begin with, we would have a template something like this:

template<int A, int T, const char* D>

class packet_doc {

  static const int __attribute__ ((section(".packet_doc"))) apid=A;

  static const int __attribute__ ((section(".packet_doc"))) fieldType=T;

  static const char* __attribute__ ((section(".packet_doc"))) desc=D;

}

We could make things fancier by keeping track of the position in the packet using more advanced template metaprogramming.

Update:
Nope, defeated. While you can use an address as a template parameter, a string literal doesn't necessarily have an address on its own. You can set up a const array with a string literal in it, and use that as the template parameter, but that starts getting way too ugly. I'll do it the old way, and use the bottom of the stack for temporary space. It isn't secure, because what happens if the stack and this buffer collide, but I'm not going to worry about that.

Interesting things about Voyager 1 and 2 launches

There is something fascinating about science. One gets such wholesale returns of conjecture out of such a trifling investment of fact.

--Mark Twain

It's amazing how much we can do with such a small amount of data. The following analysis is based solely on the dates of launch and arrival of the Voyager spacecrafts, and an ephemeris of the planets.

Voyager 2 was launched on 1977-08-20T14:29:00Z, while Voyager 1 was launched later on 1977-09-05T12:56:00Z. Voyager 1 passed Voyager 2 on the way out (the exact time depends on how you define "pass") and arrived at Jupiter on 1979-03-05T12:05:26TDB, while Voyager 2 arrived on 1979-07-09T22:29:51TDB.

Notice that while the arrival reference is a full-blown set of orbital elements, we did not use these. Instead, we use the Lambert/Gauss targeting method to plot a course which departs from the center of the Earth on the indicated time, considers only the gravity of the Sun and ignores the gravity of the Earth, and arrives at the center of Jupiter at the indicated time. There is one unique trajectory which crosses the indicated places at the indicated times and is prograde.

So, even though Voyager 1 is launched later, it arrives first. This is for two reasons:

1. Voyager 1 has a higher heliocentric energy
2. Voyager 1 is launched on the "outside track". Normally the inside track is faster, but not in this case. If both spacecraft are going to intercept Jupiter, the first to arrive must be on the outside track because Jupiter is moving from "outside" to "inside".

We end up with the following picture:

There is a lot to look at in this picture, so here are the thousand words. Red is Voyager 1, blue is Voyager 2. The small white circle is the orbit of Earth, with the sun marked as a yellow dot, Jupiter the next larger white circle, and Saturn the largest. The tick marks are 30-day intervals starting at the launch date of Voyager 2. This means that the ticks for Voyager 1 and 2 are directly comparable. We see Voyager 1 pass at about the 4th tick mark. A zoom in reveals that from this perspective directly from Ecliptic North, the trajectories actually cross twice. Normally the trajectory diagrams don't show them crossing at all.

Another interesting thing is that the orbit of Voyager 2 if it did not intercept Jupiter, would have continued about 1AU past Jupiter's orbit. However, Voyager 1's orbit would have gotten almost out to Saturn. Did they use a larger launch vehicle for Voyager 1? No, both spacecraft had almost identical launch speeds (vinf=10.951km/s for Voyager 1, 10.316km/s for Voyager 2). The main difference is just the direction of launch. Voyager 2 was launched above the ecliptic plane (departure asymtote 17 deg above the ecliptic) while Voyager 1 was launched almost in the ecliptic (departure asymtote 4 deg above ecliptic). Also, Voyager 1 was launched closer to the direction of travel of the Earth. Voyager 2 was launched 18 degrees inward, while Voyager 1 was launched 4 deg outward.

Another blast from the past

Once upon a time, I participated in the Internet Ray-tracing Competition. It turns out that they have an archive, and it turns out that most of the source code for my images are still there. I was able to recover another one today:

A modern render of an ancient image

My notes say that this took 6 hours to model and 1h30m45s to render on an AMD K6-2 300MHz 192MiB memory machine. The re-render at the same resolution takes 4.423 seconds on an Intel Xeon E3-1505M laptop with 32GiB memory. This is over 1200x faster. At HD resolution it takes about 13.7 seconds.