Tuesday, March 3, 2015

Btrfs filesystem df

Do you know how hard it is to make backups of multiple terabytes? How tall of a stack of floppies that would be? Yes, I have backups, of the important stuff I can't re-get easily, onto several different computers.

With that out of the way, I was running uncomfortably close to out of space on my main system. It only had a couple of hundred gigabytes left. (I remember when a 1.2GB system was so unimaginably vast that I split it up into 4 pieces.) However, the filesystem in question is btrfs, running as raid1 on 3x 2TB drives. I had always intended for that system to be raid5, which is why I got 3 drives for it. Since raid5 on btrfs wasn't ready for prime time, I did a raid1 and waited for the day when raid5 became usable.

With kernel 3.19, that day finally came. A quick (not really, it took many hours
) rebalance later, and raid1 became raid5 without even taking the filesystem offline. Now I go from 3TB total usable space (6TB device space divided by 2, raid1 duplication factor) to 4TB (effectively two drives data, one parity).

But do I? I know that the normal df command is basically useless for free space on btrfs, so I do the following:

root@omoikane ~
# btrfs fi df /mnt/big
Data, RAID5: total=2.48TiB, used=2.46TiB
System, RAID5: total=64.00MiB, used=348.00KiB
Metadata, RAID5: total=6.94GiB, used=3.80GiB
GlobalReserve, single: total=512.00MiB, used=3.54MiB


What gives? I expected something like 4TB of space, and no amount of TiB->TB conversion is going to cause a discrepancy that big. In fact, this says that I have less space than I did prior to converting to raid5. What did I do wrong? As it turns out, nothing. First, look at this:

root@omoikane ~
$ btrfs fi show /mnt/big
Label: none  uuid: 6bf52a05-b0cc-4b21-96db-32cc9a1bed7d
        Total devices 3 FS bytes used 2.47TiB
        devid    1 size 1.82TiB used 1.24TiB path /dev/sdb
        devid    2 size 1.82TiB used 1.24TiB path /dev/sdc
        devid    3 size 1.82TiB used 1.24TiB path /dev/sdd

btrfs-progs v3.19-rc2


Not all of the disks are even fully allocated. If you add up two of them, you get the 2.48TB expected, and you only add two because of raid5. So why aren't my disks fully allocated? Is it like mdadm, where I have to do something manually to get that space? Again, that turns out to not be the case.

To test this, I opened up two terminals. In one, I made a large file by dd'ing /dev/urandom into a file. I used that instead of /dev/zero so that I wouldn't have to be in doubt with file holes.

In the other terminal, I watched the output of btrfs fi df, in a little bit more fine-grained manner:

me@omoikane ~
$ btrfs fi df -k /mnt/big
Data, RAID5: total=2658172928.00KiB, used=2643827200.00KiB
System, RAID5: total=65536.00KiB, used=348.00KiB
Metadata, RAID5: total=7274496.00KiB, used=3990868.00KiB
GlobalReserve, single: total=524288.00KiB, used=0.00KiB

As the dd ran, I watched this report, then saw the total amount of raid5 space grow (the bold number above). I knew what was going on at that point: The system was automatically growing the data space as needed. The unallocated space reported by fi show is available automatically without any need to rebalance or manually allocate. I suppose it is held in reserve for snapshots, a feature I don't use. In any case, the space is really there and the fully allocatable space is 3.64TiB, of which 2.47TiB is used, meaning I now have over a full TiB available. The raid5 conversion worked.

Interestingly, when I deleted the big file, the amount of space allocated to raid5 data decreased.

Monday, December 15, 2014

The Useless Machine

Once upon a time, I accidentally designed and built an electronic circuit that turned itself off, therefore accidentally re-inventing the Useless Machine. My nephew saw a YouTube video of one, and fell in love with the concept. So I got one:

It's a quite elegant machine, in that it is completely powered off when closed, and runs on two switches, with no other logic, no ICs, no nothing. It's circuit is a DPDT toggle switch, which appears to be the "on/off" switch but is more accurately referred to as the "forward/reverse" switch. It also has a microswitch inside which trips when the arm pulls all the way in. When the switch is forward, the motor runs forward until the arm pops out, collides with the switch, and puts it into reverse. The reverse circuit switch is in series with the microswitch, so it only runs until the arm retracts completely. It's kind of an interesting logic problem: How do you arrange the switches to do what is desired?


Thursday, October 23, 2014

Tuesday, September 2, 2014

Mt Evans and the amorphous project

This is the benchmark on the "top" of Mt Evans. I say "top" because there is another boulder within 20 feet which is about 3-4 feet higher, and another one within 50 feet which is 2-3 feet higher still. However, this is the one with a readable altitude (if just barely). The two devices are my phone in GPS mode, and the controller for Project Yukari, running a program called "Southwest" because I intend to take it on various flights I make around the country in the future. One of the interesting things about this recorder is that it carries a pressure and temperature sensor.

Sunday, July 13, 2014

St Kwan's Campanile is complete!

My name is Kwanzymandias, king of villagers: Look on my works, ye Mighty, and despair!


This is a full-scale model of the bell tower in St Mark's Square. I searched for days, making camp three times before discovering this village in a lake, about 150m from the third camp. I created this whole world with the express intent to find such a village, and when I did, I defended it and built my base there.

It's not quite The One Eight, yet, but it does have some nifty features. It is 102m tall, including the Tower of Shininess on the roof, consisting of one obsidian and four gold blocks. Its floor plan is a square 12m on a side, with walls 2m thick, just like the real thing. It shares with the real tower a spiral ramp around an empty center. The interior space is marked by four brick pillars, with a 4m square space in between them and the 1m wide ramp around them. Unlike the real tower, this is a working building, crammed to the gills with FTB technology. There is a floor in the center space every 6m, once per lap of the ramp.

The first floor is the lobby, and has nothing.

The second floor is the power station. It features the building tesseract, used to pipe lava from the pumping station in the Nether. The power station consists of 10 magma dynamos running on the lava pumped in, and their power goes out through the tesseract to power the pump. Until the local Nether lava ocean is drained, I will have power.

The third floor is the main AE room, with a 4m MAC, along with the controller, drive, and crafting terminal and monitor. My power armor tinker table is also here.

The fourth and fifth floors are gardens, with automatic planters and harvesters. These were used to grow earth and water seeds, used to make clay, then bricks for the walls (the walls were made of dirt first). The fifth floor also has the Tinker's Construct station for fixing tools.

The sixth floor is the chicken factory. It uses Xisuma's chicken cooker design to make several cooked chickens per hour, more than enough for my needs.

The seventh floor is the Nether portal room, with a floor of soul sand for growing nether wart.

The eighth floor is the twilight and Mystcraft portal room.

The ninth floor is just below the transition from brick to marble, and has the ore processing system. More buildcraft-type machines will go here as I need them.

The tenth floor is the belfry, where the big arched windows are. Here I have the bedroom, as well as a second crafting terminal, unifier, and uncrafting table. I plan to hang some note blocks from the ceiling to simulate the bells.

The eleventh floor is where the redstone for the bells will go.

The twelfth floor is not spoken for, it is the brick area above the marble and below the roof.

The thirteenth floor is under the roof, and is also not spoken for. I had planned at one point to put a mob farm here (in the dark) and have the mobs plummet through the tower, but I don't need any mob farms at the moment.

Thursday, July 3, 2014

Cleaning up the code

"All the most important mistakes on a project are made on the first day"
--Old design adage

"There is no teacher but the enemy. No one but the enemy will tell you what the enemy is going to do. No one but the enemy will ever teach you how to destroy and conquer. Only the enemy shows you where you are weak. Only the enemy tells you where he is strong. And the only rules of the game are what you can do to him and what you can stop him from doing to you."
-- Mazer Rackham

"Reinhardt regarded the mysteries of the Universe not as indifferent questions of physics or chemistry, but as implacable, malicious foes. They were to be assaulted with science, vanquished at any cost, forced to yield their treasure house of knowledge."
-- from The Black Hole by Alan Dean Foster

In any interesting problem, you don't know enough about the problem to begin with to even form a plan for solving it. You do the best you can, and in the process of solving the problem, you learn what you should have been doing from the beginning. If time and resources weren't an obstacle, you could go back and solve the problem right from the beginning. But, usually for any interesting problem, time and resources are an obstacle, and you are stuck with what you started with. At each point in solving the problem, you only change what came before to the absolute minimum degree necessary.

As a result, sometimes legacy design choices make the solution quite a bit more complicated than a clean solution would be.

Now I have time and resources -- a whole year of time. It's time to clean house.

Sunday, June 22, 2014

A custom board for Yukari

The Loginator is a fine circuit. It is great at what it was designed for, logging. Even as a robot controller, it still spends a good chunk of time logging.

To really spiff it up as a robot controller, it needs a couple of things:

  • Two free PWM pins. PWM5 is AD6, so it is already free. PWM1 and 3 are also UART0, and PWM4 and 6 are also UART1. PWM2 is also part of SPI0, which talks to the SD card, but any GPIO can do the CS thing. All the other pins are already used, but I wouldn't mind parting with one of the I2C ports. In particular, I2C1 is used to talk to the 11DoF. It might make sense to switch that to I2C0, since SDA1 is also BSL. Then again, SDA1 and BSL don't interfere with each other. Another alternative is to give up AD7 or AD8, since I don't use the free pins much.
  • A GP2106 port. I have no complaints about the GP2106. It worked fine. It should have a port on the Loginator, either to interface with the existing breakout, or direct with the 1.8V parts on the Loginator board directly.
  • Perhaps rather than the 11DoF port, have the sensors directly attached.
An LPC2368 would go a long way towards fixing all of these things. It has 100 pins, so there is less overlap. Perhaps a Propeller would work too.

AVC 2014 Event Report

Or: Give Up! It's time for you to throw in the towel, capitulate and raise the white flag...

I made it farther than I did last time. I made it to the starting line. I turned on the robot, and watched it run straight as an arrow into the fence. It looked like it didn't even try to make the turn.

After that first run, I looked at the wiring and misread it. Since the LPC2148 is short on pins, all of the PWM outputs are multiplexed with other functions, functions it turned out I needed. PWM 1 and 3 are also UART 0, PWM 4 and 6 are also UART 1, and PWM 2 and 5 are also part of SPI 0. As it turns out, I needed all of those things, so I had to pick one to discard. I would have liked the GPS on UART 1 so that I could program the device over UART 0, but since that was the one thing that was possible to do without, I had to give it up. I had the GPS wired to UART 0, but only when it was in robot mode. When it was in bootloader mode, UART0 had to be connected to the host (through the FTDI cable). When I glanced at the board after the run, as I said, I misread it. I thought that the wiring was set up for bootloader. This would have made it impossible to do waypoint navigation. It would have resulted in the robot running straight as an arrow forever (or until it hit the fence).

As it turns out, it did try to make the turn. Like I said, I had misread the wiring and the GPS was fine. I could see from the logs that at about the right place it signalled to turn -100 (out of the full servo range of -128 to 127). It wanted to turn -600, but of course it was limited. I was let down by the steering mechanism again, but this time it wasn't a twitchy servo, it was my own circuit.

I was worried about running a solderless breadboard as the base of the robot controller, so I got some matching solder protoboards, but ended up never using them, mostly because I didn't want to commit the Loginator to this project. I didn't want to solder things down, so I ended up with two short breadboards stuck side by side. One supported the Loginator and FTDI socket, while the other supported the GPS interface and the opto-isolators for the servo control. The opto-isolator was the weakest part, partly from the sheer number of wires in such a small space, partly because I had to switch the channels at the last minute.

During the test run before the race, the thing was running long and seemed to want to turn only after I had picked it up. It might have already had this fault at this point. I have the logs and can analyze at this point. Soldering down the opto-isolator and its wires probably would have worked.

In any case, I was looking for an excuse to give up. It is one thing to surrender before you have to, and another to realize that there really is an insurmountable obstacle. During the test run on Friday, I had manually driven the thing around the track and flipped it on the finish line. The board popped out, but was apparently undamaged. I was almost hoping that the board was damaged beyond repair so that I could honestly say that I was beaten, rather than had given up.

I had convinced myself on Saturday that the steering problem was in the former  class. I had convinced myself that I needed to solder the whole thing to a board to get it to work. Thinking back on it now, if I had disassembled and reassembled it, it very well might have worked. I had the solder protoboard with me, and I could have borrowed a soldering iron. I could have done it there at the race.

This means I gave up too soon. I haven't finished. I will have to go back next year. Joseph is interested, so maybe I can use him as the motivation I need. Maybe he can help with Yukari, maybe he will help with another 'bot. Yukari is so close, I can feel it.

I did meet some nice people. Team Bloomberg was a dad with his family who had driven out from Wisconsin. Team Deep Space was an RC car with a robot controller, pretty similar to mine, except he went with odometry and a 4WD chassis. However, it had a cool hat, a flying saucer.