Archive for the ‘inventions’ Category

Intervals: clock and power on the same line

Thursday, October 25th, 2012

One of the main design considerations of Interval bars was trying to minimize the number of connections between adjacent bars. The current design has three connections - two for power and one which is a bidirectional, self-synchronizing data signal. The logic to synchronize everything ended up being a fairly complicated bit of code. It would be much simpler and faster if all the microcontrollers were driven from a single clock signal, so that all the signals could be synchronous. However, that would end up using 4 pins instead of 3.

What if instead of combining the clock signal with data, we combined it with power? Well, it's only worth doing this if we can do it without too much external hardware. A simple resistor/capacitor filter would be acceptable, but that's not enough to properly restore both power and clock signals. The DC part (across the capacitor) would be okay but then across the resistor we would get an AC signal, when what we really want is a square wave that oscillates between 0 and 5V, so we'd need some additional components to fix up the clock signal.

The worse problem, though, is that with each interval bar added to the circuit, we'd be adding an additional capacitor (in series with a resistor) between ground and (power+clock), so the clock signal is going to get degraded (it's amplitude will be reduced) with each bar. We can reduce this problem by increasing the resistance of the resistor, but then we'd have to use a higher voltage for the power line, increasing the complexity of the root box.

In the end I decided that a solution that makes the hardware simpler but the software more complicated was preferable, since the software only needs to be written once, but who knows how many times an interval bar will get built?

Zoomable family tree

Wednesday, October 3rd, 2012

There seem to be lots of people on the internet who don't really understand evolution. You can tell who they are because they ask things like "If people evolved from monkeys, why are there still monkeys?" and claim that there are things called microevolution and macroevolution which are qualitatively different. Sometimes this happens because people perceive there to be a conflict between their preconceived beliefs and the theory of evolution. Sometimes though it's just that they're taught by people who have such perceptions, which is a bit of a tragedy because a poor quality of teaching will lead to these students not being able to contribute to science in the future.

Without pretending to have any answers for the more fundamental problem of bad teaching, I do have an idea which could help people get an intuitive understanding of how evolution works.

Really what evolution says is that there ultimately there is a single family tree - we're all related if you go back far enough. Also, the blueprints for our bodies are not engraved in some unalterable stone tablet sealed in a cryogenic vacuum somewhere - they're stored as atomic-scale patterns inside our warm, wet cells and therefore change from time to time (albeit very slowly by the timescales we're used to).

So what I'm imagining is a phylogenetic tree viewer with a twist - you can zoom into a phylogenetic tree, and if you zoom in far enough you can see individuals and the parent-child relationships between them, like a normal family tree. Individuals would actually be shown as what typical individual of that lineage would have looked like (possibly with typical random variations). But these pictures would be stored in some kind of vector representation so that one could be morphed into another gradually. The point is that no matter where you zoom in, there's no place you can find where the children don't resemble their parents. You'd also be able to get an intuitive understanding of the process of speciation by seeing how a single species lineage splits into two, but on the level of individuals there's no one individual that you can point to that marks the point at which the species diverge.

By zooming in right at the top of the tree you'd be able to see (a theory of) abiogenesis (the first point at which information was passed on from parent to child).

A simple version of this program (just including a tiny minority of species) would be relatively easy to create and could still fulfill the basic educational possibilities. But the biologists collaborate and the more data that the program was fed, the more interesting things could be done with the result. Ideally it would be made as accurate as possible according to current best theories. Of course, a huge amount of the data is currently unknown and would have to be synthesized (at high zoom-in levels, almost all of the details would need to be made up). So if the program is used for scientific purposes it would be good to have some way to visualize which aspects of the image are accurate according to current best theories and which aspects were synthesized.

The data that's added to the program need not all be biological in nature - people could add their own family trees (and even those of their pets) as well (though reconciling low-level actual data with synthesized data could be tricky).

Buying up the junk

Wednesday, September 26th, 2012

I recently moved from Seattle to the UK. Part of the process of doing a big move like that is getting rid of stuff that you don't need any more. For some things that's just a question of throwing it away (we had one visit from the haulers, two trips to the tip and several weeks of extra garbage charges). But there's other stuff that, while we didn't need it any more, would have some value to someone else. Getting rid of all that stuff was a consistent feature of all our todo lists, and we still didn't manage to get rid of as much stuff as we wanted. We gave some things away to friends, sold some things on a neighborhood email list and had a (not particularly successful) garage sale.

There really ought to be a better way than this. There must be a enormous amount of value locked up in peoples houses in the form of stuff that they don't use but which still has some value and therefore they don't want want to throw away, but getting rid of it is an annoying, difficult, low priority task, so never gets done.

I think somebody could make a fortune by setting up a company that takes away your unwanted stuff. You'd request a visit from them on their website (or maybe they could just show up on a regular basis) and take away anything that you didn't want. They'd do the work of valuing it, selling it and shipping it, and then send you the proceeds (after taking their cut). If, after the valuation stage, you decided that the item was worth more than that you could reject the offer and they'd bring it back with the next visit (perhaps for a small charge to avoid the service being abused as a free valuation service). Items that might leak liquids or emit odors would probably not be accepted (the small amount of value held in such items would probably not be worth the possible damage to other items).

They'd do all the work of making sure that items were packed sufficiently well for shipping, reusing packing materials as much as possible (and eliminating a large amount of waste). If they delivered items as well (instead of relying on UPS, Fedex or similar) they could take away the packing materials on delivery, helping the environment and saving the customer from another annoying job (my workshop in Seattle often used to get cluttered up with old cardboard boxes, packing peanuts and bubble wrap).

Another nice thing about this business is that it would be really easy to bootstrap - you could start it off in just one city with a couple of people, a van and a simple website and some insurance against breaking things. Deliveries to places too far away for the van to get to (or between two different cities where the company does have vans) could be done with the existing delivery services. After visiting one house for a requested pickup they could visit other nearby houses and ask if they have any items they want to get rid of.

Another Argulator feature that never was

Sunday, August 23rd, 2009

One Argulator feature I was, at one point, absolutely convinced that I needed was a web page cache/archive. The trouble is that one often wants to link to other sources of information when making statements (for example to cite a source or backup some assertion). On the web the natural way to do that is to use a link to a URL elsewhere on the web. But URLs are fickle things - the resource they point to can change with time or disappear altogether (something I've always tried to avoid happening with this site - I believe pretty much every URL that was ever valid is still valid today and points to the same information, even if it has been tweaked a bit over the years).

If you're referring to a URL in an Argulator statement, though, that's a problem because a statement's meaning is never supposed to change once it's been created. If it could change, you're putting words into the mouths of all the people who have expressed an opinion on that statement. So I wanted a way to make sure that web pages mentioned in Argulator statements never changed or disappeared, which meant bringing them under the control of the Argulator server. The idea was that when someone mentioned a URL, the Argulator server would fetch that URL, along with the graphics and CSS required to make the page display properly, just as if it was a web browser. Then it would store these files on the server and show it's cached copy (much like Google's cache or the Internet Archive does) when a link is clicked.

But that turns out to be surprisingly difficult, and opens up a whole cannery of worms. Security is a big problem. To ensure nothing bad is ever served from the Argulator site we need to strip out all executable content (script, Java, ActiveX, flash, XUL, Silverlight etc.) from the pages. Which means parsing all the CSS and tag attributes. After that treatment the page might not render at all (especially if it contains things like protection from ad-blockers). Such a cache should also respect the directive and refuse to cache pages whose authors do not want them cached.

Then what do we do if someone requests we take down a cached page (maybe it contains illegal content, or the content owner complains on copyright grounds)? We don't want to create work for ourselves, but on the other hand we don't want to make it too easy for people to sabotage statements they disagree with by getting that statement's sources removed.

I considered the possibility of just linking to pages on the Internet Archive but that may stifle discussions of current events since the IA doesn't serve any pages less than 6 months old. But the idea of linking to an external site instead of keeping the pages on Argulator got me thinking. Such a cache may be useful for other things besides Argulator, so maybe the cache should be a separate (but associated) site. Realizing that such a site would be useful in its own right got me to wondering why nobody had done that before, which got me to realizing that maybe they had. A few google searches later I found several sites which did exactly that. I then realized that Argulator should stick to its core competency and leave the web caching to the experts. Statements can link to any URL anywhere on the internet, but it is recommended for the sake of stability to link to one of these caching services. I hope Argulatists will realize that URLs pointing elsewhere might not have the same contents as they had when the statement was originally created, and take such statements with the appropriate amount of salt.

Why Argulator requires users to create an account before creating statements

Saturday, August 22nd, 2009

One thing I wanted to do with Argulator but didn't quite manage was to eliminate the login barrier - to allow people to use any part of the site without creating an account first. The idea was to essentially put all the site's data under multi-headed version control, and then put each temporary user's changes in its own branch, so that they wouldn't affect logged-in users or other temporary users. When a temporary user did decide to create an account, then those changes would be merged into the main branch.

The trouble with that is that arbitrary merges are very difficult. What happens if two temporary users happen to create identical statements (or statements which are similar enough that they would be duplicates if they had both been created by logged-in users)? What happens if a statement is deleted in one branch and created in another? Adding UI for resolving arbitrary merge conflicts would be difficult enough, but forcing users to go through it when they create an account would be downright user-hostile.

I've therefore come to the conclusion that the principle of "you can do anything without creating an account" only works for sites where users' actions don't really affect other users (such as a shopping site).

Error handling in Argulator

Friday, August 21st, 2009

One interesting thing I had to think about for Argulator was what to do with error messages. There are three types of errors that can occur in applications such as these when classified by the action that needs to be taken.

The first type of errors are things that real users can run into, and which are not problems with the program itself. For example, visiting a statement's page and finding that it has been deleted. These deserve user-friendly error messages and are usually pretty straightforward.

The second type are validation errors, for example someone trying to attempt SQL injection. There's no point giving a user-friendly error message for these as long as we're sure real users can't run into it, for example if we're doing the same validation on the client side, so these are generally handled just with a "die" call.

The third type are programming errors on my part (assertion failures, essentially). These also deserve user-friendly error messages (without details, just a generic "uh oh, something went wrong, we're on it") but they also send me email with some details so that I can try to figure out what's going wrong.

The trouble is it's not always easy to tell what is the second type and what is the third type. Some things seem like they would have to be programming errors but maybe there's some validation missing which means that they could be triggered by invalid input. Some things seem like they could only be triggered by invalid input but perhaps there's a programming error which could cause them to be exposed to real users.

I hope I haven't made too many mistakes classifying these error messages. When choosing whether a particular error is a type 2 or a type 3, I've tried to err on the side of type 3 because the consequences of getting it wrong are less severe (a type 2 misclassified as a type 3 just results in me getting spam, a type 3 misclassified as a type 2 results in me not getting notified, and the user getting an unfriendly error message). Please let me know if you see any error messages in Argulator that don't seem particularly friendly (for example, just text on an empty page instead of a popup div with a button on top of a normal page).

Other principles and innovations in Argulator

Thursday, August 20th, 2009

Some other little bits of Argulator that I'm quite proud of:

  • Pieces of UI that would be represented as a modal dialog in a desktop application (such as the search results, or the create user dialog) use an HTML div with absolute positioning, that "floats" above the rest of the page. Behind this is a window-filling div with a semi-transparent background, which causes the rest of the page to darken, focussing the user's attention on the dialog. The popup "window" can be dragged with the mouse and scrolling still works, so the rest of the page can still be read if one needs to refer to it during the dialog interaction.
  • When there is a script error, it is caught and transmitted to the server with an AJAX request. The server then emails it to me so that I can fix the bug. If this fails, we do nothing in order to avoid an infinite loop. I wish more sites would do this - there is so much broken Javascript out there it's pretty much impossible to surf normally with Javascript debugging switched on. This means I either need to switch my browser between "debugging mode" and "surfing mode" when starting/stopping work on Argulator, or use a different browser for debugging than I use for surfing.
  • Argulator uses 96-bit random character strings (encoded as 16 characters of 6 bits each) all over the place, for cookies, pseudo-cookies (for preventing cross-site request forgery attacks), salts and email authentication. These use few enough characters that a URL containing one easily fits on one line of an email message, but have enough entropy to make them unguessable for security purposes. They are generated using /dev/urandom if available, otherwise they are generated with a cryptographic hash, an entropy pool, and the current time.

Speaking of cryptography, Argulator's main cryptographic shortcoming is that the user's password is sent across the wire in plaintext (in a POST request, so it doesn't show up in server logs or caches). I thought seriously about hashing it on the client side (implementing a cryptographic hash function in Javascript). But in the end I decided that this was too much effort for too little gain. The danger of having your password stolen comes mostly from malware and keystroke loggers on the client side, which this does nothing to protect, or having your password stored in plain text in the database (it isn't - we salt it and hash it before storing it). Lots of other sites also send passwords in plaintext. And even if I did implement this, most users would not even be able to tell - really the way to solve this is to use https instead of http for authentication, which I will do if Argulator gets sufficiently popular that I can justify the expense. Then you'll get the little padlock icon in your web browser and you know you're secure. Until then, just don't use the same password for Argulator that you use for important things like your bank account.

How Argulator presents statements

Wednesday, August 19th, 2009

Even before you get around to creating any arguments there is the problem of how to display a complex network of statements in a user-friendly way. Part of the trouble here is that in general there may be a lot of statements in play at any given time, and a lot of different relationships between different statements. How do we present these relationships in a way that makes intuitive sense? And how can you be sure that the statement you're looking it is the one you think it is?

Programmers deal with structured information like this all the time. They have two major techniques for doing so - one is using nested sets of parentheses (or other brackets) and the other is naming some substructure and then using that name elsewhere. Both of these techniques are rather unintuitive to non-programmers and require some thought to use - if you're using brackets you have the problem of mismatching them, and with naming you have the problem of "okay, so c is a+b, where is c defined again?"

My "aha" moment was coming across the inline-block display style in HTML. This clever device allows one to have a piece of HTML which acts like an inline element from the outside and block from the inside. This means that 2 such blocks can be composed in (at least) 2 different ways: side-by-side and one-inside-the-other, and the web browser does the hard work of figuring out where to put each block and how big to make it. These relationships can be composed arbitrarily, and the blocks scale very well from tiny (just enough to hold a single identifying number) to so large you have to scroll the web page to see the entire contents. The only downside is that they don't work properly in FireFox 2 - I think this is now sufficiently rare to not be worth bothering with, but there may be a hack to get inline-block working in this browsers if there is demand.

So a rectangular block with a thin raised border became my "UI cue" that says "hey, this is a statement - you can go and give your opinion about this or argue for/against it". The block's border also acts as a set of parentheses of a sort - containing everything within it. An argument block, for example, encapsulates subblocks showing the argument it is arguing for or against and the substatements that make up the argument - these statements appear right there in the block rather than being incorporated by reference (avoiding naming problems) and the parentheses are automatic, implicit and intuitive.

The next problem is making the statements visually distinctive, so you can see at a glance which statements are which, while keeping them similar enough that anyone can see that they're all variations on the same basic object. There are several ways this is done. The most obvious is the background/border colour. Most statements have a white background and a grey border, but "special" statements (those that have some meaning to Argulator itself) are coloured - green for "for" arguments, red for "against" arguments and blue for deletion statements. This colour scheme reflects a colour scheme that can be found throughout Argulator: red also means "disagree" and "refuted", green also means "agree" and "corroborated" and blue also means "neither agree nor disagree". Colour vision is not a requirement for being able to use Argulator, but the extra visual clues in the colours should be helpful for those who can see them.

Since becoming a professional software engineer, one thing I have become very familiar with is bug databases. Bug databases are great because every bug is given a unique number, and if one wants to refer to a particular bug, one can just refer to it by its number. The person you're communicating with might have to look the bug up in the database, but it's still much more concise and accurate to say "bug 123456" than "that crashing bug Bob found last week". One often gets to know the numbers of the bugs that are currently important (like the phone numbers of close friends) - when this happens, one doesn't even need to look in the bug database to see which bug is being referred to. However, as an aide memoire when I'm writing an email about a bug I always try to remember to paste in the title as well as the bug number.

Argulator has a similar principle - every statement block has a number inside it, towards its left-hand side (or top-left corner if it spans multiple lines). This number uniquely identifies the statement. If you know a statement's number N you can visit its page just by typing the URL:, refer to it in a statement you're creating by typing {N} or use it in an argument by using its number as the substatement text. You can also negate a statement's meaning by negating its number, so anywhere you can use a statement you can also use the negation of that statement.

In a statement's block, the number serves two other purposes as well as uniquely identifying which statement you're looking at:

  1. The colour that is used to render the number gives a quick indication of the "average" opinion that people have expressed on that statement. As well as red, green and blue mentioned above, the colours can combine - red and green combine to form yellow meaning "polarizing", green and blue combine to form cyan meaning "uncontroversial" and red and blue combine to form magneta meaning "unpopular". Other hues and saturations represent different proportions of "agree", "disagree" and "neither agree nor disagree" opinions. Lighter colours mean more opinions, darker colours mean fewer. This is another visual hint that the statement is the one you're looking at.
  2. The number itself is a link that takes you to the statement's page, where you can see much more information about it, express your opinion on it and argue for or against it.

There are several other mnemonic cues next in a statement's block next to the statement's number - a coloured dot showing your opinion, an eye icon if it's a statement you're watching, and a red R or a green C to show if it's been refuted or corroborated. Together, these cues are a dense source of information about a statement but their deeper purpose is give each statement a unique visual "personality" richer than just its statement number alone. I hope that these cues will make it easy to tell at a glance that a given statement is the one think it is, even if you don't remember the number exactly.

There is one more aspect of statement block display, which is expanding and collapsing. Each statement block has a small black triangle in it after the number. The triangle points left if the statement is expanded (and its text is visible) and right if it is collapsed to just the number and icons. Clicking on this triangle toggles between the two states. This allows you to see the big picture at a glance (when statements are collapsed) or drill down to see the details if you want to. This bears some similarities to the "zooming user interface", another concept from "The Humane Interface".

Argulator's user interface

Tuesday, August 18th, 2009

One of the key problems I identified when first planning Argulator was how to make an interface which was sufficiently easy to use given the fundamental complexity of the tasks involved in manipulating a complex network of statements. In particular, UI for creating an argument was a big sticking point. An argument has several parts (at least one but as many as you like) each of which is a statement. When creating an argument, you're actually creating a pile of statements at once. However, some of those statements may already exist (once Argulator's database is sufficiently well-populated, I expect most arguments will recycle at least one statement). However, chances are that they use slightly different phrasing or spelling, so just checking for identical text is a bit of a non-starter. And if the database gets full of nearly-duplicate statements, Argulating will get very annoying - one would end up refuting the same thing 20 different times with slightly different phrasing each time.

To minimize duplicates, I realized that I wanted to force users to search for existing statements before creating a new one. This means doing some things on the server for each statement. If I used classic CGI this would be quite an annoying, unresponsive user experience:

  • Type in statement. Click to search. Wait for page to reload.
  • Click on search result to go back to argument creator. Wait for page to reload.
  • Once all statements are in place, click to create the argument. Wait for page to reload.

I almost implemented this but I thought the site would suffer greatly from being so clunky. On the other hand, I didn't want to make it a desktop application either as that would make it that much more difficult for people to try out.

The answer came in the form of AJAX. This was just starting to take off, but it was a while before it seemed to be something practical for mere mortals like me to implement, rather than deep magic that could only be done in something as complex as Outlook Web Access or Google Maps. Once I realized I could implement an AJAX application myself, things started to pick up. The sequence would be the same but with no page reloads the user experience would be much smoother and more intuitive.

Having decided to use AJAX (and make it a requirement for using the site) it made sense to AJAXify everything in sight - page reloads hardly ever happen in Argulator (except when you go to a different URL like the page for a different statement, user or definition). The main manipulations (setting an opinion, creating an argument, and creating a deletion statement) all happen in-page. Even logging in and out doesn't cause a reload. Whenever I thought of a way to use Javascript to make the interface more interactive, I did so. I think and hope the resulting interface is a pleasure to use. Whatever you're trying to achieve with the site, the next step should always be obvious - please let me know if it isn't.

One other UI principle I've tried to stick with is one from Jef Raskin's book "The Humane Interface" - there are no "are you sure" confirmation dialogs in Argulator, and just about every action is easy to undo (even deleting one's account can be undone for a year).

Introducing Argulator

Monday, August 17th, 2009

A while back I wrote about an idea I had for a website which allows arguing over the internet.

Well, it's taken almost 4 years but it's finally alive! The site is at It's a little different from how I originally envisioned it in some ways (having evolved somewhat over the years) but has many of the features I originally described.

Over the next week I'll be posting a series of articles about the development of the site.