Now I switched to ebooks some time ago; I don’t read books on dead trees anymore. I use an ebook reader I bought from a French company called Bookeen, which unfortunately appears to have imploded since. I also own a Sony PRS-505, a much better reader. Ebook readers are great: they use E-Ink displays instead of LCDs. This type of display does not emit light; it relies on actual black or white particles shifting position behind the panel. if a black particle comes up, that’s a dot on the display. This means:

• The displays are monochrome for now (though color versions are on the way)
• Reading on such a display does not strain the eyes, like a CRT or LCD monitor does. It really is exactly like reading on a slice of dead tree.
• It takes a long time to draw something on the screen — easily a second to refresh the display. These aren’t photons being shifted, after all, but real blobs of stuff.
• The display needs electricity only to shift those particles; once they’re in place, no power is required to keep them there — unlike LCDs, CRTs, and so on. This is called a bistable display. But hey, battery life is great: I typically charge my reader once every couple of weeks, while using it for hours nearly every day.

This is great technology, and obviously the future of books. Some people will dispute that, of course: they’ll come up with retarded arguments like the “texture” and the “smell” of books. These are the same kinds of people that will insist that music sounds better on vinyl, and movies look better in a theater. But you can’t stop progress: the music industry had its business model turned upside down by the internet, and the movie industry is getting its lunch eaten as well.

The Dying Dinosaurs Of Print, in the words of the incomparable Mike Cane, are mostly ignoring ebooks. Where they’re not ignoring it, they’re requiring DRM to restrict your freedom on what you can do with your purchased book. Is any of this starting to sound familiar? You bet. That’s another reason why I’m sure ebooks are the future: the publishing industry is just as blind to the coming disaster as the music and movie industries were.

The smartest player in the industry is Amazon, whose Kindle reader integrates seamlessly with its online ebook store. The ebooks on Amazon are cheaper than the dead tree version, as they should be — typically $10. But that’s only in the US, and only if you have a Kindle. I’m in Europe, and I don’t have a Kindle. In fact, they’re not for sale in europe, and neither are Amazon’s ebooks.

Sony sells ebooks through its online store — for US residents only. They sell the reader in the UK as well, in partnership with Waterstone’s, whose ebook store doesn’t even HAVE that book. Even if they did, something tells me they wouldn’t sell it to us non-brit European beasties anyway. There’s also Mobipocket, whose online store only sells DRM-encumbered ebooks in the outdated mobipocket format — and they don’t have that book either.

So where do I buy this book? Quite simply, I can’t. Nobody will take my money! But of course, just as in the music and movie industry, if you fail to sell people what they want to buy at the price they’re willing to pay, they’ll just download it for free. The book is already on some torrent sites, albeit as an audio ebook. I’m not interested in that: I want the text-only ebook version. But it’s only a matter of time until it appears.

So this is what I’m going to do: I’ll wait for the book to appear as a text ebook on some torrent site. If at that time I can legally buy it online for $10 (or even €10), I’ll buy it. If not, I’ll download it for free.

And that, ladies and gentlemen, is what’s going to kill the dead tree book publishing industry.

Therein lies the rub: broadband access in Belgium is controlled by a duopoly consisting of the incumbent operator, Belgacom, and a large cable operator, Telenet, which operates only in the northern part of the country. People in the southern part of the country pretty much get it in the shorts.

European law dictates that incumbent operators must share their networks, in order to create competition in a formerly monopolized market. This has been the case since 1998, and in many countries this made a big difference. Not in Belgium, however. The government owns over 50% of the shares in Belgacom — and is also in charge of regulating Belgacom. That, of course, is not a recipe for creating a competitive telecoms market. So the politicians did what politicians do: they stacked the regulatory agency (BIPT) with a bunch of toadies that let Belgacom run wild, and every so often skimmed the cream off of Belgacom’s massive profits, generally right in time to fill a hole in the government budget. Amazing how that works.

The result is a thoroughly distorted broadband market. This has not gone unnoticed by the European Commission; its 2008 report lays out the awful state of the Belgian broadband market in quite a bit of detail — it even mentions a few of Belgacom’s nasty anti-competitive moves.

So while Belgacom is required to offer wholesale, bitstream and unbundled access to other operators, they are allowed to make all of these offers prohibitively uncompetitive, expensive and complex. The wholesale offer is ridiculously uncompetitive; if you sell it to customers at the same price at which Belgacom offers the same product, you will lose money on every DSL line. The bitstream offer is very expensive and complex to deploy — more about this below — and the unbundled offer is even worse. Practically speaking, very few operators do unbundled DSL, and none do wholesale. The bitstream offer is somewhere between the two, and while it’s complex and expensive, it’s the only real choice ISPs have. This bitstream offer is called BROBA.

So what makes deploying BROBA so hard? For starters, the whole thing is based on ATM — an outdated network technology that died years ago, in every aspect of networking except DSL. ATM equipment is expensive, and very limited in speed; where nowadays every computer comes with a gigabit ethernet interface built in, the Belgian BROBA infrastructure is basically built on STM-1 cards — those are 155Mbit/s, and ATM guarantees at least 20% overhead, so those really aren’t going to do more than 140Mbit/s of IP traffic. And that’s just one of the reasons ATM died years ago.

Another problem is that Belgacom forces ISPs to buy VPs (virtual links) to every single local exchange where they want to offer DSL service, and more VPs if you want to offer different classes of service. There are about 1000 of these, and Belgacom forces you to buy access lines to various (arbitrary) “zones” of the country to transport these VPs. Your customers then come in on a certain VC (virtual circuit) over those virtual links. You terminate these in something called a BAS, which authenticates the customer and connects him or her to the internet. This is all massively expensive, and requires a lot of expertise to boot.

In the meantime, the rest of the world has moved off of ATM, and onto ethernet. It’s a huge trend in the telecoms industry: everything IP, and everything on ethernet. There are enormous advantages to ethernet: the per-port cost is very cheap, and new standards for ever-higher speeds, quality of service, troubleshooting, and multiplexing are being developed all the time.

So Belgacom started moving off of ATM years ago, along with every other large telecom operator. But their BROBA offer didn’t move off of ATM: keeping ISPs on expensive ATM gear suited them just fine, thanks very much. They have tons of ATM gear anyway, so keeping it around for the regulated offer doesn’t cost them much. Of course, even the toothless BIPT would at some point start making noises about this; it was only a matter of time until they required Belgacom to offer BROBA on ethernet to ISPs. So about four years ago Belgacom set about designing an ethernet interconnect setup that would somehow disadvantage the ISPs.

Now there are really only two ways to multiplex ethernet links: MPLS and VLANs. MPLS is a tag-switching technology very popular in large networks; it allows various services like QoS and VPNs to exist as a kind of “overlay” onto the physical ethernet backbone. VLANs are virtual ethernet networks, very popular in smaller networks and easy to use, but don’t scale as well as MPLS. So the logical way to hand over customers from a telco providing DSL service to an ISP providing internet access for them, is simply to put each customer in his own VLAN. Easy to implement, easy for ISPs to filter, route, put in their own VPN, etc.

So Belgacom decided that this was the one thing they were NOT going to do. One of the requirements they set for the new BROBA over ethernet design was that ISPs would NOT be able to identify customers by their VLAN ID on a certain link. So that’s what they did. Amazingly, the BIPT outdid themselves in this case: they didn’t even demand BROBA on ethernet from Belgacom. This was four years ago. Seriously.

Belgacom put forward their new BROBA over ethernet design recently, and in addition to the deliberate VLAN sabotage, they also put in a few more zingers! They won’t support anything but PPPoE — so you’ll be buying new modems for most (all) of your existing customers, and swapping them out at the exact moment of migration to the new ethernet design. If you want to check if the migration worked, well, you can forget about that too, since they won’t support ethernet OAM (it’s “not possible”, according to Belgacom, in a hilariously deadpan lie).

This is not the usual Belgacom attempt at sabotaging anything that smells of competition. They’ve always tried nasty stuff, but they could only go so far. The BIPT, toothless as it is, still had to abide by European law. So while not competitive, there was in the end something of a market. But recently, Minister Q — an unusually clued in politician, thoroughly in favor of free markets — managed to get a law voted that gives the BIPT more power over operators. He then threw out the BIPT’s entire board and chairman. Replacing them takes a while, however, and many political parties will try to stack the board with more toadies that will keep Belgacom from losing ground in the market; they like filling holes in their budget, after all. So with the old board getting thrown out and the new board members getting vetted, and then no doubt taking some time to get their feet wet in such an important regulatory body, we’re looking at a period of no regulation at all.

That, presumably, is why Belgacom published their new design now. It’s a total attack on what little broadband competition there is in Belgium; they mean to kill off what little there is, and to do it while the regulator is not in fighting shape.

The broadband market in Belgium has never been healthy, but if Belgacom gets away with this there won’t be any competition at all. That’s what makes this latest nastiness on Belgacom’s part so different: it’s a flanking attack on the entire market at once.

Belgacom has always been against competition in what it regards as its domain; this is very much in its DNA as a company. No amount of timid regulation or fines will tame it; not even the larger fines the BIPT can now apply. The only way Belgium will ever have a truly healthy and competitive broadband market is if Belgacom is forced to spin off its local networks; i.e. putting the copper wire, the equipment it terminates in, and the networks connecting the buildings or street cabinets that equipment is in, into another company, which then offers only wholesale services to operators, where Belgacom is just one of the operators.

This was done in the UK, for example, where BT spun off Openreach. This was also done in Belgium for the electricity and natural gas markets, where the incumbent (Electrabel) was forced to let go of their local networks. Electrabel, you see, was not majority-owned by the Belgian government. And this is the problem, of course. The Belgian government will never neuter Belgacom like this while it’s the majority shareholder. And in Belgium’s current unstable political climate, there isn’t a soul that will even try to sell off Belgacom. Unless they need a lot of money in short order. So do we have to hope for massive budget shortfalls before this gets fixed?

Full disclosure: I am an independent consultant, and among other things I often do work in the Telecom and ISP market in Belgium. I do not, however, have any financial interests in any of them; I am simply passionate about free markets, and utterly outraged about this latest stunt Belgacom is trying to pull.

While testing, and indeed in the first few months of the service being live, I saw the update files were all pretty small — a couple of MB usually — so like the idiot that I am, I wrote the code for sending an update via the web service like this:

• In the webserver child process, read the entire file into memory
• output that content towards the webserver via CGI

Not a problem, works like a charm. Then they uploaded a 200MB update.

The child process reads in the 200MB file, thus growing to that size in memory. The webserver is apache, and it gets that 200MB of content from the child process, thus also growing to 200MB in size. For some reason I don’t quite understand, apache does NOT let go of the child process, even though it demonstrably has all its output. So I now have 400MB worth of memory in use, for a single download. Five customers started up their application, and the 2GB of memory in the server melted like snow in a name-writing contest. The memory can’t be shared, because the child process handles putting it in memory; the kernel can’t possible know it’s all the same bunch of bits. It can’t even be shared between the apache and its child process, the CGI interface is too crude for that. It’s just a pipe, forget about mmap or anything like that.

The machine eventually ran itself out of swap, and started killing processes. That’s when I noticed, of course. I threw a ton of swap at the machine to keep it going, found the code that caused the crazy memory usage, put a brown paper bag over my head, and started looking for a better solution.

The obvious way to handle this is to have the kernel send the file to the remote end, and have the userspace process — apache in this case — wash its hands of the whole thing. Linux can do this with the sendfile system call. It takes input and output file descriptors, and just sends one to the other. The calling process simply waits for sendfile() to return.

But how can I do this from an apache subprocess? It doesn’t have access to the remote end’s socket file descriptor, only apache has this. That’s where mod_xsendfile comes in. It examines headers coming from subprocesses, looking for a X-Sendfile header. If found, it takes the value (everything after the colon) to be a filename to send to the remote end. The module then discards the content sent by the subprocess, and calls sendfile() instead. So instead of reading in the whole file and outputting it back to apache, I now return only the full path to the file in an X-Sendfile header, and the whole thing is solved. Apache now needs no memory at all per update it sends out, so it’s a lot more scalable.

Apache has a fairly crufty old codebase, and it’s not the most dynamic of projects — but I still don’t understand why mod_xsendfile isn’t a standard module. This should be in every apache distribution — it solved this problem beautifully. Nevertheless, the module is available, and it works. There’s life in the old girl yet!

I should note that lighttpd has this functionality built in, as a part of the included mod_fastcgi. This is perhaps no surprise — lighttpd is specifically intended to be more lightweight and scalable, and perform better, than apache.

Decoding 1080p in real time is quite a bit of work; it takes about 80% of one of the cores in my desktop’s Intel Q6600 CPU. So how does the Popcorn Hour do it, without even a fan to cool it down? The heart of the box is the Sigma Designs 8635 chip. This is a SoC specifically designed for media centers. It has a 300Mhz MIPS core, and a bunch of DSPs used for decoding video and audio, as well as video scaling, 2D graphics acceleration, and much more. The chip has an incredible number of interfaces — everything from audio/video output ports to USB, ethernet, PCI and even ISO 7816. You can connect this chip to most things you buy in a home electronics store, and a few things in your wallet, too :-)

The PCB has some other components on it — RAM, a SATA-USB interface chip, an ethernet PHY and a HDMI 1.3a support chip — but these are all just peripherals hanging off of the 8635. The chip runs cool, evidently — it has a pretty regular heat sink on it.

The Popcorn Hour runs a piece of software called Networked Media Tank (NMT). Both the box and this software are made by a company called Syabas, which licences out the software to other manufacturers of media center boxes based on the 8635 chip. There are lots of these now; it seems most media centers supporting 1080p are in fact running NMT. They’re all based on the 8635, and all come with the same generic remote control unit.

Unfortunately, NMT is pretty awful. Coming from XBMC, the Rolls Royce of media centers, using NMT is a huge step backward. The user interface is a disaster in usability, and it’s ugly and slow. It has only very basic features, nothing fancy or interesting. To give an example: a recent update finally gave the ability to resume watching a movie from where you left off. Hot stuff!

It’s not going anywhere, either. The Syabas people are evidently concentrating on bug fixes and implementing basic features like that resume. They aren’t doing a major overhaul of the UI or feature set. This software is never going to be appreciably better than it is now.

So the obvious question is, can you run XBMC on the Popcorn Hour? That would be the best of both worlds: great software running on great hardware. Unfortunately, it’s not quite that simple. For one thing, XBMC runs only on Intel hardware (and apparently has some machine code, so it’s not just a matter of recompiling on another platform). It also relies on OpenGL support for its UI, and uses the CPU for decoding video — it has no code for offloading that type of operation to another chip. In short, XBMC was made for a very different hardware platform. The XBMC developers don’t appear to be very interested in porting to a MIPS architecture either. To top it all off, driving the DSPs in the 8635 appears to be somewhat of a closed affair: Sigma Designs doesn’t even publish a datasheet for it. An open source effort to make decent software for the box is thus not just a lot of work, but may actually involve reverse engineering the video decoding hardware… a non-trivial task.

So an improved UI for my media center doesn’t seem likely. Ah well, it wasn’t all that expensive, and when something better comes out I’ll look at that. So what would a next-generation media center be? Sigma Designs may come out with a new version of their chip which supports 3D acceleration, and somebody else might then write a decent UI for it. But I dislike closed platforms, and I don’t think I’ll throw more money at a media center that is based on a Sigma Designs chip. As long as it’s closed, we’re going to be stuck with crap like NMT. After all, if Sigma Designs opened their chip up, their product would have more software written for it, and Syabas would have to clean up their act. Yes, open source is good for competition.

There is light at the end of the tunnel, however. Nvidia just came out with the ION platform. This is a low-power SoC featuring much the same sort of built-in peripheral interfaces as the Sigma Designs chip, except this one doesn’t have a CPU core on board — it has a GeForce 9400M graphics chip instead. Nvidia combines this with an Intel Atom CPU. The Atoms are Intel’s low-power x86 CPU, marketed towards netbooks and such. Unfortunately, where CPU performance is concerned, they’re crap. However, Nvidia has announced an upcoming version of ION that uses VIA’s Nano CPU — a direct competitor to the Atom.

The interesting bit about the ION platform is the hardware video decoding on the GeForce portion of it. It can decode 1080p in real time, and I assume (considering its target market) the whole setup can be passively cooled. Nvidia’s track record with opening up specs for driving their chips is spotty at best — their linux graphics drivers are still closed, but they did come up with CUDA (also supported on the ION, by the way — you can encode video with it!). We’ll have to see if they’re more open with this. If they are, an XBMC port to the platform seems likely.

So I thought I’d look into it, and see if it can be hacked in some way or another. I didn’t succeed, unfortunately, and it’s not really worth spending a huge amount of time on, but I thought I’d at least document what I did find out.

The DVD has a regular ISO9660 filesystem on it. On the file system are a bunch of .KWI files, and a directory called IDX. KWI stands for KIWI, which is an open standard that describes the format map media in car navigation systems. The standard is managed by a consortium called Kiwi-W, originally made up of Japanese car manufacturers but which now has a wider membership. The standard describing the KIWI file format(s) can be found here.

A couple of things to note:

• The map, routing and POI data on the DVD is not encrypted. You can just strings them and see places and such fly by.
• The firmware for the navigation system is also on the DVD, in the LOADING.KWI file.

Every KWI file on the DVD has its own binary format; other than a set of defined data types, they don’t really have much structure in common. The names are standard, however. Since we’re looking for the code that implements the nanny features, the LOADING.KWI file is what we need: that’s where all the code is. It’s essentially a container format, much like a ZIP file. It contains a series of modules with some metadata thrown in. I’ve written a tool (see below) to view or extract the modules from a LOADING.KWI file. This is what’s in my DVD’s LOADING.KWI file:

number of systems found: 1
System 1
    Manufacturer ID: 0f 56 a3 00 3c 3c 8a 00 07 00 01 66
    Number of modules in this system: 5
    Module 1:
	Name: V206
	Version: 100J
	Category: Program
	Valid from: 2006-07-17
	Address: 0x800
	Size: 15.2 MB

    Module 2:
	Name: V214
	Version: 100K
	Category: Program
	Valid from: 2006-07-17
	Address: 0xF3F000
	Size: 15.7 MB

    Module 3:
	Name: V308
	Version: 100E
	Category: Program
	Valid from: 2006-06-09
	Address: 0x1EE9000
	Size: 20.5 MB

    Module 4:
	Name: V309
	Version: 100E
	Category: Program
	Valid from: 2006-06-09
	Address: 0x336D000
	Size: 20.4 MB

    Module 5:
	Name: V312
	Version: 100E
	Category: Program
	Valid from: 2006-06-08
	Address: 0x47DC800
	Size: 18.9 MB

The manufacturer ID shown consists of a number of fields that describe the location, in latitude and longitude, of the navigation DVD’s manufacturer’s HQ. The 13th byte, 07, means they’re on the 7th floor of the building at that location. You can’t make this stuff up.

Anyway, if we look at a hexdump of the start of the first module, called V206, this is what we see:

00000000   2E 4C 44 52  0D 0A 56 32  30 36 31 30  30 4A 0D 0A  .LDR..V206100J..
00000010   4A 75 6C 20  31 34 20 32  30 30 36 00  00 00 0D 0A  Jul 14 2006.....
00000020   32 30 3A 33  35 3A 31 37  00 00 00 00  00 00 0D 0A  20:35:17........
00000030   41 49 53 49  4E 20 41 57  20 63 6F 2E  4C 54 44 00  AISIN AW co.LTD.
00000040   00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00  ................

So this is some kind of header, and Aisin AW Co Ltd. is clearly the bunch on the 7th floor that made this navigation DVD. They’re also a leading member of the Kiwi-W consortium. The modules contain other interesting stuff: there are icons in there, in regular BMP format. I extracted some (you can find them by looking for the letters BM — 42 4d in hex — in the modules). They were icons I’ve seen on the navigation display.

Some of the code in there was apparently compiled with symbols. There is also lots of text to be found: debug output, various strings — and then there’s this:

00891070   67 61 E7 E3  6F 2E 00 00  44 72 69 76  65 20 73 61  ga..o...Drive sa
00891080   66 65 6C 79  20 61 6E 64  20 6F 62 65  79 20 74 72  fely and obey tr
00891090   61 66 66 69  63 20 72 75  6C 65 73 2E  0A 57 61 74  affic rules..Wat
008910A0   63 68 69 6E  67 20 74 68  69 73 20 73  63 72 65 65  ching this scree
008910B0   6E 20 77 68  69 6C 65 20  76 65 68 69  63 6C 65 20  n while vehicle
008910C0   69 73 0A 69  6E 20 6D 6F  74 69 6F 6E  20 63 61 6E  is.in motion can
008910D0   20 6C 65 61  64 20 74 6F  20 61 20 73  65 72 69 6F   lead to a serio
008910E0   75 73 20 61  63 63 69 64  65 6E 74 2E  0A 4D 61 6B  us accident..Mak
008910F0   65 20 73 65  6C 65 63 74  69 6F 6E 73  20 6F 6E 6C  e selections onl
00891100   79 20 77 68  69 6C 65 20  73 74 6F 70  70 65 64 2E  y while stopped.

That’s the disclaimer, alright. The “I Agree” bit is there as well. So what you do to find the code that puts up the button and waits for the user to hit it, is note down where in the file that string is, and then look for code that references something at that location. And NOP it out.

Alas, it’s not that simple in this case: the module itself is clearly in some format I don’t know. It may well be an executable format, like ELF, or it may be a container format containing several executables and a bunch of icons. I’m not even sure what CPU the navigation system uses, so randomly looking for references isn’t going to do much good.

I don’t really have any more time to spend on this, and I really don’t want to rip the navigation unit out of the car to look at what CPU is on the motherboard. If anyone has more information, let me know. The car is a Lexus IS 220d, but other Toyota/Lexus models are likely to have the same unit.

The tool I wrote to view/extract modules from the LOADING.KWI file is called “kiwi”, and can be downloaded here. You’ll need python on your system.

mount -t debugfs none_debugs /sys/kernel/debug
modprobe usbmon

Then you can just cat USB traffic like this:

cat /sys/kernel/debug/usbmon/1u

It all comes out in an ASCII dump format which is easily parsed. Every USB bus also has a device file where you can sniff the raw packets straight off the wire. More info in the usbmon documentation.

But while it’s all easily parsed if you need it, there aren’t really any tools around that do it for you. That is… except for libpcap. Libpcap is the power behind the throne of the venerable tcpdump tool. Tcpdump is not much more than a command line parser and pretty-printer of various network protocols. The heavy duty lifting is done by libpcap, not least by providing a cross-platform API for sniffing devices, something that is otherwise non-standard and different on every platform. It’s great, I’ve used it before (in capstats) and it’s very easy to use.

Libpcap on linux supports usbmon sniffing, which means you can use tcpdump to sniff a USB port and write this to a capture file. But best of all: wireshark, the all-singing all-dancing network analyzer that uses tcpdump capture files, has USB support as well.
So this is the result:

The screenshot shows a filter applied to only see device 18 on the sniffed USB bus. That’s an arduino, i.e. an FTDI USB-serial chip. The FTDI chips send status updates to the USB host system every 16ms (!). The status update consists of a two-byte message (described here). This is actually present in every packet coming in from the FTDI chip; status updates just don’t have any other data. So for a clean sniffing session from the arduino, we want to filter out any packets that are < 3 bytes in length.

The end result is serial data which the arduino sent to the host system. The screenshot shows a session on my arduino shell, arsh. This is great stuff – wireshark includes a massive amount of analysis tools and lots of options for filtering and otherwise massaging your captured data.

You need relatively recent versions of libpcap, tcpdump and wireshark for this. I compiled all three of these out of their respective repositories (easy compile all). On my ubuntu system, the libpcap version was particularly old. Tcpdump doesn’t have a pretty-printer for USB data yet, so you can only dump to a capture file for processing by wireshark.