Research, reverse engineering and rants

Tackling Power Over Ethernet

Sun, 08 Nov 2020 22:08:03 +0100

Linux has no support for Power over Ethernet (PoE) in any way. There are many ethernet switches on the market that support PoE, and many of those switches run Linux, but none of those are supported in the mainline kernel, and their PoE support has not been upstreamed.

This situation is changing with the Realtek RTL83xx project. This series of SoCs is very popular, found in dozens of distinct models of many different vendors. The kernel code is coming together nicely. Many systems built on these chips support PoE, and this will need to be handled like any other feature on those chips. And there is much to support about PoE: number of ports, power modes, power budget, priority, and much more.

An opportunity thus represents itself: to create a new kernel subsystem and accompanying userspace tools, and have it automatically become the standard way of supporting PoE on Linux.

Let’s take a look at what’s needed.

Hardware

The voltage on current-carrying lines of PoE ports is in a fundamentally different range, anywhere from 44V to 57V, than what a switch uses internally. The chip on the switch’s motherboard that provides this voltage is called the Power Sourcing Equipment (PSE) controller. It lives in a different power domain than the rest of the switch, typically with its own hookup to the system’s power supply. That means communication with these PSEs has to go via an electrically isolated path.

Generally the SoC doesn’t talk to the PSE directly; there’s a dedicated microcontroller of some sort that sits between them, mediating what the SoC can drive (often only UART) with what the PSE needs, such as I2C. In some cases these two form a product pair, such as the Microsemi PD69100/PD69108 management/PSE controllers, respectively. But usually the management microcontroller is something more generic, such as an STM32F100.

We’ll need to know the protocols the SoC has to speak to these different microcontrollers if we’re going to control PoE. It’s not always documented, but we’ve got a good handle on it. Indeed, the firmware running on these microcontrollers may turn out to be an interesting target for replacement with an open source version; crazier things have been done.

Kernel

A PoE kernel subsystem should do the following:

Provide a framework for drivers that know about the protocol which the PSE, or its intermediate microcontroller, needs. Since the other endpoint of that protocol is dependent on the switch model, this information needs to live in the device tree. The drivers will also need to be aware of the firmware version on the other end, at least until we can replace it.
The device tree should also provide information about the model’s PoE capabilities, in order to mediate requests from userspace. Managing the power budget, for example, should be done here – no need to force every userspace client to keep track of this.
A userspace API, either via an ioctl on a device special file /dev/poe or perhaps a more modern sysfs interface. It’s easy enough to work out what this API should provide: whatever the PSE/MCU protocols provide must have an equivalent in this API. Additionally we can add discoverability to that: the userspace application should be able to find out which PoE capabilities the system has, on which ports.

Userspace

While a userspace API is all good and well, using it involves awkward calls to ioctl, with lots of symbols pulled in from kernel headers, or lots of error-prone messing with sysfs files. That’s pretty laborious, and extra hard for scripting languages to tackle. A better way to handle this is to write a small library to sit in front of the awkward operations and symbol soup. A good example if this is libgpiod, which sits in front of the userspace interface to the kernel gpio subsystem. A library like this is also pretty easy to write language bindings for.

A quick and easy way to use PoE functionality from the shell is also a good idea. You need such a shell tool to test the library as it’s developed, anyway.

Writing a web interface that can work the system’s PoE functionality is then a matter of using the language bindings of the library as well. Easy!

I’m working on lots of different aspects of the Realtek RTL83xx kernel code at the moment, but this is an interesting enough opportunity that I think I need to get started on it – before somebody else does!

In Through the Out Door

Sun, 17 Nov 2019 20:29:24 +0100

In the previous episode of this reverse engineering effort, we finally found a good way to get hold of the real hardware register addresses, and we extracted the UART registers to begin with.

What’s needed for a minimal booting kernel is first interrupts, meaning information on how to drive the SoC’s interrupt controller, and timers, also a facility supplied by the SoC. We really only need one timer: the system timer, which supplies Linux’s “ticks” – it’s what drives the scheduler. When you enable the system timer, you set it to fire at a set interval; firing means generating an interrupt which you’ve linked it to.

The interrupt controller in the GK7101 is called the VIC (Vectored Interrupt Controller); there are two of them, each handling up to 32 interrupts. ARM provides a standard interrupt controller facility with its CPU cores, called the PrimeCell VIC. As it turns out Goke’s VIC is almost, but not quite, compatible with it. This is turning into a theme; we saw the same oddity with its not-quite-16550 UART.

No matter, we have the SDK source code to figure out how to work it. First let’s get the registers from the SDK header, and run them through our HAL emulator:

/****************************************************/
/* Controller registers definitions                 */
/****************************************************/
#define VIC_IRQ_STA_OFFSET      0x30
#define VIC_FIQ_STA_OFFSET      0x34
#define VIC_RAW_STA_OFFSET      0x18
#define VIC_INT_SEL_OFFSET      0x0c
#define VIC_INTEN_OFFSET        0x10
#define VIC_INTEN_CLR_OFFSET    0x14
#define VIC_SOFTEN_OFFSET       0x1c
#define VIC_SOFTEN_CLR_OFFSET   0x20
#define VIC_PROTEN_OFFSET       0x24
#define VIC_SENSE_OFFSET        0x00
#define VIC_BOTHEDGE_OFFSET     0x08
#define VIC_EVENT_OFFSET        0x04
#define VIC_EDGE_CLR_OFFSET     0x38

The vendor driver thinks the VIC lives on the AHB at 0xf2000000, with the two VICs at offset 0x8000 and 0x9000 respectively. So the VIC1 status register would be at 0xf2008030. Running these through the HAL for translation:

sumner: for reg in 30 34 18 0c 10 14 1c 20 24 00 08 04 38; do ./emu-hal halcode-fromdevice 0xf20080$reg; done
0xf2008030 -> 0xf0003000
0xf2008034 -> 0xf0003004
0xf2008018 -> 0xf0003008
0xf200800c -> 0xf000300c
0xf2008010 -> 0xf0003010
0xf2008014 -> 0xf0003014
0xf200801c -> 0xf0003018
0xf2008020 -> 0xf000301c
0xf2008024 -> 0xf0003020
0xf2008000 -> 0xf0003024
0xf2008008 -> 0xf0003028
0xf2008004 -> 0xf000302c
0xf2008038 -> 0xf0003038

Hey, look at how these are suddenly sorted! And how they suddenly match ARM’s PrimeCell VIC layout! It makes me wonder whether there’s an extra license charge for using ARM’s UART, VIC and so on.

The timer subsystem officially lives at the APB base 0xf3000000 + timer offset (which is actually 0), and translates to post-MMU addresses like this:

sumner: for reg in 0c 00 04 08 14 18 1c 20 24 28 30 34 38; do ./emu-hal halcode-fromdevice 0xf30000$reg; done
0xf300000c -> 0xf100b030
0xf3000000 -> 0xf100b000
0xf3000004 -> 0xf100b008
0xf3000008 -> 0xf100b00c
0xf3000014 -> 0xf100b010
0xf3000018 -> 0xf100b018
0xf300001c -> 0xf100b01c
0xf3000020 -> 0xf100b020
0xf3000024 -> 0xf100b028
0xf3000028 -> 0xf100b02c
0xf3000030 -> 0xf100b004
0xf3000034 -> 0xf100b014
0xf3000038 -> 0xf100b024

Making a long story short: I implemented interrupt controller and timer drivers in a mainline kernel, and it didn’t work. Fortunately we’ve got that early_print() facility. By inserting those liberally, I figured out the kernel hung in init/calibrate.c:calibrate_delay_converge(). That’s the first kernel function to use the timer: it makes some precise measurements for delay loops and such, and does this by busy-looping waiting for jiffies to change. Jiffies is updated only by the system timer; if that never fires, you’ve got a hang.

The big question here is what’s broken: either one of interrupts or timer not working would break this function.

On the assumption I’d overlooked some hardware initialization somewhere, I dug deeper into the SDK source and decompiled vendor kernel… and found exactly that. There is some hardware setup in, of all places, the pre-kernel decompressor. From the SDK source:

gk_gpio_clrbitsl(GK_PA_GPIO0 + 0x3c, 0x00000001);
#ifdef CONFIG_PHY_USE_AO_MCLK
gk_rct_writel(GK_PA_RCT + 0x024, 0x00124021);
gk_rct_writel(GK_PA_RCT + 0x078, 0x00555555);
gk_rct_writel(GK_PA_RCT + 0x084, 0x00000004);
gk_rct_writel(GK_PA_RCT + 0x080, 0x00000001);
#endif

//misc clock configure
// SFLASH ioctrl
gk_rct_writel(GK_PA_RCT + 0x0198, 0x00000011);

// Sensor ioctrl
gk_rct_writel(GK_PA_RCT + 0x019C, 0x00000012);

The write calls in the PHY_USE_A0_MCLK block (which is defined), certainly look like they might be related to a system clock. Alas, this made no difference: still a hang at the calibration function.

This is rather hard to debug. I don’t have JTAG; the GK7101 may well have it, but it’s certainly not broken out on this camera board. The SoC is a BGA package, so there’s not going to be any probing pins for JTAG. The Linux kernel certainly supports this class of CPU, so interrupt vectors and handlers should just work here. I’ve gone over my code lots of times, and can’t find anything wrong.

On the assumption I must therefore still be driving either the interrupt controller or timer facility wrong, the way forward is clear: go over the running vendor kernel again until I find what else it’s doing. You can only stare at the same bits of code so many times though.

What I’d really like to do is get a log of all the hardware register accesses the vendor kernel does, so I can compare that to my code. But of course the Linux kernel doesn’t have anything like a register-level tracepoint, these are just memory access reads/writes after all.

But… the vendor kernel does have such a facility: that horrible HAL! It sits exactly where we need a trace facility: between drivers and their register-level access. What if we could get the HAL to not just do its read/write thing, but also log what it did, and which function asked it to?

We’d have to intercept calls to the HAL. That’s not too hard: we know where it lives, where its jumptable is, and we have access to it at U-Boot time. After all, U-Boot puts it in memory before the kernel is called. It turns out that the HAL jumptable – the hw_ops structure – is actually populated by the call to hal_init(). So we need to intercept the call to hal_init() by the vendor kernel’s decompressor as well. Why not do both at once: change hal_init() to call some code of ours after it’s populated the jumptable, so we can then:

Save a copy of that jumptable somewhere.
Populate the jumptable with our own functions, which log the call and then call the original function from the saved jumptable.
Boot the vendor kernel, and grab the log from memory.

Of course memory is rather a difficult facility at this level. It turns out that hal_init() is called twice:

Once by the decompressor, with arguments that make the HAL translate from physical addresses, since the MMU is off at decompression time.
A second time during proper kernel startup, this time with arguments that are virtual addresses.

That means our logging code will need to figure out whether it’s running with or without MMU, and adjust its log memory pointer accordingly. This is normally done by a call to the MMU coprocessor, but there’s an easier way: the decompressor runs from the 0xc1000000 block (where U-Boot put it), but it decompresses the kernel into the 0x80000000 block, from where the kernel then runs. So checking the caller function’s high byte will tell us where main memory lives. This is especially easy on ARM: the caller’s address is in the lr register. You don’t even have to go find it on the stack!

As to memory location regardless of MMU prefix: we’ll just drop it in the middle of the RAM range. All things being equal, a kernel boot shouldn’t use so much memory that it would reach halfway its RAM, even on this board’s measly 16MB.

This is all low-level enough that it needs to be done in machine code. The code is here. You’ll have to excuse the crudity of the code: it’s the first time I’ve played with ARM machine code. Incidentally, this technique of inserting a jump right into some other function, and having that function end with running the instruction that was lost and then jumping back, is called a wedge. At least that’s what it was called in the Commodore 64 days; over in serious business PC land they called it a TSR (Terminate and Stay Resident). TLAs make everything better, don’t they?

From the U-Boot shell we can use the tftpboot command to load the compiled blob of our wedge at 0xc0016000, and call it with “go c0016000”. The wedge is now inserted at the end of hal_init(), where it will switch the jumptable around as explained above. Booting the vendor kernel will populate the log. I’ve made a custom root filesystem with the awesome buildroot, and made the vendor kernel load that off a microSD card. We can then use memtool to grab the log from memory. Piping it to gzip and scp’ing it over the network for analysis was the quickest way to get the log.

Each call to a HAL function saves four 32-bit items:

The caller’s address.
An integer denoting which function in the jumptable was called, i.e. a number from 0 to 17.
The first argument to the call – for read functions this is the address, write functions have the value to be written here
The second argument; only used for write calls, is the address to be written.

Here’s a sample:

0000000: 080a 00c1 0400 0000 3c90 00a0 0000 0000  ........<.......
0000010: 140a 00c1 0700 0000 0000 0000 3c90 00a0  ............<...
0000020: 280a 00c1 0700 0000 2040 1200 2400 17a0  (....... @..$...
0000030: 3c0a 00c1 0700 0000 0000 0000 7800 17a0  <...........x...
[...]

The first entry shows an instruction at 0xc1000a08 called hw_ops->hw_readl(0xa000903c). The second one translates to hw_ops->hw_writel(0, 0xa000903c).

A little Python makes quick work of this:

0xc1000a08 hw_readl(0xa000903c)             
0xc1000a14 hw_writel(0x0, 0xa000903c)       
0xc1000a28 hw_writel(0x124020, 0xa0170024)  
0xc1000a3c hw_writel(0x0, 0xa0170078)       
0xc1000a50 hw_writel(0x4, 0xa0170084)       
0xc1000a64 hw_writel(0x1, 0xa0170080)       
0xc1000a78 hw_writel(0x11, 0xa0170198)      
0xc1000a90 hw_writel(0x112032, 0xa017008c)  
0xc1000868 hw_readl(0xa0005014)             
0xc1000884 hw_writel(0x55, 0xa0005004)      
0xc1000868 hw_readl(0xa0005014)             
0xc1000868 hw_readl(0xa0005014)             
0xc1000884 hw_writel(0x6e, 0xa0005004)      
0xc1000868 hw_readl(0xa0005014)             
0xc1000868 hw_readl(0xa0005014)             
0xc1000884 hw_writel(0x63, 0xa0005004)

See those hw_writel() calls with 0xa0005004 as the address? That’s the UART input/output register, and it’s writing 0x55, 0x6e and 0x63 – that’s ASCII for “Unc”, the start of “Uncompressing Linux…".

The caller address is interesting to see, but looking up which function that’s in is going to be a bit of work. Fortunately we have a shell on this running kernel, and Linux publishes its function symbol table in /proc/kallsyms – not for the decompressor, but for the kernel proper. If we feed those symbols into a Ghidra disassembly of the decompressed kernel, it can give all those functions names in its disassembly.

Ghidra also has a very, very elaborate API (it’s in Java, so overengineered to the gills). We can use that to feed the caller address to an API function that determines the function that address is in, and show the function name next to the caller.

Also, the log output shows the addresses the vendor drivers think they’re writing to, but those are of course pre-HAL. But since we have our handy HAL emulator, we could enrich our log output still further by adding the real address it ends up using – very handy if you’re writing drivers that don’t use this HAL nonsense. Here’s what the enriched log looks like, starting right after the decompressor handed over to the kernel proper:

0x804ea6e0 gk7101_map_io                  get_version(0xc0015ad0)           -> 0x0
0x804ea8d0 gk7101_init_irq                hw_writel(0x0, 0xf2008000)        -> 0xf0003024
0x804ea8e4 gk7101_init_irq                hw_writel(0x0, 0xf2008008)        -> 0xf0003028
0x804ea8f8 gk7101_init_irq                hw_writel(0x0, 0xf2008004)        -> 0xf000302c
0x804ea914 gk7101_init_irq                hw_writel(0x0, 0xf2009000)        -> 0xf0010024
0x804ea928 gk7101_init_irq                hw_writel(0x0, 0xf2009008)        -> 0xf0010028
0x804ea93c gk7101_init_irq                hw_writel(0x0, 0xf2009004)        -> 0xf001002c
0x804ea950 gk7101_init_irq                hw_writel(0x0, 0xf200800c)        -> 0xf000300c
0x804ea964 gk7101_init_irq                hw_writel(0x0, 0xf2008010)        -> 0xf0003010
0x804ea978 gk7101_init_irq                hw_writel(0xffffffff, 0xf2008014) -> 0xf0003014
0x804ea98c gk7101_init_irq                hw_writel(0xffffffff, 0xf2008038) -> 0xf0003038
0x804ea9a0 gk7101_init_irq                hw_writel(0x0, 0xf200900c)        -> 0xf001000c
0x804ea9b4 gk7101_init_irq                hw_writel(0x0, 0xf2009010)        -> 0xf0010010
0x804ea9c8 gk7101_init_irq                hw_writel(0xffffffff, 0xf2009014) -> 0xf0010014
0x804ea9dc gk7101_init_irq                hw_writel(0xffffffff, 0xf2009038) -> 0xf0010038
0x80017d44 gk7101_irq_set_type            hw_readl(0xf2008000)              -> 0xf0003024
0x80017d64 gk7101_irq_set_type            hw_readl(0xf2008008)              -> 0xf0003028
0x80017d80 gk7101_irq_set_type            hw_readl(0xf2008004)              -> 0xf000302c
0x80017e20 gk7101_irq_set_type            hw_writel(0x0, 0xf2008000)        -> 0xf0003024
0x80017e34 gk7101_irq_set_type            hw_writel(0x0, 0xf2008008)        -> 0xf0003028
0x80017e48 gk7101_irq_set_type            hw_writel(0x100000, 0xf2008004)   -> 0xf000302c
0x80017b4c gk7101_ack_irq                 hw_writel(0x100000, 0xf2008038)   -> 0xf0003038
0x80017be4 gk7101_enable_irq              hw_writel(0x100000, 0xf2008010)   -> 0xf0003010
0x80019fa0 get_apb_bus_freq_hz            hw_readl(0xf3170014)              -> 0xf1170000
0x80019fb4 get_apb_bus_freq_hz            hw_readl(0xf3170118)              -> 0xf1170118
0x800177ac gk7101_ce_timer_set_mode       hw_readl(0xf300000c)              -> 0xf100b030
0x800177b8 gk7101_ce_timer_set_mode       hw_writel(0x400, 0xf300000c)      -> 0xf100b030
0x800175e8 gk7101_timer_offset            hw_readl(0xf3000020)              -> 0xf100b020
0x8001764c gk7101_ce_timer_set_mode       hw_readl(0xf300000c)              -> 0xf100b030
0x80017658 gk7101_ce_timer_set_mode       hw_writel(0x400, 0xf300000c)      -> 0xf100b030
0x80019fa0 get_apb_bus_freq_hz            hw_readl(0xf3170014)              -> 0xf1170000
0x80019fb4 get_apb_bus_freq_hz            hw_readl(0xf3170118)              -> 0xf1170118
0x80017678 gk7101_ce_timer_set_mode       hw_writel(0xa8750, 0xf3000020)    -> 0xf100b020
0x8001768c gk7101_ce_timer_set_mode       hw_writel(0xa8750, 0xf3000038)    -> 0xf100b024
0x80019fa0 get_apb_bus_freq_hz            hw_readl(0xf3170014)              -> 0xf1170000
0x80019fb4 get_apb_bus_freq_hz            hw_readl(0xf3170118)              -> 0xf1170118
0x8001770c gk7101_ce_timer_set_mode       hw_writel(0x0, 0xf3000024)        -> 0xf100b028
0x80017720 gk7101_ce_timer_set_mode       hw_writel(0x0, 0xf3000028)        -> 0xf100b02c
0x80017734 gk7101_ce_timer_set_mode       hw_readl(0xf300000c)              -> 0xf100b030
0x80017740 gk7101_ce_timer_set_mode       hw_writel(0x411, 0xf300000c)      -> 0xf100b030
0x80017754 gk7101_ce_timer_set_mode       hw_readl(0xf300000c)              -> 0xf100b030
0x80017760 gk7101_ce_timer_set_mode       hw_writel(0x401, 0xf300000c)      -> 0xf100b030
0x80017774 gk7101_ce_timer_set_mode       hw_readl(0xf300000c)              -> 0xf100b030
0x80017780 gk7101_ce_timer_set_mode       hw_writel(0x501, 0xf300000c)      -> 0xf100b030
0x801dce78 serial_gk7101_set_termios      hw_writel(0x10, 0xf3005018)       -> 0xf100500c
0x801dced8 serial_gk7101_set_termios      hw_writel(0xd, 0xf3005004)        -> 0xf1005000
0x801dceec serial_gk7101_set_termios      hw_writel(0x0, 0xf3005000)        -> 0xf1005004
0x801dcf04 serial_gk7101_set_termios      hw_writel(0x3, 0xf3005018)        -> 0xf100500c
0x801dcf48 serial_gk7101_set_termios      hw_readl(0xf3005000)              -> 0xf1005004
0x801dcf54 serial_gk7101_set_termios      hw_writel(0x5, 0xf3005000)        -> 0xf1005004
0x801dc004 serial_gk7101_set_mctrl        hw_readl(0xf300500c)              -> 0xf1005010
0x801dc07c serial_gk7101_set_mctrl        hw_writel(0x4, 0xf300500c)        -> 0xf1005010
0x801dc1ec serial_gk7101_console_write    hw_readl(0xf3005000)              -> 0xf1005004
0x801dc204 serial_gk7101_console_write    hw_writel(0x5, 0xf3005000)        -> 0xf1005004
0x801dc158 serial_gk7101_console_putchar  hw_readl(0xf3005014)              -> 0xf1005014
0x801dc17c serial_gk7101_console_putchar  hw_writel(0x5b, 0xf3005004)       -> 0xf1005000
0x801dc158 serial_gk7101_console_putchar  hw_readl(0xf3005014)              -> 0xf1005014
0x801dc158 serial_gk7101_console_putchar  hw_readl(0xf3005014)              -> 0xf1005014
0x801dc17c serial_gk7101_console_putchar  hw_writel(0x20, 0xf3005004)       -> 0xf1005000
0x801dc158 serial_gk7101_console_putchar  hw_readl(0xf3005014)              -> 0xf1005014
0x801dc158 serial_gk7101_console_putchar  hw_readl(0xf3005014)              -> 0xf1005014

That first call to get_version() is used to print the “hal version = 20151223” line in the console log we saw earlier. After that a bunch of IRQ initialization stuff, followed by timer initialization.

After that it’s some console initialization output; the putchar calls are the first real kernel console output. So that’s all as expected; as a matter of fact that matches exactly what my interrupt and timer drivers put into the registers. Yet the vendor kernel makes it through the calibration function. It’s not long before the log starts showing this:

0x8000845c gk7101_vic_handle_irq          hw_readl(0xf2008030)              -> 0
0x80017b4c gk7101_ack_irq                 hw_writel(0x100000, 0xf2008038)   -> 0
0x800175a0 gk7101_ce_timer_interrupt      hw_writel(0x100000, 0xf2008038)   -> 0

That’s a timer interrupt coming in, getting ack’ed, and getting handled. It just never happens in my kernel.

And so we’re stuck again!

Emulate Your Way to Success

Sat, 16 Nov 2019 19:59:01 +0100

In the last episode of unlocking the Goke GK7101 SoC, we found ourselves faced with a big obstacle: a HAL layer in the form of I/O read/write calls that translated on-board peripherals’ register locations to their real addresses. The HAL’s underlying code is convoluted and much too hard to parse – it’s a large maze of twisty little if-then-elses, all alike. And since this SoC has tons of functionality, there are hundreds of register addresses to find.

But then, a surprise: a wild SDK appears! Somebody uploaded an official Goke SDK tarball to a certain open source repository site, and it has tons of code: the full Linux kernel as hacked up by Goke, their U-Boot source, a build system to make a full working system including root filesystem, and even some example applications that use their kernel drivers.

Weirdly, all of it is marked either GPL or, in a very few cases, public domain. You have to wonder why they’re not just putting this thing up for download; it’s literally all they have to do to abide by the GPL.

Sure enough, lots of information about the HAL is to be found. The struct with the read and write calls is called hw_ops:

struct hw_ops
{
    int (*get_version)(void);
    unsigned int (*reserved)(unsigned int );

    unsigned char (*hw_readb)(unsigned int );
    unsigned short (*hw_readw)(unsigned int );
    unsigned int (*hw_readl)(unsigned int );

    void (*hw_writeb)(unsigned char , unsigned int );
    void (*hw_writew)(unsigned short , unsigned int );
    void (*hw_writel)(unsigned int , unsigned int );

    unsigned int (*flash_read)(void);
    void (*flash_write)(unsigned int);

    unsigned char (*usb_readb)(unsigned int ptr, unsigned int offset);
    unsigned short (*usb_readw)(unsigned int ptr, unsigned int offset);
    unsigned int (*usb_readl)(unsigned int ptr, unsigned int offset);
    void (*usb_writeb)(unsigned int ptr, unsigned int offset, unsigned char value);
    void (*usb_writew)(unsigned int ptr, unsigned int offset, unsigned short value);
    void (*usb_writel)(unsigned int ptr, unsigned int offset, unsigned int value);

    unsigned int (*dma_readl)(unsigned int ptr);
    void (*dma_writel)(unsigned int ptr, unsigned int value);

#if SPI_API_MODE
    unsigned char (*spi_readb)(unsigned int ptr);
    unsigned short (*spi_readw)(unsigned int ptr);
    unsigned int (*spi_readl)(unsigned int ptr);
    void (*spi_writeb)(unsigned int ptr, unsigned char value);
    void (*spi_writew)(unsigned int ptr, unsigned short value);
    void (*spi_writel)(unsigned int ptr, unsigned int value);
#endif
};

Keeping in mind that all these are 32-bit function pointers, our 0x10 and 0x1c offsets correspond to hw_readl and hw_writel, respectively.

The first function, get_version, looks like this in Ghidra:

                ********************************************
                *                 FUNCTION                 *
                ********************************************
                uint32_t __stdcall get_version(void)
     uint32_t     r0:4      <RETURN>
                get_version                       XREF[2]: hal_init:c00120b0(*), 
                                                           c0012160(*)  
c00121a8  ldr   r0, [DAT_c00121b0]                                       = 20151223h
c00121ac  bx    lr
                DAT_c00121b0                      XREF[1]: get_version:c00121a8(R
c00121b0  unde  20151223h

That simply returns the number 0x20151223. We’ve seen that before, in the vendor kernel’s boot log:

[    0.000000] hal version = 20151223

What about the source to the actual HAL code? That gets put into memory by U-Boot, where the kernel then fences it off with the MMU, and uses it in-place. The U-Boot source has this:

static inline unsigned int gk_hal_init (int cp_flag)
{
    unsigned int rval = 0;
    unsigned int *haladdress;

    haladdress = (unsigned int *)CONFIG_GK_HAL_ADDR;
    *(volatile u32 *) (CONFIG_U2K_HAL_ADDR) = (u32)haladdress;
    if(1==cp_flag)
    memcpy(haladdress,hal_data,sizeof(hal_data));

    hal_function_t hal_init = (hal_function_t) (haladdress) ;

    g_hw = (struct hw_ops *)hal_init (0, 0, 0x90000000, 0xA0000000, 0) ;

    return rval ;
}

It’s copied from hal_data to CONFIG_GK_HAL_ADDR, which is defined to 0xc0012000. So what’s hal_data?

int hal_data[]={
0xe92d45f8,0xe1a04000,0xe1a05001,0xe1a06002,0xe1a0a003,0xe59d7020,0xe3570000,0x0a00000e,
0xe59f8148,0xe58802f0,0xe5881320,0xe59832f8,0xe3530000,0x0a000002,0xe2880e2f,0xe3a01001,
...

For whatever reason, the HAL source is missing. Nice to have their kernel source, but we’re no closer to solving this massive HAL problem.

Breaking it down, getting the real register locations involves:

identify “fake” register address.
call the HAL’s hw_readl() function.
find out what that function translated the address to.

Now that we have source code, it’s at least gotten a lot easier to get a list of fake registers – and names to describe them, which is very nice to have. Here’s the UART definition:

#define UART_RB_OFFSET          0x04
#define UART_TH_OFFSET          0x04
#define UART_DLL_OFFSET         0x04
#define UART_IE_OFFSET          0x00
#define UART_DLH_OFFSET         0x00
#define UART_II_OFFSET          0x08
#define UART_FC_OFFSET          0x08
#define UART_LC_OFFSET          0x18
#define UART_MC_OFFSET          0x0c
#define UART_LS_OFFSET          0x14
#define UART_MS_OFFSET          0x10
#define UART_SC_OFFSET          0x1c    /* Byte */
#define UART_SRR_OFFSET         0x88
[...]
/* UART[x]_LS_REG */
#define UART_LS_FERR            0x80
#define UART_LS_TEMT            0x40
#define UART_LS_BI              0x20
#define UART_LS_FE              0x10
#define UART_LS_THRE            0x08
#define UART_LS_DR              0x04
#define UART_LS_PE              0x02
#define UART_LS_OE              0x01

That matches the code in the vendor kernel’s decompressor exactly, including the LSR’s DR bit being at bit 2.

But how can we call hw_readl() for all of those registers, and get the results back in a way we can use in our code? We have the HAL code, but it’s not like we can run the HAL without the GK7101… or can we?

It would actually be possible to run the code on any other ARM core sharing that ISA, if not for the fact that it interacts with actual memory locations: these HAL functions don’t just return the locations – they perform the actual reads and writes. So what we need is an ARM platform that will let us call the function but abort before doing the actual read operation.

Of course an architecture with an MMU makes that easy: by simply not mapping any memory except the part that runs the code, memory accesses to these unmapped locations generate an exception – that is, they run the instruction located at the ARM vector called Data Abort.

This would be even easier to do without having to mess with actual exception vectors on raw hardware, with custom handlers that somehow log what happened. What we need is something like QEMU – which supports ARM – to start up a little VM that only loads the HAL code, calls one function in it, and reports in some detail about a Data Abort exception. That’s actually a bit messy for QEMU; it’s intended for a much higher level of control. We need something lower-level than QEMU, but that’s somehow still easy to use and control. That seems like a big ask – would something like that even exist?

Enter the magnificent Unicorn. The authors took the basic low-level VM code in QEMU, and built a very different thing around it. Unicorn takes the form of a library, with an API that lets you create a VM, map memory into it, set callbacks to your code for exceptions, run code with a granularity down to one instruction, and read/write registers. It’s utterly easy to use, well documented, and exactly what we need. The API is in C, but has bindings for tons of languages, notably Python.

What we need to do, in order:

Create an ARM VM.
Map some memory into it, obviously not in the region where I/O might be. We’ll use 0xc0000000, since the HAL lives in that region.
Assign some of that memory to the stack, and set the virtual CPU’s stack pointer (the sp register).
Drop the HAL code in there, at address 0xc0012000 – we have no guarantees that all that code is relocatable, so this will avoid problems.
Set up a callback for the Data Abort exception: the exception called when code tries to access memory the MMU doesn’t have a mapping for. We’ll just print the “bad” address from there, and that’ll give us the final address as determined by the HAL.
Call the hal_init() function – recall it sets up some variables used by the address translation stuff later.

We’ll also need a little bit of code to call hw_readl() with a supplied address we want translated. Something like this should do it:

ldr r4, [pc, #4]	// load hw_readl() pointer
ldr r0, [pc, #4]	// load argument to hw_readl()
blx r4				// call hw_readl()
// --- code ends here ---
.long 0		// pointer to hw_readl() goes here
.long 0		// test address goes here

The Unicorn application is here. It’s a bit messy, with lots of debugging code, but it works. You give it a HAL code blob as extracted from the device and a test address:

sumner: ./emu-hal halcode-fromdevice 0xa0005014
0xa0005014 -> 0x70005014

Using the UART register offsets we found in the source code, we can thus map the whole UART block:

sumner: for reg in 04 00 08 18 0c 14 10 1c 88; do ./emu-hal halcode-fromdevice 0xa00050$reg; done
0xa0005004 -> 0x70005000
0xa0005000 -> 0x70005004
0xa0005008 -> 0x70005008
0xa0005018 -> 0x7000500c
0xa000500c -> 0x70005010
0xa0005014 -> 0x70005014
0xa0005010 -> 0x70005018
0xa000501c -> 0x7000501c
0xa0005088 -> 0x70005088

Success! We’ll need to extract register offsets for all subsystems from the source code, but that’s a luxury problem: a bit of parsing with a Python script is peanuts compared to having to extract them from decompiled drivers.

Next up, we’ll figure out the registers for the interrupt and timer subsystems, so we can get a kernel to boot. We can use the early_print() facility to debug that, since a proper UART driver won’t work until we have interrupts.

Hal of Horrors

Fri, 15 Nov 2019 21:13:38 +0100

In the previous post we found the GK7101 SoC’s UART base address and a few registers by decompiling their version of Linux’s decompressor:

UART base address is 0xa0005000
The input/output register is at offset 0x04
Offset 0x14 holds flags:
- Bit 6 needs to be high before sending.
- To drain the input buffer, read from offset 0x04 until bit 2 goes low.

This all looks suspiciously like a standard 16550 UART, but not quite: offset 0x04 corresponds to the 16550’s RHR/THR register (Read/Transmit Holding Register), except the 16550 has it on offset 0x00. Offset 0x14 matches the LSR (Line Status Register), with bit 6 matching TEMT (Transmitter Empty), but bit 2 doesn’t quite match – it’s the equivalent of bit 0 in the LSR.

So it’s not a 16550-compatible UART, just a vague attempt at one. Here’s my version of the 4 macros to work this thing:

#define UART0_TH		0x04
#define UART0_LS		0x14
#define UART0_LS_DR		(1 << 2)
#define UART0_LS_TEMT	(1 << 6)


#ifdef CONFIG_DEBUG_UART_PHYS
        .macro	addruart, rp, rv, tmp
        ldr	\rp, =CONFIG_DEBUG_UART_PHYS
        ldr	\rv, =CONFIG_DEBUG_UART_VIRT
        .endm
#endif

        .macro	waituart,rd,rx
1001:		ldr	\rd, [\rx, #UART0_LS]
        tst	\rd, #UART0_LS_TEMT
        beq	1001b
        .endm

        .macro	senduart,rd,rx
        str	\rd, [\rx, #UART0_TH]
        .endm

        .macro	busyuart,rd,rx
1001:		ldr	\rd, [\rx, #UART0_LS]
        tst	\rd, #UART0_LS_DR
        bne	1001b
        .endm

To use this low-level console facility, you need to define the following in your kernel’s configuration:

CONFIG_DEBUG_LL=y
CONFIG_DEBUG_UNCOMPRESS=y
CONFIG_DEBUG_LL_INCLUDE="debug/gk710x.S"
CONFIG_EARLY_PRINTK=y
CONFIG_DEBUG_UART_PHYS=0xA0005000
CONFIG_DEBUG_UART_VIRT=0xf3005000

That turns on the facility, enables it in the decompressor, and tells the build system where to find the code. You’re supposed to create that file and define the macros in there. CONFIG_EARLY_PRINTK creates a function called early_print(), which uses these macros as well. Handy to have while you’re debugging your real UART driver.

The last two definitions are used in the addruart macro above, defining the UART base address in physical and virtual (post-MMU) memory. How did I come up with the virtual address? Remember this output from the vendor kernel bootup?

[    0.000000] AHB: 0x90000000  0xf2000000  -- 0x1000000
[    0.000000] APB: 0xa0000000  0xf3000000  -- 0x1000000
[    0.000000] PPM: 0xc0000000  0xc0000000  -- 0x200000
[    0.000000] BSB: 0xc4800000  0xf5000000  -- 0x200000
[    0.000000] DSP: 0xc4a00000  0xf6000000  -- 0x3600000
[    0.000000] hal version = 20151223

That looks to me like the MMIO mapping: first the physical address, then the virtual address it gets mapped to, followed by the length of the block. APB refers to the Advanced Peripheral Bus, a standard facility on ARM chips. It’s a memory-mapped I/O bus for SoC designers to hook low-speed peripherals to, exactly where you’d expect to find a UART. The virtual address for the UART block within the APB is this 0xf3005000.

That should be all that’s needed to get a mainline kernel’s decompressor output working on this chip.

And yet… it doesn’t work. The decompressor outputs nothing. Diddling those UART registers from the U-Boot shell (which has basic memory monitor commands) also does nothing.

But the vendor kernel didn’t really write to those addresses directly, did it? It used that weird struct with the read and write function pointers, feeding it the addresses we found – and that somehow did work.

This is some sort of abstraction between I/O reads/writes and the actual hardware. In other words, it’s a HAL (Hardware Abstraction Layer). Hardware companies often stick these between their drivers and the operating system they have to run on, in the hope it will save them time and effort. The idea is they can write one driver, and just have a HAL abstract out whether that driver needs to talk to Linux or Windows, for example.

The Linux kernel never accepts drivers that come with a HAL. It makes for an unnecessary internal API – the HAL layer – that nothing else will use, and it creates a lot of unnecessary code. Anyone that wants to mainline code based on a vendor HAL typically gets to rewrite the whole thing, using the vendor code as nothing more than a reference to registers and such. Of course, vendor code even without a HAL also tends to get rewritten from scratch before it goes into mainline, because it’s a pile of dung. But I digress.

Incidentally, not every operating system refuses HAL-based code: the Zephyr project happily accepts such code, for all the wrong reasons. That’s because it’s a project run by a consortium of competitors, managed by the Linux Foundation. In other words there isn’t a clue to be found. This is why we need Linus, folks.

Still, this particular HAL seems unusual – it doesn’t sit between a driver and the OS, but between I/O reads/writes and the registers!

Ghidra didn’t quite manage to extract the address of that struct, and it was quite the struggle to find, but it turns out to live at 0xc0015ad0. U-Boot has a severe misfeature in that asking it to dump memory in 32-bit-sized chunks will make it flip those chunks from little-endian to big-endian. However this comes in handy if you’re on a wrong-endian system and you happen to be human. Here’s the struct:

GK7101 # md.l c0015ad0 20
[PROCESS_SEPARATORS] md.l c0015ad0 20
c0015ad0: c00121a8 c0012904 c0012b80 c0012b0c    .!...)...+...+..
c0015ae0: c00127e8 c0012a90 c0012a14 c001295c    .'...*...*..\)..
c0015af0: c0012250 c0012268 c0012280 c00122a0    P"..h"..."..."..
c0015b00: c00122c0 c00122e0 c0012300 c0012320    ."..."...#.. #..
c0015b10: c0012340 c001235c c0015ad0 00042400    @#..\#...Z...$..

Those are clearly all pointers to the same region, roughly 0xc0012000 - 0xc0015000. We saw the read function at offset 0x10, and write at 0x1c. Let’s take a look at read, at 0xc00127e8. Remember it takes a single 32-bit unsigned integer argument (the address) and returns the value at that address, also uint32_t.

uint hw_readl(uint *address)

{
    undefined *puVar1;
    uint *puVar2;
    uint uVar3;
    code *pcVar4;
    uint *puVar5;
    undefined4 local_14;
	
    if (param_1 == DAT_c00128ec) {
LAB_c0012840:
        uVar3 = *(uint *)(*DAT_c0012900 + 0x5014);
        return uVar3 & 0xffffffc0 | 8 | (uVar3 & 6) >> 1 | (uVar3 & 1) << 2 | (uVar3 & 0x18) << 1;
    }
    if (param_1 < DAT_c00128ec || param_1 == DAT_c00128ec) {
        if (param_1 != DAT_c00128f0) {
LAB_c0012874:
            local_14 = 0;
            puVar5 = param_1;
            puVar1 = FUN_c00127b8((uint)param_1);
            if (puVar1 == NULL) {
                if (((uint)param_1 & 0xff000000) == 0x60000000 ||
                    ((uint)param_1 & 0xff000000) == 0x70000000) {
                    return 0;
                }
                return *param_1;
            }
            puVar2 = (uint *)(**(code **)(puVar1 + 4))(param_1,&local_14);
            if (puVar2 == NULL) {
                return 0;
            }
            pcVar4 = *(code **)(puVar1 + 0x10);
            if (pcVar4 == NULL) {
                return *puVar2;
            }
            uVar3 = (*pcVar4)(local_14,*puVar2,0,pcVar4,puVar5);
            return uVar3;
        }
    }
    else {
        if (param_1 != DAT_c00128f4) {
            if (param_1 == DAT_c00128f8) goto LAB_c0012840;
            goto LAB_c0012874;
        }
    }
    return *(uint *)(*DAT_c00128fc + 0x16000);
}

This is very much spaghetti code, no doubt at least partly due to the decompiler’s distorted view: the machine code it translates is always spaghetti-shaped. Translated into proper C and then into pseudo-code it does this:

If the address is 0xA016018C or 0xF316018C, return 0x00474b37. That’s actually the string “7KG”. Similarly, if the address is 0xA0160188 or 0xF3160188, return 0x00474b37 – the string “1010”. Note the 0xa0000000 and 0xf3000000 address ranges here – so these are definitely related, and this code is apparently meant to work on both physical and virtual addresses.
If the address is 0xA0005014 or 0xF3005014 – hey, that’s the UART LSR! – then read from an unknown prefix + 0x5014 and shift the various bits in there around to match the almost-not-quite-16550 we found earlier. The unknown prefix turns out to be set by an earlier call to 0xc0012000, some sort of initialization of this HAL.
Any other address, a function at 0xc00127b8 is called with the address, and a number of some sort is returned. That number is used as an index into a jumptable of functions; the indexed function is then called with the address and returns the “real” address. The read is then done on that address, and the result returned.

So all we have to do is figure out which index we get for our 0xa0005000 registers, decompile the function that handles this block, and see what it returns. The real physical address for the UART block turns out to be 0x70005000. The flags of the LSR register at 0x70005014 are totally different, but the code up there shows the mapping. Oh, and the real RHR/THR register is actually at offset 0x00 – so this HAL doesn’t just translate peripheral base addresses, but also the ordering of registers within their blocks.

The problem is that this is a big jumptable, with a large number of functions. They’re all constructed as a series of if-then-else statements on the address, no doubt in an effort to make it all go fast. Clearly, this was written before tree traversal algorithms were invented.

The upshot is that it’s 16K of really dense machine code, a serious chore to decompile, parse and grok – it would just takes ages, and mistakes would be easy to make. Yet I have to do all of them, or I won’t have the real addresses of the SoC’s various features.

So with the physical address of 0x70005000 the UART works. That’s one problem solved, but now I’ve got hundreds of problems just like it.

This is horrible! What to do?

Getting Started on a New Board

Wed, 13 Nov 2019 12:39:48 +0100

In part 1 of this series we found an interesting little board, and located the built-in UART pins. Let’s take a look at the output on boot:

console init done


U-Boot 2012.10 (Dec 07 2016 - 13:48:53) for GK7101 rb imx222 v1.00 (GOKE)

HAL:  20151223 
DRAM:  128 MiB
Flash: 16 MiB
16 MiB
NAND:  SPINAND MID = 0xff, DID = 0xffff, Data = 0x1ffffff !spinand_board_init[1581]: No support this SPI nand!
SF: Detected GD25Q128C with page size 256 B, sector size 64 KiB, total size 16 MiB
In:    serial
Out:   serial
Err:   serial
Net:   arm_freq(600MHz)..............0x112032
use int MII..............
gk7101
Hit any key to stop autoboot:  5

Lots of useful information there. It’s an ancient version of Das U-Boot, the most commonly used bootloader for embedded Linux systems. Sure enough, it’s running Linux:

put param to memory
mem size (70)
bsb size (2)

the kernel image is zImage or Image
entry = 0xc1000000 
## Transferring control to Linux (at address c1000000)...

Starting kernel ...

machid = 3988 r2 = 0xc0000100 
Uncompressing Linux... done, booting the kernel.
[    0.000000] Booting Linux on physical CPU 0
[    0.000000] Linux version 3.4.43-gk (root@ubuntu) (gcc version 4.6.1 (crosstool-NG 1.18.0) ) #92 PREEMPT Wed Dec 7 16:55:36 CST 2016
[    0.000000] CPU: ARMv6-compatible processor [410fb767] revision 7 (ARMv7), cr=00c5387d
[    0.000000] CPU: VIPT aliasing data cache, VIPT aliasing instruction cache
[    0.000000] Machine: Goke GK7101 RB_IMX222 board V1.00
[    0.000000] Memory policy: ECC disabled, Data cache writeback
[    0.000000] AHB: 0x90000000  0xf2000000  -- 0x1000000
[    0.000000] APB: 0xa0000000  0xf3000000  -- 0x1000000
[    0.000000] PPM: 0xc0000000  0xc0000000  -- 0x200000
[    0.000000] BSB: 0xc4800000  0xf5000000  -- 0x200000
[    0.000000] DSP: 0xc4a00000  0xf6000000  -- 0x3600000
[    0.000000] hal version = 20151223 
[    0.000000] Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 17780
[    0.000000] Kernel command line: console=ttySGK0,115200 noinitrd mem=70M rw mtdparts=gk7101_flash:256K(boot),64K(bootenv),2560K(kernel),7168K(rootfs),1024K(rom),5312K(APP) rootfstype=squashfs root=/dev/mtdblock3 init=linuxrc ip=192.168.1.254:192.168.1.112:192.168.1.1:255.255.255.0:"gk7101":eth0 mac=3C:97:0E:22:E1:14 phytype=0
...

The system boots right into a BusyBox shell. Since the system has a microSD slot, the easiest way to take a good look at the firmware is to stick in a microSD card and just copy the built-in storage over to it. This way we can examine the goods with a much better set of tools than will be available on the camera.

The board has an SPI flash chip on it:

GigaDevice 16MB SPI flash

This isn’t like a hard disk or SSD, with a partition table of some sort to help the system figure out how the storage is structured. The Linux kernel needs to be told about the partitions. This version of U-Boot has this hardcoded, and it passes it along to the Linux kernel as part of the command line:

mtdparts=gk7101_flash:256K(boot),64K(bootenv),2560K(kernel),7168K(rootfs),1024K(rom),5312K(APP)

These partitions show up as /dev/mtdblock0, /dev/mtdblock1 and so on. Das U-Boot is on the first partition, its configuration in the second, Linux kernel on the next one followed by the root filesystem. The last partition (“APP”) contains the main application the camera runs, and the “ROM” partition contains configuration.

The mainline kernel has support for the basic ARM platform the GK7101 SoC uses, ARM1176: Selecting CONFIG_ARCH_MULTI_V6 gets us working code, but of course no I/O of any kind. The first thing the kernel outputs when U-Boot calls it is normally the decompressor. It prints “Uncompressing Linux... done, booting the kernel.” to the console, then jumps into the kernel proper, at its new location. The console, of course, is not so easy here: there is no BIOS that will output this on a VGA port, like in X86. Instead, Linux has a very low-level facility to have the decompressor use a memory-mapped UART as an output-only console.

Normally, driving a UART would entail having an interrupt fire when e.g. a byte has arrived on the UART. At decompression time, however, the CPU has no interrupts, timers, or even MMU: U-Boot disables all that before handover. Therefore the decompressor’s UART driver can’t just write characters to a buffer, knowing that some timer will fire and empty the buffer into the UART registers as it clears itself. Instead, the driver has to check if the UART’s output register is clear, and only then write a single character.

This facility can also be used by the kernel once it’s up and running, but by that time it will have a working MMU – and the I/O registers for the UART may have moved to a different address. The kernel solves all of this by allowing you to define 4 macros for your platform, and it uses these to construct a putc() function.

From arch/arm/boot/compressed/debug.S:

ENTRY(putc)
    addruart r1, r2, r3
    waituart r3, r1
    senduart r0, r1
    busyuart r3, r1
    mov  pc, lr
ENDPROC(putc)

The addruart macro should load the base address for the UART registers – the physical address in r1, and the corresponding virtual address in r2 (r3 is just a scratch register it can use).

waituart should implement a busy loop waiting for the UART’s line status register to show the output register is empty. senduart puts its first argument into the output register, and busyuart again should implement a busy loop waiting for output to clear – a flush() function.

Note how addruart loads the physical address into r1, and the other macros then use r1. This is specific to the decompressor, since it runs pre-MMU. The kernel facility that uses these macros instead calls them with addruart's second argument.

So what’s the UART’s physical base address, and what do the registers look like? Only one way to find out: disassemble the vendor’s decompressor, and see what addresses it uses.

There are many disassemblers available, starting of course with GNU objdump. This is as good an occasion as any to try out a new tool that came out: Ghidra. It’s a reverse engineering suite in the style of the venerable IDA Pro. Unlike IDA, it’s open source – and very unlike anything, it was written and is maintained by the NSA. The code is right here on GitHub. It’s a little surreal to be using NSA code, but as it turns out the tool is very good. It supports a ton of architectures, and has a great decompiler – which, due to Ghidra’s internal architecture, automatically works on all supported CPUs.

The camera vendor’s version of Linux is a hacked-up 3.4.43. So compiling that version from source and comparing the decompiled output with the source code should go a long way to understanding the vendor’s code as well. The first thing that gets sent to console is “Uncompressing Linux...”. This is done in arch/arm/boot/compressed/misc.c, in function decompress_kernel(). The call is to putstr(), a wrapper around the putc() function we saw defined earlier. Following that should get us the macros we’re after.

Unfortunately the compiler inlined the putc function, so the loop has all the low-level I/O smeared into it Here’s the source code:

static void putstr(const char *ptr)
{
    char c;

    while ((c = *ptr++) != '\0') {
        if (c == '\n')
            putc('\r');
        putc(c);
    }

    flush();
}

This is Ghidra’s decompiled version of the vendor code:

void putstr(char *ptr)
{
    byte bVar1;
    uint uVar2;
    int *piVar3;
	
    piVar3 = *(int **)(PTR_DAT_c10008c0 + DAT_c10008c4 + -0x3efff7e4);
    while( true ) {
        bVar1 = *ptr;
        if (bVar1 == 0) break;
        if (bVar1 == 10) {
            do {
                uVar2 = (**(code **)(*piVar3 + 0x10))(DAT_c10008c8);
            } while ((uVar2 & 0x40) == 0);
            (**(code **)(*piVar3 + 0x1c))(0xd,DAT_c10008cc);
        }
        do {
            uVar2 = (**(code **)(*piVar3 + 0x10))(DAT_c10008c8);
        } while ((uVar2 & 0x40) == 0);
        (**(code **)(*piVar3 + 0x1c))((uint)bVar1,DAT_c10008cc);
        ptr = (char *)((byte *)ptr + 1);
    }
    while (uVar2 = (**(code **)(*piVar3 + 0x10))(DAT_c10008c8), (uVar2 & 4) != 0) {
        (**(code **)(*piVar3 + 0x10))(DAT_c10008cc);
    }
    return;
}

The DAT_ variables are references to memory addresses containing values, which the instructions dereference. The ARM architecture has fixed 32-bit instructions, so cannot handle direct register loads of 32-bit addresses. So we’d expect to find these UART base addresses in variables like this.

That last while loop is the flush() function i.e. busyuart. Cleaned up and with the variables filled in, it looks like this:

while (uVar2 = (somestruct + 0x10)(0xa0005014), (uVar2 & 4) != 0) {
    (somestruct + 0x10)(0xa0005004);
}

That looks like it reads from 0xa0005014 and waits for bit 2 to go low, reading from 0xa0005004 while it’s not. In other words, that bit indicates there’s stuff in the input buffer (it has one!), and drains it. So 0xa0005004 is the input buffer register.

But why is this using a function? The struct is somewhere in memory, and evidently the member at offset 0x10 is a pointer to a read function.

There’s only one line that sends putstr()'s string argument anywhere, so that’s got to be the senduart macro’s implementation:

(**(code **)(*piVar3 + 0x1c))((uint)bVar1,DAT_c10008cc);

Cleaned up:

(somestruct + 0x1c)(bVar1, 0xa0005004);

So the mystery struct has a pointer to a write function at 0x1c, and the UART output register is evidently 0xa0005004 as well.

Right before that is a busy loop, clearly the waituart macro:

    do {
        uVar2 = (**(code **)(*piVar3 + 0x10))(DAT_c10008c8);
    } while ((uVar2 & 0x40) == 0);

This reads 0xa0005014 in a loop, waiting for bit 6 to go high.

There is no implementation of addruart, as all address are hardcoded in memory variables. But we have the addresses, and some bitfields. That should be enough to make a kernel that boots and tells us it’s decompressing itself.

Progress!

This Little Camera

Fri, 08 Nov 2019 14:18:00 +0100

Some time ago I came across a strange product on Banggood. It was a small camera, with a base and separate camera head on top. The camera head supports pan and tilt, like a bobblehead.

Digoo DG-M1Z

The specs aren’t half bad: supports 1080p, has wifi and ethernet on board, and has a microSD card slot. These are intended for home security. The idea is you set these down somewhere and it streams live video over wifi. Sounds great!

The strange thing was the price: these cameras go for $15-$20 on Banggood and similar Chinese sites. Turns out that’s not the complete price however: you’re supposed to buy a monthly subscription for the “cloud service”. The camera uploads video to this cloud, and you can watch it there with your paid account.

The cloud in question is some Chinese service, and of course the camera cannot work without it: you can’t just stream your own video off of your own camera. So it needs a working internet connection as well.

So for $15 + a monthly fee you buy a spyware device that uploads your house’s video to China. You’d have to be insane to think that’s a good deal; I think it’s downright offensive.

It occurred to me that these things would be awesome if you could only put your own software on them, so it wouldn’t connect to anywhere – just store or stream video locally. That really would get you a $15 camera without the Chinese mass surveillance feature.

I decided to look a little closer at the hardware, and see how doable this would be. As expected at this price, the system is mostly a single chip – the GK7101 – with only the barest essentials to support it. This very capable chip is an SoC made by Goke Micro, a Chinese company. The chip has an older ARM 1176 CPU core and a ton of peripherals:

I set about finding a serial port of sorts on the board. There’s usually one there for debugging, but sometimes it take a little effort to find it. No problem this time though:

X marks the spot

As usual for hardware of this caliber, it runs Linux. Also as usual, none of the changes made to the Linux kernel for this hardware have been upstreamed. There are a ton of kernel modules, all of which are marked as GPL. The kernel itself is an old 3.4.43 version, also with many changes.

It seems to me I’d need to, in order:

get a mainline kernel running on the board
port the modules one by one into new drivers, reverse engineering their functionality as I go
get a basic root filesystem going and look at implementing a streaming API, and whatever else seems handy

Of course the sharp-eyed reader will have noticed that, in fact, points 1 and 2 (porting the kernel) are actually optional. After all, the userspace software is the bit that locks you in to this Chinese cloud stuff; if that software can use the video coming out of that custom kernel, I can write software to do the same.

But you know what? Porting the kernel is, to me, by far the most interesting part of this. It’s a pretty big project, since the SoC has so much functionality on board.

It’s also really hard, because there is no datasheet available for the chip. I’ve tried contacting various people at Goke, but the only person that replied was a sales guy – who stopped replying the moment he realized I wasn’t going to buy 10k units. Needless to say requests for their GPL’ed code didn’t get anywhere.

There is an SDK they give to customers, but that is also not publicly available.

What a challenge this is. Let’s go!