Kkboot—A Boot Loader Based on Kexec

kboot—A Boot Loader Based on Kexec

Werner Almesberger

werner@https://www.wendangku.net/doc/55115431.html,

Abstract

Compared to the“consoles”found on tradi-tional Unix workstations and mini-computers, the Linux boot process is feature-poor,and the addition of new functionality to boot loaders of-ten results in massive code duplication.With the availability of kexec,this situation can be improved.

kboot is a proof-of-concept implementation of a Linux boot loader based on kexec.kboot uses a boot loader like LILO or GRUB to load a reg-ular Linux kernel as its?rst stage.Then,the full capabilities of the kernel can be used to locate and to access the kernel to be booted,perform limited diagnostics and repair,etc.

1Oh no,not another boot loader! There is already no shortage of boot loaders for Linux,so why have another one?The moti-vation for making kboot is simply that the boot process of Linux is still not as good as it could be,and that recent technological advances have made it comparably easy to do better.

Looking at traditional Unix servers and work-stations,one often?nds very powerful boot en-vironments,offering a broad choice of possible sources for the kernel and other system?les to load.It is also quite common to?nd various tools for hardware diagnosis and system soft-ware repair.On Linux,many boot loaders are much more limited than this.

Even boot loaders that provide several of these advanced features,like GRUB,suffer from the problem that they need to replicate functional-ity or at least include code found elsewhere, which creates an ever increasing maintenance burden.Similarly,any drivers or protocols the boot loader incorporates,will have to be main-tained in the context of that boot loader,in par-allel with the original source.

New boot loader functionality is not only re-quired because administrators demand more powerful tools,but also because technologi-cal progress leads to more and more complex mechanisms for accessing storage and other de-vices,which a boot loader eventually should be able to support.

It is easy to see that a regular Linux system hap-pens to support a superset of all the functional-ity described above.

With the addition of the kexec system call to the2.6.13mainline Linux kernel,we now have an instrument that allows us to build boot load-ers with a fully featured Linux system,tailored according to the needs of the boot process and the resources available for it.

Kboot is a proof-of-concept implementation of such a boot loader.It demonstrates that new functionality can be merged from the vast code

28?kboot—A Boot Loader Based on Kexec

base available for Linux with great ease,and without incurring any signi?cant maintenance overhead.This way,it can also serve as a plat-form for the development of new boot concepts.

The project’s home page is at http:// https://www.wendangku.net/doc/55115431.html,/

The remainder of this section gives a high-level view of the role of a boot loader in general, and what kboot aims to accomplish.Additional technical details about the boot process,includ-ing tasks performed by the Linux kernel when bringing up user space,can be found in[1]. Section2brie?y describes Eric Biederman’s kexec[2],which plays a key role in the op-eration of kboot.Section3introduces kboot proper,explains its structure,and discusses its application.Section4gives an outlook on fu-ture work,and we conclude with section5. 1.1What a boot loader does

After being loaded by the system’s?rmware,a boot loader spends a few moments making it-self comfortable on the system.This includes loading additional parts,moving itself to other memory regions,and establishing access to de-vices.

After that,it typically tries to interact with the user.This interaction can range from checking whether the user is trying to get the boot loader’s attention by pressing some key, through a command line or a simple full-screen menu,to a lavish graphical user interface. Whatever the interface may be,in the end its main purpose is to allow the user to select,per-haps along with some other options,which op-erating system or kernel will be booted.Once this choice is made,the boot loader proceeds to load the corresponding data into memory,does some additional setup,e.g.,to pass parameters to the operating system it is booting,and trans-fers control to the entry point of the code it has loaded.

In the case of Linux,two items deserve special mention:the boot parameter line and the initial RAM disk.

The boot parameter line was at its inception in-tended primarily as a means for passing a“boot into single user mode”?ag to the kernel,but this got a little out of hand,and it is nowadays often used to pass dozens if not hundreds of bytes of essential con?guration data to the ker-nel,such as the location of the root?le system, instructions for how certain drivers should ini-tialize themselves(e.g.,whether it is safe for the IDE driver to try to use DMA or not),and the selection and tuning of items included in a generic kernel(e.g.,disabling ACPI support). Since a kernel would often not even boot with-out the correct set of boot parameters,a boot loader must store them in its con?guration,and pass them to the kernel without requiring user action.At the same time,users should of course be able to manually set and override such pa-rameters.

The initial RAM disk(initrd),which at the time of writing is gradually being replaced by the initial RAM?le system(initramfs),provides an early user space,which is put into memory by the boot loader,and is thus available even be-fore the kernel is fully capable to interact with its surroundings.This early user space is used for extended setup operations,such as the load-ing of driver modules.

Given that the use of initrd is an integral part of many Linux distributions,any general-purpose Linux boot loader must support this functional-ity.

2006Linux Symposium,V olume One ?29

Boot

process

Hard? and firmware New device drivers New protocols

Combination of services

New file systems Convenience

Compatible "look and feel"

Administration User experience Figure 1:The boot process exists in a world full of changes and faces requirements from many directions.All this leads to the need to continuously grow in functionality.1.2

What a boot loader should be like

A boot loader has much in common with the operating system it is loading:it shares the same hardware,exists in the same administra-tive context,and is seen by the same users.From all these directions originate require-ments on the boot process,as illustrated in ?g-ure 1.

The boot loader has to be able to access at least the hardware that leads to the locations from which data has to be loaded.This does not only include physical resources,but also any protocols that are used to communicate with devices.Firmware sometimes provides a set of functions to perform such accesses,but new hardware or protocol extensions often require support that goes beyond this.For example,al-though many PCs have a Firewire port,BIOS support for booting from storage attached via Firewire is not common.

Above basic access mechanisms lies the do-main of services the administrator can combine more or less freely.This begins with ?le system

formats,and gets particularly interesting when using networks.For example,there is noth-ing inherently wrong in wanting to boot kernels that happen to be stored in RPM ?les on an NFS server,which is reached through an IPsec link.The hardware and protocol environment of the boot process extends beyond storage.For ex-ample,keyboard or display devices for users with disabilities may require special drivers.With kboot,such devices can also be used to interact with the boot loader.

Last but not least,whenever users have to per-form non-trivial tasks with the boot loader,they will prefer a context similar to what they are used to from normal interaction with the sys-tem.For instance,path names starting at the root of a ?le system hierarchy tend to be easier to remember than device-local names pre?xed with a disk and partition number.

In addition to all this,it is often desirable if small repair work on an unbootable system can be done from the boot loader,without having to ?nd or prepare a system recovery medium,or similar.

30?kboot—A Boot Loader Based on

Kexec

kboot ?f Kernel memory (before rebooting)

Kernel memory

(while and after rebooting)

Jump to kernel setup

1

file

Figure 2:Simpli?ed boot sequence of kexec.

The bottom line is that a general-purpose boot loader will always grow in functionality along the lines of what the full operating system can support.1.3

The story so far

The two principal boot loaders for Linux on the i386platform,LILO and GRUB,illustrate this trend nicely.

LILO was designed with the goal in mind of being able to load kernels from any ?le system the kernel may support.Other functionality has been added over time,but growth has been lim-ited by the author’s choice of implementing the entire boot loader in assembler.1

GRUB appeared several years later and was written in C from the beginning,which helped

1LILO

was written in 1992.At that time,32-bit real

mode of the i386processor was not generally known,and the author therefore had to choose between programming in the 16-bit mode in which the i386starts,or implement-ing a fully-featured 32-bit protected mode environment,complete with real-mode callbacks to invoke BIOS func-tions.After choosing the less intrusive of the two ap-proaches,there was the problem that no suitable and rea-sonably widely deployed free C compiler was available.Hence the decision to write LILO in assembler.

it to absorb additional functionality more quickly.For instance,GRUB can directly read a large number of different ?le system formats,without having to rely on external help,such as the map ?le used by LILO.GRUB also offers limited networking support.

Unfortunately,GRUB still requires that any new functionality,be it drivers,?le systems,?le formats,network protocols,or anything else,is integrated into GRUB’s own environment.This somewhat slows initial incorporation of new features,and,worse yet,leads to an increas-ing amount of code that has to be maintained in parallel with its counterpart in regular Linux.In an ideal boot loader,the difference between the environment found on a regular Linux sys-tem and that in the boot loader would be re-duced to a point where integration of new fea-tures,and their subsequent maintenance,is triv-ial.Furthermore,reducing the barrier for work-ing on the boot loader should also encourage customization for speci?c environments,and more experimental uses.

The author has proposed the use of the Linux kernel as the main element of a boot loader in [1].Since then,several years have passed,some of the technology has ?rst changed,then

2006Linux Symposium,V olume One?31

matured,and with the integration of the key el-ement required for all this into the mainstream kernel,work on this new kind of boot loader could start in earnest.

2Booting kernels with kexec

One prediction in[1]came true almost immedi-ately,namely that major changes to the bootimg mechanism described there were quite prob-able:when Eric Biederman released kexec, it swiftly replaced bootimg,being technologi-cally superior and also better maintained. Unfortunately,adoption of kexec into the main-stream kernel took much longer than anyone expected,in part also because it underwent de-sign changes to better support the very elegant kdump crash dump mechanism[3],and it was only with the2.6.13kernel that it was?nally accepted.

2.1Operation

This is a brief overview of the fundamental as-pects of how kexec operates.More details can be found in[4],[5],and also[3].

As shown in?gure2,the user space tool kexec?rst loads the code of the new kernel plus any additional data,such as an initial RAM disk,into user space memory,and then invokes the kexec_load system call to copy it into kernel memory(1).During the loading,the user space tool can also add or omit data(e.g., setup code),and perform format conversions (e.g.,when reading from an ELF?le).

After that,a reboot system call is made to boot the new kernel(2).The reboot code tries to shut down all devices,such that they are in a de?ned and inactive state,from which they can be instantly reactivated after the reboot.

Since data pages containing the new kernel have been loaded to arbitrary physical locations and could not occupy the same space as the code of the old kernel before the reboot any-way,they have to be moved to their?nal desti-nation(3).

Finally,the reboot code jumps to the entry point of the setup code of the new kernel.That kernel then goes through its initialization,brings up drivers,etc.

2.2Debugging

The weak spot of kexec are the drivers:some drivers may simply ignore the request to shut down,others may be overzealous,and deac-tivate the device in question completely,and some may leave the device in a state from which it cannot be brought back to life,be this either because the state itself is incorrect or irrecoverable,or because the driver simply does not know how to resume from this speci?c state.

Failure may also be only partial,e.g.,VGA of-ten ends up in a state where the text does not scroll properly until the card is reset by loading a font.

Many of these problems have not become vis-ible yet,because those drivers have not been subjected to this speci?c shutdown and reboot sequence so far.

The developers of kexec and kdump have made a great effort to make kexec work with a large set of hardware,but given the sheer number of drivers in the kernel and also in parallel trees, there are doubtlessly many more problems still awaiting discovery.

32?kboot—A Boot Loader Based on Kexec

udev

dropbear

kexec

etc.

kboot utils

kboot shell

uClibc/glibc

Lean kernel

(sh, cat, mount, ...)

BusyBox Figure 3:The software stack of the kboot envi-ronment.

Since kboot is the ?rst application of kexec that should attract interest from more than a relatively small group of developers,many of the expected driver con?icts will surface in the form of boot failures occurring under kboot,af-ter which they can be corrected.

3Putting it all together

Kboot bundles the components needed for a boot loader,and provides the “glue”to hold them together.For this,it needs very little code:as of version 10,only roughly 3’500lines,about half of this shell scripts.Already LILO exceeds this by one order of magnitude,and GRUB further doubles LILO’s ?gure.2Of course,during its build process,kboot pulls in various large packages,among them the entire GCC tool chain,a C library,Busy-Box,assorted other utilities,and the Linux kernel itself.In this regard,kboot resembles more a distribution like Gentoo,OpenEmbed-ded,or Rock Linux,which consist mainly of

2These

numbers were obtained by quite unscienti?-cally running wc-l on a somewhat arbitrary set of the ?les in the respective source

trees.

kexec

Main system

("booted environment")

Figure 4:The boot sequence when using kboot.meta-information about packages maintained by other parties.3.1

The boot environment

Figure 3shows the software packages that con-stitute the kboot environment.Its basis is a Linux kernel.This kernel only needs to support the devices,?le systems,and protocols that will be used by kboot,and can therefore,if space is an issue,be made considerably smaller than a fully-featured production kernel for the same machine.

In order to save space,kboot can use uClibc [6]instead of the much larger glibc.Unfortunately,properly supporting a library different from the one on the host system requires building a dedi-cated version of GCC.Since uClibc is sensitive to the compiler version,kboot also builds a lo-cal copy of GCC for the host.To be on the safe side,it also builds binutils.

After this tour de force,kboot builds the appli-cations for its user space,which include Busy-Box [7],udev [8],the kexec tools [2],and

2006Linux Symposium,V olume One?33

dropbear[9].BusyBox provides a great many common programs,ranging from a Bourne shell,through system tools like“mount,”to a complete set of networking utilities,including “wget”and a DHCP client.Udev is responsible for the creation of device?les in/dev.It is a user space replacement for the kernel-based devfs.The kexec tools provide the user space interface to kexec.

Last but not least,dropbear,an SSH server and client package,is included to demonstrate the ?exibility afforded by this design.This also of-fers a simple remote access to the boot prompt, without the need to set up a serial console for just this purpose.

3.2The boot sequence

The boot sequence,shown in?gure4,is as follows:?rst,the?rmware loads and starts the?rst-stage boot loader.This would typi-cally be a program like GRUB or LILO,but it could also be something more specialized, e.g.,a loader for on-board Flash memory.This boot loader then immediately proceeds to load kboot’s Linux kernel and kboot’s initramfs. The kernel goes through the usual initialization and then starts the kboot shell,which updates its con?guration?les(see section3.5),may bring up networking,and then interacts with the user.

If the user chooses,either actively or through a timeout,to start a Linux system,kboot then uses kexec to load the kernel and maybe also an initial RAM disk.

Although not yet implemented at the time of writing,kboot will also be able to boot legacy operating systems.The plan is to initially avoid the quagmire of restoring the?rmware envi-ronment to the point that the system can be booted from it,but to hand the boot request back to the?rst stage boot loader(e.g.,with lilo-R or grub-set-default),and to reboot through the?rmware.

3.3The boot shell

At the time of writing,the boot shell is fairly simple.After initializing the boot environment, it offers a command line with editing,com-mand and?le name completion,and a history function for the current session.

The following types of items can be entered:?Names of variables containing a com-

mand.These variables are usually de?ned

in the kboot con?guration?le,but can also

be set during a kboot session.3The vari-

able is expanded,and the shell then pro-

cesses the command.This is a slight gen-

eralization of the label in LILO,or the

title in GRUB.

?The path to a?le containing a bootable

kernel.Path names are generalized in

kboot,and also allow direct access to de-

vices and some network resources.They

are described in more detail in the next

section.When such a path name is en-

tered,kboot tries to boot the?le through

kexec.

?The name of a block device containing the

boot sector of a legacy operating system,

or the path to the corresponding device

?le.

?An internal command of the kboot shell.

It currently supports cd and pwd,with the

usual semantics.

3In the latter case,they are lost when the session ends.

34?kboot—A Boot Loader Based on Kexec

Syntax Example Description

variable my_kernel Command stored in a variable

/path/boot/bzImage-2.6.13.2Absolute path in booted environment //path cat//etc/fstab Absolute path in kboot environment path cd linux-2.6.14Relative path in current environment device hda7Device containing a boot sector

/dev/device/dev/hda7Device?le of device with boot sector device:/path hda1:/bzImage File or directory on a device device:path hda1:bzImage(implicit/dev/)

/dev/device:/path/dev/sda6:/foo/bar File or directory on a device

/dev/device:path/dev/sda6:foo/bar(explicit/dev/)

host:/path server:/home/k/bzImage-a File or directory on an NFS server http://host/path http://server/foo File on an HTTP server

ftp://host/path ftp://server/foo/bar File on an FTP server

Table1:Types of path names recognized by kboot.

?A shell command.The kboot shell per-

forms path name substitution,and then

runs the command.If the command uses

an executable from the booted environ-

ment,it is run with chroot,since the shared

libraries available in the kboot environ-

ment may be incompatible with the expec-

tations of the executable.

With the exception of a few helper programs, like the command line editor,the kboot shell is currently implemented as a Bourne shell script.

3.4Generalized path names

Kboot automatically mounts?le systems of the booted environment,on explicitly speci?ed block devices,and—if networking is enabled—also from NFS servers.Furthermore,it can copy and then boot?les from HTTP and FTP servers.

For all this,it uses a generalized path name syntax that re?ects the most common forms of specifying the respective resources. E.g.,for NFS,the host:path syntax is used,for HTTP,it is a URL,and paths on the booted environment look just like normal Unix path names.Table1 shows the various forms of path names.

Absolute paths in the kboot environment are an exception:they begin with two slashes instead of one.

We currently assume that there is one principal booted system environment,which de?nes the “normal”?le system hierarchy on the machine in question.Support for systems with multiple booted environments is planned for future ver-sions of kboot.

3.5Con?guration?les

When kboot starts,it only has access to the con-?guration?les stored in its initramfs.These were gathered at build time,either from the user (who placed them in kboot’s config/direc-tory),or from the current con?guration of the build host.

This set of?les includes kboot’s own con?g-uration/etc/kboot.conf,/etc/fstab,

2006Linux Symposium,V olume One?35

Figure5:Some of the con?guration?les used by kboot.

and/etc/hosts.The kboot build process also adds a?le/etc/kboot-features containing settings needed for the initialization of the kboot shell.

Kboot can now either use these?les,or it can, at the user’s discretion,try to mount the?le system containing the/etc directory of the booted environment,and obtain more recent copies of them.

The decision of whether kboot will use its own copies,or attempt an update?rst,is made at build time.It can be superseded at boot time by passing the kernel parameter kboot=local.

3.6When not to use kboot

While kboot it designed to be a?exible and ex-tensible solution,there are areas where this type of boot loader architecture does not?t.

If only very little persistent storage is available, which is a common situation in small embed-ded systems,or if large enough storage devices would be available,but cannot be made an inte-gral part of the boot process,e.g.,removable or unreliable media,only a boot loader optimized for tiny size may be suitable.

Similarly,if boot time is critical,the time spent loading and initializing an extra kernel may be too much.The boot time of regular desktop or server type machines already greatly exceeds the minimum boot time of a kernel,which em-bedded system developers aim to bring well be-low one second[10],so loading another kernel does not add too much overhead,particularly if the streamlining proposed below is applied. Finally,the large hidden code base of kboot is unsuitable if high demands on system reli-ability,at least until the point when the kernel is loaded,require that the number of software components be kept to a minimum.

3.7Extending kboot

The most important aspect of kboot is not the set of features it already offers,but that it makes it easy to add new ones.

New device drivers,low-level protocols(e.g., USB),?le systems,network protocols,etc.,are usually directly supported by the kernel,and need no or only little additional support from user space.So kboot can be brought up to date with the state of the art by a simple kernel up-grade.

Most of the basic system software runs out of the box on virtually all platforms supported by Linux,and particularly distributions for em-bedded systems provide patches that help with the occasional compatibility glitches.They also maintain compact alternatives to packages where size may be an issue.

Similarly,given that kboot basically provides a regular Linux user space,the addition of new

36?kboot—A Boot Loader Based on Kexec

ornaments and improvements to the user inter-face,which is an area with a continuous de-mand for development,should be easy.

When porting kboot to a new platform, the foremost—and also technically most demanding—issue is getting kexec to run. Once this is accomplished,interaction with the boot loader has to be adapted,if such inter-action is needed.Finally,any administrative tools that are speci?c to this platform need to be added to the kboot environment.

4Future work

At the time of writing,kboot is still a very young program,and has only been tested by a small number of people.As more user feed-back arrives,new lines of development will open.This section gives an overview of cur-rently planned activities and improvements.

4.1Reducing kernel delays

The Linux kernel spends a fair amount of time looking for devices.In particular,IDE or SCSI bus scans can try the patience of the user,also because they repeat similar scans already done by the?rmware.The use of kboot now adds another round of the same.

A straightforward mechanism that should help to alleviate such delays would be to predict their outcome,and to stop the scan as soon as the list of discovered devices matches the pre-diction.Such a prediction could be made by kboot,based on information obtained from the kernel it is running under,and be passed as a boot parameter to be interpreted by the kernel being booted.

Once this is in place,one could also envision con?guring such a prediction at the?rst stage boot loader,and passing it directly to the?rst kernel.This way,slow device scans that are known to always yield the same result could be completely avoided.

4.2From shell to C

At the time of writing,the boot shell is a Bourne shell script.While this makes it easy to integrate other executables into the kboot shell,execution speed may become an issue, and also other language properties,such as the dif?culty of separating name spaces,and how easily subtle quoting bugs may escape discov-ery,are turning into serious problems. Rewriting the kboot shell in C should yield a program that is still compact,but easier to maintain.

4.3Using a real distribution

The extensibility of kboot can be further in-creased by replacing its build process,which is very similar to that of buildroot[11],with the use of a modular distribution with a large set of maintained packages.In particular Open-Embedded[12]and Rock Linux[13]look very promising.

The reasons for not reusing an existing build process already from the beginning were mainly that kboot needs tight control over the con?guration process(to reuse kernel con?gu-ration,and to propagate information from there to other components)and package versions(in order to know what users will actually be build-ing),the sometimes large set of prerequisites, and also problems encountered during trials.

4.4Modular con?guration

Adding new functionality to the kboot environ-ment usually requires an extension of the build

2006Linux Symposium,V olume One?37

process and changes to the kboot shell.For common tasks,such as the addition of a new type of path names,it would be desirable to be able to just drop a small description?le into the build system,which would then interface with the rest of kboot over a well-de?ned interface. Regarding modules:at the time of writing, kboot does not support loadable kernel mod-ules.

5Conclusions

Kboot shows that a versatile boot loader can be built with relative little effort,if using a Linux kernel supporting kexec and a set of programs designed with the space constraints of embed-ded systems in mind.

By making it considerably easier to synchro-nize the boot process with regular Linux devel-opment,this kind of boot loader architecture should facilitate more timely support for new functionality,and encourage developers to ex-plore new ideas whose implementation would have been considered too tedious or too arcane in the past.

References

[1]Almesberger,Werner.Booting Linux:

The History and the Future,Proceedings

of the Ottawa Linux Symposium2000,

July2000.

https://www.wendangku.net/doc/55115431.html,/

cv/papers/ols2k-9.ps

[2]Biederman,Eric W.Kexec tools and

patches.http://www.xmission.

com/~ebiederm/files/kexec/

[3]Goyal,Vivek;Biederman,Eric W.;

Nellitheertha,Hariprasad.Kdump,A

Kexec-based Kernel Crash Dumping

Mechanism,Proceedings of the Ottawa

Linux Symposium2005,vol.1,pp.

169–180,July2005.http://www.

https://www.wendangku.net/doc/55115431.html,/2005/

linuxsymposium_procv1.pdf [4]P?ffer,Andy.Reducing System Reboot

Time with kexec,April2003.http://

https://www.wendangku.net/doc/55115431.html,/archive/andyp/

kexec/whitepaper/kexec.pdf [5]Nellitheertha,Hariprasad.Reboot Linux

Faster using kexec,May2004.

https://www.wendangku.net/doc/55115431.html,/

developerworks/linux/

library/l-kexec.html

[6]Andersen,Erik.uClibc.

https://www.wendangku.net/doc/55115431.html,/

[7]Andersen,Erik.BUSYBOX.

https://www.wendangku.net/doc/55115431.html,/

[8]Kroah-Hartman,Greg;et al.udev.

https://www.wendangku.net/doc/55115431.html,/pub/

linux/utils/kernel/hotplug/

udev.html

[9]Johnston,Matt.Dropbear SSH server

and client.http://matt.ucc.asn.

au/dropbear/dropbear.html [10]CE Linux Forum.BootupTimeResources,

CE Linux Public Wiki.

http://tree.celinuxforum.

org/pubwiki/moin.cgi/

BootupTimeResources

[11]Andersen,Erik.BUILDROOT.http:

//https://www.wendangku.net/doc/55115431.html,/ [12]OpenEmbedded.

https://www.wendangku.net/doc/55115431.html,/ [13]Rock Linux.

https://www.wendangku.net/doc/55115431.html,/

38?kboot—A Boot Loader Based on Kexec

Proceedings of the Linux Symposium V olume One

July19th–22nd,2006

Ottawa,Ontario

Canada

Conference Organizers

Andrew J.Hutton,Steamballoon,Inc.

C.Craig Ross,Linux Symposium

Review Committee

Jeff Garzik,Red Hat Software

Gerrit Huizenga,IBM

Dave Jones,Red Hat Software

Ben LaHaise,Intel Corporation

Matt Mackall,Selenic Consulting

Patrick Mochel,Intel Corporation

C.Craig Ross,Linux Symposium

Andrew Hutton,Steamballoon,Inc.

Proceedings Formatting Team

John W.Lockhart,Red Hat,Inc.

David M.Fellows,Fellows and Carr,Inc.

Kyle McMartin

Authors retain copyright to all submitted papers,but have granted unlimited redistribution rights

to all as a condition of submission.