Monday, May 30, 2011

Weekly Update

Hindered once again by being sick (e.g. I slept 18 of the last 24 hours) I didn't get quite as much work done this week as I hoped for. Below I have outlined many of the elements of the Android Emulator that will need to be merged into mainline QEMU.

Essential Elements
All the goldfish hardware. Additionally this needs to be supported by new command line arguments to properly configure the new devices. The command line arguments will ideally be done by parsing the hardware configuration file for an Android Virtual Device (AVD) with a new command line argument (-android-hw ?) as opposed to manually configuring an AVD via the command line.

As far as I can tell, this is the only thing essential to booting an Android Image, as other elements, such as Networking and additional hardware sensors can be disabled. The CPU emulation in target-arm/ shouldn't need to be modified, as it runs off the standard ARM CPU.

Important Elements
Networking. The Android Emulator contains it's own heavily modified version of net.c and the slirp/ directory, as well as the addition directories telephony/ and proxy/ that don't exist in mainline QEMU. Without merging this, the booted Android Image would be pretty useless for most purposes. Because this code must be shared with the other target architectures (unless we set up a specific target-android build), and it contains a large number of changes to get it to work, merging the Networking portion may take up a significant amount of time.

Elements to Disable
There are several elements that need to be disabled for Android as they are not supported, i.e. migration and usb support. Additionally, there are several areas where code is disabled during an Android build that will have to be dealt with some other way since the current plan is to integrate it into the target-arm build.

Unknowns
The file 'aio.c' was modified with the addition of iolooper (a wrapper for select()) but I'm still not quite sure why. Iolooper is utilized in several other android files so it can be merged when necessary, though since it is only used in android specific areas it is a trivial merge.

There are several other areas where there were heavy changes made, such as in 'qemu-char' that I do not yet understand.

Conclusions
I still believe that the main high-level elements to be merged are only the goldfish hardware and the networking elements. In addition to these, there were a number of changes made to the helper subsystems that will be difficult to determine, and I'm sure that I'll end up viewing a lot of years old commits to separate upstream merges from necessary changes.

What worries me is the number of changes made for Android that are almost certainly incompatible with the other QEMU targets. This may necessitate a new 'target-android' build in mainline QEMU or there would be some possible performance degradation with a global 'android_guest' variable.

Friday, May 27, 2011

Current Focus

I've been looking through the source of the Android Emulator (AE) and comparing it to mainline QEMU. I'm currently stepping through the AE's wrapper main function to figure out how much I can strip out and how to manually emulate an Android Virtual Device (The Android SDK specially sets up AVD's for use with the AE, hiding the image files from the user). Some things of note:

Goldfish
The hardware used in the Android Emulator is essentially self-contained. While this is probably the most essential piece to port over for basic operations, I don't think it will be as difficult as I originally conceived due to the modular design of the hardware component of QEMU.

Networking
The AE heavily modified/added on to the networking components of QEMU. They created their own copy of slirp/ under slirp-android/ and modified it for their own use, though I don't understand what changes they've made as of yet. They've also added support for the phone components under telephony/, including support for SMS and GSM.

I don't believe that the specialized networking components will be necessary to get Android to boot on QEMU, so I will plan to port these components after those necessary to boot.

Wednesday, May 25, 2011

QEMU Dev Documentation

The one thing I've found lacking in the QEMU project is source code documentation. There are several papers on the technology behind the emulation features in QEMU, but not a lot of information is out there about how each piece is implemented except for the source code (which for the most part lacks useful comments). This creates a bit of a barrier to entry for new developers on the project, and is actually one of the reasons I did not start contributing to QEMU during the school year, as I didn't have the time to devote to get up to speed.

As I've begun to explore this myself, I decided that I'd make a list of links that I've found helpful in getting up to speed. Since I've found that there really are now good sources for organizational details of the source code other than the source code itself, I decided that I will do my part and contribute to some of that documentation later on in the summer.

http://wiki.qemu.org/Manual :
This is the official documentation for QEMU that mostly focuses on users, but also has a little bit of information for new developers and some useful links.

http://old.lugatgt.org/articles/qemu_internals/downloads/slides.pdf :
These are some slides from a presentation given by Chad Kersey on QEMU Internals that focuses mainly on dynamic translation and basic block chaining. It also notes where some of those internal features are located in the source code, so it's easy to start browsing the code.

http://qemu.weilnetz.de/qemu-tech.html :
This link is generated from the qemu-tech.texi file located in the source tree. While this copy may be up to date, it would be safer to generate this document yourself from the source code tree. The qemu-tech document describes the internals of QEMU, while the other .texi files in the source tree provide user documentation.

http://wiki.qemu.org/Download
The QEMU source code. All the documentation is there in the form of source code.


A technical paper by Fabrice Bellard discussing the original internals of QEMU. This paper is from 2005, so the details may have changed significantly.**Edit: As noted in the comments below, this is extremely out of date, so don't read it unless you're interested in the project's history.

Monday, May 23, 2011

Setting up Android Environment

Most of the information I am about to relay can be found here at the official walk through for setting up a Build Environment for the full Android system. I wrote up a short shell script to do all the set up work and start a full compilation of the system for the efficient that I have made available here, though I will point out it is necessary to read through it and make some small changes to customize it for your own folder naming conventions.

I found that I hit the least snags using the latest Ubuntu x86_64 version, so the script installs dependencies for that distro. Something else could be substituted in but all the Android testing is done on Ubuntu systems, and the build process will stop and complain if you try x86.

After the compilation finishes, all of the output binaries will exist under the "out/" subdirectory. Usage for the emulator is described on this website, although more options are available by querying the binary. The emulator can be built individually by running 'make emulator' in the main Development Environment folder or with more control by following the instructions in 'external/qemu/INSTALL'.

I chose to build the full Android environment so I could have more control over building an Android Kernel as well as having full access to all the necessary emulator bits.

Sunday, May 22, 2011

Beginning

Here we are, the official beginning of coding time for GSOC 2011, and I'm already behind where I hoped to be. I absolutely hate it, but at this point it can't be helped. Some of the blame lies with me for not buckling down and working just after my graduation, and some of it lies with a viral infection that my doctor says is probably mono. Due to my sudden obsession with sleeping half the day and being tired and achy the other half, my usual schedule was disrupted and my productivity dropped to almost nothing. I had hoped that what I proposed for the "Initial Ramp-Up" stage would be completed before the official start, so I am already a week behind what I had hoped to be my schedule. I seem to be getting over my illness so I estimate that I will be roughly 75-80% productive this first week, and I'll work a bit more than planned over the following weeks if necessary.

As of now I do not have any vacations planned that would disrupt my work, though that will probably change at some point and I may lose up to a week. As soon as plans are made I will update the schedule to reflect any necessary changes.

Proposed Rough Schedule

The coding period for GSOC runs from May 23 to August 22, a period of 13 full weeks. Additionally, a mid-term evaluation is scheduled for July 11, which is the beginning of the 10th week. I've divided the 13 week summer into 4 separate stages, each of which has their own completion criteria.

Initial Ramp-Up - 1 Week (May 23 - May 29)
For the first week of the sumer, I will be doing a couple of things. First off, I will be comparing the Android Emulator with Mainline QEMU to determine, and make a list of, the broad areas I need to merge into the Mainline to get it running an Android image. While most of the changes appear to be in the addition of the "goldfish" hardware emulation, there are several other areas that need to be checked to determine what needs to be merged. The current Android Emulator can't be compared only to QEMU release 0.8.2 because some later patches to QEMU have been selectively added back into the Android Emulator, which complicates this process. After this process I will have a better understanding of how the timeline for the summer will go and will update the schedule.

During this stage I will also complete some of the due diligence that I've fallen behind on, including setting up a public git repository for my branch of Mainline QEMU and posting setup instructions for building the Android Emulator (I've got this in a script for Ubuntu 64-bit which is running right now, so this shouldn't take long).

Merging Changes - 9 Weeks (May 30 - July 31)
As already mentioned I will probably be changing the plans for this section of the schedule after I finish determining the necessary changes to be made during the Initial Ramp-Up period. This period is budgeted at 9 weeks of work, and will be entirely devoted to merging the portions of the Android Emulator into my branch of Mainline QEMU in order to enable the running of an Android image. My current schedule of changes to be made divides the necessary modifications into two segments, "goldfish" hardware and everything else. As it seems most of the changes are in the "goldfish" hardware, I'm hoping to get that done in the first 5 weeks and use the remaining 4 to work on the rest.

The end of this stage occurs when all of the necessary merges listed during the Initial Ramp-Up period are completely merged into my branch. The mid-term for GSOC occurs one week after I plan to be finished merging "goldfish", so even if I fall behind my current project schedule I should be finished with that portion by the mid-term. Finishing that will be my main goal to be on track at mid-term evaluations.

Testing and Debugging - 2 Weeks (August 1 - August 14)
The two weeks devoted to testing and debugging may seem excessive, but I'm trying to allow some leeway for any unforeseen issues. If everything works fine then I can focus on fixing everything up and cleaning up the documentation. Since everything is not going to work fine and/or the previous stage will take more time than anticipated, these two weeks are being set aside for

At the end of this period, no more code merging should be taking place. The only anticipated code modifications would be resolving diff conflicts with Mainline.

Final Stages - 1 Week (August 15 - August 21)
During the final week I plan on doing mostly clean up work, catching up on documentation and hopefully getting the diffs integrated back into Mainline QEMU. I may be crunched for time moving into a new apartment and preparing for grad school this week (my school year begins the day GSOC ends) so I'm planning on having most everything done before.

Monday, May 16, 2011

First Day

For those of you coming here unaware, allow me to summarize the project.

Background
When Google wanted to create an emulator for their Android OS they chose to base their design off of one of the most popular, QEMU, to save on design work. They forked the project from the 0.8.2 release and made modifications to it to enable the emulation of an Android image. Additionally, some design improvements introduced to mainline QEMU may have been cherry picked and inserted into the Android Emulator project which could provide for some confusion down the line. Now that this Android Emulator has matured and is viable, mainline QEMU wants to include all of the necessary bits to run Android as well, leading to this Summer of Code Project.

Tasks
The task laid out for this summer is straightforward in its description, select the appropriate changes made to QEMU 0.8.2 necessary to run Android images and import those changes into the mainline QEMU. To do this I must analyze the current Android Emulator code for the necessary core components and then work on porting the components to the mainline. As I get deeper into my analysis and understanding of the system throughout this week, I'll be providing a planned schedule of my goals expanding on these thoughts.

Current Work
Due to school and other issues I have only just begun to work on the project, which will give me a full week of preparation before the official start. I have been setting up my development environment complete with build scripts and I will post an instructional for doing so in a day or two. Additionally I will be mapping out a schedule for the summer and hopefully getting to know the community a bit during this last week.

I'll be logged in under my username 'pjack' and lurking on #qemu and #qemu-gsoc on irc.oftc.net whenever I can remember to do so. And I'm a bit new to using IRC, so don't be too mean ;)