Ryan 2.0

The Road of HBase

2010-04-13T15:46:00.000-07:00

A short update about HBase. Still busy finishing up the last finishing touches on 0.20.4. Also looking at the road towards durability in HDFS. Some great patches upcoming. Performance is always lurking everywhere.

I'll be at the Big Data Workshop this month. See you all there!

HBase 0.20

2009-06-24T23:38:00.000-07:00

My HBase colleagues and I have been working hard on the next generation of HBase - the upcoming 0.20 release! We recently published an alpha release and are working on stabilizing and testing for the final 0.20 release candidate.

The release sports many new and cool features:
- New file format offers improved efficiency and features
- New read paths are faster and less memory intensive
- Zookeeper integration - allows multiple masters
- New API, better delete semantics, improved caching, and more!

In addition to code, I've also taken a bit of time out to speak at a recent conference here in SF. There were some other great projects speaking, all videotaped and uploaded. Check it out, lots of smart people talking on some fascinating subjects.

HBase update and 0.20

2009-05-06T23:03:00.000-07:00

Thanks to a long list of contributions at many levels in HBase, the Hadoop Project Management Committee has voted to approve my committer status. This is reflects important contributions to making HBase 0.20 stable and performant.

In the mean time, HBase 0.20 is on track. Recent changes involved porting HBase to use Hadoop 0.20, and the introduction of the real-time Lzo compression library. Initial tests indicate that using Lzo over GZ can nearly double read and write speed.

But why use compression? Surely the extra time to actually compress and decompress all records would make things too slow? With the use of the lzo "real time" compression library, one can achieve reasonable compression but at a very low latency/cpu cost. The benefits include a more efficient use of bandwidth and disk. Instead of reading a 64k block, that block compressed might be 30k instead to represent the same data (a 2.1 compression ratio seen in my data with lzo). Now your reads from HDFS are twice as fast, and double the amount of data you'll actually be able to store.

The next steps in HBase 0.20 are a series of fundamental changes to how deletes are stored in the store files (HFile), and how reads and scanners work. These should improve performance via better algorithms and a more efficient processing of insert and delete entries in multiple HFiles.

In the mean time, HBase trunk works and is stable. If you have a data set that is small values or you need more speed, give it a shot. It's not a release quality and future commits may invalidate your entire installation, but I have loaded hundreds of gigs of data and successfully read it out again. Maybe it will work for you?

C - A New Golden Age

2009-03-27T21:00:00.001-07:00

I've been noticing a new trend lately - more people seem to be eschewing C++ and moving to C. There seems to be several major reasons why this is happening, and it's an interesting story of the evolution of programming.

One major reason is simplicity - C is substantially simpler than C++. Furthermore, given the level of complexity, including ABI issues, there are very real arguments as to if the complexity is worth the benefits. At least one start-up I know had made a policy to use C over C++ for their real-time systems. Given the need for memory intensive, CPU bound processes, one is often forced to use non-VM languages like C, C++. Given the practical choice between the two, choosing the former can be a better choice!

The other reason is interoperability - C has a standardized application binary interface (ABI) that is the system-default binary interface on Unices. Furthermore, C's ABI does not change, and will not change. Compare and contrast to C++ which does not have cross-vendor compatibility (soon apparently), and may not be compatible across point releases (GCC 4.1 not compatible with GCC 4.2 for example).

As a corollary to the last note, many people are opting to use more 'scripting' or higher level languages - such as Python, Ruby, Lisp or OCaml. All these languages natively interface directly with C, but not always with C++. If they do support C++ they support a subset - such as no exceptions - thus forcing one to wrap C++ libraries to play nice. One can write the memory-intensive, CPU bound parts in C, and write the higher level logic in Python, thus getting the benefits of both worlds when necessary.

It seems like now is the time for a new Renaissance of C - there are many libraries out there which provide highly advanced functionality wrapped up in C (eg: gtk++, gnome, libapr). Basing your next performance sensitive project on C may be the seemingly anachronism your team needs to escape C++ compile hell.

HBase 0.20 Dev Preview

2009-03-13T15:37:00.000-07:00

As a major contributor to HBase, I have had the privilege of seeing some of the new features of the next release in action before anyone else, in part because I developed them. I can say that 0.20 is shaping up to be an amazing release that will wow many people (I hope!).

In my last blog post I went into the historical of how a new file format came about. Well as we stand a few weeks later, history is now fact, and HBase 0.20 is now based on HFile (H for HBase, they rejected RFile). With a minimal integration, HFile provides a 5x baseline speedup compared to HBase 0.19.x. This is the simplest integration - not changing how HBase stores keys, or doing anything else. Additional performance improvements must come at the cost of more intrusive fixes.

We estimate that there is another 5x speed improvement by doing more intrusive fixes - the first of which is to change the way HBase stores keys in the store. While doing that we pave the way to avoid object allocation and copying bytes around. This also involves turning lots of functions from taking byte[] to taking byte[],int,int. It also means instead of copying bytes in and out of new objects, often just to compare 2 keys, instead create and use 0-copy pointers into existing arrays.

This also preps HBase for a new RPC that pushes the serialization out to the client. The client will eventually serialize keys and values into a format that will allow the server to shuffle bytes directly in and out of HFiles.

This exercise has been also been about how to make Java CPU and memory efficient. There has been many lessons on how exactly does Java use memory, what are per-object overheads, hidden APIs (look at java.lang.instrument) and just general bad-ass Java programming. I encourage anyone who has an interest in fast and efficient Java programming to visit us on irc.freenode.net channel #hbase.

Scalability: What are YOU doing about it?

2009-02-13T15:21:00.000-08:00

I've been working on HBase lately. The system is cool, integrates nicely with Hadoop and HDFS and has an awesome community. But there is one elephant in the room: raw seek and read speed of HBase.

Due to the speed issue, hbase is ruled as interesting but too slow in some blog posts. Since serving a website directly out of HBase is on the todo list for many users (including me), I took some time out to do something about it.

At the recent HBase LA Hackathon, a number of core system implementers and commiters met and hung out with users and new users. On the agenda was planning the next 0.20 release.

It became apparent that there were two massively needed features:

ZooKeeper integration for reliability

Faster HBase for website serving

While both of these are very interesting to me, the faster HBase was of a more critical need for me. I sat with Stack one of the lead developers to do some performance testing and profiling on a new file format.

When retrieving data from disk, the file format is a performance linchpin. It provides the mechanisms for turning blobs of data on disk back into the structured data the clients expect.

We were evaluating a new proposed file format (known as TFile), and while the raw numbers on the file format seemed good, profiling exposed some worrysome trends. The details are too detailed, but my intuitive guess is the layered stream micro-architecture of the new file format was not leveraging explicit block efficiency, and instead was relying on the layers of streams to block read instead. The problem is it becomes hard to control, and loss of control reduces performance in this case.

Furthermore our tests were being done on local disk which has the benefit of OS block caching. When you move to HDFS, things change - data gets are expensive network RPCs, and you want to cache as much as possible as well as retrieve as much as possible at once.

So in parallel with Stack, he went forward and ripped out stream layers from TFile, and I decided to start anew and write a new block-oriented file format. Once both got to a sufficient space, Stack ran performance tests, and the results were conclusive, the new format, dubbed HFile, is vastly superior to existing formats (Mapfile).

My new file format HFile (previously RFile) is block oriented (you read block chunks at a time) and has an explicit block cache via an interface. Reading, all the data is stored and referred to by ByteBuffer reducing unnecessary duplication. The streaming features of other formats were removed, since with HBase it is necessary to store the key/value in memory at least once during the write (into the so-called "memcache"). Reducing that complexity while explicitly managing blocks has resulted in a file format that is faster, reduces memory overhead (See: ByteBuffer) and improves in-memory block caching all in one go.

The current status is that HFile is on track to be the new official file format for HBase 0.20. Development is active, and while there is nothing for people to use yet, you can see the glorious details at github: http://github.com/ryanobjc/hbase-rfile/commits/rfile

Join us on IRC as well - #hbase on freenode.

Performance of HBase importing

2009-01-09T20:52:00.000-08:00

My latest project has been to import data in to HBase. The data profile is very simple, 6 integers, serialized using Thrift and the natural key is 3 of those 6 integers.

The table design is simple, a single column family, and row key is "%s:%s:%s" - the stringified version of the business key integers. The stored data is a simple 6 integer thrift struct serialized using the binary serialization. The data source is flat files.

Loading data is not entirely trivial since there a number of options. The native HBase client is in Java and is fairly thick and intelligent. Since our data is stored on multiple servers, the client has to do the work of finding which server and then talking to that server.

If you are not interested in using the Java client, the option is to use a Thrift gateway. This is a Java thrift server that provides a thrift API and uses the HBase client to talk to the HBase cluster to get things done.

So my initial attempt was a Python client that talks via the Thrift gateway. This allowed me to load about 20 million records in 6 hours.

My next step was to directly interface with the HBase API using Jython. Jython is only up to CPython 2.2, but a 2.5 release is in the works. The roadmap talks about 3.0, which is nice.

My first attempt at a Jython/HBase/JVM program netted me a 5 times speedup, allowing me to load 20 million records in 70 minutes. I was getting about 4 inserts per milliseconds. This scales with multiple writers as well.

Thanks to a hint by Stack on IRC I added these lines of code:

table.setAutoFlush(False)
table.setWriteBufferSize(1024*1024*12)

And now I achieved another speedup. Currently with these lines I can import between 30-50 rows per ms. This allows me to import 20 million records in 12 minutes! This also scales up to at least 6 times parallel - meaning I can import 180-300 rows per ms. That is Millisecond, aka 1/1000th of a second.

This represents a net 30x speedup from using the Thrift Gateway to using the native HBase API and tweaking buffering settings.

The Path of a Programmer

2008-11-17T18:42:00.001-08:00

The path of a programmer is a varied and twisted one. You start out on a problem, and before you know it, you've ended up in the deepest, darkest dungeon.

I started out in the Green Fields of Python code. I should enhance this script I thought. But, it's complex and critical, so I should test this before I write more. I know it works now, but it might stop working in the future.

So, before I can test, I need to restructure to make it into more "bite sized chunks". Each chunk is easier to test. Futhermore, Python forces you to do so if you want to test. Ok, now I have to worry about module structures and where to put files. A moment of consternation, then moving on.

Now, let's test... test code is going ok. Wait, what is this 'global' keyword?

And now, a few hours later, I am finding myself annoyed that Python doesn't have lexical closures. I had the exact same problem as this person here.

I was remarking to my co-workers that as a programmer you are always chasing after that perfect language. You want the balance between clean design, all the features you want, and modernity. Roughly speaking the language choices I've pondered about are:

- Python: Very clean, modern, but missing strange features. The white-space sensitivity makes it a hard without a full editing environment - not the greatest embedded scripting language.
- Javascript: Elegant, has good features, and severe gotchas. More of a web dev language, and not as well established as a server-side programming language. Lacks advanced features (think OS-level functions).
- Ruby: Frankly I don't know enough, but I do know the lack of unicode support is a major concern - with UTF8, multi-lingual support becomes easy.
- Java/C++/C: Too static. You end up writing too much code.
- Erlang: Cool concurrency model, but the language seems to lack "programming in the large" features.
- Common-Lisp: Inspired by Paul Graham perhaps? CL is fairly old and has wonder features from the early 80s, such as mono-case symbols. Even major Lispers don't recommend using Lisp for serious stuff.
- Haskell: Very weird, and is it practical to get things done in a pure functional language?
- PHP: Too hacky at a language level, and lacks many features. Passing function names as strings? Ouch.

I've obviously left off a huge number of languages, these are the ones that are foremost in my mind. If you feel like I've done a major disservice and my life could be transformed, let me know!

Practically speaking, my life is ruled by Python and PHP right now. PHP has quirks, but it's reasonably dynamic and not too bogus. Python is one of the better languages, but things like lexical closures and the limited-lambda support really chafes me. I want uber power, and I'm not sure I'm getting it.

Election over, can we get back to code please?

2008-11-05T18:40:00.001-08:00

Well the elections are over. With an epic battle for the white house, it's like a slow-moving car crash - how can you NOT look?

But it is now time to get back to the code. Time to crank out those functions, write those new classes, debug those files of twisted goto-ridden lines.

Recently at work we were having a discussion about what kinds of new systems we might need to build. Some people were advocating trying to make do with existing systems, but tweak them more. Others were arguing that this might not be enough, and we'll have to end up coding new persistent-daemons anyways, we might as well get a start now.

Both arguments are pretty convincing, the first one is summed up by "don't over-engineer, only build what you need." The best bug you can have is the one you don't write. The second argument would be "if you can easily predict what you need, get a start early so you can build experience with it." If you know you'll need a database, why not set it up early and build expertise in-house?

The first argument seems to be a perfect counterbalance to the second. More code vs less code, in this structure, the less code would seem to have to win. If you take a longer term approach, you can easily come to the conclusion that you may end up with more code in the long term - sometimes the best time to make changes is when things are flexible. Less cruft to work around later, less data to migrate. Build expertise when things are relatively low volume.

These two approaches are seemingly intractable. Stubborn developers in the #1 camp can drag out debate and ultimately "win". But there is a third way.

What if a new system wasn't 50,000 lines of Java code? What if a new persistent-server-process was 500 lines of code? What if a complicated transaction-logged persistent engine was in reality 2000 lines of code? A good developer can make radical changes to 2000 lines of code in a weekend easily.

What if you can build the new system in an afternoon? There is practically no reason not to toss out that system, after all, afternoons have been wasted on even more trivial things - OS problems, expense reports, the DMV. The sooner you can build and throw out code, the more you'll learn about the nature of your problem.

But how do you build and throw out code quickly? The ultimate limit is how fast a human being can type - say 60 words per minute. If we assume that one line of code is 4 words, then we have 15 lines per minute, or about 1000 per hour. One afternoon would be about 4000 lines. And practically speaking most programmers would be lucky to get a solid 4 hours of coding a day.

So if you can only realistically get a few thousand lines of code out of a programmer a day, you better make sure those lines really count. By increasing the level of abstraction, coders write less words yet get more done. This is a fundamental argument that is difficult to dispute - no one would argue that programmers are more productive in C vs assembly, or in Python vs C. The power generally comes from what you do not have to do as you code, vs what you do. Memory management, structure definition, fundamental libraries, loops and so on. All of these things improve the levels of abstraction, and thus improve the level of programmer productivity - you simply go farther with those 4000 lines of code.

This is a popular minority argument to make. I think one of the better ones is done by Paul Graham. Nearly every language jockey takes this position - some languages are more powerful than others, and why would you use the lessor of two powers?

But in the end, programmers are human, and humans are creatures of habit. And so we only get a new language once every 7-10 years. C in the 80s, C++ in the early 90s, Java from about 1997-2005, and now... what?

Election night photos

2008-11-05T00:05:00.000-08:00

There was an epic election results party in the Castro tonight. Tempered a bit perhaps due to the Prop 8, but still, very exciting and happy. Here are some photos:

Android OTA Upgrade

2008-10-30T02:23:00.000-07:00

It's pretty late but I just got the notification that my G1 wanted an update. Early adopter is early adopter so I let it go.

It's a pretty slick updater, completely hand free and wireless. One nice thing about the G1 vs the iphone is the degree of independence. The iphone is essentially a teathered device. You can't get all data off without syncing with itunes. Unless you fork over for mobile me. At least gmail and gcal are free.

I really hope they maintain a good update velocity. I can think of a dozen features that I'd love to see.

Android Stuff

2008-10-28T16:51:00.001-07:00

Well it's been a week since I started using my G1. Overall, this is a great device, and I won't be switching back to my iPhone original flavor. So here is my new list of things I really love about this device:

3G speed - 800 kbps in tests. EDGE can come in at up to 200 kbps too!
GPS - the 50' accuracy rules!
IM - It's great to be able to instant message on the device.
MMS - picture messages, they are just so handy when you need them.
Push Gmail - this is just outstanding.
Gmail interface - finally a compelling email application for a mobile device. If you are a heavy Gmail user, this is worth the price of admission.
Auto-sync contacts and calendar. Don't worry about losing this device!
Full text keyboard is fast and awesome.
Scroll ball is surprisingly useful - it's great for navigating tiny links on webpages.
Notification system and swipe-down status bar is surprisingly awesome and usable!

Now, on to some things that need improving:

Regular email app is substandard.
Surprisingly: I miss the portrait virtual keyboard of the iPhone. Flicking open the keyboard all the time for quick SMSes isn't always handy. I'm in luck though: this feature is on the short-term Android roadmap!
Android Market - needs more apps. Needs the incentive of pay-apps to up the overall platform quality.
Apps in general - you need to cache more (on the SD card?) and avoid using the network.
Battery life... This is probably the darkest note here. Battery life can be dismal. I've heard of people getting more life as the battery gets calibrated.

The G1 is a definate buy. If you have the 3G iPhone and are happy with AT&T you probably won't benefit from the move. But if you hate AT&T or are unwilling to buy into the iPhone phenonema, consider the G1. This is also a great device for programmers - you can code apps up quickly and without Apple's approval. I cannot emphasize this enough, since I am dealing with a multi-week iPhone dev-sign up flow right now and it is so annoying.

Now, on to photos... The G1 has a very respectable 3 megapixel camera, but it won't be causing me to leave my Canon 5D at home anytime soon. Here is a sample photo:

Nighttime photos are tricky, but no more so than under the iPhone. Bonus: it you are tricky with the physical button you can take candid shots.

My Android review

2008-10-21T23:10:00.000-07:00

Well the Android arrived today. First off, I noticed that someone at Google is taking box design cues from Apple. As soon as you open it up, there is a little spot for the phone and you can yank it right out. Check it out:

This was a nice touch, and I'm glad someone was paying attention. The phone is not featured because it is taking the photo itself!

Setting up was a breeze. I moved my tmo sim from my old iPhone to the Android device, then it asked me to enter my Google address. A few minutes later and it was all done. Next step is syncing my address book entries (from the iPhone), I'll let you know how that goes.

Instead of a big blob of blog text, I'm going to list the things I really love so far about the Android:

Physical buttons = maximizing of screen real estate.
Unlock screen is cool and fun.
The GPS is really awesome and really accurate when you drive around.
The notification system with the slide-down status bar is a fantastic UI system
Push GMail rules
Oh the GMail client is as great as the web client
Instant Messaging - it's as integrated as SMS
MMS again - that was tiresome iPhone

One interesting thing about the Android OS is they are trying to break down the wall between applications and move towards activities/screens. For example, in the maps application you can change some system settings (GPS receiver on/off) - but instead of opening up the settings app, it looks seamless and when you hit the 'back' button you end where you were. In the iPhone you would have to remember to switch back to the maps application after changing the setting.

Overall, the Android is not as shiny-slick as the iPhone, but it is slick in it's own way, and very very fast which is a huge plus. It also feels like using a portable computer again, not just a pre-defined gadget (the iPhone).

Android... it's alive!

2008-10-18T01:07:00.000-07:00

Well I finally got an updated shipment status from UPS:

Looks like my previous hope/assumption that UPS had the item but wasn't scanning it just wasn't true. For some reason T-Mobile send the shipping number to UPS 3 days prior to its actual shipping.

What is really interesting is how the package went from Kentucky to California in just 5 hours. The tracking says 'Mather, CA' which Google maps puts on the junction of several forestry roads. Ok, so poorly named tracking point, since Sacramento's airport is called "Sacramento Mather Airport." Air freight from Kentucky to California. I only hope that UPS doesn't dilly dally over the package until Tuesday because that is when it is due. I've seen them do things like that before.

Android shipping update...

2008-10-17T00:59:00.000-07:00

Er... there is no update! Since I last posted breathlessly that the Androids have started to ship, nothing has happened for my specific android. UPS still reports "billing information received." I guess I won't have a new phone on Friday.

Next week it is. In the mean time, there are a bunch of reviews hitting the street. Personally I want to see how much faster the 3G network is - AT&T's 3G iPhone network appears to be a speedy 2x over the slow EDGE network. Go go power 200 kbits/sec.

It's hard to tell exactly how fast the G1 will be on 3G, but some people think it will have HSDPA which should give it nearly megabit speeds. This all remains to to be seen.

However in the mean time, the 3G network has been switched on here in San Francisco. I'll let you know how it goes as soon as it hits my hands.

This just in - Androids start to ship!

2008-10-15T00:27:00.001-07:00

I've been anxiously awaiting my Android, so to my pure JOY and excitement I am proud to say that my particular Android, pre-ordered from T-Mobile, SHIPPED today!

Photographic evidence:

Now I have two things to wait for

2008-10-13T23:13:00.000-07:00

On the heels of the much-anticipated Android launch is another amazing product launch. This one already had it's press moment, and you probably missed it.

What you missed is Canon's next revision of the 5d, the Mark II. The original 5D (which I have) is a 12 megapixel wonder, with a full frame sensor, allowing you to get the full wide range out of your lenses.

Rewind a bit... Full frame sensor? Why is this important? Well in Canon's (and Nikon's) camera system (known as EOS) the lenses were built with a 35mm piece of film in mind. Since building sensors (which are essentially large slabs of silicon) at the full size of 24×36 mm was impracticable, camera manufacturers would cut some corners (literally!) and use a smaller size. This size is generally known as APS-sized.

What happens is the camera only uses a square in the middle of the available light from the lens. This is known as the cropping factor, and it effectively makes your lenses appear "longer" - that is the field of view is narrower as if the focal length was larger. If you take a film negative and chop off the edges, that is the equivalent effect as demonstrated by this blog post.

So your beautiful 50mm lenses become a less practical 80mm. If you want ultra-wide lenses, well you are out of luck, because an amazing 17mm is actually a boring 27mm lens. Lower numbers mean wider field of view - think those amazing panoramics.

The 5D was one of the first affordable full-frame sensor, allowing you to extract the maximum wide end of your lenses. Due to the large sensor size, the 5D had an amazingly smooth image, with great low light performance. Most digital cameras have a totally unusable 1600 ISO setting (higher settings offer better low light performance at the expense of a noisier photo). The 5D is noisy at 1600 ISO but it is both "good looking" noise and fairly reasonable. The original 5D was a winner.

But the new and improved 5D Mark II is going to be even bigger. Canon has brought it's 21 megapixel full frame sensor from the 1Ds Mark III to the 5D. Hugely expanded ISO range of up to 6400 ISO standard, or up to 25k ISO in "expanded" mode. It remains to be seen how good 25k looks, but if you need it, you'll have it. Live LCD view - yep, finally the 5D gets the feature every cheapo point and shoot has.

Another nice feature is the anti-dust feature. Dedicated 5D shooters know it as a very dusty camera. I have hundreds of photos nearly ruined by endless dust on the sensor. A combination of substandard environmental sealing and frequent in-the-field lens changes really did a number to the internal bits. Well the new revision has a long overdue feature that helps you minimize sensor-dust and lets you measure and compensate for it in-camera.

But what is even more amazing is the 5D's ability to capture 1080p high definition video! A photographer in New York managed to crank out an amazing video in a weekend with a pre-production unit. Sadly due to the FAT32 limitation, video sizes are limited to 4 GB or 12 minutes at the highest resolution. Choose your scenes carefully!

Most point and shoot video cameras have a video mode, but they are usually crippled by an extremely low resolution - you get 640x480 on the top end models. This might be good enough for some vacation videos, but it's nothing ultra impressive.

However, this new video mode of the Canon 5D Mark II is really pushing up against the highest end pro digital video systems. The 5D's sensor has 10x the pixels of 1080p HD video (which is slightly above 2 megapixels) and by downscaling, the quality of the output should improve. Combined with excellent lenses - the 24-105 L zoom is particularly great - a whole new generation of videographers should be kicking out some awesome HD content with a kit priced at under $4,000!

The 5D has a launch date in November sometime, and some online shops are allowing preorders. Canon is probably still tweaking their production line, so I figure we won't hear anything until the end of the month.

In the mean time, the original 5D is no slouch. Here is a spur-of-the-moment photo at Twin Peaks. I didn't have a tripod so the stone ledge was my impromptu tripod.

Photo Filler

2008-10-13T02:14:00.000-07:00

While we all wait for Android updates, here is a picture from San Francisco:

Welcome to Ryan 2.0

2008-10-13T01:38:00.000-07:00

Instead of doing a post about my bio, I just jump in to it.

The following two things are holding my fancy right now. The first is the new season of the Sarah Silverman show. The opening episode started out strong, easily one of the best of the whole series. One of the strong elements is the depiction of mild drug use - pot smoking to be specific. Generally standards and practices don't really allow the demonstration of pot smoking as anything but negative. A classic example is "That 70s Show" which had to cut out the consequence-free drug-doing as the show got more popular. It's just a breath of fresh air to see things displayed in a truthful manner - the truth being that millions enjoy recreational use of pot with little or no consequences, except for more frequent trips to 7-11 of course.

The second thing is the upcoming T-Mobile G1 launch. I ordered the G1 on the first pre-order date. I am keeping an eye out on UPS's ship track website... Doubly so since the original promise was an arrival date of October 22nd, which is fast approaching. The double part is the upcoming Thursday 9pm event with The Raconteurs. This suggests perhaps an early ship date for the 1.5 million of us who preordered.

Finally, keep an eye out on Tuesday for new Apple notebooks. Rumours are heavy that a new single metal piece manufacturing technique and casing are to be on display. I am happy with my new Mac Book Pro from work, so I won't be buying any new hardware myself.