Fazal Majid's low-intensity weblog

Fuji GF670 first impressions

Fri, 29 May 2009 07:23:26 GMT

I just received my Fuji GF670 from Dirk Rösler at Japan Exposures. This is a folding medium-format rangefinder camera, an anachronism in many respects, but I regret not getting a G690 when they were still made and since this is a limited edition (apparently quite a popular one at that), I went ahead. I have not yet shot a roll, but here are my first impressions:

The unfolding mechanism is a bit finnicky. You have to be careful to get the front standard aligned with the film plane. Once deployed it seems fairly stable. Folding it back is also quite tricky.
The meter indicator LEDs and controls are very reminiscent of the Epson R-D1, not surprising since both are actually made by Cosina.
The leaf shutter is amazingly quiet. It makes a Leica sound like a clunker in comparison.
The camera is quite light for MF, it feels lighter than a R-D1 (even though it weighs nearly twice as much) and is not that much larger.
It does not exude quality like the Fuji-manufactured TX-2 (Hasselblad XPan II).
The rangefinder patch is bright and clear. The RF base length is very short as in a VC Bessa, and will probably not be as precise as a Leica, XPan or Zeiss Ikon.
The film loading mechanism is very easy to use, and built as well as other Fuji MF cameras such as the G617.
You have to remember to reset the lens to infinity focus in order to fold it.
You get a choice of 6x6 and 6x7, 120 and 220.
The optional case is a snug fit. I wish it included a belt loop.

In grand old techno-fetishistic tradition, I put up an unboxing gallery.

Anthony's Cookies grand opening

Sun, 26 Apr 2009 00:44:24 GMT

Another gourmet treats shop joined the burgeoning scene in the Mission. Anthony's Cookies opened today to a line that stretched around the corner.

As one of the officials present said, it takes courage to start a business in this economic climate. Specially in as business-hostile a city as San Francisco, if I may add.

Inside the store was a buzzing hive of activity, with the eponymous proprietor busy preparing batches of free cookies for the awaiting hordes. At $5 for a half dozen, these cookies are a steal. I tried the double chocolate chip, it came fresh from the oven and had a strong chocolate aroma and the right texture. All in all, a great addition to a neighborhood that already has more than its share of good places to indulge a sweet tooth. I added him to my Google map of recommended bakeries, ice cream parlors and sweet shops in San Francisco.

Garbage-collecting your Mac OS X Address Book

Fri, 27 Mar 2009 22:48:36 GMT

After years of Outlook and Palm synchronization, when I fully switched over to the Mac, I moved my contacts database over to Mac OS X's Address Book (painfully due to Outlook's roach motel tendencies, but that's another story). For the most part I am satisfied, its data model is far more powerful than Outlook's, and most Mac apps feature excellent Address Book integration. I can call someone using Skype just by right-clicking on a phone number, for instance.

I just noticed, however, that my scripted Address Book backups were pushing the bounds of the reasonable: over 60MB zipped. For a mere 357 contacts, that seems a tad excessive. Upon further inspection, I realized my Address Book directory ~/Library/Application Support/AddressBook was pushing 152MB, the bulk of it in the Images subdirectory.

It turns out there are two causes for this problem:

When you drag and drop an image for a contact, then crop it, Address Book keeps the full-size image around, presumably in case you want to change the crop later. In most cases this is unnecessary and wastes space.
Address Book does not seem to remove the images for a contact when you delete it. Worse, those images get carried over into manual backups and thus backing up, blowing away your Address Book directory and restoring from backup will not get rid of the cruft.

I wrote the short shell script below to back up the AB directory, extract the list of contacts and delete any image that fits in the two categories above. This took me down to a much more reasonable 11MB. You can download the zipped script here. Disclaimer: I tried my best to make this as generic as possible, but I cannot be held responsible if running this script causes you to lose data, so I would advise you to perform your own backup prior to running it.

#!/bin/sh
backup=$HOME/ab_clean_backup.$$
in_ab=$backup/in_ab
all=$backup/all
datadir="$HOME/Library/Application Support/AddressBook"
db=`echo "$datadir"/*.abcddb`

exit_ab() {
  echo killing AddressBook
  ps -u `whoami` | grep "Address Book" | grep /Applications | awk '{print $2}'|xargs -n 1 kill -9
}

backup_ab() {
  echo "Backing up address book directory $datadir to backup $backup"
  rm -rf  $backup > /dev/null 2>&1
  mkdir $backup
  ditto "$datadir" "$backup"
}

remove_old() {
  echo extract list of contacts in AB from SQLite
  sqlite3 "$db" "select zuniqueid from zabcdrecord"|cut -d: -f 1|sort > $in_ab

  cd "$datadir"
  cd Images

  # comment out the next two lines if you want to keep full-size originals
  echo removing full-scale images
  rm -f *.jpeg

  echo finding all the images
  ls -1|grep -v '\.jpeg$'|sort > $all

  echo removing images with no associated AB record
  comm -13 $in_ab $all | xargs rm
}

exit_ab
backup_ab
remove_old

Nehalem Mac Pro first impressions

Wed, 18 Mar 2009 07:00:25 GMT

Some people use laptops as their primary computing environment. I am not one of them. Desktop replacement laptops like the MacBook Pro are heavy, and truly portable ones like my MacBook Air are too limited. Even the desktop replacement ones have limited expandability, slow drives, poor screens, lousy keyboards. My workhorse for the last 5 years was a dual 2GHz PowerMac G5. I am surprised I kept it so long, but I guess that says something about the durability of Macs and how you are not required to go on the hadware upgrade treadmill with each release of the OS. To paraphrase Borges, each increasingly bloated version of Windows makes you regret the previous one. I am also surprised at how much residual value the G5 has kept on eBay.

That said, the G5 was showing its age. Stitching panoramas made from 22MP Canon 5DmkII frames in AutoPano Pro is glacially slow, for instance. I was not willing to switch to the Mac Pro until today because the archaic shared bus on previous Intel chips is a severe bottleneck on multi-processor and multicore performance, unlike the switched interconnect used by PowerPC G5 and AMD Opteron processors, both of which claim descent from the DEC Alpha, the greatest CPU architecture ever designed. The new Xeon 3500 and 5500 Mac Pros use Intel's new Nehalem microarchitecture, which finally does away with the shared bus in favor of a switched interconnect called QuickPath and on-chip memory controllers (i.e. Intel cribbed AMD's Opteron innovations).

I splurged on the top of the line Mac Pro, with eight 2.93GHz cores, each capable of running two threads simultaneously, and 8GB of RAM. The standard hard drive options were completely lackluster, so I replaced the measly 660GB boot drive with an enterprise-class Intel X25-E SSD. Unfortunately, at 32GB it is just enough to host the OS and applications, so I complemented it with a quiet, power-efficient yet fast 1TB Samsung SpinPoint F1 drive (there is a WD 2TB drive, but it is a slow 5400rpm, and the 1.5TB Seagate drive has well-publicized reliability problems, even if Seagate did the honorable thing unlike IBM with its defective Deathstars).

I originally planned on using the build-to-order ATI Radeon HD 4870 video card upgrade, but found out the hard way it is incompatible with my HP LP3065 monitor (more below) and had to downgrade back to the nVidia GeForce GT 120. It would have been nice to use BootCamp for games and retire my gaming PC, but I guess that will have to wait. The GT120 is faster than the 8800GTS in the Windows box, in any case.

In no particular order, here are my first impressions:

The "cheese grater" case is the same size as the G5, but feels lighter.
The DVD-burner drive tray feels incredibly flimsy.
Boot times are ridiculously fast. Once you've experienced SSDs as I originally did with the MacBook Air, there is no going back to spinning rust.
I have plenty of Firewire 800 to 400 cables for my FW400 devices (Epson R1800, Nikon Super Coolscan 9000ED, Canon HV20 camcorder) so I will probably not miss the old ports and probably not even need a hub (Firewire 800 hubs are very hard to get).
The inside of the case is a dream to work with. The drive brackets are easy to swap, the PCIe slots have a spring-loaded retention bar that hooks under the back of the card, and the L brackets are held with thumbscrews, making swapping the cards trivial, with no risk of getting a marginal connection from a poorly seated card.
The drive mounting brackets have rubber grommets to dampen vibrations, a nice touch. There also seems to be some sort of contact sensor in the rear, purpose unknown.
There are only two PCIe power connectors, so you can only plug in a single ATI 4870 card even though there are two PCIe x16 slots. The GT 120 does not require PCIe power connectors so you would have to expand capacity with one of these. Considering the GT 120 is barely more expensive than the Mini DisplayPort to Dual-Link DVI adapter cable, it makes more sense to get the extra video card if you have two monitors.
The entire CPU and RAM assembly sits on a daughterboard that can be slid out. This will make upgrading RAM (when the modules stop being back-ordered at Crucial) a breeze.
Built-in Bluetooth means no more flaky USB dongles.
No extras. The G5 included OmniGraffle, OmniOutliner, Quickbooks, Comic Life and a bunch of other apps like Art Director's Toolkit. No such frills on the Mac Pro even though it is significantly more expensive even in its base configuration.
The optical out is now 96kHz 24-bit capable, unlike the G5 that was limited to 44kHz 16-bit Red Book audio. I have some lossless 192kHz studio master recordings from Linn Records, so I will have to get a USB DAC to get full use out of them. I am not sure why Apple cheaped out on the audio circuitry in a professional workstation that is going to be heavily used by musicians.
The G5 was one of the first desktop machines to have a gigabit Ethernet port. Apple didn't seize the opportunity to lead with 10G Ethernet.
The annoying Mini DisplayPort is just as proprietary as ADC, but without the redeeming usability benefits of using a single cable for power, video and USB. DisplayPort makes sense for professional use with high-end monitors like the HP Dreamcolor LP2480zx that can actually use 10-bit DACs for ultra-accurate color workflows. There is no mini to regular DisplayPort adapter, unfortunately. Well, the thinness of the cable is a redeeming feature. Apple has always paid attention to using premium ultra-flexible cables everywhere from the power cord to Firewire.
Transferring over 800GB of data from the old Mac is utterly tedious, even over Firewire 800 using Target Disk mode on the G5...
As could be expected, the Mac Pro wipes the floor with the G5, as measured by Xbench. A more interesting comparison is with the MacBook Air, which also uses a SSD, albeit a slowish one.

Note (2009-03-17):

The BTO upgrade ATI Radeon HD 4870 video card I initially ordered won't recognize my HP LP3065 30" monitor, at least not on the Dual-link DVI port, which essentially renders it useless for me.

Update (2009-03-18):

I went to the Hillsdale Apple Store. The tech was very helpful, but we managed to verify that the ATI Radeon HD 4870 card works fine on an Apple 30" Cinema Display (via Dual-link DVI) and on a 24" Cinema LED display (via mini-DisplayPort). The problem is clearly an incompatibility between the ATI Radeon HD 4870 and the HP LP3065.

I am not planning on switching monitors. The HP is probably the best you can get under $3000, and far superior in gamut, ergonomics (tilt/height adjustments) and connectivity (3 Dual-link DVI ports) to the current long-in-the-tooth Apple equivalent, for 2/3 the price. My only option is to downgrade the video card to a nVidia GeForce GT 120. I ordered one from the speedy and reliable folks at B&H and should get it tomorrow (Apple has it back-ordered for a week).

Update (2009-03-19):

I swapped the ATI 4870 for the nVidia GT120. The new card works with the monitor. Whew!

Update (2009-03-29):

I have just learned disturbing news about racial discrimination at B&H. For the reasons I give on RFF, I can no longer recommend shopping there.

Parallelizing the command-line

Sun, 28 Sep 2008 05:15:11 GMT

Single-thread processor performance has stalled for a few years now. Intel and AMD have tried to compensate by multiplying cores, but the software world has not risen to the challenge, mostly because the problem is a genuinely hard one.

Shell scripts are still usually serial, and increasingly at odds with the multi-core future of computing. Let's take a simple task as an example, converting a large collection of images from TIFF to JPEG format using a tool like ImageMagick. One approach would be to spawn a convert process per input file as follows:

#!/bin/sh
for file in *.tif; do
  convert $file `echo $file|sed -e 's/.tif$/.jpg/g' &
done

This does not work. If you have many TIFF files to convert (what would be the point of parallelizing if that were not the case?), you will fork off too many processes, which will contend for CPU and disk I/O bandwidth, causing massive congestion and degrading performance. What you want is to have only as many concurrent processes as there are cores in your system (possibly adding a few more because a tool like convert is not 100% efficient at using CPU power). This way you can tap into the full power of your system without overloading it.

The GNU xargs utility gives you that power using its -P flag. xargs is a UNIX utility that was designed to work around limits on the maximum size of a command line (usually 256 or 512 bytes). Instead of supplying arguments over the command-line, you supply them as the standard input of xargs, which then breaks them into manageable chunks and passes them to the utility you specify.

The -P flag to GNU xargsspecifies how many concurrent processes can be running. Some other variants of xargs like OS X's non-GNU (presumably BSD) xargs also support -P but not Solaris'. xargs is very easy to script and can provide a significant boost to batch performance. The previous script can be rewritten to use 4 parallel processes:

#!/bin/sh
CPUS=4
ls *.tif|sed -e 's/.tif$//g'|gxargs -P $CPUS -n 1 -I x convert x.tif x.jpg

On my Sun Ultra 40 M2 (2x 1.8GHz AMD Opterons, single-core), I benchmarked this procedure against 920MB of TIFF files. As could be expected, going from 1 to 2 concurrent processes improved throughput dramatically, going from 2 to 3 yielded marginal improvements (convert is pretty good at utilizing CPU to the max). Going from 3 to 4 actually degraded performance, presumably due to the kernel overhead of managing the contention.

Another utility that is parallelizable is GNU make using the -j flag. I parallelize as many of my build procedures as possible, but for many open-source packages, the usual configure step is not parallelized (because configure does not really understand the concept of dependencies). Unfortunately there are too many projects whose makefiles are missing dependencies, causing parallelized makes to fail. In this day and age of Moore's law running out of steam as far as single-task performance is concerned, harnessing parallelism using gxargs -P or gmake -j is no longer a luxury but should be considered a necessity.

Large sensor compact cameras finally on the horizon

Sat, 30 Aug 2008 17:37:21 GMT

I have stated on the record that my dream camera is a digital Contax T3 with an APS-C size sensor (or larger). Sigma launched the DP1, the first large-sensor compact this year, but it is a flawed camera, very sluggish, with a slow f/4 lens, and its Foveon sensor tops out at ISO 800, making it in practice a less capable low-light camera than my Fuji F31fd.

A few weeks ago, Olympus and Panasonic announced the Micro Four Thirds specification, which would allow for interchangeable-lens compact cameras with a larger sensor than the nasty tiny and noisy ones used on most compacts. Unfortunately it seems the whole misguided Four Thirds effort is destined to flounder, just as APS did compared to 35mm, despite the undeniable convenience. The 18x13.5mm sensor size has almost half the area of an APS-C sensor and all Four Thirds camera made so far have predictably poor low-light performance.

In a much more promising development, Samsung announced today that since it is finding it very hard to dislodge Canon and Nikon from their top position in DSLRs or even make a dent, they are going to create an entire new segment of professional quality compact cameras using the same APS-C sensors as their DSLRs, and due for 2010. Samsung uses the Pentax lens mount for its DSLRs, and has a long established relationship with Schneider Kreuznach. Pentax makes some very nice pancake lenses that combine high optical quality with small size. The only other company is Olympus, but the 25mm f/2 is saddled with the aforementioned Four Thirds sensor with all the limitations that entails.

At the same time, Thom Hogan has echoed rumors of an APS-C size Coolpix compact from Nikon. It looks like the big camera manufacturers can no longer afford to ignore the pent-up demand for this category, as demonstrated by the brisk sales of the DP1 (No. 49 on Amazon's Digital SLR chart).

r n m restaurant

Sun, 13 Jul 2008 06:39:19 GMT

I have just eaten what is hands-down my best meal of the year at r n m restaurant (their capitalization, not mine), on Haight & Steiner in the Duboce Park/Lower Haight district of San Francisco (not to be confused with the formerly raffish and now utterly commercialized Haight-Ashbury).

The restaurant is named after the chef-owner's father, Robert Miner, a co-founder of Oracle. The food was so good I am almost ready to forgive Oracle for their sleazy extortion tactics...

I started with the Parisian style tuna tartare with waffle chips, microgreens and a quail egg, a very classic dish (and one too often botched by careless chefs), given a little pep with a slight acidity. It was followed by an absolutely outstanding pan-roasted local halibut on ricotta gnocchi with asparagus and morel mushroom ragout, meyer lemon vinaigrette and mâche. The halibut was crisp outside, flaky inside. The ragoût was simply wonderful, a deep, rich and tangy broth, also slightly acidulated, with a generous helping of precious black morels. To top it off, the dessert, a Peach and cherry crisp with home-made blueberry gelato combined two of my favorite summer fruit in an unbeatable combination.

Be advised the parking situation in that neighborhood is particularly nightmarish, even by SF standards. If I had realized they offer valet parking, I wouldn't have had to park halt a mile away (after seeking a place in vain for nearly half an hour).

A contrarian take on Delicious Library 2

Mon, 02 Jun 2008 05:29:40 GMT

On Friday I yielded to the hype, and after cursory testing, I purchased a copy of Delicious Library 2. The clincher was the new feature that allows you to inventory your physical posessions like electronics or cameras, and publish them in HTML format for insurance purposes.

Unfortunately, after some slightly less cursory use of the product, it is deeply unsuited to this purpose. To think I actually upgraded my home Powermac G5 from Tiger to Leopard just to use this software...

For starters, on my dual 2Ghz G5, when running in a window on my secondary 30" monitor, the program is slow as molasses. With a library with only 3 items total, entering data fields is a one character per second tar pit. Moving the window back to the primary 23" monitor helped only a bit.

Secondly, the data model is simplistic. For all practical purposes, gadgets are treated just like books, with some repurposing of fields. The all-important serial number can't even be displayed in column view.

Third, even basic tasks are not handled properly. I have a Symbol CS1504 pocket scanner, which is one quarter the price or size of the Bluetooth scanner Delicious Monster sells, and has a 500 barcode memory, so you can actually use it away from your computer. Using my Python driver, I scanned some books' bar codes, dumped a text file of ISBNs and imported it into DL2. The import mapper allows you to specify which fields of the tab-separated text file go into which field of the DL2 data model. You would expect it to retrieve book detail automatically, but it does not do so. Worse yet, the "retrieve book details from the Internet" menu is grayed out when you select one of the imported books.

There are also some fit-and-finish issues. Right-clicking to get the context menu and selecting the "View in Amazon" option does not do anything. Perhaps this is due to the fact Camino is set to be my default browser, but the other way to view book details on Amazon works (hovering the mouse over the book cover thumbnail, then clicking on the overlaid eye icon that appears when you hover).

On the plus side, the HTML export works quite well, and the loan manager probably does as well, but given the shortcomings of the current version, I would not advise using it unless all you want to manage is books, CDs and DVDs, and you can afford to buy their expensive Bluetooth scanner to use in wireless semi-tethered mode.

Christopher Elbow chocolates

Wed, 14 May 2008 05:32:27 GMT

A few months ago, a new chocolate shop opened in Hayes Valley. Christopher Elbow chocolates is based in Kansas City, not a place that immediately springs to mind when the Great American Chocolate Renaissance is discussed. I had bought some of their products from Cocoa Bella, however, and knew they were good, if pricey.

They sell moderately expensive chocolate bars (the No. 10 41% milk chocolate with hazelnuts is pretty good), drinking chocolate, and bouchéees. The latter are a little too bleeding edge for my taste (spices do not belong in chocolate), but the Bourbon Pecan is to die for, a light and moist, pecan marzipan, almost creamy despite the deliberately roughly chopped texture, and topped with ganache. Not surprisingly, it is usually sold out at the other outlets..

The real draw, as far as I am concerned, is the hot chocolate. Dark, rich, creamy and thick, specially if you ask them to blend it with genuine praline, it is absolutely delicious. You can enjoy it in the twee little salon in the corner of the store before a concert at the nearby Symphony, or shopping in Hayes valley. If you are in the neighborhood, try also Miette Confiserie.

The value of over-the-counter service

Fri, 25 Apr 2008 00:42:05 GMT

My primary computer is a dual 2GHz PowerMac G5 until I can upgrade it with a Nehalem Mac Pro, most likely around the end of the year or early next year. I bought it in 2004, along with a 23" Apple Cinema HD (the old pinstripe plastic bezel kind with an ADC connector). Unfortunately, about a year ago the CCFL backlight on the monitor started turning pink from old age, and thus unusable in a properly color-managed photographic workflow.

I used that as an excuse to splurge on a humongous (and agoraphobia-inducing) HP LP3065 30 inch LCD monitor after reading the glowing reviews. The two features that sold me were the enhanced color gamut (the only way to improve that would be to get a $6000 Samsung XL30, something I am not quite prepared to do), and the fact it has 3 built-in DVI ports, so it can easily be shared by multiple computers (assuming they support dual-link DVI, which unfortunately my basic spec Sun Ultra 40 M2 does not). The fact it was 25% less expensive than the Apple 30" Cinema Display helped, of course.

About 6 months ago, I discovered there was a fine pink vertical line running across the entire height of the monitor, roughly 25 centimeters from the left. Since I primarily use that monitor for photo (the primary monitor for Mail, web browsing or terminals remains the Apple), at first I worried there was a defect with my camera. I managed to reproduce the problem with my MacBook Pro (they have dual-link DVI, unlike lesser laptops), and called HP support (the 3 year HP warranty was also an important consideration when I purchased).

My first support call in November 2007 went well, and the tech told me I would be contacted to arrange for an on-site exchange. This is a seriously heavy monitor and I did not relish the idea of lugging it back to FedEx, so getting premium support for a business-class monitor sounded an attractive proposition. Unfortunately, they never did call back, and as I had other pressing matters to attend to involving international travel, I just put it out of my mind (it is a very subtle flaw that is not even always visible).

I only got around to calling them back a few weeks ago. Unlike in November, I was given the run-around with various customer service reps in India until I was finally routed to a pleasant (and competent) tech in a suburb of Vancouver (the US dollar going in the direction it is, you have to wonder how much longer before HP outsources those call centers back to the US). The problem is not with Indian call centers, in any case, all but one of the CSRs were very polite (I suspect Indians learn more patience as they grow up than pampered Americans or Europeans would). The problem is poorly organized support processes and asinine scripts they are required to go through if they want to keep their jobs. In any case, the Canadian rep managed to find the FRU number and also told me someone would call to schedule an appointment. Someone did call this time, to let me know the part was back-ordered and they would call me when it becomes available.

This morning, as I was heading for the shower, my intercom buzzed. It was a DHL delivery man with the replacement monitor. I had to open the door to him in my bath robe. Naturally, nobody at HP bothered to notify me and had I left earlier, I would have missed him altogether.

One of the great things about Apple products is that if you live near an Apple store, you can just stop by their pretentiously-named Genius bars and get support for free (though not free repairs for out-of-warranty products, obviously). I now have a fully working HP monitor again, so I suppose I can't complain too loudly, but the Apple monitor with the sterling support looks like the true bargain in hindsight.

Backing up is hard to do (right)

Thu, 10 Apr 2008 23:45:47 GMT

You can never overstate the importance of backups. Over the last year I have put quite a bit of effort in making sure my data is backed up properly. The purpose of this article is not to describe backup best practices (that is a vast subject, there are other, better resources available on the web, and in any case there is no one-size-fits-all solution). I am just documenting my setup, the requirements that drove it, and possibly give readers some ideas.

The first part in planning for backup is to do an inventory of the assets you are trying to protect. In my case, in order of priority:

1.5GB of scans of important documents: birth certificates, diplomas, invoices, legal documents, bank statements, and so on. This data is very sensitive, and should be encrypted.
150GB of digital photos and scans
My address book, which lives on my laptop
My source code repositories
My personal email, approximately .75GB
The contents of this website, about 5GB
190GB of music (lossless rips of my CD collection)
My Temboz article database

Thus the total storage capacity required for a full backup is reaching the 400GB mark. This in itself precludes DVD-R or even tape backup (short of buying an expensive LTO-4 tape drive or an autoloader, that is).

The second step is to devise your threat model. In my case, by decreasing order of likelihood:

Human error
Hard drive failure
Software failure (e.g. filesystem corruption)
Silent data loss or corruption, e.g a defective disk
Theft
Fire, earthquake, natural disaster, etc.

Third, some general principles I believe in:

Do not use proprietary backup formats. The best format is plain files on a filesystem identical in structure to the original.
Do not rely on offline media for backups. The watched pot does not boil over, online data is much less likely to go bad without my noticing until it is too late.
A backup plan needs to be effortless to be successful. Plugging in external drives when backups are needed, or rotating drives between home and office is something I have tried, but not stuck to.
Backups should be verified — they should generate positive feedback, so that the absence of feedback can alert to problems
For all types of data, there should be one and only one reference machine that holds the authoritative copy. Multi-master synchronization and replication is possible using tools like Unison, but is much harder to manage and increases the risk of human error.

With these preliminaries out of the way, here is my system:

My primary backups reside on my home server, a Sun Ultra 40 M2 workstation, running Solaris 10. This machine is very quiet, so I can keep it running in the room next to my bedroom without disturbing my sleep. It is also relatively power-efficient at 160W with seven hard drives.
One of the seven drives is the 160GB boot drive, and the other six are 750GB Seagate drives configured in a 3TB ZFS RAID-Z2 storage pool.
With large SATA drives, reconstruction after a drive failure is long and the risk of another drive failing due to the stress of rebuilding is not negligible. RAID-Z2 can tolerate two drives failing, unlike RAID 5 which can only tolerate a single drive failure. This level of data protection is higher than RAID 1 since RAID 1 won't protect you if two drives that are the mirror of one another fail. You can get the same level of protection in RAID 6 or RAID-DP.
I have scripts to take ZFS snapshots daily, equivalent to the auto-snapshot service. The daily snapshots are kept for the current month, then I keep only monthly snapshots. Snapshots are the primary line of defense against human error.
Snapshot technology consumes only as much disk space as required to store the differences between the snapshot and current versions of a file, and is much more efficient than schemes like Apple's Time Machine where a single byte change to a multi-gigabyte file like a Parallels virtual disk image will cause the entire file to be duplicated, wasting storage. Because snapshots are taken near instantly and cost almost nothing, they are an extremely powerful feature of a storage subsystem.
I backup from my various machines to the Sun via rsync over ssh. An incremental backup of my PowerMac G5, which has most of the 400GB in my backup set, takes less than 5 minutes over Gigabit Ethernet, despite the ssh encryption.
ZFS is probably the best filesystem, bar none, but it is not perfect, as demonstrated by the Joyent outage and you still need another copy for backup in case of ZFS corruption.
Every night at 2AM a cron job on my old home server (2x400GB, ZFS RAID 0), that I now I keep at work, pulls updates from the Sun using rsync over ssh (the company firewall won't let me push updates to it from the Sun). Another cron job at 8AM kills any leftover rsync processes, e.g. if there are more data changes to transfer than fit in the 1-2 GB that can be transferred in 6 hours over my relatively pokey 320-512kbps DSL uplink (no thanks to AT&T's benighted refusal to upgrade its tired infrastructure).
My cron jobs use verbose output which generates an email sent back to me. I could suppress those messages, but then I would lose the ability to detect errors.
A last line of defense is to back up my server at work to a D-Link DNS-323 NAS box using rsync over NFS. This cute little unit holds two Western Digital Green Power 1TB drives in RAID 1, which slide right in, no tools required. It consumes next to no power or desk space. Since it runs Linux and is easy to extend using fun-plug, I could conceivably run the cron and rsync from there. As a bonus, the built-in mt-daapd server streams my entire music collection to iTunes over the LAN so I can listen to any of my CDs at work.
It can take a few days for this data bucket brigade to catch up with a particularly intense photo shoot, but it will eventually and is never too far behind. This provides me with near continuous data protection and disaster recovery.

Push recruiting

Thu, 20 Mar 2008 23:33:30 GMT

As I was debugging why feedparser is mangling the GigaOM feed titles, I found this easter egg on the Wordpress hosted site:

zephyr ~>telnet gigaom.com 80
Trying 72.232.101.40...
Connected to gigaom.com.
Escape character is '^]'.
GET /feed HTTP/1.0
Host: gigaom.com

HTTP/1.0 301 Moved Permanently
Vary: Cookie
X-hacker: If you're reading this, you should visit automattic.com/jobs and
apply to join the fun, mention this header.
Location: http://feeds.feedburner.com/ommalik
Content-type: text/html; charset=utf-8
Content-Length: 0
Date: Thu, 20 Mar 2008 23:36:17 GMT
Server: LiteSpeed
Connection: close

Connection closed by foreign host.

Knowing how to issue HTTP requests by hand is one of my litmus tests for a web developer, but I had never thought of using it in this creative way as a recruiting tool...

Macworld 2008 round-up

Wed, 16 Jan 2008 06:29:41 GMT

The MacBook Air was what I was waiting for (I pre-ordered the SSD version just before the online Apple Store buckled under the load). I have a MacBook Pro 15", and because of its weight I end up leaving it at work and not carry it with me at all times (the MacBook is hardly any lighter). Sure, the Air has drastically limited connectivity (the lack of Gigabit Ethernet is probably what I will regret most, even though I clocked my Airport Extreme at 90 true Mbps throughput). Other minuses include the glossy screen (instead of an anti-reflective one), the MacBook-like chiclet keyboard rather than the much nicer MacBook Pro keyboard), or the sealed non user-replaceable battery.

I suspect people deriding it are people whose main machine is a laptop. My main machine is a tower desktop, and no laptop is ever going to compete in terms of capacity and expandability. The drive on the laptop is merely a cache for the desktop where the real data lives. The compromises the Air makes are acceptable ones in exchange for a machine that is light enough for me to carry all the time. I was considering getting an Asus Eee PC prior to the show, and the MacBook Air is a vastly more capable and polyvalent machine.

Apart from that, the show was a relatively quiet one with few truly noteworthy new products. Here are the main highlights:

Matias did not have the Tactilepro 2.0 keyboard on display. I love mine (a version 1 with the ALPS keyswitch) and would like to get a spare, but apparently they have parted ways with the manufacturer of the new Matias-designed keyswitches and are working on a 3.0 version for later this year.
Fujitsu were demonstrating an ultra-small, bus-powered document scanner, the S300M. Unfortunately, once again for reasons due to licensing of the bundled software, they could not release a single SKU that would work with both PCs and Macs.
The German company Project Wizards was demonstrating Merlin, a project management program similar to Microsoft Project. The scheduling and load-leveling algorithms look at least as capable as Project 2000, and they told me the next version will allow team members to report on task advancement by simply contacting a built-in web server. Looks like a promising product.
Samsung showed the CLP-300 which they bill the world's smallest color laser printer. Indeed it looks roughly the same size as my monochrome HP LaserJet 1320, and much smaller than my bulky HP 2605dn, that's quite an achievement. I am wary of Samsung lasers since buying the CLP-500 for Kefta a few years back. The print quality was fine, but it was ludicrously slow, taking something like 5 minutes per color page to print. The CLP-300 seems reasonably fast, faster than the 2605dn at any rate.
Samsung was also showing off the gorgeous XL30 30" LED-backlit LCD monitor. LED backlight is more environmentally friendly, does not shift colors as it ages unlike a TFT backlight, and gives a wider color gamut. Unfortunately, its price is a princely "between $6000 and $7000".
Microsoft was showing off Office 2008, emphasizing ease of use and productivity rather than features for features' sake for a change.

They even set up a bloggers-only salon to curry favor, complete with Internet cafe and snacks.
I tried Nikon's humongous AF-S VR Nikkor 200mm f/2G IF-ED lens. Very heavy but impressive piece of gear.
Canon was showing off the new Flash-based HD camcorders they introduced at CES. They are not that much smaller than the HDV ones. The HV30 replaces the excellent HV20, but the only real improvements are 1080p30 mode and an articulating LCD.

A San Francisco local's advice to Macworld attendees

Mon, 14 Jan 2008 02:38:56 GMT

I have been living and working in downtown San Francisco for almost eight years now. Until a month ago, my office window (right) used to overlook Third Street and the Moscone center. San Francisco is a popular convention destination (one wonders why proctologists seem to prefer it to, say, Detroit) but Macworld Expo is definitely the biggest show in town. Restaurants and hotels are taken by storm, taxis become scarce, traffic gets even snarlier and the lines at Metron eateries cross the threshold of ludicrousness. So here are a few tips for Macworld attendees to have a better time and not caught in tourist traps.

Transportation

Driving in San Francisco is a non-starter. Traffic is horrendous, parking is scarce and you would lose far too much time just getting around. SF Muni is a pretty good public transport system (at least by admittedly paltry US standards) and their 1, 3 or 7 day Passport passes are good value.

Cars are mostly useless inside the city, but nice if you want to drive to make a Fry's run or a day trip to Marin across the Golden Gate. If you must drive, the friendly folks at Reliable Rent-a-Car will give you decent rates on Toyotas. Until I bought a car last month, they were my go-to place for when I needed a car.

Lunch

San Francisco has the best food in the United States, but you wouldn't know if from the overpriced eateries in a three block radius. The Firewood Cafe and Buckhorn Grill in the Metron are actually reasonably decent, but the throngs of convention-goers mean long lines. Mo's Grille has excellent burgers (I recommend the aptly named "Belly Buster"), and since access to it is a little tortuous, you have a fighting chance (it is literally just above the Moscone South).

Ranging a little further, Nova has decent burgers and a lovely lobster quesadilla, and the new Westfield Mall three blocks to the west has a decent food court. Some good local chains are Bistro Burger, S.F. Soup Co. or Café Madeleine (official birthday cake purveyor to Kefta).

That said, the best lunch experience is to take the historic F line streetcar to the Ferry Building Marketplace with its wide variety of gourmet food stores and eateries. I heartily recommend the clam chowder at Ferry Plaza Seafood (it used to be my Friday lunch of choice) or the eclectic fare at Boulette's Larder. Chocolates from Michael Recchiuti or fresh-pressed olive oil from Stonehouse make for great (and edible) souvenirs.

Staying hydrated is important when you expect to spend an entire day on the show floor. There is a Whole Foods store a mere block away where you can buy any required provisions.

Dining

Dining in San Francisco is an embarrassment of riches, it would be a shame to settle for overpriced hotel food. A word to the wise: most of the better places are hooked into the OpenTable reservation system which makes finding a good place with availability a much less hit-and-miss affair. This year Macworld coincides with the annual Dine About Town event where participating restaurants will offer specially discounted menus.

Equipment

Murphy's law will strike at the worst possible moment. If you need help with your Mac, the geniuses at the San Francisco Apple Store (or the smaller Chestnu St or Stonestown locations) can help. It's also good to keep in mind the Apple stores all offer free WiFi connectivity.

If you need commodity spare parts like a USB hub in a hurry, Central Computers is a mere block away and carries a wide assortment, albeit PC-centric.

If you are an attendee and have questions I have not answered, please feel free to email me, my contact info is at the right.

She said Yes

Sun, 14 Oct 2007 02:00:00 GMT

I flew into London today. I took the adorable Shaheen B. to dinner at The Ledbury.

Between courses, I asked her to marry me. She said Yes. The only plausible explanation for this lapse in judgement is temporary insanity. I couldn't be happier.

P.S. the ring is guaranteed to be De Beers-free and not a blood diamond, courtesy of the nice folks at Brilliant Earth, who by the greatest of coincidences are just across the street from my office.

iPhone first impressions

Mon, 09 Jul 2007 19:47:53 GMT

I thought I would escape the frenzy of iPhone hype by filtering out any mentions of it from my feed reader. In fact, I was quite resentful of the way the iPhone launch pushed out the release of OS X 10.5 Leopard to October 2007. On my way to my cousin's wedding on Friday June 29th, I passed by the San Francisco Apple Store and saw the line. It was actually fairly tame, as it only went halfway around the block (when the store originally opened, the line went all the way around and spilled over into Market Street).

Of course, when I came back, I had to see one. One of the petty annoyances with my Nokia E62 was how it would take several seconds for the address book to load. The iPhone, despite having a much more computationally intensive user interface, still manages to have lightning-quick responsiveness to user input. That itself convinced me to buy one.

The iPhone mostly meets or even exceeds the hype. The user interface is exceptionally good, let alone for a version 1.0 product. Some quick notes from a Nokia E62 switcher (my previous phone was also using Cingular/AT&T):

Email and web are very snappy. The SSL implementation on the E62 would take forever to negotiate with my home IMAP server (as in several minutes), the iPhone's is instant. The E62's web browser, despite being based on the same WebKit code base as Apple's Safari, could not run two concurrent AJAX XMLHttpRequest concurrently, Safari has no such problems.
The battery life is very short, well under 2 days, and it takes a long time to fully charge.
The glass screen does not scratch, but it does show fingerprints and smudges.
The virtual keyboard is surprisingly effective. This was the single biggest area where I thought it would fall short, but it actually performs far better than the E62's chiclet keys. Part of the reason is that the E62's keys actually wobble when you press them, which doesn't make for precise typing, and they are so tiny anyway that it's hard to type accurately without pressing other keys in the process. The iPhone's magnification effect as well as the fact you can slide your finger to correct a misregistered virtual keypress, makes for much faster typing. The predictive text engine is also far superior to schemes like Symbian's, or T9. T9 is unbearably annoying in the same vein as Microsoft Word's noxious autocorrect functionality or Clippy, I always disable T9 on any phone that has it, the iPhone's system is unobtrusive and eminently usable in comparison.
The sound quality on the iPhone is not at the same level as the E62, specially for the speakerphone.
No voice recorder. A rather silly omission.
The calendar does not support To-Do list items from iCal. This is ridiculous.
You cannot use iTunes music files as either the ring tone or alarm sound. This was probably to appease AT&T and the RIAA, who seem to believe they have a divine right to make you pay over again and again for the same music. Even if I were prepared to accept their racketeering and pay the obscenely expensive charge for a ring tone, I seriously doubt they would have what I used for mine on the E62, the finale theme from Sibelius' Kullervo op. 9.
The recessed phone jack is incompatible with most earphones like my ER-4P, but it works just fine with B&O A8, whose jack is actually a fairly thin molded connector wrapped in a rubber jacket that easily slips off to accommodate the iPhone jack.
Safari has no option to remember passwords for you, unlike the desktop version, and it does not recognize the standard http://login:password@site/ convention either, which makes logging onto Temboz harder than it has to be.
The Bluetooth functionality in the iPhone is pretty minimal, limited to using Bluetooth wireless headsets and not much more. You cannot beam business cards or photos. Unlike the E62, I cannot use it as a modem for either my MacBook Pro or my Nokia N800. Since there is no SSH client on the iPhone, this could bite me when I need emergency access. Then again, the $20 unlimited data plan for iPhones is half the price of my previous $39.99 unlimited data plan.
Not supporting Java or Flash is a feature, not a bug.
The camera, as could be expected, is mediocre. We all know the only purpose is to snap facsimiles of notes, billboards, flyers. whiteboards and the like.
The calculator is minimal. It does not support RPN and does not have either scientific or financial capabilities.
You can specify 24-hour time format, but there is no way to specify ISO date format.
The iPhone seems incompatible with my SendStation PocketDock Line Out USB, and thus I cannot connect it to my Ray Samuels Hornet pocket headphone amplifier and full-size Sennheiser and AKG headphones. It is also incompatible with Apple's own universal AV dock, and displays a warning message telling you so. Then again, since it is a GSM phone, the annoying pulsating buzz induced by GSM would make such an arrangement impractical.

Update (2007-07-13):

It must be the Friday 13th effect at work... My iPhone seems to have developed a defective proximity sensor. The phone works as a speakerphone, but no longer turns the headset speaker on when I bring it to my ear. Resetting and even restoring the phone does not help, it's probably a hardware issue.

Fortunately, the SF Apple Store Genius Bar let me in this evening without an appointment, and swapped it for a new one. This was the first time they had seen this particular problem, and they told me Apple's policy for the first month is to do full replacements and collect field failures for analysis. The repair process afterwards seems to be still up in the air. I would recommend they have swap or loaner units on hand, as people are less likely to tolerate not having a phone than not being able to listen to music for a week.

MacBook Pro 3G first impressions

Tue, 12 Jun 2007 02:35:19 GMT

I upgraded my MacBook Pro to the third-generation model so I can bump up my RAM to 4GB. Aperture and CS3 are very resource-intensive, and the 2GB upper limit of my first-generation MBP was somewhat constraining.

I just transferred my files over using Apple's migration utility and target firewire mode. The process, while not 100% automated (it did not transfer X11, for instance, or some of the preferences), is far smoother than any Windows equivalent. Here are my first impressions on the new model:

The new, environmentally friendly mercury-free LED backlight is definitely more blue in tone than the pinkish cold-cathode fluorescent backlight on the old model. The default ColorSync monitor profile does a good job of compensating for this, however. There is some vignetting on the 15" screen (darkening in the corners). I wonder how the 17" model fares, and whether they had to add additional LEDs for a more even backlight illumination.
This machine is fast. It blows my dual-2GHz G5 PowerMac out of the water in all benchmarks other than disk I/O. Unsurprisingly, it is also much faster than the first-generation machine, specially on graphics but also on disk I/O.
It does not heat up quite as much as the older Core Duo model, the heat level, while high, never reaches a potentially dangerous temperature. This is probably due to improved power management, as running two Parallels virtual machine will bring it up to the same pant-scorching levels as the Core Duo.

Adobe "Creative" Suite 3, a mixed bag

Fri, 11 May 2007 04:53:02 GMT

I installed Adobe Creative Suite 3 on my home PowerMac and my MacBook (the license allows you to install it on two computers as long as they are not in simultaneous use). The only real reason I upgraded is to get a native Intel version. I have barely started using it already and I already have peeves:

Bridge looks butt-ugly, is even slower than before and with a more amateurish interface than ever
The install procedure is incredibly annoying and Windows-like. There is no justification for an install procedure that chokes if the beta was not uninstalled officially (although I have to give some brownie points due to the fact the cleanup script is written in Python).
The icons are aesthetically bankrupt. What kind of credibility does Adobe think it has with creative people with such an astoundingly lackluster effort?
Barely installed and already in need of software updates. The widespread availability of fast Internet connections is no excuse for shoddy release management or a "we'll patch it post-release" mentality. Speaking of which, the only proper time to interrupt users with a software update dialog is as they are quitting the application, not by getting in the way of whatever task they are trying to get done by starting up the app.
Don't clutter my hard drive with legal drivel in twenty different languages. It's called "Creative Suite", not "Boilerplate Suite".
All the tie-ins to paid add-on services like Adobe Stock Photos or Acrobat Conferencing are incredibly obnoxious, just like those for MSN or .Mac.
JavaScript in Acrobat is a big security and privacy risk, and should be disabled by default.
On the plus side, thanks for making a "Design Basic" edition without all the despicable Flash garbage in it. I would actually pay more for the Basic version than for the supposedly premium one infected with Flash and Dreamweaver.

Update (2008-01-01):

It seems Adobe has also crossed a serious ethical line by building in spyware to track on whenever a user starts a CS3 application.

As far as I am concerned, this is the last straw and I will actively start looking for substitutes for Adobe products as soon as I return from my vacation.

Update (2008-01-02):

It seems Adobe does not collect the serial number after all. The apps should nonetheless never call on the Internet except possibly to check for updates. For people like myself who have static IPs, the IP address itself could be used to correlate the analytics with personal information.

Slava Rostropovich, 1927-2007

Fri, 27 Apr 2007 10:53:43 GMT

Legendary cellist and all-around good guy Mstislav "Slava" Rostropovich passed away in Moscow today. He was a friend and supporter of Shostakovich, Prokofiev, Britten and many others like Dutilleux, and many of the greatest works for cello of the 20th century, indeed of all time, were dedicated to him.

I had the opportunity to hear him conduct Shostakovich's "Babi Yar" a year ago (when I took this photo) and a few years earlier as a cellist the Dvořák Cello Concerto (sadly in replacement of the far superior Shostakovich First Cello Concerto that was dedicated to him).

The world of music has suffered a grievous loss. None of the current generation of cellists (Ma, Gastinel) is of the same caliber. As a conductor, his legacy is more mixed, as his Shostakovich interpretations often lack fire, but his Prokofiev cycle with Erato is marvelous, specially the Fourth in its original version.

Eery similarities

Sat, 07 Apr 2007 20:31:18 GMT

Is it just me or is the Finale of Sibelius' Second Symphony, Op. 43 (1902) eerily reminiscent of the first movement of Saint-Saëns' Third Piano Concerto, Op. 29 (1869)?