Over the years I've seen those stories about travelers who make the mistake of using mobile data services outside of their home countries and end up with ridiculously big phone bills. But I didn't realize how easily it could happen until my colleague, Nilofer Merchant (link), got her phone bill last week.
Nilofer recently made a business trip through Canada. While waiting in the Toronto airport for her flight home, she fired up her laptop. The WiFi service in the airport costs about $10 a day, so she decided to use her AT&T data card instead.
Big mistake. Big, big, big mistake.
When her phone bill arrived last week, the total charge for that airport session was $10,609.27.
Needless to say, this was a topic of pretty intense discussion at the office last week.
The stories I'd heard about people with big roaming bills were usually about someone on vacation who used data for a couple of weeks. I was very surprised to find that you could run up such a huge bill in just a few hours. Here's how it happens:
AT&T's Data Connect plan gives five gigabytes of data transfer a month, with additional data priced at five cents per megabyte. But when you're roaming in Canada, there is no prepaid data allowance, and the charge is $15 per megabyte -- 300 times higher than the charge at home.
Nilofer supposedly transferred 707 megabytes in that airport session, which adds up to over $10,000.
I'm not sure how she could have generated that much traffic in a few hours. She said she was doing normal business tasks, not watching videos. It's barely theoretically possible for someone to use that much data in the time available. PC World reports that AT&T's network can transfer about 1,400 kilobits per second (link), which means 707 megabytes could theoretically be transferred in a little bit over an hour. But that assumes a single continuous connection, running at full speed for the entire time. You're not likely to get that in a real-world browsing session, which is full of starts and stops.
You'd think AT&T would warn a customer when they're building up this sort of charge, but that's not the case. The only notification was two form e-mails AT&T sent after the data session was already over. The first said her data service had been shut off due to excessive charges, and the second --dated one minute after the first one -- warned of high usage. That message claimed that "AT&T has sent your end user multiple text messages regarding their high level of international data usage." No text messages were ever received, and in fact I don't know how you would send text messages to a data card.
But the biggest question is not how much data was transferred, or why AT&T doesn't notify customers properly; it's why the roaming charges are so high in the first place. There are a lot of excuses given for that by the operators, but what it comes down to is a cooperative effort between the operators to fleece each-others' customers when they roam. When AT&T customers roam to Canada, they pay 300 times the home rate for data. Meanwhile, when Canadians on the Rogers network roam to the US, they pay 200 times the home rate for data -- unless they have Rogers new One Rate plan, which eliminates roaming charges in the US (link).
The interesting thing about the new Rogers plan, which was introduced last month, is that it proves it's possible to create reasonable roaming charges throughout North America. The operators just choose not to. Because they make a lot of money from it.
The data roaming charges, and their impact on your bill, are not completely hidden by the operators, but they come pretty close. The information is scattered in several locations, and little or no effort is made to explain what the charges mean in practical terms. To find the charge at Rogers, you have to look on their website here, click on the "Legal Disclaimer" link in the tiny type at the bottom of the page, and scroll down to footnote 4. Unless you're technical enough to understand the difference between MB and kb, you may not even realize that roaming costs extra.
AT&T's site is almost as obscure. The company's page with tips on international roaming is here. It discloses the charges for roaming, but doesn't explain how those charges compare to home-country charges. As is the case with Rogers, you're expected to spot the KB vs. MB distinction, and know what it means.
AT&T also provides a helpful map showing its coverage in the US and Canada. Nowhere does it warn of the roaming charges for data in Canada.

The closest thing I could find to a warning about charges was another window with "laptop travel tips" that contains this message: "Your LaptopConnect service provides access to email, Web browsing, and VPN applications that can use a significant amount of data, so remember -international data roaming can get expensive quickly."
A more honest notification might be printed in big red letters, and would say something like this:

"Using wireless data outside your home country is about as smart as juggling chainsaws. In a single day, you can build up charges large enough to buy us a new car. We're constantly amazed that people keep falling for the roaming thing, but you know what PT Barnum supposedly said about suckers. If after reading this you're still stupid enough to use roaming data, please stop by our headquarters the next time you're in Dallas and we'll buy you a drink (although knowing you, we can probably stick you with the tab for that as well.)"

There's a continuous buzz online from people who have been caught by the roaming trap.
Adam Savage, co-host of the television show Mythbusters, had a similar incident last year (link). After he raised a stink on Twitter, the charges were dropped.
You can read some more examples here and here and here and here and here. It makes you wonder why some politician hasn't taken up this issue. A nice round of Congressional hearings would be fun (I'm looking at you, Nancy Pelosi).
In response to complaints in Europe, the EU recently regulated roaming fees and capped roaming data charges at one Euro per megabyte, about a tenth of AT&T's charge (link). As a rugged, individualistic American, I'm generally skeptical of the EU's reliance on the dead hand of regulation. But in this case, I congratulate my friends in Europe, and I say bring on the bureaucrats.

Fuji is now making cameras that can locate dog and cat faces within the frame for improved autofocus.  The company has prepared a helpful guide to choosing a dog or cat breed.

A few weeks ago, my daughter Marina, who's 12, asked me to start mentioning her by name on this website, and when I link to her blog, photos of her on Flickr, the new blog she just set up with her sister, and so on.

Until now, I've been pretty careful about just calling her "M" or "Miss M," because while I'm personally comfortable putting my own name and information on the Web, that's not a decision I should have been making for my kids, especially before they were able to understand what its implications are. (For similar reasons, here on the blog I generally refer to my wife by her nickname Air, at her request.)

But Marina has started to find that annoying, because when she searches for "Marina Miller," she nearly always finds other people instead. She's starting to build herself an online profile—and the first component of that is establishing her online existence.

I was online around that age too, but at the turn of the 1980s it was a very different thing. In fact, no one expected to be themselves: we all used pseudonyms, like CB radio handles. And it was a much smaller, geekier community—or rather, communities. I had no Internet access until the decade was over, so connections were local, and each bulletin board system (BBS) was its own island, accessed by dialup modem, often by one person at a time. The Web hadn't been invented, and the concept of a search engine or a perpetual index of my online life was incomprehensible.

On a recent episode of CBC Radio's "Spark," Danah Boyd, who researches these things, noted that today's adults often look at our online exposure in terms of what can go wrong, while our younger compatriots and children look at it in terms of its benefits, or what can go right. It's not that they don't care about privacy, but that they understand it differently.

Marina is now closer to adulthood than toddlerhood, and her younger sister, at 10, is not far behind. I think that's a bit hard for any parent to accept, but in the next few years both our daughters have to (and will want to) learn to negotiate the world, online and offline, on their own terms. Overprotective helicopter parenting is a temptation—or today, even an expectation—but it's counterproductive. Just like we all need to learn to walk to school by ourselves, we all need to learn how to live our lives and assess risks eventually. I'd rather not wait until my kids are 18 or 19 and only then let them sink or swim on their own.

I think I share the more optimistic view about being myself on the Web because, unlike many people over 40 today, I have been online since even before my teens, and I've seen both the benefits and the risks of being public there. I hope my experience can help Marina and her sister L (who hasn't yet asked me to go beyond her initial) negotiate that landscape in the next few years.

That is, if they continue to want my help!

I talked Friday with Dwight Merriman, founder of 10gen (the MongoDB company). He more or less convinced me of his definition of NoSQL systems, which in my adaptation goes:

NoSQL = HVSP (High Volume Simple Processing) without joins or explicit transactions

Within that realm, Dwight offered a two-part taxonomy of NoSQL systems, according to their data model and replication/sharding strategy. I’d be happier, however, with at least three parts to the taxonomy:

• How data looks logically on a single node
• How data is stored physically on a single node
• How data is distributed, replicated, and reconciled across multiple nodes, and whether applications have to be aware of how the data is partitioned among nodes/shards.

After talking with Dwight, and also with Cassandra project chair Jonathan Ellis, I feel I’m doing decently in understanding the first of those three areas. But there’s a long way yet to go on the other two.

In Dwight’s opinion, as I understand it, NoSQL data models come in four general kinds.

• Key-value stores, more or less pure. I.e., they store keys+BLOBs (Binary Large OBjects), except that the “Large” part of “BLOB” may not come into play.
• Table-oriented, more or less. The major examples here are Google’s BigTable, and Cassandra.
• Document-oriented, where a “document” is more like XML than free text. MongoDB and CouchDB are the big examples here.
• Graph-oriented. To date, this is the smallest area of the four. I’m reserving judgment as to whether I agree it’s properly included in HVSP and NoSQL.

As Dwight sees it, JSON (JavaScript Object Notation) is the emerging markup standard for the document-oriented data models, and to some extent the BLOB part of key-value models as well. Reasons seem to include:

• JSON is something web developers are likely to know anyway.
• JSON, unlike XML, is schema-less. In the NoSQL world, that’s perceived as a good thing.
• Perhaps for both these reasons, JSON is perceived as easier to use than XML.

Except as noted, I’m not aware of anything that solidly contradicts the above.

Dwight went on to say that there are two main NoSQL replication/sharding models, in line with the seminal papers to which I previously linked:

• Based on or resembling Dynamo. The core idea here is accepting eventual consistency among nodes as being good enough, even if that means you sometimes read dirty data. The benefit is that you never are blocked from writing. By way of contrast, systems that enforce true inter-node consistency (think of a two-phase commit) can shut you down from writing if consistency guarantees aren’t being confirmed in a timely manner. Thus, in a Dynamo-like scheme you write data to multiple nodes, via consistent hashing; then when the time comes you read one or more nodes, and hope that what you’re getting back is a correct result.
• Based on or resembling BigTable. In this model you’re trying to keep the nodes fully consistent in the usual way, e.g. by synchronous replication. Indeed, what’s being kept consistent is both data itself, and metadata about the data’s location. Details surely vary a lot from implementation to implementation.

I’m fuzzier on this stuff than on the data models, because to date nobody has ever explained to me how an actual live system (MongoDB, Cassandra, whatever) implements its replication strategy. Also, while I think that in both these models applications are allowed to be ignorant of the replication/sharding strategy, I’m not as sure of that as I’d like to be.

If we stop here, we already have something useful. MongoDB has a document data model, and is in the BigTable-like replication camp, at least at first. Cassandra has a table-like data model, and is on the Dynamo-like eventual consistency side. But to say those are the only differences that matter would be like saying that all shared-disk RDBMS (e.g., Oracle and Sybase IQ) are essentially alike. That, of course, would be nonsense.

So a third dimension needed in this taxonomy is how the systems actually bang data on and off of disk (or silicon, as the case may be). I don’t yet have an overview of that. I know something of how Cassandra does it, and will write about same in a future post, but that’s about it. So please stay tuned.

Sure, throwing a steak on a fire-hot grill feels pretty manly, but don't pretend it's easy to cook it just right. Cooking steak in a sous-vide water bath is all the rage now, and will get you perfectly cooked beef every time. More »

For the last six years, marketing teams and Web services have pointed to GMail's invite-only approach as on of the most successful examples of driving user demand in a time of scarcity. At one point, GMail invites were so sought after that account-holders were selling off their spare invites on eBay. In fact, that's where I bought mine, as well as a never again used account ID for Orkut, way back in 2004. Today, at SXSW, we learned the move to make GMail an invites-only platform was not a marketing strategy, but instead one driven by fear from engineering, who thought they might not be able to scale under tremendous potential demand.
As is well-known, Gmail's debut on April 1, 2004 was not a massive April Fool's joke that promised a gigabyte of Web-based e-mail at a time when competitors like Yahoo! offered a comparatively measly 50 megabytes. Combined with a new Web-based interface that echoed the quality of desktop apps, and integrated anti-spam, the team thought potential customer demand for the new platform could outstrip available resources - and the engineers pushed for the invite only system.
"People were selling their souls on the Internet for invites," said Arielle Reinsten, Product Marketing Manager for GMail, at SXSW Interactive today. "But it wasn't a marketing idea at all (to offer invites). We were worried about capacity. It was an engineering decision that was seen as marketing."
With GMail now having passed the initial time of scarcity, open to all, the company now focuses on how they can leverage word of mouth from happy customers, and push for activities like viral videos and fan-building activities. Reinstein did say GMail was advertised widely in 2007, in part of a wide push to highlight it's built-in spam protection, but "it was a drop in the bucket compared to the organic growth and the viral growth GMail is known for," she said.

The old standby for recording historical system activity is sar - system activity reporter. There are many alternatives, both free and commercial, but sar has the advantage that it comes with the OS, and pretty much any version of any (unix-like) OS.
Because it's there, we use sar, saving it's output into a big archive and using tools like sar2rrd to produce charts. (It's not the only thing we use, of course.)
The problem is that, particularly on Solaris, sar is terrible. The data it collects is woefully incomplete - network data is the worst, being completely absent, but there's much more missing. Some of what is present is aggregated away so that much of the details is lost. And the list of what's present is fixed, so the whole framework is completely non-extensible.
So, I'm fed up with that, and need to do better. Note that most tools out there don't help with capturing all the data, as they have their own preconceived notions of what data might be useful (although they are generally far more complete than sar).
Enter kar - the kstat activity reporter. This is really amazingly simple. Given that (almost) all the performance data you want is obtained from kstats, simply save all the kstats on a regular basis. The implementation I have here is to save kstat -p output into files inside zip archives. Now, that's not ideal, but it has some advantages: it's almost zero effort, it gives complete coverage, and it's naturally extensible. If it works out and is found to be useful, more optimal mechanisms could be defined.
I've said it a couple of times above, but I'm going to say it again: the key advantage here is that the data is complete and thereby naturally extensible. I don't want to enhance sar by trying to cherry-pick interesting statistics (and we could all argue for months about what might go on the list). By saving everything you automatically pick up anything new that's added. And you let consumers decide which of the statistics are interesting when you get to the post-processing phase. Say I wanted to look at the historical behaviour of the zfs ARC - no problem, it's all there in the kstats.
Using kstat -p is a convenient shortcut, but does have other advantages. Because the output is textual, all your favourite analysis tools - awk, sed, perl, grep, python, whatever - can munge the data with no effort. And you can chuck the data into your graphing application of choice.
If that wasn't enough, jkstat 0.35 has support for reading in the output of kar in both the browser and chart builder.

./jkstat browser -z /var/adm/ka/ka-2010-03-01.zip

or

./jkstat chartbuilder -z /var/adm/ka/ka-2010-03-01.zip

will do the trick.

Piled Higher & Deeper by Jorge Cham		www.phdcomics.com
title: "The Rising Cost of Public Higher Education" - originally published 3/12/2010 For the latest news in PHD Comics, CLICK HERE!

by Jean-Louis Gassée

This week, no iPad disquisition, no large companies engaged in contorted Kama Sutra embraces, no Google-Apple-Microsoft love triangle. We’ll revisit these topics in due course but, for the time being, let’s go back to a geeky topic unadulterated by geopolitics or markitecture: software and brakes. Last month, we looked at the software invasion of automotive braking systems. More specifically, we looked at the interplay between braking and kinetic energy recovery in the Toyota Prius.

Today, we’re going back to “soft brakes” for another set of applications: differentials and stability control.

Differentials. Fifty years ago, they wore out, they made noises, they had to get careful periodic checks and special lubrication. Now, with progress in metal allows, high-precision machining and modern lubricants, they’re rarely seen or heard of. Yet, they perform and important role and their basic design suffers from one no less critical flaw.

(Here, we’ll assume a rear-wheel drive car. The concept applies to all drive configurations.)

The important role goes like this: when the car turns, the two wheels on the same axle draw circles of different radii, smaller radius for the inner wheel, larger for the outer. As a result, the outer wheel must turn faster that the inner one. This is no problem for the front axle whose wheels are “free”, not driven. But, for the rear axle, we’re in trouble: the drive shaft attached to the gear box connects to the axle through a 90 degrees angle gear. This causes each wheel to be driven at the same speed. This is fine in a straight line but causes wheel slippage when we turn as the wheels must rotate at different speeds.

This was the arrangement when the very early automobiles mimicked horse carts. On carts, wheels on the same solid axle did slip in a turn, but said wheels didn’t have to provide any traction, the horse did. In a car, the axle provides traction and wheel slippage works against stability and comfort, to say nothing of tyre wear.

So, the differential was invented. It’s a little counterintuitive at first but it works beautifully.

See this touchingly kitsch Chevrolet video. Or this learned Wikipedia article. Relatively simple, once you get the hang of the planet gear’s role. And universal.

But trouble starts right away.

The differential provides “on-demand” difference in rotating speed. Great, but what happens if one wheel is in the mud, or snow and the other on solid ground? The differential is tricked in “believing” the wheel on slippery terrain needs more speed, all the speed. We end up with the wheel on solid ground not rotating at all and the wheel in the mud spinning madly. Just the opposite of the required solution: stop the madly spinning wheel and direct all the power to the “good” wheel, the one on stable terrain.

As you can imagine, the problem was discovered right away.
One crude but effective solution: temporarily locking the differential, that is preventing the “differentiating” planet gear (see the illustration in the Wikipedia article) from rotating. Many real 4 by 4 SUVs offer locking differentials. The crudest arrangements force you to get out of the vehicle to lock a wheel hub, but it gets you out of the hole.
More refined, automatic or semi-automatic arrangements are called limited-slip differentials. A mechanical device, a kind of centrifugal speed governor “notices” the excessive difference of speed between the two wheels. By construction, we know the minimum turn radius for the car, this gives us the maximum difference in wheel rotation. Anything more, we must intervene. The Limited Slip Differential was born. Helpful but complicated, fragile, expensive and, a more recent concern, adding a bit of weight to the car.
Enter software modulated brakes.

As discussed in the February 7th Monday Note, modern cars feature individual wheel speed sensors and fast electrohydraulic actuators reducing or applying brake pressure.

The differential’s birth defect is now a “mere matter of software”. The central computer detects wheel spin and, instead of letting the differential be fooled, the software applies brake pressure to the spinning wheel. The “good” wheel keeps receiving torque and we get out of the hole. As you can imagine, programmers delight in applying delicate variations to the braking action, thus providing a barely visible intervention.
Software being “free” we have a neat, inexpensive solution, not as good as straight mechanical locking in extreme off-road situations but good enough for many everyday situations.

Stability control. By this we mean helping drivers stay on the road. Oversteer is when the “tail” comes out, when the rear-end of the car gets outside the intended trajectory. This can end up in a spin, in the ditch or into the inner guard rail. It depends more on the car than on the driver whose instinctive reaction can make things worse. (“Lifting”, that is reducing throttle, can result in weight transfer from rear to front that exaggerates the spin.)

The same applies to understeer, when the car “pushes”, that is when it turn less than what the driver wants. Here, instinctive driver reactions are a little less problematic as they usually increase traction for the drive wheels; this is the origin of the “safer but duller” reputation of front-wheel drive cars.
Enter more sensors.

Modern cars have accelerometers sensing yaw (horizontal let-right movement) and steering wheel angle sensors. Using these, the central computer can compare intention, where the driver points the car, and result, what actually happens to the cars direction, oversteer or understeer.

For example, if the nose doesn’t want to come in, understeer, we apply braking to the rear wheel “inside” the intended turn. To complicate things, the computer also plays (gingerly, think weight transfer trouble) with the electronic throttle.

Over time, “resolution” has increased everywhere: sensors capture more delicate nuances, actuators offer finer control steps, software models of car motion become more detailed and Moore’s Law makes electronics less expensive. That’s why all modern cars feature some permutation of the devices we just discussed. This annoys some drivers who think it deprives them of fun, of the opportunity to sharpen and demonstrate their driving skills. Manufacturers react by providing ways to turn these “electronic nannies” off. Or, so they say, because attorneys are watching. In fact, the Off switch ought to be labelled Partially Off, even in German sports cars…

There is more, much more, as in amazing feats of hardware and software used to improve engine efficiency and cleanliness. Fodder for a future Monday Note.

—JLG@mondaynote.com

Related columns:

1. Soft Brakes on the Prius Once upon a time, I took my Wehrmacht staff car to the Palo Alto service shop. As I mentioned a...
2. Cars, computers and politicians Last time we had a big, big problem with cars, computers came to the rescue. This was after the second...
3. Software: how do you compete with free? That’s the question Steve Ballmer, Microsoft’s CEO, is trying to answer every morning when he goes to work. On the...

1. Potato Chips date back to 1853. An unhappy customer at a restaurant in Saratoga Springs, NY kept returning his fried potatoes to the chef, requesting they be prepared much thinner. The chef sliced them so thin that they could not be eaten with a fork, and thus was born the potato chip.

2. It wasn’t until the early 20th century that potato chips expanded from restaurant food and started selling in bags.

3. Flavored chips were born in the 1950’s by a small independent manufacturer who then sold the concept to the larger corporations.

4. As junk food goes, potato chips contain only 3 ingredients (potatoes, oil, salt) and are considered by some nutritionists the lesser of savory evils.

5. Potato chips were sold for over $15B (!!!) last year worldwide. They tally up about one third of all savory snacks.

6. While we call them potato chips, our overseas friends use the word crisps.

7. A single serving of potato chips, 1 ounce, contains only 150 calories, 10 grams of fat and 180mg of salt. That’s less than 10% of the daily calories of most people, about 15% of the fat, and 8% of the maximum sodium intake.

8. Problem is that most people don’t stop at one serving. Honestly, are 11 chips enough for you? People can wolf down 5 times that amount in sitting, dip not included…

9. One of the most popular flavored potato chip varieties is Sour Cream and Onion. Compared to just 3 ingredients in the original version, this one has 20 ingredients including MSG, Palm oil, and artificial colorings.

10. Just in case you were wondering, this national holiday does not appear in congressional records. It’s just another made up holiday by snack food industry marketing geniuses.

What to do at the supermarket:

Potato chips don’t have to be a daily ritual with your lunch sandwich. However, if you do buy them, choose potato chips that have been minimally processed -  Stay with the plain, unflavored chips as they have no additives and colorings. Buy single serve 1 oz. bags of chips so that you won’t be tempted to eat more than one serving at a time.

New! Choose a better breakfast with CerealScan™ by Fooducate

This is not your average journal. Made for artists the Fabriano Artist’s Journal does well with a variety of media. But does it play well with fountain pens and inks? The paper has some tooth but it still works well with most nibs and every ink loved it. There was absolutely no bleed through or show through. [...]

As part of the 2010 Atlas Obscura Day, Join Stanford University Hospital chief engineer Leander Robinson on a tour of one of the largest pneumatic tube networks in the world. Snaking through the medical center’s walls are four miles of tubes that shuttle specimens and paperwork around the facility at 18 miles per hour. Robinson will explain how this incredible system works.

Meet at the fountain in front of the Stanford Medical Center at 300 Pasteur Dr. at 10am.

• RSVP here: http://obscuraday-stanford.eventbrite.com/
• Cost: Free, RSVP is required

Obscura Day is an international celebration of unusual places taking place around the world on March 20, 2010. It is sponsored by Atlas Obscura, a guide to the world’s wonders, curiosities, and esoterica.

Join us at the Computer History Museum in Mountain View for a demonstration of their amazing Victorian Computer, the Difference Engine #2. Regarded by many as the first example of a mechanical computer, the Difference Engine was only one of many inventions by the mathematician Charles Babbage. Due to his difficult personality and the enormous technical challenges involved in building his designs, the machines were never completed in his lifetime.

Meet inside the museum at the Difference Engine display. The demonstration will begin at 2pm, and lasts for about 1/2 hour.

• RSVP here: http://obscuraday-babbage.eventbrite.com/
• Cost: Free

Obscura Day is an international celebration of unusual places taking place around the world on March 20, 2010. It is sponsored by Atlas Obscura, a guide to the world’s wonders, curiosities, and esoterica.

Microwaves have gotten a bad rap with some people because they just don't cook some things as they should be cooked—food blog Savory Sweet Life attempts to squash this misconception with a lesson on cooking potato chips in the microwave. More »

Sources say Dan Dobberpuhl, PA Semi's founder and pre-acquisition chief executive, has jumped ship to work at chip-related start-up Agnilux.

There's no shortage of cutesy portable speakers for listening to music when you're out and about. But if you want something that doesn't feel like it's going to snap in two, the Audio-Technica BoogieBox will do you a solid. More »

I’m loving WebPagetest.org. In Even Faster Web Sites I said, “[WebPagetest] hasn’t gotten the wide adoption it deserves.” It got a boost after Matt Cutts mentioned WebPageTest.org in his interview with WebProNews. But I still meet people who aren’t aware that this great performance tool is out there, so let me bang their drum some more.

Pat Meenan and Eric Goldsmith are the team behind AOL Pagetest and WebPagetest.org. AOL Pagetest is the Windows tool that works in IE. Pat took that and put it behind a web server running in his basement and called it WebPagetest. That was two years ago. He took the Open Source route and now there are instances of WebPagetest running in Virginia, California, UK, China, and New Zealand hosted by AOL as well as Strangeloop Networks, Aptimize, and Daemon Solutions.

The power of WebPagetest.org is that it’s web-based – you don’t have to do any installs and you can run it on any OS and browser. On the backend, WebPagetest runs the page in either IE7 or IE8 and displays the results. This might be a limitation for some folks – if you want to test a page on Mac OS X using Safari you’ll have to do that with some other tool. But the fact that IE 7&8 are the dominate browsers means you can see the most typical experience regardless of what platform you’re currently working on. Since many developers work on Mac using Safari or Firefox, and more are moving to Chrome, it’s important that they can easily see how their web pages load for a majority of their users.

New Test

Here’s how it works: Go to the New Test tab. Enter the URL you want to test and click submit. Pretty easy! 90% of the time that’s what I do, but there are other options you can tweak:

• pick a geo location – VA, CA, UK, CN, or NZ
• choose IE7 or IE8
• choose a connection speed – Dial, DSL, FIOS (Dial and DSL are done via throttling)
• test just the first (empty cache) page load or first and repeat
• repeat the test up to 10 times to get a bigger sample size
• opt to keep your test results private if desired

Results

The results page shows summary stats (page load times, bytes downloaded, # of HTTP requests) and a mini performance analysis (compression, image optimization, concatenating scripts and stylesheets). But the piece I love is the waterfall chart.

If you click on the mini waterfall, it takes you to a larger view where you can do additional customizations. I’ve been relying on this for my current series of blog posts on P3PC (performance of 3rd party content). And Pat even added a few options I requested. You can set the size of the image, remove certain requests (for example, I sometimes remove favicon.ico), and whether to show the extra bits on CPU and bandwidth utilization. I end up with clean waterfall charts like this:

And there’s more – Video!

When you create a new test, you can opt to record a video of the page loading (go to the Video tab under “Step 4 – Test Options” in the figure above). WebPagetest generates a filmstrip of images as well as a video. The image filmstrip shows what the page looks like as it loads. You can choose different time increments (0.1, 0.5, 1 and 5 seconds). Wikipedia is pretty straightforward as it loads. Here’s the filmstrip for FOX Sports. Content arrives in the 0-5 second range, more images (like the logo) are filled in from 5-10 seconds, and flash arrives by the 15 second mark. You can also view the video.

Pat does a lot of this work on his own time and all the video features are in Alpha, so be tolerant. I use WebPagetest.org daily and it has become one of my favorite performance tools. Definitely give it a try.

Browserscope is an open source project based on my earlier UA Profiler project. The goal is to help make browsers faster, safer, and more consistent. This is accomplished by having categories of tests that measure how browsers behave in different areas. Browserscope currently has these test categories: Network, Acid3, Selectors API, Rich Text, and Security.

The project is led by Lindsey Simon. Today he blogged about updates to the Browserscope security tests. The security test category was created by Collin Jackson (CMU) and Adam Barth (UC Berkeley). They worked with David Lin-Shung Huang and Mustafa Acer (both from CMU) on today’s release of tests for HTTP Origin Header, Strict Transport Security, Sandbox Attribute, X-Frame-Options, and X-Content-Type-Options. Check out their blog post for more details on what these tests actually do.

There are other new features in today’s release. We’ve updated the list of “top” browsers (notice we dropped IE 6). Lindsey added a dropdown menu to each test category for easier navigation. I run the Network test category. In that area I broke the overloaded parallel script loading test into four more specific tests that measure whether external scripts load in parallel with images, stylesheets, iframes, and other scripts. Brian Kuhn (Google) contributed a test for measuring whether the SCRIPT ASYNC attribute is supported.

One of the key aspects of Browserscope is that all the data is crowdsourced. This is critical. It allows the project to run without requiring a dedicated test lab. And the data is gathered under real world conditions. But to be successful, we need people in the web community to participate. When you’re done reading this post, point your browser to the Browserscope test page and click “Run All Tests”. It’ll only take a few minutes and you can sit back while it walks through all the tests automatically. We’re all in this together. Join us in making the web experience faster, safer, and more consistent.

I love my iPhone. That sentiment doubled the day I followed Nat Torkington’s pointer to Ten Second iPhone Tethering. Later that month I flew to Boston and basked in the freedom of having an Internet connection at the airport and hotel without paying wifi fees.

Then it all came crashing down. iPhone 3.1 came out. I had to choose between visual voicemail and tethering or consider jailbreaking my iPhone. Tech support in my household is limited (me) so I said goodbye to tethering. I’m back to paying hotels $10 per day to use their wifi, or signing up for a day of T-Mobile Hotspot usage at Starbucks.

Then I got my Nexus One. I really like it. It’s a huge improvement over the G1 I got last year. iPhone is still my dominant phone, but I carry the Nexus One and Palm Pre with me and am spending more time on the Nexus One.

I’m gearing up for some travel so revisited the topic of tethering. I was stunned when I spoke to AT&T tech support two days ago and they told me they support tethering. How did I miss this?! Then the guy said I had to jailbreak my iPhone. It seems weird to have tech support recommend jailbreaking. I guess that’s a result of the AT&T/Apple love/hate relationship. Same story with Palm Pre – gotta jailbreak it.

My hopes rose when I found articles saying you could tether with Nexus One. I installed PdaNet. That went smoothly. It works on Mac and Windows. I’m Mac at home but when I travel I take my Windows laptop, so that’s the critical platform for tethering. I’m always wary of new installations bogging down Windows, but PdaNetPC.exe is only 17M of memory and 0% of CPU when not in use, so I’m fine with that running in the background.

I tested it last night at home, but the real test was this morning. I stopped for coffee at Peets, booted up Windows, tethered my Nexus One, opened a ssh session, and drove to work. At every stoplight I verified my ssh session session was still active. I was reading email, surfing the Web.  It was exhilarating. I know that’s incredibly geeky to say, but I revel in the freedom it gives me. +1 for tethering without jailbreaking. All smartphones should do this.

A quick and dirty analysis based on tracking order IDs indicates that customers ordered the iPad in excess of an estimated 50,000 units in the first two hours since Apple begun accepting pre-orders at around 8:30 Eastern time. The basis for this estimate is the order ID, a unique numerical identifier assigned to track each order [...]

New Zealand's government-run Internet filtering system is now running, and two ISPs are already using the system. Seven thousand websites are on the list, most dealing with child sexual abuse, bestiality, and other illegal content, as classified by the country's official censors (you too can be a censor for a day). Such material has been illegal offline in New Zealand for years, so the expansion of the program to the Internet isn't a big surprise. But will it work?

The government runs the filter, but ISP participation remains voluntary. Currently, Maxnet and Watchdog are confirmed to be using the filter, though other ISPs are said to be interested. Maxnet CEO John Hanna explained his company's position to Computerworld New Zealand: "Filtering out child pornography is also very much in line with our company values—our customers would be disappointed to hear if we weren’t participating. So participation for us has always been a no-brainer."

Read the comments on this post

When I was younger I thought busy people were more important than everyone else. Otherwise why would they be so busy? I had busy bosses, busy parents, and always I just thought they must have really important things to do. It seemed an easy way to see who mattered and who didn’t. The busy must matter more, and the lazy mattered less.

This is the cult of busy. That simply by always seeming to have something to do, we all assume you must be important or successful.

It explains the behavior of many people at work. By appearing busy, people bother them less, and  simultaneously believe they’re doing well at their job. It’s quite a trick.

I now believe the opposite to be true. Or the near opposite. Here’s why:

• Time is the singular measure of life. It’s one of the few things you can not get more of. Knowing how to spend it well is possibly the most important skill you can have.
• The person who gets a job done in one hour will seem less busy than the guy who can only do it in five.  How busy a person seems is not necessarily indicative of the quality of their results. Someone who is better at something might very well seem less busy, because they are more effective. Results matter more than the time spent to achieve them.
• Being in demand can have good and bad causes.  Someone with a line of people waiting to talk to them outside their office door at work seems busy, and therefore seems important. But somehow the clerk running the slowest supermarket checkout line in the universe isn’t praised in the same way, it means they’re ineffective. People who are at the center of everything aren’t necessarily good at what they do (although they might be). The bar of being busy falls far well below the bar of being good.
• The compulsion to save time may lead nowhere. If you’re always cutting corners to save time, when exactly are you using the time you’ve saved? There is this illusion some day in the future you get back all the time you’ve squirreled away in one big chunk. I don’t think time works this way. For most Americans it seems most of our time savings goes straight into watching television. That’s where all the time savings we think we get actually goes.
• The phrase “I don’t have time for” should never be said. We all get the same amount of time every day. If you can’t do something it’s not about the quantity of time. It’s really about how important the task is to you. I’m sure if you were having a heart attack, you’d magically find time to go to the hospital. That time would come from something else you’d planned to do, but now seems less important. This is how time works all the time. What people really mean when they say “I don’t have time” is this thing is not important enough to earn my time. It’s a polite way to tell people they’re not worth your time.

This means people who are always busy are time poor. They have a time shortage. They have time debt. They are either trying to do too much, or they aren’t doing what they’re doing very well. They are failing to either a) be effective with their time b) don’t know what they’re trying to effect, so they scramble away at trying to optimize for  everything, which leads to optimizing nothing.

On the other hand, people who truly have control over time have some in their pocket to give to someone in need. They have a sense of priorities that drives their use of time and can shift away from the specific ordinary work that’s easy to justify, in favor of the more ethereal, deeper things that are harder to justify. They protect their time from trivia and idiocy. These people are time rich. They provide themselves with a surplus of time. They might seem to idle, or to relax, more often then the rest, but that may be a sign of their mastery not their incompetence.

I deliberately try not to fill my calendar. I choose not to say Yes to everything. For to do so would make me too busy, and I think, less effective at what my goals are.  I always want to have some margin of my time in reserve, time I’m free to spend in any way I choose, including doing almost nothing at all. I’m free to take detours. I’m open to serendipity. Some of the best thinkers throughout history had some of their best thoughts while  going for walks, playing cards with friends, little things things that generally would not be considered the hallmarks of busy people. It’s the ability to pause, to reflect, and relax, to let the mind wander, for when it returns it’s often sharper and more efficient, but most important perhaps, happier than it was before.

This post was inspired by Marrissa Bracke’s essay Why I stopped working with busy people.

Also see:

• In praise of being idle, Bertrand Russel
• How to be idle: a loafers manifesto, Tom Hodgkinson
• Work vs Progress

Google's CEO went to Abu Dhabi this week and preached. He sermonized about Google's exceptional virtue — its indifference to profit and supreme trustworthiness. His speech should have been shocking. Except that delusional self-righteousness is now routine at Google. More »

It's not a front-facing camera, but Apple has added something to the iPad since announcing it: A new button on the side that locks the screen orientation. More »

We do a decent amount of social media coverage here at Ars, but not everything that happens with Facebook, Twitter, and the like is worth its own story. Sometimes, though, we happen across things that make us say "huh, that's interesting." It turns out there are a lot of things we thought we knew about social media users, but not all of them are true. Here are a few tidbits we gathered that may surprise some of you.

Read the comments on this post

My first author copies of "The Trade of Queens" arrived this morning; that's one of them, sitting on top of the pile of its predecessors on the step-stool. (Parenthetically, this means that copies should be showing up in warehouses and book stores over the next week, and in the mailboxes of folks who placed advance orders very soon thereafter.) The stack you're looking at is the culmination of eight years' work; I began work on these books in 2002, if I remember correctly. It's about 70-100 pages longer in total than "War and Peace". And I'd like to talk about them for a bit.

Back in early 2005, I wrote a somewhat cynical essay on an earlier incarnation of this blog, titled "Five rules for cold-bloodedly designing a fantasy series". Web rot and a change of publishing platform have made that essay somewhat hard to find, so I'm going to reproduce it in edited form right here (below the cut), with updates and further insights gathered on the road.

Mon, 21 Feb 2005:

I've just returned the page proofs of the paperback edition of 'The Family Trade', due out in May in the US, and my thoughts turn to the history of the book: why and how it got written, and how things have turned out. For some reason this doesn't seem to be a topic novelists discuss much in public, so I thought I'd jot down some notes here.

The story begins in late 2001. I'd had a breakthrough year; in addition to being nominated for one of the major awards in the SF field for the first time, I'd acquired a literary agent who successfully sold my first two SF novels to Ace. I was working like a dog, trying to write at least one and a half novels a year on top of a workload of freelance computer journalism — in early 2000, when the bottom dropped out of the dot-com boom, events had caught me between stools and I ended up writing two magazine columns and numerous features every month to make ends meet. Writing books looked like a less stressful way out (at least you get to measure your cash flow in months rather than weeks), and I'd sold two, so why not try to sell some more?

Note for the uninitiated: a literary agent is the jobbing novelist's white knight. Your agent takes a cut (typically 15%) of your earnings. But if you don't earn, you don't get paid, and neither do they, so their job is to figure out how to get you as much money as possible. As a recent survey shows, agented novels get significantly higher advances, on average, than unagented; like an accountant, a good agent should earn you a lot more money than they cost. So when talking business (as opposed to art), your agent is the first person you turn to — they'll shoot down unsalable ideas before you waste six months pursuing them, and provide helpful advice on how to make your good ideas sell better. (But note the qualification about talking business, as opposed to art.)

When I raised the idea of writing some more books with my agent, her first comment was, "you realize that 'Singularity Sky' probably won't be in print for two to three years? And 'Iron Sunrise' won't be out for a year after that? Ace have a backlog, and they've also got an option on your next SF novel." (An option clause means you've got to send the next SF novel to your existing publisher, who have to reject it or sit on it for an inordinate length of time before you're free can send it elsewhere.) "On the other hand, if you really want to write for a living, can you do something that isn't specifically SF, so we can sell without breach of contract? Like, say, a big fat fantasy series?"

This made me stop and think hard. The thing is, I've read a lot of extruded fantasy product in my time, and I don't much like it. Fantasy and Science Fiction are co-marketed in most bookstores, but this conceals the fact that they're actually radically different genres in outlook. Loosely speaking, if Science Fiction is often a literature of disruption (in which change is, if not good, at least embraced), Fantasy is frequently a literature of consolation: a warm feather-bed of social conservativism disguised as nostalgic escapism, a longing for feudal certainties. While there's nothing intrinsically wrong with Fantasy, the marketing mechanism applied to it tends to promote those aspects of it that I really don't like: the hordes of marching sub-Tolkien clones. (I'm with China Mieville on this.) And besides, Robert Jordan is still alive and selling.

Rule 1: Don't steal from living authors, their ecological niche in the publishing jungle is already occupied. (Alternatively: nobody needs another Robert Jordan.)

If I was going to write extruded fantasy product, I'd have to write it from the point of view of the young lad growing up with poor but honest folks somewhere in middle earth who discovers that he's destined to grow up to be the Dark Lord, overthrow the established order, and start a revolution. Because? I'm a native of a nation that has a hereditary aristocracy and a monarchy, and it's a lot less romantic in real life than in fiction. As long as they're constitutionally reigned in and kept busy opening supermarkets and holding garden parties a monarchy isn't too toxic, but if you go back a century or two what you get is basically a hereditary dictatorship (complete with secret police) dressed up in fancy clothing. If you want a modern cognate, you need look no further than Kim Jong-Il.

I said as much to my agent and she sighed (inasmuch as one can sigh in email) and said "don't do that, the readers will hate you." Readers who hate authors do not buy their books. I saw her point, and shelved the idea. Score one for commercial pressure over art. (Note from 2010: These days I'd probably push a bit harder, and see if there was a way to square the circle — I'm more inclined to take commercial risks. But nothing concentrates the mind like a magazine going bust oweing you for three months' work, which was also happening at the same time back then ...)

Idea number two: I've been interested in alternate history as a sub-field of SF for a while. There are a couple of ways of writing alternate history; you can do it straight (as an historical novel set in a history that never happened) or if you bend the rules enough to allow for a visitor from our own world to get a tourist visa to the universe next door, you can use it as a tool to poke at our conceptions of how our own world operates.

First I took a stab at designing a straight alt-hist novel. (Elevator pitch: "I'm going to cross the streams of The IPCRESS File and Heart of Darkness in a universe where the first world war ended in 1919 with allied tanks sitting in the wreckage of Berlin, and the decaying British empire went on to invent fascism in the 1940s. It's 1962, and two OSS agents are injected into British-dominated Europe to trace the underground railroad that is funneling abducted/brainwashed American scientists east. Our two spooks, "Wild" Bill Burroughs and his swivel-eyed Californian sidekick Philip K., follow the trail — by way of a sleazy S&M nightclub in Hamburg presided over by ageing queen Adi and his boyfriend Rudi Hess — to Ceylon, where in the guts of a hollowed-out mountain they confront the jackbooted, monocle-wearing Air Commodore Arthur Clarke and his program to build an atom-bomb powered space dreadnought.) My agent shot it down as "too weird". With 20/20 hindsight, I think she may have had a point.

Next I went to look at mode #2 of alternate history: visiting the world next door.

One fly in the ointment is that AH fiction is often marketed as SF. (See "contract option" above.) But I happened to recall a precedent for doing it in fantasy — noted SF/fantasy author Roger Zelanzy's masterwork, the Chronicles of Amber, all ten books of it, featured a family of rather paranormal protagonists who could walk between worlds. The Amber books sold like bandits, but since Zelanzy's untimely death in 1995 the ecological niche has been empty. And "magic" makes for an end-run around an option on an SF novel ...

Rule 2: Steal from the best. (There's no point stealing from the worst.)

Why not, I thought, take the basic premise (a family of folks who can walk between worlds) and strip off all the superstructures Zelazny added to the mix? Reboot it in the context of a coherent alternate history set-up and see where it goes. Maybe even (being mischievous) add the "child of poor but honest folks who grows up to be the [thematic] dark lord" sub-plot to anchor it more firmly in the marketing soil of the contemporary extruded fantasy series while laying the groundwork for a later refutation of the key thesis of consolatory return? I could get to have my cake (a long fantasy series) and eat it (the intellectual challenge of doing something new).

Rule 3: If you steal an entire outfit from one writer's wardrobe, people will mock you for being imitative. So steal from at least two, and mix thoroughly.

The mere theme of a bunch of relatives who can walk between time lines does not a novel make. You've given them the means, but not the motive or method. Luckily it's not virgin territory; other writers have been here before, and it's always worth looking at the prior art. In the SF field one author in particular stands out — H. Beam Piper. Dead since 1964, his books are nevertheless still in print: a sure sign that he had something to set him apart from the majority of writers (who go out of print for good within two years of their demise). Among his most enduring works are a handful of stories and a short novel ('Lord Kalvan of Otherwhen') about the Paratime Police — an agency established by an imperialist time line that ruthlessly exploits the resources of its neighbours. While I really didn't like his key ideological assumptions, his technique was another matter. So, in accordance with Rule 3, I decided to use Piper as my other source.

Now, here's an important point: I was planning a series. Conventional publishing wisdom is that you can only publish one book a year in a given genre — if you publish more, you risk cannibalizing your own sales (unless you have an avid fan base). So if I was going to get to grips with this project, I was going to be in it for the long haul. How long is long? Well, I didn't particularly want to limit it to a trilogy — what I'd decided to look from the attic of ideas was a background and a basic premise, not a story, and I had some big ideas to explore. It would take at least four big, fat books to get to grips with it.

The first book: thesis. We're introduced to the world-walking folks, get to see why they engage in this activity (which, on the face of it, is personally risky). It's probably the oldest reason of all — economics. They do it to get rich.

This series is going to be sold as fantasy, so a mediaevalist or at least very non-contemporary setting is pretty much mandatory. This has Implications. If the family of world-walkers come from a society that's backward and primitive by our standards, that puts a whole new spin on the premise. Usually, in this sub-genre, visitors from other time lines have Advanced Super Science mojo, which invites unwelcome plot non-sequiteurs. In contrast, making them primitive is (as far as I know) a first.

For our protagonist, I can use the "child of poor-but-honest folks coopted into the aristocracy" cliche, only, like, inverted, so that being coopted into the aristocracy is bad. They find it stifling and unpleasant — a big clash of cultures. They rebel. (Hey, I'm back to the disruptive protagonist theme again!) But a poor-but-honest character from a society dominated by aristocratic time-line traders is going to be at a marked handicap. How about making them a long-lost by-blow who's grown up in our world, and gets sucked in against their will? And who's pre-wired with a curious urge to look in dark corners? A journalist, say. Who starts digging places they shouldn't, is forced to go on the run, and then has to desperately struggle to build a secure power base for themselves before the assassins close in ...

And that's how 'The Family Trade' gets the first inkling of a plot skeleton. I scoped it at around 200,000 words, or 600 pages.

The second book: antithesis. The first book sets loose a whole flock of pigeons. Pigeons shit everywhere, get eaten by hawks, and lay eggs: they have side-effects. Somewhere down the line, the consequences of our protagonist's arrival are going to start making themselves known. They're from a relatively advanced culture and they've been dropped into a relatively backward — but not politically unsophisticated — one; shades of the old time-travel classic Lest Darkness Fall by L. Sprague de Camp. Meanwhile certain other sub-themes (that fell out of the first novel outline) suggested themselves, which I'm not going to go into here. Truth and consequences: I scoped out 'The Clan Corporate' at around 250,000 words, or 750 pages.

There were two more books in the original series pitch I prepared and sent to my agent. I figured I had enough loose ends to tie myself up in enjoyably for five years — but no longer. I'd already been through sequel hell in writing 'Iron Sunrise', and figured out that no series should outlive the author's interest:

Rule 4: When choosing the themes to pilfer, only pick ones that you, personally, find interesting — if you pick something boring you'll only have yourself to blame if it's successful and you end up chained to the desk to write more of it for the next decade.

I sent the pitch to my agent, and she said, "huh, I think I can do something with this. Want to write the first book?"

So I did. The first draft ran to 155,000 words, was written in a twelve-week frenzy, and had an ending that sucked mud through a straw. My test readers told me this, so I re-wrote it and the manuscript bloated to 190,000 words. I'd run out of energy at the end of the first draft. The second worked. It's still the longest book I've ever written.

During the writing of the book a whole bunch of extra ideas occured to me. It acquired a lot more texture and complexity, and the series outline mutated in line with it. This is a good thing. I may have spent the first 90% of this essay writing a frank endorsement for deriving all your ideas from your predecessors, but it's one thing to steal the floor plan and another thing entirely to steal the wallpaper and silverware as well:

Rule 5: However much you're stealing, make sure it doesn't look stolen. Genre publishing is a beauty show, and originality wins prizes (but not too much originality).

All writers are periodically asked "where do you get your ideas?" Our dirty little secret is that ideas are cheap. You've got ideas. Your pet cat has probably got ideas. You can find ideas in the back-catalog of authors who died forty years ago, or you can go sit in a cave for forty days and nights and bring back ideas. Or you can slavishly ape Roger Zelazny's technique. What matters isn't the ideas, but what you do with them. I managed to take a grab-bag of ideas pioneered by other writers, and by inverting a couple of assumptions and hybridizing a handful of unrelated strains I came up with something new that, as far as I know, hasn't been done before.

My agent took the book and sold it to Tor. Where David Hartwell gave it a thorough editorial working over (in the course of which it swelled to just under 200,000 words). Then the dread words came down from on high: "can you split this into two volumes?" This is my sole apology to those readers who are annoyed at the abrupt ending of 'The Family Trade' — it's the first half of the original book, splitting them so that the series would run in 300-page chunks (rather than 600-750 page doorsteps) wasn't my idea (in fact, I protested it), but in the final analysis I can only tell my publisher where to get off if I'm willing to get off (and go find another publisher — after acquiring a reputation for being "difficult to work with"). I understand the reasoning behind the decision, and indeed if I'd been working with the publishers before I wrote the book it would have fitted the form factor they wanted — but that's not how the business works, and these are the breaks. At least the second half of the story will be in the shops in roughly twelve weeks' time.

Back to March 2010:

The original four-book outline now maps out to four story arcs. The first, "The Family Trade", was split into two books ("The Family Trade" and "The Hidden Family"). The second, "The Clan Corporate", would have needed two and a half books if it had been written to the 300-pages-per-volume constraint from the start. However, no plan survives contact with the enemy.

I was already 60,000 words into "The Clan Corporate", with about eight months to go to my deadline, when the form factor was declared from on high. 60,000 words into a projected 250-300,000 word novel is the intro and set-up. (The original plan was 250,000 words, but sub-plots bloat; adding 20% in the development process isn't unusual.) I was faced with a dilemma: tear it all up and redesign the rest of the series story arc from scratch and write a new 100,000 word novel, or just plough on with what I had, add another 40,000 words, and try to make it work. And while eight months might sound like plenty of time, I was overcommitted: I had another book that I was due to hand in two months later! Realistically, the throw-it-away-and-restart-from-scratch option wasn't an option. Which is why "The Clan Corporate" reads slowly, with little direct action happening until the very end — it's the setup sequence of a much longer book.

I also discovered a new ailment of the serial novel at this point. It's this: you have to spend some time at the beginning tying each new volume into what has gone before, and you ned to spend some effort making sure there's at least a nod in the direction of giving the book a climax. All of this is overhead, and the rule of thumb I've learned is that it adds about 33% to the length of the story. By taking a 250-300,000 word book and splitting it into 100,000 word episodes, I ended up having to add an extra 130,000 words — bringing it up to four books.

Midway through book #4 ("The Merchants' War") I got some good news. My agent had just sold two more SF novels to Ace, and had written in a change to the contract: the option held by Ace on my SF novels now explicitly excluded "Merchant Princes" books. This was a huge relief; it meant that I could begin writing in the back-story behind the Clan's world-walking capability. In the first three books it was presented as a huge non-sequiteur, implicitly magical; by book six it should be fairly obvious that the series is SF in fantasy drag, and as the series expands the breakdown and decay of fantasy tropes continues.

So, with six books on sale, I've actually only completed the first two of four planned story arcs. The first, "The Family Trade", can be summarized as "Miriam discovers her disturbingly-talented family; kicks back: releases a whole bunch of pigeons." The second, "The Clan Corporate" (in four volumes) can be summarized as "Miriam's pigeons return, doing what pigeons do — they shit everywhere and there's an unholy mess". The third story — that would be books 7-9 inclusive, and I've just aboutworked out how to make them fit the 100,000 words-per-volume format. But I'm not about to begin writing them for a while; I want a couple of years off!

Final note: you're probably wondering what the Merchant Princes is about.

Paul Krugman nailed it: it's implicitly about sociology and economics, and more specifically, about the development trap. If you're reading this blog entry on a computer of your own, you're probably a native of the developed world. But what is the developed world? What does it mean to be undeveloped? More importantly, why do some societies develop rapidly, and others fail? Compare and contrast South Korea or Japan with Thailand or Burma; the former two are richer on a per-capita basis than Germany, but only developed in the past sixty years. The latter two ... aren't. Why is that? Again, consider the middle east, and the parlous state of science, industry, and economic productivity in most of the countries there. Some of them are clearly trying to develop; others ... aren't. In the Merchant Princes books, by setting up a bunch of time lines with divergent histories I was able to establish an artificial development scenario, and examine development traps; how it is that the ruling elite of a very poor country can live an imported developed-world lifestyle, but fail to spark economic development in their general population. And there's also a thesis buried in there about the toxic effects of ruling elites, and the suppression of free trade, free speech, and human rights that ruling elites are prone to.

Some of you may be wondering why I needed to write a real political figure into the series at all. There are a couple of answers to this question. Firstly, a huge problem any writer faces with an ongoing serial novel is that you can't go back and redraft earlier chapters if you realize too late that you've gotten elements of the plot tangled up: they're already in print. As it turns out, I'd implicitly written a very high-ranking US government figure, in cahoots with the Clan, into the books right from the start; it was only in the proess of writing book #4 that I realized I needed to bring this person front-and-centre, lest the plot peter out embarrassingly or otherwise expose me as not having a clue. Secondly: one of the failure modes to which constitutional democracies are prone is the usurpation of power by hard-liners, when confronted by an external threat. It was obvious that the exposure of the Clan would necessitate a drastic response from the US government — so why not explore the boundary condition where the response is dictated by the most outspoken national security hawks? Thirdly and finally: these novels are set in an alternate present. There are clues, from book 4 onwards, that this is not our world. (If you haven't read them yet, keep an eye open for Paris Hilton and "Chemical" Ali.)

Oh, and finally: there are no unambiguous "good guys" in this series. I've had some indignant mail from readers who don't like my treatment of, on the one hand, the Clan, and on the other, of WARBUCKS. The former ... in their own world, they're arrogant aristocrats; in our world, they're narcoterrorists. And as for the latter, at least in these novels there's a reason he's a paranoid arsehole. There is a tragedy wrapped in an enigma here: the tragedy of the ordinary people, living in interesting times. And I think that's probably going to be the overarching theme of the third story line, when I start writing it ...

A Japanese company has developed a tilt-shift adapter for Micro Four Thirds cameras.

Read more and comment »

The two-day festival after the 159th Annual St. Patrick’s Day parade provides a great opportunity for attendees to learn more about Irish history and culture while having fun. A full two days of activities are planned for the St. Patrick’s Day Festival at Civic Center Plaza and on Grove Street, Polk to Larkin Street.

The colorful festivities surrounding the parade will showcase Irish Culture through live performance and entertainment, arts and crafts exhibitors, and food and beverage concessions, children’s rides and inflatables, cultural displays, a petting zoo and pony ride and a number of non-profits booths representing the Irish community.

The Festival will take place, before, during and after the Parade. The Parade begins at 11:30 AM at the corner of Second and Market, where more than 5,000 participants from all over the state, will reel about with laughter and revelry all the way to City Hall.

San Francisco’s 159th Annual St. Patrick’s Day Festival
Saturday & Sunday, March 13th & 14th, 2010
Civic Center Plaza ~ San Francisco
10:00 AM to 5:00 PM
Parade Saturday, March 13th at 11:30 AM

There’s no question that Dunhill knows to supply men with premium gear, thus we’re almost used to receive extravagant releases from them just like their newly introduced Chassis...

There's an informal assumption that spinach and other leafy greens lose nutrients the longer they sit on grocery store shelves, and it's actually backwards. A plant physiologist in Texas says supermarket lighting actually boosts the vitamin content in your salad fixings. More »

There are two kinds of companies I really like. One that ignores the competition entirely. And one that picks a fight. Method, and their new laundry detergent line, is a great example of the latter.

The new Method laundry product eschews the standard awkward, heavy, messy jug for a svelte, light, one-handed, easily stored, pump-powered dispenser bottle. It’s so much better.

They claim it works better too, but I’m not concerned about that for this post. Even if it works just the same, the form factor is a huge win.

I’ve run out of laundry detergent so many times because I haven’t felt like lugging home one of those big jugs when I was at the store (I often walk home and one of these jugs weighs nearly as much as everything else I’m buying). I’m always like “I’ll get it another time” and then it’s too late. But the new Method bottle is just like a bottle of water. There’s no barrier to carry.

The pump dispenser is perfect fit for laundry detergent. My pour spout detergent bottles almost always leak, drip, or get dirty from dust and grime that is attracted to the gooey viscous liquid. The pump ends all that. Four pumps for a normal load and you’re good to go.

Yes, there are more important problems in the world than laundry detergent, but I’m still glad Method picked a fight and kicked ass. This is a wonderfully designed product with a form factor has been taken for granted for too long. Good for them.

One of the scourges of modern life may have been profoundly misunderstood

BEING fat is bad for you. On that, almost everyone agrees. It is just possible, though, that almost everyone is wrong. In fact, getting fat may be a mechanism that protects the body. The health problems associated with fatness may not be caused by it but be another consequence, another symptom, of overeating.

That is the heretical proposal of Roger Unger and Philipp Scherer. Dr Unger and Dr Scherer, who work at the University of Texas, in Dallas, have been reviewing the science of what has come to be known as metabolic syndrome. This is a cluster of symptoms such as high blood pressure, insulin resistance and fatness that seem to increase the risk of heart disease and strokes, late-onset diabetes and liver disease. Metabolic syndrome is found in a sixth of the American population. ...

One of the major milestones for ZFS Storage appliance with 2010/Q1 is the ability to dedup data on disk. The open question is then : What performance characteristics are we expected to see from Dedup ? As Jeff says, this is the ultimate gaming ground for benchmarks. But lets have a look at the fundamentals.

ZFS Dedup Basics

Dedup code is simplistically a large hash table (the DDT). It uses a 256 bit (32 Bytes) checksum along with other metata data to identify data content. On a hash match, we only need to increase a reference count, instead of writing out duplicate data. The dedup code is integrated in the I/O pipeline and is done on the fly as part of the ZFS transaction group (see Dynamics of ZFS, The New ZFS Write Throttle ). A ZFS zpool typically holds a number of datasets : either block level LUNS which are based on ZVOL or NFS and CIFS File Shares based on ZFS filesystems. So while the dedup table is a construct associated with individual zpool, enabling of the deduplication feature is something controlled at the dataset level. Enabling of the dedup feature on a dataset, has no impact on existing data which stay outside of the dedup table. However any new data stored in the dataset will then be subject to the dedup code. To actually have existing data become part of the dedup table one can run a variant of "zfs send | zfs recv" on the datasets.
Dedup works on a ZFS block or record level. For a iSCSI or FC LUN, i.e. objects backed by ZVOL datasets, the default blocksize is 8K. For filesystems (NFS, CIFS or Direct Attach ZFS), object smaller than 128K (the default recordsize) are stored as a single ZFS block while objects bigger than the default recordsize are stored as multiple records Each record is the unit which can end up deduplicated in the DDT. Whole Files which are duplicated in many filesystems instances are expected to dedup perfectly. For example, whole DB copied from a master file are expected to falls in this category. Similarly for LUNS, virtual desktop users which were created from the same virtual desktop master image are also expected to dedup perfectly.
An interesting topic for dedup concerns streams of bytes such as a tar file. For ZFS, a tar file is actually a sequence of ZFS records with no identified file boundaries. Therefore, identical objects (files captured by tar) present in 2 tar-like byte streams might not dedup well unless the objects actually start on the same alignment within the byte stream. A better dedup ratio would be obtained by expanding the byte stream into it's constituent file objects within ZFS. If possible, the tools creating the byte stream would be well advised to start new objects on identified boundaries such as 8K.
Another interesting topic is backups of active Databases. Since database often interact with their constituent files with an identified block size, it is rather important for the deduplication effectiveness that the backup target be setup with a block size that matches the source DB block size. Using a larger block on the deduplication target has the undesirable consequence that modifications to small blocks of the source database will cause those large blocks in the backup target to appear unique and not dedup somewhat artificially. By using an 8K block size in the dedup target dataset instead of 128K, one could conceivably see up to a 10X better deduplication ratio.

Performance Model and I/O Pipeline Differences

What is the effect on performance of Dedup ? First when dedup is enabled, the checksum used by ZFS to validate the disk I/O is changed to the cryptographically strong SHA256. Darren Moffat shows in his blog that SHA256 actually runs at more than 128 MB/sec on a modern cpu. This means that less than 1 ms is consumed to checksum a 128K and less than 64 usec for an 8K unit. This cost is online incurred when actually reading or writing data to disk, an operation that is expected to take 5-10 ms; therefore the checksum generation or validation is not a source of concern.
For the read code path, very little modification should be observed. The fact that a reads happens to hit a block which is part of the dedup table is not relevant to the main code path. The biggest effect will be that we use a stronger checksum function invoked after a read I/O : at most an extra 1 ms is added to a 128K disk I/O. However if a subsequent read is for a duplicate block which happens to be in the pool ARC cache, then instead of having to wait for a full disk I/O, only a much faster copy of duplicate block will be necessary. Each filesystem can then work independently on their copy of the data in the ARC cache as is the case without deduplication. Synchronous writes are also unaffected in their interaction with the ZIL. The blocks written in the ZIL have a very short lifespan and are not subject to deduplication. Therefore the path of synchronous writes is mostly unaffected unless the pool itself ends up not being able to absorb the sustained rate of incoming changes for 10s of seconds. Similarly for asynchronous writes which interact with the ARC caches, dedup code has no affect unless the pool's transaction group itself becomes the limiting factor. So the effect of dedup will take place during the pool transaction group updates. Here is where we take all modifications that occurred in the last few seconds and atomically commit a large transaction group (TXG). While a TXG is running, applications are not directly affected except possibly for the competition for CPU cycles. They mostly continue to read from disk and do synchronous write to the zil, and asynchronous writes to memory. The biggest effect will come if the incoming flow of work exceed the capabilities of the TXG to commit data to disk. Then eventually the reads and write will be held up by the necessary write (Throttling) code preventing ZFS from consuming up all of memory .
Looking into the ZFS TXG, we have 2 operations of interest, the creation of a new data block and the simple removal (free) of a previously used block. ZFS operating under a copy on write (COW) model, any modification to an existing block actually represents both a new data block creation and a free to a previously used block (unless a snapshot was taken in which case there is no free). For file shares, this concerns existing file rewrites; for block luns (FC and iSCSI), this concerns most writes except the initial one (very first write to a logical block address or LBA actually allocates the initial data; subsequent writes to the same LBA are handled using COW). For the creation of a new application data block, ZFS will then run the checksum of the block, as it does normally and then lookup in the dedup table for a match based on that checksum and a few other bits of information. On a dedup table hit, only a reference count needs to be increased and such changes to the dedup table will be stored on disk before the TXG completes. Many DDT entries are grouped in a disk block and compression is involved. A big win occurs when many entries in a block are subject to a write match during one TXG. Then a single 1 x 16K I/O can then replace 10s of larger IOPS. As for free operations, the internals of ZFS actually holds the referencing block pointer which contains the checksum of the block being freed. Therefore there is no need to read nor recompute the checksum of the data being freed. ZFS, with checksum in hand, looks up the entry in dedup table and decrement the reference counter. If the counter is non zero then nothing more is necessary (just the dedup table sync). If the freed block ends up without any reference then it will be freed.
The DEDUP table itself an an object managed by ZFS at the pool level. The table is considered metadata and it's elements will be stored in the ARC cache. Up to 25% of memory (zfs_arc_meta_limit) can be used to store metadata. When the dedup table actually fits in memory, then enabling dedup is expected to have a rather small effect on performance. But when the table is many time greater than allotted memory, then the lookups necessary to complete the TXG can cause write throttling to be invoked earlier than the same workload running without dedup. If using an L2ARC, the DDT table represents prime objects to use the secondary cache. Note that independent of the size of the dedup table, read intensive workloads in highly duplicated environment, are expected to be serviced using fewer IOPS at lower latency than without dedup. Also note that whole filesystem removal or large file truncation are operation that can free up large quantity of data at once and when the dedup table exceeds allotted memory then those operation, which are more complex with deduplication, can then impact the amount of data going into every TXG and the write throttling behavior.

So how large is the dedup table ?

The command zdb -DD on a pool shows the size of DDT entries. In one of my experiment it reported about 200 Bytes of core memory for table entries. If each unique object is associated with 200 Bytes of memory then that means that 32GB of ram could reference 20TB of unique data stored in 128K records or more than 1TB of unique data in 8K records. So if there is a need to store more unique data than what these ratio provide, strongly consider allocating some large read optimized SSD to hold the DDT. The DDT lookups are small random IOs which are handled very well by current generation SSDs.
The first motivation to enable dedup is actually when dealing with duplicate data to begin with. If possible procedures that generate duplication could be reconsidered. The use of ZFS Clones is actually a much better way to generate logically duplicate data for multiple users in a way that does not require a dedup hash table.
But when the operating conditions does not allow the use of ZFS Clones and data is highly duplicated, then the ZFS deduplication capability is a great way to reduce the volume of stored data.
The views expressed on this blog are my own and do not necessarily reflect the views of Oracle.

Referenced Links :

• ZFS Dedup
• Bobn's first Look
• Dedup Community
• Dynamics of ZFS
• Write Throttle
• Recordsize
• SHA256 Performance

2010/Q1 delivers 200% more Exchange performance and 50% extra transaction processing

One of the great advances present in the ZFS Appliance 2010/Q1 software update relates to the block allocation strategy. It's been one the most complex performance investigation I've ever had to deal with because of the very strong impact previous history of block allocation had on future performance. It was maddening experience littered with dead end leads. During that whole time it was very hard to make sense of the data and segregate what was due to a problem in block allocation from author causes that leads customer to report performance issues.

Executive Summary

A series of changes to ZFS metaslab code lead to 50% improved OLTP performance and 70% reduced variability from run to run. We also saw a full 200% improvement on MS Exchange performance from these changes.

Excruciating Details for aspiring developer "Abandon hope all ye who enter here"

At some point we started to look at random synchronous file rewrite (a la DB writer) and it seemed clear that the performance was not what we expected for this workload. Basically, independent DB block synchronous writes were not aggregating into larger I/Os in the vdev queue. We could not truly assert a point where a regression had set in, so rather than threat this as a performance regression, we just decided to study what we had in 2009/Q3 and see how we could make our current scheme work better. And that lead us on the path of the metaslab allocator:

• ZFS Block Allocation
• Space Maps

As Jeff explains, when a piece of data needs to be stored on disk, ZFS will first select a top level vdev (a raid-z group, a mirrored set, a single disk) for it. Within that top level vdev, a metaslab (slab for short) will be chosen and within the slab a block of Data Virtual Address (DVA) space will be selected. We didn't have a problem with the vdev selection process, but we thought we have an issue with the block allocator. What we were seeing was that for random file rewrite the aggregation factor was large (say 8 blocks or more) when performance was good but dropped to 1 or 2 when performance was low. So we tried to see if we could do a better job at selecting blocks that would lead to better I/O aggregation down the pipeline. We kept looking at the effect of block allocation but it turned out the source of problem was in the slab selection process.
So a slab is a portion of DVA space within a metaslab group (aka a top level vdev). We currently divide VDEV space into approximately 200 slabs (see vdev_metaslab_set_size). Slabs can be either loaded in memory or not. When loaded, the associated spacemaps are active meaning we can allocate space from them. When slabs are not loaded, we can't allocated space but we can still free space from them (ZFS being copy-on-write or COW, a block rewrite frees up the old space). In this case we just log to disk the freed range information. As load and unload of spacemaps are not cheap and we insure we minimize such operation.
So each slab is weighted according to a few criteria and the slab with the highest weight is selected for allocation on a vdev. The first criteria for slab selection is to reuse the same one as the last one used: basically don't change a winner. We refer to this as the PRIMARY slab. The second criteria for slab selection is the amount of free space. The more the better. However, lower LBA (logical block addresses) which maps to outer cylinders will generally give better performance. So we weight lower LBA more than inner ones at equivalent free space. Finally, a slab that has already been used in the past, even if currently unloaded, is preferred to opening up a fresh new slab. This is the SMO bonus (because primed slabs have a Space Map Object associated). We do want to favor previously used slabs in order to limit the span of head seeks : we only move inwards when outer space is filled up.
The purpose of the slabs is to service a block allocation, say for a 128K record. So when a request comes in, the highest weighted slab is chosen as we ask for a block of the proper size using an AVL tree of free/allocated space. There was a problem we had to deal with in previous releases which occurred when such allocation failed because of free space fragmentation. Then the AVL tree was then not able to find a span of the requested size and was consuming CPU only to figure out there was no free block present to satisfy an allocation. When space was really tight in a pool we walked every slab before deciding that the allocation needed to be split into small chunks and a gang block (a block of blocks) created. So the spacemaps were augmented with another structure that allowed ZFS to immediately know how large an allocation could be serviced in a slab (the so called picker private tree organized by size of free space).
At that point we had 2 ways to select a block, either find one in sequence of previous allocation (first fit) or use one that fills in exactly a hole in the allocated space: so called best fit allocator. We also decided then to switch from best fit to first fit as a slab became 70% full. The problem that this created, we now realize, is that while it helped the compactness of the on-disk layout, it created a headache for writes. Each new allocation, got a taylored-fit disk area and this lead to much less write aggregation than expected. We would see that write workloads to a slab slowed down as it transitioned to 70% full (note this occurred when a slab was 70% full not the full vdev nor the pool). Eventually, the degraded slab became fully used and it would transition to a different slab with better performance characteristic. Performance could then fluctuate from an hour to the next.
So to solve this problem, what went in 2010/Q1 software release is multifold. The most important thing is: we increased the threshold at which we switched from 'first fit' (go fast) to 'best fit' (pack tight) from 70% full to 96% full. With TB drives, each slab is at least 5GB and 4% is still 200MB plenty of space and no need to do anything radical before that. This gave us the biggest bang. Second, instead of trying to reuse the same primary slabs until it failed an allocation we decided to stop giving the primary slab this preferential threatment as soon as the biggest allocation that could be satisfied by a slab was down to 128K (metaslab_df_alloc_threshold). At that point we were ready to switch to another slab that had more free space. We also decided to reduce the SMO bonus. Before, a slab that was 50% empty was preferred over slabs that had never been used. In order to foster more write aggregation, we reduced the threshold to 33% empty. This means that a random write workload now spread to more slabs where each one will have larger amount of free space leading to more write aggregation. Finally we also saw that slab loading was contributing to lower performance and implemented a slab prefetch mechanism to reduce down time associated with that operation.
The conjunction of all these changes lead to 50% improved OLTP and 70% reduced variability from run to run (see David Lutz's post on OLTP performance) . We also saw a full 200% improvement on MS Exchange performance from these changes.
The views expressed on this blog are my own and do not necessarily reflect the views of Oracle.

Even in developing countries, most people in the world now rely on mobile phone service. Not surprisingly, the largest mobile carriers in the world generally service the largest countries in the world. However, the data reveals a few surprises. Here’s a look at who is using mobile service around the world, which providers are used and how people are using their mobile devices.

Click to Enlarge

www.paris-26-gigapixels.com

Today we welcome Sherri Kruger, editor of Zen Family Habits, as a guest writer on Unclutterer. She also has a personal development site dedicated to sharing simple tips to enjoy life.

Last July, Erin wrote “10 uncluttering things to do every day.” I was proudly doing a few things on her list, but as usual there were a couple I hadn’t considered. This got me thinking about what other things I could do daily to reduce the clutter around our home.

Here are 10 more uncluttering things you can do each day.

1. Reset your home each evening. This doesn’t have to take long, but it’s really effective. Spend 5 or 10 minutes on a quick run-through of your home. Straighten books and knickknacks, return dishes to the kitchen, and hang up jackets. Don’t strive for perfection, this is just a quick pick up.
2. Never leave a room empty handed. Look around you. Are there things that don’t belong? When you leave the room, for whatever reason, be sure to grab a glass and return it to the kitchen, or whatever the case may be.
3. When you’re done with something, put it away. Right away. Clutter arises when we take something out, use it for awhile and neglect to return it to it’s proper home. Remember the Unclutterer’s gospel, “A place for everything, and everything in it’s place.”
4. Hit the laundry basket. Every time. It may seem easier to simply let your clothes fall where they may, but this only creates clutter. Take 30 seconds to hang up your clothes or put them in the laundry basket. Erin recommends getting ready for bed an hour before you plan so you’re not exhausted when handling your clothes.
5. Take out the garbage. Perhaps garbage day occurs only once a week, but emptying the garbage nightly, even if not entirely full, is a great habit start. Over-flowing bins are not attractive.
6. Vacuum everyday. Vacuuming ensures everything is up off the floor. Essentially, you’re doing a nightly reset during the day making it even easier to keep on top of clutter.
7. Clear out your e-mail inbox. Hundreds of e-mail messages in your inbox can be incredibly overwhelming. Take time at the end of each day to clear out your inbox. When you come back in the morning, it’ll be a lot less daunting.
8. Cut out the non-essentials. Re-evaluate the necessity of your involvement in groups, clubs, committees or boards. Limit yourself to participating in things that are important to you and make you happy.
9. Do just one thing each day. Pick a drawer, closet, or shelf that’s driving you nuts. Focus on doing one little thing to move yourself closer to the clutter free state you’re Seeking. Ask yourself: Is this really important? Can I get this again relatively easily?
10. One thing out everyday. Walk through your home with a critical eye. Look for one thing you don’t need, use, or want. Keep a couple of boxes by the garage or front door for temporary storage.

I hope this inspires you to do a little bit every day to keep ahead of the clutter and move toward a calmer and simpler life.

Order a lot of stuff online? You know what an infection foam packing peanuts can feel like in your house. Keep a few handfuls of the little buggers handy, though, for home projects that involve nails and screws. More »

This paper on a new fault-based attack on RSA has been making the rounds (Pellegrini, Bertacco, and Austin, "Fault-Based Attack of RSA Authentication). The general idea here is that you have a system that is doing RSA signatures (e.g., an SSL/TLS Web Server). You induce faults in the signature computation by reducing the power to the processor, which causes the process to produce invalid signatures which can then be analyzed by the attacker to recover the private key. They demonstrate this attack on OpenSSL.

Theoretically, this is interesting, but I'm not sure how much practical impact it has: in order to mount this attack, you need direct physical access to the machine in order to control the input voltage supply. Unless you're working with a computer that is fairly heavily secured, physical access generally translates into being able to take control of the device and extract the private key in any case. Second, the attack as implemented was performed on a FPGA-based SPARC implementation, and the researchers seem to have directly controlled the input power to the processor. In most computers (though DC-based datacenters may be different) the power to the chip is pretty heavily controlled by the power supply, and so it's at least an open question if you would be able to get good control over the chip input voltage by manipulating the AC line voltage. So, it's not like there are a huge number of environments in which this attack would be feasible.

Based on my reading of this paper, because the attack relies on invalid signatures, the simple countermeasure is just to check signatures before you emit them, which OpenSSL doesn't currently do (I'm not sure I agree wih the authors call OpenSSL's failure to do this a "serious vulnerability", but I'm not sure I agree with this characterization, since my understanding is that it's pretty standard practice not to do so). Because RSA signature verification is about 20x faster than RSA signature generation, adding this additional check would not cause significant performance overhead. However, even without this countermeasure, this doesn't seem like a significant risk to most uses of RSA.

This post is about my attempts to figure out how to best view Twitter and Facebook updates. I still haven't figured it out, and though I've been meaning to write this post for a while, but I was hoping to have developed some sort of solution or system to expound upon, but I don't.

Let me, however, expound on what I've noticed so far.

First, let me describe how I use Twitter, and to a lesser extent, Facebook. With Twitter, I follow, at this moment, 245 accounts. None of these accounts have been added for any reason except for me thinking that what that person has tweeted in the past is interesting, and that I want to hear what they say. In other words, I haven't followed someone just because they followed me. Depending on your point of view, 245 people may seem like a lot or a little, but when you add in industry people, official accounts, friends, co-workers and contacts, it adds up rather quickly. (And I'll tell you the truth, I'm not a very social person. I'm sure the real "networkers" out there are following a lot more than this.)

So because I use my own custom news reader, I've ended up storing my Twitter feed of updates into a personal database which I can then use to tweak my Tweet-reading experience, and to look at from a historical perspective. Since last November (the last time I purged the DB for whatever reason), I've gotten over 98,000 updates. In more specific terms, every 15 minutes, 24 hours a day, for the past 130 days, I've been storing every tweet I would normally get online.

98k tweets / 130 days = 743 tweets a day on average.

That's a lot of updates in 5 months, (and that's not counting the Facebook updates, which I also read). In fact, if you look at the chart I made above, the average doesn't really show the real story - the number of tweets can vary between 600 on a slow day to well over 1,000 on a busy day. Except for the Christmas holidays, when everyone took a break it seems. Some tweeters are more verbose like Michael Gartenberg (who's tweeted 3850 times in that period, or around 30 times a day average), and others are much less so, but regardless, the total number of tweets stays north of the 600 mark daily.

Let me say, it's basically *impossible* to keep up with 600+ updates a day. I've tried.

I'm talking about reading each and every tweet from each of my contacts. Not filtering in any way, but simply re-organizing and re-formatting the tweets to see if there's some magic way of keeping up. The best I've been able to do so far is to organize the Tweets on their own page, sorted by contact. Here's a snapshot of what that looks like. By organizing tweets and Facebook updates by person, they become almost blog-like in their nature. It definitely helps for skimming, so if someone I know that hasn't posted for a while shows up, I'll see their pic and can stop and read more closely. I can also see when someone is just being overly verbose, and skim all their tweets all at once.

I've also tried:

• To organize the tweets based on hourly and half-hourly increments.
• Adding Javascript so I don't see an individual's tweets unless I click to expand that section.
• Highlighting the updates that have links, calling them out for better scanning
• Formatting tweets like one big paragraph per person (getting rid of the date)

In addition to ways of seeing *everything*, I've also tried filtering:

• Based on "favorites", ranked from 1 to 5, with the top ranked users sections expanded, and the rest collapsed.
• Filtering to show only those tweets with links
• Filtering based on keywords I find interesting (mobile, etc.)

None of these options really works, for various reasons. Either you miss context, or you miss the flow of a conversation, or you end up just focusing a fewer amount of users and should just unfollow the rest anyways. So, if it's impossible to *really* keep up with a moderate amount of users, the question in my mind is, how the hell is Twitter and Facebook continuing to do so well, and increase in popularity?

Well, I've come up with a couple of terms that I think explains how these services actually work. The first is what I'm calling "Phased Attention" - a time period in which a user views a stream of updates filled with "Transient Information" - data that by it's very nature is expected to come and go.

To be more specific, "Phased Attention" is pretty much how anyone who uses a Twitter client deals with the constant flow of updates - they turn it on, participate, and then turn it off. Anything posted outside that time period is of little concern. Unlike my attempts to archive and read every tweet from every user, every day, it seems the only real way to consume the constant flow of data from contacts, data sources, etc. is during the time when that information is freshest, not at later.

"Transient Information" is information that you don't necessarily need to worry about. If it was, it would (and should be) sent by different means than a status update stream. This is an important concept that I think we're starting to see is fundamental to Twitter and Facebook. First, because trying to force-fit important information into the stream is just a *bad* idea, and secondly, because we're learning how much transient information is actually out there. What my friend ate for lunch, what someone's opinion of a TV show is, or even stock prices and sports scores are all bits of info that are valuable only for a certain amount of time, to a certain audience.

It may seem obvious, but I think these concepts are a fundamental change from how we've viewed communication and information consumption up until now. Email is expected to be durable for the most part, and messages build up in your inbox until you actively do something about them. This is similar to the way news feeds have been read up until now, as well. Emails and news items don't get deleted after a certain amount of time, do they? (Though many of us wish they would). However, that is essentially what happens to tweets when they disappear "below the fold" - they might as well be gone forever.

On the flip side of this, chat - including group chats and IM - expect that all participants actively and synchronously take part - you wouldn't bother typing if there was no one around to respond to you. (Though IRC might be the exception here). These systems expect the *participants* to durable, if someone drops from a IM during the middle of a conversation, it usually means the end of the conversation.

Basically, these two ideas means that Twitter and Facebook is fundamentally different in that users they don't expect either the participants nor the messages themselves to be durable.

Let's imagine how a person signs up to Twitter and starts using it for the first time. The 245 people I follow average about 5 posts a day - so assuming this new user has typical friends, the first few days or so of using Twitter is very similar to other services she may have used before. She posts an update, her friends see it, and they respond, etc. After she adds her 10th friend, however, she's now getting nearly 50 updates a day, and she's noticing that her friends don't always respond to her tweets like they did at the beginning. But enough do so that she keeps on updating her status - it's fun and cathartic, thinking that there's all these people out there that are interested in what she's doing.

Soon however, she's following or friended a few dozen people, and a few dozen are following her back, including some services like CNN, or her favorite singer or actor. The number of updates have skyrocketed to nearly 200 a day or more. Every once in a while she'll "catch up" to what a friend wrote over the weekend or something, but in general, more and more messages fly by without her seeing them. But that's ok because when she's online, she responds or retweets the ones she thinks are interesting, and also sees responses from one or two people that's following her. This keeps her interest going like a Pavlovian randomized reward and the process continues.

The key takeaway from all this analysis is simply a better understanding of the true nature of these new data streams:

• You can't consume all the data in a typical update stream. There's basically no way of organizing, sorting or prioritizing it that would let you see it *all*. The best you can do is filter, which by its very definition means you're missing something.
• Any stream which contains mostly Transient Information - like status updates - can't replace any medium that contains vital information. The chances that the updates will be missed - *even by those subscribed to them* - is just too high.
• Phased Attention is the 21st century's answer to Information Overload. As the data volume increases, more of it will simply be skimmed, missed or outright ignored outside a set time-frame or context. What's different is that unlike in years past, this is becoming an accepted, and expected way of dealing with communication and information, and it's bound to bleed back into other technologies such as email.

I have a suspicion that the novelty of the transient information stream is going to wear off. I'm not sure when, but it feels to me like a pyramid scheme at the moment, driven by the power of numbers of people who are involved. For every one person who becomes disillusioned with Twitter due to the lack of interaction or return on time invested, there are two people who replace them. But there comes a point when that growth has to end because there really is a finite number of people out there. For example, I've got roughly 1200 people following me on Twitter. If I update enough, eventually a few of them will actually see my tweets and respond. If they don't, it's pretty much like a write-only medium, which wouldn't be compelling. Retweets, and favorites, etc. help this a bit, as it lowers the bar in terms of effort to give feedback, but even this eventually is going to fade in value.

So after all this, what's next? I don't know. I'm still struggling to understand the growth of social networks, and every day I'm starting to wonder if the emperor has any clothes on or not. Something doesn't seem right, I just can't figure out exactly what it is. The idea that people are able and willing to use a Twitter client all day, with multiple accounts and windows, subscribing to hundreds of people's updates, posting 30 or more times a day, and are still able get anything done in their life seems far fetched to me. Are all those people really content with missing 90% of the information streaming through? I guess maybe they are, because that's the only way you'd be able to manage it.

Or maybe it's just me. I'm just not sure...

-Russ

You probably experience this on a regular basis: a client sends you an illustration or a logo they'd like to use in a project, but it's a low-res bitmap or a flat image file with a background texture. Or both, if you're really lucky. Sure, you can try and ask for a vector version, but more often than not what they originally sent was the best copy they had on hand.

I seem to have made something of a hobby out of trying to coax useful results out of this type of file. Over the years I've stumbled across far better ways of doing it than manually clipping the background with the the magic wand or various selection tools.

For the next time you're handed a less-than-ideal source image, here are a few of my tricks for isolating the part of the file I want to work with in Photoshop.

I'll use this Firefox promo image as an example. Yeah, I know I can find a vector version of this particular logo online, but it makes for a fairly good demo thanks to the complexity of the background.

Let's start with the text. A single, flat colour ends up being fairly easy to extract. The first step is using the Hue/Saturation sliders [Ctrl / Cmd + U] to drop the saturation all the way down to a value of -100:

Then trim away the surrounding pixels to focus on the area you want:

Next use the levels sliders [Ctrl / Cmd + L] to increase the contrast on either side so that the background is black and the text is white. The trick is doing it in a way that doesn't completely kill the anti-aliasing between the text and the background, while at the same time making sure the foreground/background colours are pure black and white. You don't want stray pixels on either side, otherwise nasty things will happen later on when you try using your results. If you just can't find a balance, use the brush tool to fill in the strays manually.

The best results come from images where the text is clearly a different shade than the background behind it. In this case it is, so I'm able to get a reasonably high-contrast result:

At this point you have two choices. If you don't need the ability to change the colour of the text or apply layer effects, you can stop now and set the layer's blending mode to Screen or invert it (Ctrl / Cmd + I) and set the mode to Multiply. With the either of those blend modes you should be able to place this high-contrast layer over top of a background and it'll be treated as isolated text. The key being placing it on top of non-transparent background; if there are no pixels beneath for the layer to blend with, you won't be able to save it out the same way it renders in Photoshop.

For that extra control you need to take it further. By holding down Ctrl / Cmd and clicking on the image's layer in the Layers palette you can select just the outline; copy that to the clipboard, open the Channels palette, create a new Channel, and then paste the clipboard into that channel. If you've done it right it should look something like this:

The same way you selected the outline of the image in the Layers palette, you can Ctrl / Cmd + click in the Channels palette and the selection will change to just the border between the background and foreground. If you switch back to the Layers palette and create a new Solid Color fill layer:

The text now sits on an isolated layer that allows you to easily change the colour, apply layer effects, etc.

The icon is more complicated. We've got a strong orange foreground, but the globe and the background share some tones so extracting is going to be trickier. This is a fairly common scenario in the wild, but the good news is that it's still doable in most cases.

Let's be a little more selective about what we're trying extract this time. Start by focusing on the orange fox for now, we'll come back for the globe. The Select > Color Range tool allows you to click an area and select all colours similar to that selection point, and the Fuzziness slider controls how loosely or tightly the selection matches that colour. Selecting the blue area instead of the orange produces a much higher-contrast outline of the orange area:

Once you hit okay you're left with marching ants. The selection isn't useful yet, so back to the Channels palette we go. Create a new alpha channel and fill the selection with white:

Now you can just pull up the Levels palette and, with the same high-contrast result in mind as the previous text example, adjust the levels until the fox is pure black and the background is as pure white as you can make it:

If you hit Ctrl / Cmd + click on the channel then head back to the Layers palette and select the original, you can click the Add Layer Mask button in the bottom of the palette and have the selection mask the image to just the orange areas we've managed to extract:

If you end up with too many fringe pixels from the background, you can hit Ctrl / Cmd + select the mask again, and use the Select > Modify > Contract tool to decrease the selection 1px on all sides. Invert this smaller selection with Ctrl / Cmd + Shift + I, then select and fill with black the layer mask to apply it. This will eat up a one pixel outline around the object, which should take care of the fringe without tossing out too much image data. Though it does depend on the size of the source image.

We now want to add the globe back in, so duplicate the layer and remove the mask to show the original image again. Manually trace the outline of the globe with the circular marquee tool, select the masked layer and fill that selection with white within the mask. And there we go, an isolated version of the Firefox icon that you can manipulate as you please:

Extraction this way seems like a lot of work, but once you get the hang of it you'll find it's fairly quick work to go from a source image to something usable.

For relatively simple jobs like the text, the process is straightforward and can be done in a couple of seconds. For more complicated extraction as in the case of the icon, creative alpha masking combined with a bit of manual tweaking will usually get the job done.

(And I don't have to remind you to always use this for good and never for evil, do I?)

It's a little hard to believe that it's been only fifteen months since we shipped our first product. It's been a hell of a ride; there is nothing as exhilarating nor as exhausting as having a newly developed product that is both intricate and wildly popular. Especially in the domain of enterprise storage -- where perfection is not just the standard but (entirely reasonably) the expectation -- this makes for some seriously spiked punch.

For my own part, I have had my head down for the last six months as the Technical Lead for our latest software release, 2010.Q1, which is publicly available as of today. In my experience, I have found that in software (if not in life), one may only ever pick two of quality, features and schedule -- and for 2010.Q1, we very much picked quality and features. (As for schedule, let it be only said that this release was once known as "2009.Q4"...)

2010.Q1 Quality

You don't often see enterprise storage vendors touting quality improvements for a very simple reason: if the product was perfect when you sold it to me, why are you talking about how much you've improved it? So I'm going to break a little bit with established tradition and acknowledge that the product has not been perfect, though not without good reason. With our initial development of the product, we were pushing many new technologies very aggressively: not only did we seek to build enterprise-grade storage on commodity components (a deceptively daunting challenge in its own right), we were also building on entirely new elements like flash -- and then topped it all off with an ambitious, from-scratch management stack. What were we possibly thinking by making so many bets at once? We made these bets not out of recklessness, but rather because they were essential elements of our Big Bet: that customers were sick of paying monopoly rents for enterprise storage, and that we could deliver a quantum leap in price-performance. (And if nothing else, let it be said that we got that one very, very right -- seemingly too right, at times.) As for the specific technology bets, some have proven to be unblemished winners, while others have been more of a struggle. Sometimes the struggle was because the problem was hard, sometimes it was because the software was immature, and sometimes it was because a component that was assumed to have known failure modes had several (or many) unanticipated (or byzantine) failure modes. And in the worst cases, of course, it was all three...

I'm pleased to report that in 2010.Q1, we turned the corner on all fronts: in addition to just fixing a boatload of bugs in key areas like clustering and networking, we engaged in fundamental work like Dave's rearchitecture of remote replication, adapted to new device failure modes as with Greg's rearchitecture around resilience to HBA logic failure, and -- perhaps most importantly -- integrated critical firmware upgrades to each of the essential components of the I/O path (HBAs, SIM cards and disks). Also in 2010.Q1, we changed the way the way that we run the evaluation of the software, opening the door to many in our rapidly growing customer base. As a result, this release is already running on more customer production systems than any of its predecessors were at the time that they shipped -- and on many more eval and production machines within our own walls.

2010.Q1 Features

But as important as quality is to this release, it's not the full story: the release is also packed with major features like deduplication, iSER/SRP support, Kerberized NFS support and Fibre Channel support. Of these, the last is of particular interest to me because, in addition to my role as the Technical Lead for 2010.Q1, I was also responsible for the integration of FC support into the product. There was a lot of hard work here, but much of it was born by John Forte and his COMSTAR team, who did a terrific job not only on the SCSI Target Management facility (STMF) but also on the base ALUA support necessary to allow proper FC operation in a cluster. As for my role, it was fun to cut the code to make all of this stuff work. Thanks to some great design work by Todd Patrick, along with some helpful feedback from field-facing colleagues like Ryan Matthews, I think we came up with a clean, functional interface. And working closely with both John and our test team, we have developed a rock-solid FC product. But of course (and as one might imagine), for me personally, the really gratifying bit was adding FC support to analytics. With just a pinch of DTrace and a bit of glue code, we now have visibility into FC operations by LUN, by project, by target, by initiator, by operation, by SCSI command, by size, by offset and by latency -- and by any combination thereof.

As I was developing FC analytics, I would use as my source of load a silly disk benchmark I wrote back in the day when Adam and I were evaluating SSDs. Here for example, is that benchmark running against a LUN that I named "thicktail-bench":

The initiator here is the machine "thicktail"; it's interesting to break down by initiator and see the paths by which thicktail is accessing the LUN:

(These names are human readable because I have added aliases for each of thicktail's two HBA ports. Had I not added those aliases, we would see WWNs here.) The above shows us that thicktail is accessing the LUN through both of its paths, which is what we would expect (but good to visually confirm). Let's see how it's accessing the LUN in terms of operations:

Nothing too surprising here -- this is the write phase of the benchmark and we have no log devices on this system, so we fully expect this. But let's break down by offset:

The first time I saw this, I was surprised. Not because of what it shows -- I wrote this benchmark, and I know what it does -- but rather because it was so eye-popping to really see its behavior for the first time. In particular, this captures an odd phase I added to this benchmark: it does random writes across an increasing large range. I did this because we had discovered that some SSDs did fine when the writes were confined to a small logical region, but broke down -- badly -- when the writes were over a larger reason. And no, I don't know why this was the case (presumably the firmware was in fragmented/wear-leveling/cache-busting hell); all I know is that we rejected any further exploration once the writes to the SSD were of a higher latency than that of my first hard drive: the IBM PC XT's 10 MB ST-412, which had roughly 95 ms writes! (We felt that expecting an SSD to have better write latency than a hard drive from the first Reagan Administration was tough but fair...)

What now?

As part of our ongoing maturity as a product, we have developed a new role here at Fishworks: starting in 2010.Q1, the Technical Lead for the release will, as the release ships, transition to become the full-time Support Lead for that release in the field. This means many things for the way we support the product, but for our customers, it means that if and when you do have an issue on 2010.Q1, you should know that the buck on your support call will ultimately stop with me. We are establishing an unprecedented level of engineering integration with our support teams, and we believe that it will show in the support experience. So welcome to 2010.Q1 -- and happy upgrading!

Mike Taylor: Whatever Happened To Programming?

Today, I mostly paste libraries together.  So do you, most likely, if you work in software.  Doesn’t that seem anticlimactic?  We did all those courses on LR grammars and concurrent software and referentially transparent functional languages.  We messed about with Prolog, Lisp and APL.  We studied invariants and formal preconditions and operating system theory.  Now how much of that do we use?  A huge part of my job these days seems to be impedence-matching between big opaque chunks of library software that sort of do most of what my program is meant to achieve, but don’t quite work right together so I have to, I don’t know, translate USMARC records into Dublin Core or something.  Is that programming?  Really?  Yes, it takes taste and discernment and experience to do well; but it doesn’t require brilliance and it doesn’t excite.  It’s not what we dreamed of as fourteen-year-olds and trained for as eighteen-year-olds.  It doesn’t get the juices flowing.  It’s not making.

I do a lot of work to paste libraries together too, but I do get to do a fair amount of exciting code as well. Artisanal software matters: it's our way to stop having to sleep on the cold floor of the software factory.

One of the toughest aspects of staying productive is overcoming the ups and downs of motivation. Spending 10 minutes getting organized in the morning, then 15 minutes again in the evening, can help even out your daily go-get-'em energy. More »

Microsoft did something right with Windows XP. While hardly an exciting operating system by modern standards, Microsoft still managed to deliver a flawlessly realized OS at the time… so well executed that it has happily met the needs of computer users for over a decade, even as hardware has radically changed. It’s been so popular [...]

Here are a few things you don't have time to do when your car plunges into an icy lake: remove a Leatherman multitool from your glove compartment; unfold it; cut through your seatbelt; refold it; smash through your window.

Thankfully there's the ExiTool, a clever little gadget that attaches to your seat belt for quick access when your shit goes "glug, glug, glug." It includes a high-carbon stainless steel slicer, a tungsten carbide smasher, and, just for good measure, an LED light.

Sure, having an open blade attached to your seat belt all the time isn't ideal, but it's definitely more ideal than being trapped in your car at the bottom of some murky body of water.

The ExiTool will be available soon for $27, so if you're the type of person that worries about this thing it's probably a worthwhile investment. [CRKT via The Awesomer]

Cisco did have some tricks up its sleeve for its new core router, but as you'd expect, other vendors are quick to pounce on it

Five years after announcing its development, and following a month-long online campaign trailing the launch, Pentax has finally unveiled its much anticipated 645D medium format digital camera. The first digital version of the company's 645 medium format camera system, it features a 40MP, 44 x 33 CCD sensor, 921k dot 3.0" LCD and is compatible with the existing 645 system lenses. The camera will initially be available only in the Japanese market at a suggested retail price of ¥850,000 (~ US $9,400) from May 2010.