ongoing by Tim Bray

ongoing fragmented essay by Tim Bray

Records and Lenses 6 Oct 2019, 9:00 pm

Sunday included more fun than the recent average — at my age, chilling is often more attractive than partying. Sunday featured vinyl, vintage lenses, Southern guitar boogie, and a photo-assignment. With pictures! (Which may be a little puzzling, but stay with me.)

Record convention poster

What happened was, I noticed that there was a Fall 2019 Record Convention and on impulse took the e-bike crosstown to the glamorous (not really) Croation Cultural Centre. I strolled in the door of the big room with all the people and the nice lady said “You want the record convention or the camera convention? This is the cameras.” Who knew? So I went over to the other side and ended up buying five records: Willie Nelson’s Teatro (with Emmylou and Daniel Lanois), Howlin’ Wolf’s Big City Blues, Yellowman and Fathead’s Bad Boy Skanking, James Blood Ulmer’s Free Lancing, and Sons of Kemet’s Your Queen is a Reptile.

It was just a high-school-gym-sized room with a big square of tables facing outward facing the other big square against the walls facing inward, all the tables full of boxes full of LPs. I posted an appropriately lo-rez video on Instagram. A few of the vendors had organizing principles, ranging from genre to alphabet to price. Lots didn’t.

What a bunch of stoners. Except for a few fresh-faced hipsters and the odd grizzled geek like yours truly. Good clean fun.

Main Street by night


Since I’d come all the way, I coughed up $5 for the camera show, which was pretty well maximum steampunk. Old cameras! Some with bellows and bulbs, some the size of your head. I have enough cameras but you can never have too many lenses. There wasn’t much chance of anything Fujifilm-related (it’s from this millennium) but I have a Pentax K-mount adapter and as I wrote in that context, “there are an astonishing number of excellent K-mount lenses in every imaginable length and width on the market.” So I walked around the tables saying “K-mount?” to anyone showing lenses and lots had some.

Main Street by night

Protip: Nifty fifty

If you’re stingy and want to take lovely portraits it turns out that for basically any camera in the world you can get a K-mount adapter for your camera cheap, and then go to a show like this (or eBay or whatever) and pick up a classic Pentax 50mm F1.4 for almost no money and with a little practice and love, you’ll be making your friends and loved ones look better than they really do.

But I have a nifty fifty already. I’d got two thirds of the way around the camera trail and saw nothing interesting, but then there was this Mittel-Euro dude with an astonishingly small and slim 100mm F/2.8. He wanted $120 Canadian. I walked away and completed the orbit without turning up any gold, so I went back and offered him $100 and he took it. It turns out to be an M 100/2.8, i.e. probably 40 years old.

Fujifilm XT-30 with Pentax M

Isn’t it adorable?


Sunday evening I had tickets to see the North Mississippi Allstars. So I strapped the 100mm on the Fujifilm and thought I’d try for some electric-stage drama. Only the security guy took one look and said “gotta coat-check professional cameras.” Professional?! Uh…

Anyhow, the Allstars were just fine, post-Allmans southern boogie, more on the countrified side with a killer washboard solo (no, really) and really a lot of very solid guitar. Plus they played Down By The Riverside (“Ain’t gonna study war no more”) which always gets me choked up. I’d go if they came near you, assuming you like electric white-boy music.


It turns out that at Amazon Vancouver we’ve got a little photocritique social group, where someone deals out a weekly theme and we all post pix and suggest improvements to each other. This week’s theme is “long exposure”. So as soon as I got in the taxi home (For the Vancouver-savvy: up Main from the DTES to Riley Park), with a couple of drinks under my belt and a head full of electric guitar grace, I rolled down the backseat window, set the shutter to ⅛ second and the aperture to F/4, and took pictures of pretty well everything. Three made me smile.

Main Street by night

And I think to myself… what a wonderful world.

Discogs Pain 5 Oct 2019, 9:00 pm

As I continue slowly ingesting the 900 classical LPs I inherited, I’ve developed a relationship with Discogs. It’s a good place to track your collection (here’s mine, in-progress). This is the story of my first attempt to use it to buy music, a laughable failure. It’s by way of public service, just leaving a marker, a warning others to be careful about charging into the marketplace.

There’s a lot to like about Discogs. The screens are unfussy and fast and their coverage of everything ever recorded is just awesome. The LP collection I’m working my way through has plenty of obscurities by no-hit wonders, but I’ve been through a couple hundred records now and only hit one that it didn’t already have (for your amusement: Music from The Golden Age of Silent Movies played by Gaylord Carter at the Mighty Wurlitzer Theater Organ).

Also, they’ve apparently taken a mere $2.5M in venture investment, show the world a human face, and seem to genuinely care about old recordings in a tone that’s enthusiast rather than unicorn-wannabe.

Fast Freight

Back when I was doing Songs of the Day, I wrote about Fast Freight, a standout on a generally-pretty-corny 1958 mono album by the Kingston Trio that I inherited from my Dad. It’s a lovely performance which I also use as an audio showpiece; when a technical friend looks at my audio setup and snottily wonders why I still play LPs, I use this to show what the technology can do. Which usually works pretty well

The Kingston Trio

Only Dad’s copy’s getting kind of ratty, and it occurred to me that maybe I could use Discogs to replace it. Sure enough, there were plenty on sale, so I picked one on the basis that it was advertised as having “near-mint” surfaces and was offered by a seller with a good rep.

So here’s my sad story:

  1. I jumped through the Discogs hoops, not terrible at all, and placed the order.

  2. Almost instantly, the site came back saying the status was “Invoice sent”. I didn’t see any invoice on the screen or in my Discogs account or my email in-basket. I decided to wait, on the assumption it would show up.

  3. A few days later, Discogs popped up a notice saying that payment was due and unless I did so within a day or so my reputation would suffer.

  4. I fired off a barrage of messages to Discogs support and the seller saying, in effect “WTF, I want to pay, how do I pay?” Discogs was pretty prompt getting back, said “probably best to reach out to the seller.”

  5. The seller eventually got back to me and said if I wasn’t set up on Discogs’ internal payment system, he could take PayPal. I poked around and couldn’t figure out how to set up to use their payment system (still can’t). He asked me to send it marked “friends and family” so “PayPal wouldn’t hold it”.

  6. I sent him the money and PayPal did indeed put it on hold, and had deducted a big service fee. So he refunded it to me. This led to a hilarious, where by “hilarious” I mean “really irritating”, sequence of attempts to pay him, by splitting the payment up, by marking smaller amounts “Friends and family”, and I forget the rest. None of it worked, PayPal was determined not to let my money get through to him. This leads me to wonder how he’d got himself on PayPal’s hold-the-payments list.

  7. The seller also said that somewhere along the way I’d accidentally left negative feedback, could I please remove it? I couldn’t find any feedback anywhere.

  8. Finally, by routing PayPal to a different email address, he got the money and let me know he’d sent the record.

  9. A reasonable amount of time later, the record showed up. It was all scratched to ratshit and there was a big cut in the middle of Fast Freight so that the needle kept skipping back.

  10. My Discogs account still has a permanent notification that payment is outstanding for this order and my reputation is in danger.

I’m Doing It Wrong

That’s the only conclusion I can draw. I need to go and research how to buy things on Discogs without getting taken for a ride. I suspect it’s possible but it should be easier.

At the Climate Strike 28 Sep 2019, 9:00 pm

I went and so did lots of others, but many couldn’t so I thought I’d try to share the scale and the feel.

Help Save Our Winters

(Mounted on a hockey stick)

I’m pretty old and cynical and still, this felt like it mattered. Even though it was good-humored and consciously funny. Partly it was just the scale — one of our big north-south thoroughfares full of streaming strikers it seemed forever, but they say the peak of the parade only lasted an hour and a half. The police estimated a hundred thousand but they are prone to undercounting based on an institutional fear of too many people in one place going the same direction. Having said that, the cops were pretty exellent, see below.

Many spectated.

Spectators at the Climate Strike

The crowd was young, people whom they say don’t vote, and anyhow many were too young to vote. But their good-humored passion could not fail to lift hearts. Well, mine anyhow.

This Pipeline Shall Not Pass

Not that I was the only greybeard. If you’d subtracted the kids it still would have been a pretty big deal.

Don’t Care? Would Be A Lot Cooler If Ya Did

I didn’t get a picture of the kid with the sign saying “I’m skipping Fortnite to be here!”

Photographers at the Climate Strike

Once we got downtown, all the balconies and windows were full of people watching.

This Is Not A Single-Use Planet

The most popular chant was “What do we want?” “Climate justice!” “When do we want it?” “Now!” My voice is hoarse today. That chant is a common variation on a very old theme but it was the first time I’ve been part of it. After a little practice, the crowd gets into a more practiced and powerful rhythm; in particular the “Now!” becomes an explosion, elating to be part of and maybe a little scary for the intended audience. I hope so anyhow.

There Is No Planet “B”

There was this skinny little guy walking along, well under 10, whose chant was “Want to play in the snow? Stop global warming?” Pretty pedestrian, but he had a surprising volume and amazing endurance, he kept going and going and going and eventually the rhythm was fascinating and voices around him stilled.

Some People Just Wanna Watch The World Burn

I was shocked that Justin Trudeau, pipeline promoter, had the gall to appear at the Montréal strike. His smarmy-robot mantra about “The environment and the economy” isn’t fooling anyone any more. The best route to a sound twenty-first century economy requires creative destruction centered on the fossil-fuel sector. There’s plenty of capital and demand and opportunity to fill in behind, but first we have to step away from petroleum extraction, big-time.

I’m Not A Sign Guy

Towards the end of the route, the crowd started piling up and I couldn’t make any forward progress. Somewhere up ahead I could hear speeches, but the point had been made and my feet were killing me, so I took the train home. Hats off to Vancouver’s cops and TransLink’s helpers who kept the throngs moving through City Center station and damn did they ever have those trains shuttling through on the double-quick. I admire calm-faced competence when I see it.

A Very Happy Young Girl

Everybody was self-congratulatory about the nonviolence and good cheer, and unlike other recent events, there are no counter-demonstrators in evidence. The contras are all behind the walls of the glass towers our chants were echoing off of, and those guys are way too busy to pay attention.

When the waters start rising and the crops start failing, the glass will start breaking.

We Aren’t Witches

A few signs here and there were really obscene, a repeated theme of, uh, let’s phrase it as inappropriate sexual relations involving a female parent. 100% of the people carrying these were women.

We’re Coming For Your Money

And of course, there were a variety of other causes looking for listeners in what smells like the progressive mainstream. Communists, vegans, co-operative housing advocates, I even saw a sign for the IWW, which I thought as a sort of historically-significant exotic fringe thing when I was a hot-headed young leftist 40 years ago. Seems the Wobblies have staying power.

I’m An Edgy Teen

The cops had been smart, blocked access to the Cambie bridge and its feeders well in advance. Things went a little sideways downtown where a lot of people were trying to commute or go shopping and unfortunately got on the wrong cross-street and ended up motionless for a couple of hours while the strikers streamed past. I hope nobody’s life was seriously damaged, but a lot of lives are going to be seriously damaged if we don’t get our climate-change act together.

The buses were blocked on Granville Street.

Halted Buses at the Climate Strike

You’ve probably heard of airplane boneyards, places in the desert where they park hundreds of disused unwanted jet planes? When sanity sets in and it becomes just too expensive for people to waste energy the way internal-combustion cars do, there’ll be auto boneyards too.

On Sharding 25 Sep 2019, 9:00 pm

If you need to handle really a lot of traffic, there’s only one way to do it: sharding. Which is to say, splitting up the incoming requests among as many hosts (or Lambda functions, or message brokers, or data streams) as you need. Once you get this working you can handle an essentially unlimited request volume. Of course, you have to make choices on how you’re going to divide up the traffic among the shards. I’ve had intense exposure to the options since I came to work at AWS.

Random spray

This is the simplest thing imaginable. For each message, you make a UUID and use it as the Partition Key (if your downstream uses that for sharding), otherwise just pick a target shard using your favorite random number generator.

There’s a lot to like about it. Your front-end can be dumb and stateless and fast. The load will get dealt out evenly among the shards. Spoiler alert: I think this is a good choice and you should do it if you can possibly can.

A common variation on this theme involves auto-scaling. For example, if you have a fleet of hosts processing messages from an SQS queue, auto-scaling the fleet size based on the queue depth is a fairly common practice. Once again, admirably simple.

“Smart” sharding

The idea is that you do extra work to avoid problems, for example some shards getting overloaded and others being idle. Another kind of problem is one of your upstream sources sending “poison pill” messages that cause the receiving shard to lock up or otherwise misbehave

Load-sensitivity is one “smart” approach. The idea is that you keep track of the load on each shard, and selectively route traffic to the lightly-loaded ones and away from the busy ones. Simplest thing is, if you have some sort of load metric, always pick the shard with the lowest value.

I’ve seen this done, and work OK. But it isn’t that easy. You have to figure out a meaningful load metric (queue depth? how much traffic received recently? CPU load?) and communicate that from each shard to the front end.

If poison pills are your worry — this is not rare at all — the smartest approach is usually shuffle sharding. See Colm MacCárthaigh’s awesome thread on the subject, which you should go read if you haven’t already. He has pretty pictures (see below for one I stole) and math too!

Shuffle sharding by Colm MacCárthaigh


Also known as “session affinity”. Another term you sometimes hear is “sticky sessions”; for a discussion see this article over at Red Hat. The idea is that you route all the events from the same upstream source to the same shard, by using an account number or session identifier of some sort as a partition key.

Affinity is attractive: If all the clickstream clicks from the same user or state-change events from the same workflow or whatever go to the same host, you can hold the relevant state in that host’s memory, which means you can respond faster and hit your database less hard.

Ladies, enbies, gentlemen, and others: I’ve had really lousy luck with affinity in sharding. Lots of things can go wrong. The most obvious: When traffic isn’t evenly distributed among upstream sources.

inverse-square curve

One time I was working with a stream of AWS customer events, and I thought it’d be smart to deal them out to Kinesis shards by account number. That was bad — the messages per second per account rate was distributed across account numbers in a classic inverse-square curve like the one in the margin. To make this concrete, in one of the early tests I noticed that the top 10 account numbers accounted for half the traffic. Ouch. (I don’t think this was a rare situation, I think it’s probably common as dirt.)

The general lesson is that if you unfortunately send too many “hot” upstream sources to the same shard, it’s easy to overload it. The worst case of this is when you get a single “whale” customer whose traffic is too much for any single shard.

So in this situation I’m talking about, we switched to random-spray, and the whole system settled down and ran nice and smooth. Except for, processing each message required consulting a database keyed by account number and the database bills got kind of high.

So I got what I thought was a brilliant idea, hid in a corner, and wrote a “best-effort affinity” library that tried to cluster requests for each customer on as few shards as possible. It seemed to work and our database bills went down by a factor of six and I felt smart.

Since then, it’s turned into a nightmare. There’s all sorts of special-case code to deal with extra-big whales and sudden surges and other corner cases we didn’t think of. Now people roll their eyes when smoke starts coming out of the service and mutter “that affinity thing”. They’re usually too polite to say “Tim’s dumb affinity code”.

That’s not all, folks

Here’s another way things can go wrong: What happens when you have session affinity, and you have per-session state built up in some host, and then that host goes down? (Not necessarily a crash, maybe you just have to patch the software.)

Now, because you’re a good designer, you’ve been writing all your updates to a journal of some sort, so when your shard goes pear-shaped, you find another host for it and replay the journal and you’re back in business.

To accomplish this you need to:

  1. Reliably detect when a node fails. As opposed to being stuck in a long GC cycle, or waiting for a slow dependency.

  2. Find a new host to place the shard on.

  3. Find the right journal and replay it to reconstruct all the lost state.

Anyone can see this takes work. It can be done. It is being done inside a well-known AWS service I sit near. But the code isn’t simple at all, and on-call people sigh and mutter “shard migration” in exactly the same tone of voice they use for “Tim’s dumb affinity code.”

But the database!

So am I saying that storing state in shards is a no-no, that we should all go back to stateless random spray and pay the cost in time and compute to refresh our state on every damn message?

Well yeah, if you can. That fleet running my dumb affinity code? Turns out when we reduced the database bills by a factor of six we saved (blush) a laughably small amount of money.

What about latency, then? Well, if you were using a partition key for sharding, you can probably use it for retrieving state from a key/value store, like DynamoDB or Cassandra or Mongo. When things are set up well, you can expect steady-state retrieval latencies in the single-digit milliseconds. You might be able to use a caching accelerator like DynamoDB’s DAX and do a lot better.

In fact, if the sessions you might have tried to shard on have that inverse-square behavior I was talking about, where a small number of keys cover a high proportion of the traffic, that might actually help the caching work better.

But the cache is a distraction. The performance you’re going to get will depend on your record sizes and update patterns and anyhow you probabl don’t care about the mean or median as much as the P99.

Which is to say, run some tests. You might just find that you’re getting enough performance out of your database that you can random-spray across your shards, have a stateless front-end and auto-scaled back-end and sleep sound at night because your nice simple system pretty well takes care of itself.

The Cloud and Open Source 21 Sep 2019, 9:00 pm

This is eventually about the public cloud and Open Source, but — apologies in advance — takes an indirect and long-ish path.

In AWS engineering, we develop stuff and we operate stuff. I think the second is more important.

We have good hardware and software engineers, and infrastructure that feels pretty magic to me (faves: the racks and networking gear, the consensus manager underlying QLDB, and the voodoo that makes S3 go). But, like Bill Joy said, “Wherever you work, most of the smart people are somewhere else”, so I’m not gonna kid myself that we’re magically unique at programming.

But on the operations side, the picture is really unique. First of all, there are very few places in the world where you can get operational experience at this scale. Second, AWS doesn’t run on SRE culture; the same engineers who write the code live by the dashboards and alarms and metrics that try to reflect and protect the customers’ experience (not perfectly, but we make progress).

The obsessive focus on operational excellence isn’t subtle and it’s not a secret. There’s been a re:Invent presentation about how we run our ops meetings and we even open-sourced the AWS Ops Wheel.

But it’s not all meetings. We build and deploy a lot of technology with no direct connection to any feature or function or API that a customer will ever see. These are all about having the right dashboards, and being able to extract the key ratio from petabytes of logs, and predicting what might melt down before it even gets warm.

The asshole ratio

I’ve already written that at AWS, it’s lower than I experienced at other BigTech outfits. Here’s why this is relevant: There is plenty of evidence that you can be a white-hot flaming asswipe and still ship great software. But (going out on a limb) I don’t think you can be an asshole and be good at operations.

Because ops requires being humble in the face of the evidence, acknowledging fallibility, assuming that the problem is your problem even when quite likely it’s not, and always eager to investigate theories B, C, and D even when you’re pretty sure your current theory A is right-on. Since problems in complex services are almost never solved by a single individual’s efforts, you have to be good at working with people under pressure.

Those LPs

I have a hypothesis about that good ratio and it involves the Amazon Leadership Principles (we just say LPs). I’ve gotten flack from friends who think having such things is lame and corny. But in practice they turn out to be useful, and to explain how I’m going to take side-trip into modern clinical medicine.

There’s this guy Atul Gawande, a surgeon and writer whose work I’ve admired, mostly in The New Yorker, for years. I recommend pretty well anything he writes but in particular I recommend The Checklist Manifesto. Do me (and yourself) a favor, follow that link and read the Malcolm Gladwell review excerpt. From which:

“…the routine tasks of surgeons have now become so incredibly complicated that mistakes of one kind or another are virtually inevitable: it’s just too easy for an otherwise competent doctor to miss a step, or forget to ask a key question or, in the stress and pressure of the moment, to fail to plan properly for every eventuality.“ [Sounds just like updating a million-TPS Web Service. -Tim] “Gawande then visits with pilots and the people who build skyscrapers and comes back with a solution. Experts need checklists–literally–written guides that walk them through the key steps in any complex procedure.”

Well, one insanely-complex routine task that we do all the time is hiring. You know what the LPs are at hiring time? A checklist. Now even the typical all-day interview marathon isn’t gonna reliably dig into every LP, but we do an acceptable job of taking a close look at enough of them. I believe that’s very helpful in bringing down the asshole ratio.

Open Source

Which brings me to the touchy subject of the relationship between Cloud Providers and Open Source. We and our competitors have made a good business of infrastructure operations, keeping service-oriented software servicing; reliably, durably, 24/7/365. The core EC2 business is about operating Linux boxes and IP networking at extreme scale, efficiently enough that we can rent them out at an attractive price and still make a buck.

In recent Open-Source years, some very gifted people have created wonderful pieces of software — Kafka, ElasticSearch, Mongo — and taken a new course, launching VC-financed companies to monetize with service and support. Then sometimes they find themselves competing with multiple public-cloud providers.

I have a load of sympathy for the virtuoso engineers who created these wonderful pieces of work. But here’s the thing: I have at least as much for the customers who (let’s take Kafka for an example) just need reliable high-performance streaming. A direct quote: “I’ll cheerfully pay monthly to never worry about Zookeeper again.”

On the other hand, I have little sympathy with modern VC-driven business models.

It’s like this: The qualities that make people great at carving high-value software out of nothingness aren’t necessarily the ones that make them good at operations. This has two unfortunate effects: They don’t necessarily have the right skills to build and run a crack operations team, and they might not manage to get a job at an operations-obsessed company.

I have recent personal experience with failing to hire a senior committer to a well-known OSS project, and also with paying an “open-source company” for tech support when we were spinning up a service around a package we didn’t know very well. Both of these left me unhappy.

Jack and Jonathan

Let me tell you a story. Sometime around 2008, I and Jonathan Schwartz, then the CEO of Sun Microsystems, made a sales call on Jack Dorsey at Twitter. Sun had acquired MySQL and Twitter was using the hell out of it. We wanted them to start paying us for support; after all, they were existentially dependent on this technology and everyone knew that serious Enterprises would never use unsupported software.

Jack was nice, and listened to our pitch, but we didn’t get the business.

And while, as a career software guy, I entirely love open-source culture and technologies and methods, the hypothesis that Open Source in and of itself constitutes a business model is not well supported by the evidence.

Which way forward?

Google Cloud’s recent Open Source partnerships are interesting. I look at that list of companies and it’s not obvious to me that they’re going to offer better operational excellence than Google’s, but maybe I’m wrong. It’s an interesting and probably useful experiment.

At the end of the day I’m not that worried. Most of us who’ve open-sourced stuff love the creative process for its own sake; touching and improving other engineers’ lives. The skillset evidenced by having done so will probably help you get really good jobs. Yeah, you might not get to be a Bay Area Unicorn. But you probably weren’t going to anyhow.

CL XXXIX: Island Wildlife 8 Sep 2019, 9:00 pm

Our probably-last Cottage-Life weekend of 2019 featured cetacean encounters and rodent rage. But I didn’t manage to photograph any of that, so just the usual trees and sunsets.

Like this:

Howe Sound sunset


This morning I was alone at the breakfast table, considering the sea as one does, when I saw them and was yelling “whales!” at the family. There were five killer whales, one an adorable juvenile who put a little hop into every surface-to-breathe. They weren’t in a hurry, stayed for a while.

“Orcas” is more common but I say “killer whales”. Scientists prefer that too, although “orca” has etymological standing, per Wikipedia. A whale specialist told me “orca” was popularized by the marketing group at SeaWorld; they didn’t like having “killer” in their big stars’ names; family values, y’know.

Keats Island forest

Anyhow, our local killer whale population is in trouble, and that trouble is about to get worse since the Government of Canada has in its wisdom decided to bless the tar-sands pipeline that will run a tanker more or less every day through their home waters. So a random visit is a precious gift, one that might never be repeated.

Here’s today’s only picture with an animal: A feral kid scrambling up a alien-flavored tree construct.

Kid climbing tree and stump on Keats Island

Rodent rage

What happened was, I came up the ramp to the front of the cabin and there, where we have the barbecue and the recycling, was a big chunky raccoon with a beautiful silky coat, bursting with health and vigor, looking for leavings. I thought “Let’s put on a show here and disincent repeat visits” so I yelled “Hey, get outta here” and charged him. He skedaddled to the corner of the cabin and turned left, but had traction problems on the deck so I rounded the corner right behind him. He opened a lead on the dirt trail alongside the cabin and turned left again. I chased him down the path at the cabin’s back but it wasn’t close; he turned left yet again but was out of sight by the time I got to that third turn.

I thought I’d made my point and sauntered back up to the front door — and there was the raccoon again, finishing his foraging. I kind of lost it, shrieking “Gimme a **** break you **** sleazy ****!” and sprinting like a teenager.

Genuine rage makes all the difference. He headed straight sideways into the woods, never looking back. I thought he was running before, but this time he turned on his warp drive, like when they say “Engage!” on Star Trek.

He probably skulked back when we sailed for home this afternoon but we keep the place battened down in absentia. I hope we’ll be less likely to see him on future occasions when we’re in residence.

Portraits of Puppets 21 Aug 2019, 9:00 pm

If you happened to check out my Twitter feed on the weekend, you’ll know that I attended a pair of dueling rallies outside a train station in central Vancouver. On one side, a crowd in black supporting the Hong Kong protests; on the other a red-clad flag-festooned squad bringing Beijing’s message. I was dressed in black and took pictures of the other side.

Pro-Beijing demonstrators in expensive cars

The issue

It’s a no-brainer. Hong Kong isn’t perfect but it’s a civilization, with laws and with access to the world. China is a big hulking cut-off-from-the-world prison for the mind, built on systemic brutality and corruption. I admire the Hong Kongers’ courage and fear for their future. I can’t protect them from the PRC but at least I can show where I stand, and who knows, it might even make a difference if enough other people do too and the Beijing bastards decide that crushing HK might be bad for business.

People in red

I didn’t take pictures of the pro-HK side because you can bet the other side wouldn’t hesitate to use such things against them. It was probably superfluous since the Beijingers were loaded with cameras.

Pro-Beijing demonstrators

Now you’ve seen all the signs they had. It was all very uniform and organized on the Beijing side, everyone was waving the same thing. On the HK side there was an explosion of hand-lettered signs among a scattering of HK and Canadian flags. In the picture above, I particularly liked the worried-looking dude looking left through glasses, and got a nice picture of him when the sun came out.

Pro-Beijing demonstrator

He didn’t seem to be having much fun, but that’s probably a little misleading because there were definitely people on that side who were into it.

Pro-Beijing demonstratorPro-Beijing demonstratorPro-Beijing demonstrator

These two dudes were definitely full of that old school spirit, mind you one of them had his little camera rolling non-stop.

And you have to ask who these people were? I suspect they fell into three baskets. First, committed pro-Party people, maybe from the Consulate, maybe with less official standing, genuinely on the tyrants’ side — the rewards are good. Second, Chinese folk here in Vancouver who’ve stayed inside the Party-line bubble, there are media offerings to help. Third, people who don’t like the Party or (more likely) don’t like politics, who’ve had effortless-but-irresistable family or professional pressure applied.

Let’s just call them all puppets, because that’s how the people pulling the strings think about them. Here’s the puppeteers’ infrastructure:

Pro-Beijing demonstrators

Through the crowd, you can see the table where puppets can get their placards and posters and flags. I’d just love to know who organized that table and paid for the printing.

The shouting contest

That’s what the demonstrations turned out to be. The size of the red and black crowds was roughly equal — maybe a few more on the black side? — and the police did a good job of keeping space between them; it helped that nobody I saw apparently wanted to start a fight.

Disclosure: I thought how satisfying a sudden charge across the open space at the puppets would have been, but fortunately I’m grown-up enough to keep my fantasy life where it belongs.

In terms of faces and if you ignored the colors, a lot of the people on either side could have been transplanted to the other without anyone noticing. But the black side was a little older and more grizzled and a whole lot more spontaneous and cracked better jokes and the signs were better and by the way were on the side of freedom.

Coda, with hot cars

I was kind of in the middle of the black demo and noticed that every few minutes, there’d be a roar of approval from the puppet side, countered by a thunder of booing from ours. By watching where people were looking, I traced the source to the road going by. What was happening was that a few bright Beijing sparks were driving their expensive sports cars round and round the block waving PRC flags.

Pro-Beijing demonstrators in expensive carsPro-Beijing demonstrators in a Ferrari

Who’s the white dude driving the Ferrari, I want to know.

Which I think kind of underlines the key point. Like Orwell said, the object of power is power. A chief pleasure of power is showing it off, and driving around in Lambos and McLarens and Ferraris is a pretty satisfying way to do that. Particularly when you can soak up applause from the plebeians on your side and jeers from your enemies.

It’s pretty simple

The people of Hong Kong don’t want to be censored, tortured, imprisoned, and killed by those whose asshole kids are driving supercars around West Coast cities across the Pacific. I’m with them.

Talking Hong Kong Blues 18 Aug 2019, 9:00 pm

I’m imagining a discussion that might have taken place in Baidaihe at some point this month at the annual CCP summer offsite.

“Getting ugly in Hong Kong, and I’m not sure our Ms Lam is moving things in the right direction.”

“I hear from the people on the spot that what the good people want is just peace and quiet, this is just a bunch of teenage assholes making trouble.”

“Nobody wants to give their boss bad news. Haven’t you watched the BBC coverage? Maybe you’re hearing good things from your staff, but let’s suppose the gweilo TV is right? What are we going to do?”

“We’re doing one thing that’s working, going after the troublemakers’ bosses. We took down Hogg at Cathay Pacific. That’ll make every ambitious manager in HK go on the warpath to keep their employees in the office and off the damn streets. Hong Kong, it’s about three things: Money, money, and money.”

“Except for, the bad guys are getting 20% of the population out in the streets. That’ll include people who work for every fucking bank and real-estate developer and shipping company, are we gonna get every CEO in South China fired?”

“But the police say there were only 128,000 people out!”

“The HK police are idiots and in case you hadn’t notice, they’re losing in the streets.”

“I think they’re winning. There haven’t been any arrests or violence at the last three days of protests.”

“You think that’s good?! If the word starts going around that you can get away with large-scale activism as long as you keep it peaceful… do you like the idea of four million people out on the streets of Shanghai? Or a couple of million in Guangzhou?”

“What do you mean about the word getting around? The people of China are well-protected from dangerous foreign ideas, they’re not going to watching those shitty BBC liars.”

“Don’t you look at tourism figures? Fifty one million people from our side visited Hong Kong last year. They’ll all be talking to their friends and relations.”

“Yeah, well that’s maybe ten million people, a lot of them visiting every week on business. And, let’s be honest, they’re the same ones who travel overseas and already have lots of exposure to fake news from people who hate the Party. They probably all have VPNs already.”

“On top of which, those people are making good money and they owe it to us and they know it. They’ll bloody well watch what they say.”

“You guys, this is the same kind of thinking that got our 1989 leadership into trouble, letting those ‘innocent’ students stay in Tienanmen until they thought they owned it and we had to go in with the tanks and machine guns!”

“He’s right. We have the muscle all built up in Shenzen, they can be holding down Central and Tsim Sha Tsui in 72 hours and there’ll be no more of those fucking umbrellas. On top of which, the good people there will throw flowers at our guys and go back to making money in peace and quiet.”

“Suppose they don’t. Suppose there are a quarter million assholes dressed in black yelling ‘Gaa Yau!’ at each other and ‘Two Systems!’ at us and, flashing lasers and the real fringe throwing molotovs, and all with masks so we can’t ID many, and fading away into the MTR, and then another quarter million out the next day?”

“Brother, if it really comes down to them versus us, it’ll be us. Just like in 1989. It’s not just riot-control equipment waiting there in Shenzen. And any solo hero standing in front of a PLA tank this time is going to be ashes before he gets on CNN.”

“Screw CNN. It’ll be live on YouTube and Instagram and Twitter with a couple of billion people watching, and highlights of PLA tanks squishing Hong Kong patriots waiting for people who were asleep at the time.”

“So what? The people who matter need to do business with us, what do they care what kids watch on Instagram? Are they going to walk away from the chance?”

“Well, Google did.

Here’s another thing. Suppose they’re holding out in Mong Kok and every other skeezy neighborhood away from Central and there are people in all those buildings throwing shit at us from the 3rd through 20th floors, and they turn trucks sideways in those awful little streets, how are we going to get them out?”

“The PLA is not going to be stopped by a bunch of acne-faced cockroaches! Whose side are you on?”

“Western politics is weird. They eventually turned their backs on people from their own tribe in Rhodesia and South Africa in favor of a bunch of black people!

“We have our people getting our side of the story out in every Western capital; the right kind of students marching, shouting down the local HK troublemakers.”

“Give me a break, those clueless princelings haven’t the vaguest what they’re living among. I see their latest brilliant idea is to drive around in their Lambos and McLarens waving Chinese flags. Are you really really sure you want to make that bet?”

“Look, our economy is less about imports and exports every year. If the world doesn’t want us any more, then we don’t need them! We’ll just turn our backs and China will be China for Chinese, and it’ll be great.”

“Yeah, well I don’t want to miss aprés-ski in Zermatt or my place with that view in West Vancouver.”

“You might have to, because if those HK cockroaches prove they can tell us to fuck off and go on having a decent life and making money… you talk about bringing in muscle from Shenzen, what I worry about is people there starting to dress in black.”

“Yep, let’s just keep the PLA ready to roll, and hope it doesn’t have to.”

“Hope is not a strategy.”

Jeanneau 795 Review 6 Aug 2019, 9:00 pm

In Europe this boat is called the Merry Fisher 795 and in the New World, the NC 795. I’ve owned it for a few months and improved it a bit and taken it a few places and feel like sharing.

Why review?

At this point, regular readers are thinking WTF, boat review?!? I’ve only been on a handful, I’ve only owned one since 2012, I’m still occasionally baffled by nautical jargon, and my command of knots remains imperfect.

Here’s why. When you go shopping for a refrigerator or car or coffee-maker or TV or (especially) camera, there are loads of excellent detailed skeptical-voiced reviews you can read before you cough up the money. Boats (which cost more money than most of those things) are different. All the online reviews seem to be from dealers or magazine-writers on the comp, and are by and large paeans of praise.

Jeanneau 795

There are owners’ forums out there, but they tend to focus on specific problems and solutions. What’s especially missing is “I have one of these, here are the good parts and bad parts.”

I’m not completely unqualified. It’s been at my dock for a few months, I’ve installed improvements, I’ve piloted twenty-plus hours on it, motored through extreme beauty and nasty scary rough water, taken guests on pleasure cruises and a grouchy family on a tired commute, and used it as an office for a few afternoons.

So I’ll see if I can beg a few links from other 795 owners on the forums and get this a bit of GoogleJuice with the aim of better equipping other boat shoppers like me.


Jeanneau has been making boats since 1957. That link is to French Wikipedia; the English version is mostly empty and I should fix it up. The interesting, complicated story is nicely told in English by Malcolm Perrins on a Jeanneau-owners community site. The company, since its founding by Henry Jeanneau, has been sold multiple times to US and French companies and is now owned by Beneteau.

The dealer told me that the Jeanneau powerboats are built in Poland — this made him happy because for some obscure reason it leads to favorable import-tax treatment. The Jeanneau America site says “Built in America” and the first version of this piece doubted that, but a reader from Michigan wrote “We have the NC 895. It is built in Cadillac Michigan. They took the old Fourwinns plant.”

Our boat’s curtains are labeled “Made in France” and the appliances such as chargers and thrusters and fridges are Eurobuilt and their manuals have Italian or French as the first language, with English further back in the book. So I’m inclined to believe the France/Poland story in this case.

People who are buying a boat care a lot about dimensions because one of the hardest parts is finding a place that’s big enough and deep enough to park it. The 795 is 7.34m or or 24’4" long, and 2.99m or 9’9" wide, with a hull depth of a mere 0.56m or 1’10" — that’s with the outboard hoisted, which is how you normally park it.

The 795 comes with a Yamaha outboard, either 150 or 200hp, and lots of options. It’s got a modest-sized berth in the bow, a tiny but functional head (as in bathroom), and similarly tiny stove and fridge. What electronics you get apparently depends on the dealer.

Good: Engine

We have the Yamaha F200 and since it’s an outboard, there’s more room inside the boat. I’d never really been aware of this line of motors but now when I walk around any marina I see that somewhere between a third and a half of the powerboats are wearing them. So, right in the middle of the mainstream.

It’s got a very decent little electronic control screen on the dashboard and the docs are clear and comprehensive.

We set it at 4500RPM and it pushes the boat along at a little over 40km/h, depending on wind and waves. If you open it wide up on smooth water you can get up well over fifty clicks but the experience is not relaxing, or cheap either.

Good: Comfort

Not just good, excellent. The pilothouse has room for a driver and two more people in comfort, four if they’re not chunky or need extra personal space. (Protip: The aft bench is way more comfy.) The cockpit out back has forward-facing seating for three with a cushion to lean back on, and then a couple more benches but they’re less comfy. We’ve been out for a slow cruise on a warm night to watch fireworks with seven aboard and it was just fine.

Fireworks in English Bay, photographed from a Jeanneau 795

The pilothouse is really the best feature. It has a sliding “Alaska bulkhead” which means a glass door that closes, leaving the motor and its racket outside; inside, you can have a civilized conversation without shouting.

Good: Swimming platform

It’s just big enough and has a nice practical swimming ladder. We’ve used it every time we’ve been to the cabin. I shot that fireworks picture above sitting on the platform dangling my feet in the Pacific; very relaxing.

Bad: Living quarters

While they advertise two berths, realistically there’s just not enough space for more than one couple and they’d better be intimate. What with the tiny fridge and stove, I don’t think this is the boat for a lengthy family cruise up a wild coastline.

Good: Windshield

And I mean awesome. This puppy’s front glass is the size of a small European nation and when you’re sailing home with the sun behind you in a long Canadian sunset with the mountains filling the sky in front, well, there just aren’t words for that.

Vancouver through Jeanneau 795 windshield

Coming into a Vancouver from a weekend at the cottage; about two thirds of the windshield are shown. That’s the West End at the left and the Burrard Street Bridge behind the wiper. The little grey screen on the left is the Yamaha engine readout; some timing thing prevents the Pixel 2 from photographing it properly.

The wipers’ coverage isn’t that great, leaving swathes of uncleaned glass in dirty weather, but you can see the important stuff. And it comes with a windshield-washing squirter system just like your car’s, which turns out to be brilliant when you hit big waves and they splash up and want to leave sticky salt crystals where you’re trying to look out. You load it with windshield fluid from the gas station.

It’s worth mentioning the side windows too, which open and close easily and let loads of fresh air in at cruising speed without blasting your head off, and seem completely rain-proof too.

Good: Bow thruster

This is magic. We have a nice easy tie-up along the side of a dock, not crammed into a little slip, but it’s on the left as you come in and the boat wants to be tied up with its right side to the dock, so a 180° turn in tight quarters is called for. With the thruster and a light touch, it’s reasonably straightforward. The thruster is also useful as compensation for any dumb piloting errors around the dock — of course, these never happen when I’m at the wheel.

Good: It’s hackable

In the Jeanneau owners’ community I found an active boat-improvement culture; they’re always adding something or replacing something else. By dint of extensive research from primary sources, by which I mean watching YouTube videos, I have learned how to attach things to fiberglass (Protip: Get a countersink bit for your drill) and have so far improved ours by fastening the fire extinguisher to a handy bulkhead, equipping the head with a toilet-paper rod, and installing a garbage-bag holder. Call me Ishmael.

There are a variety of surfaces suitable for equipping with electronic upgrades or just decorations. We’ve decorated a couple with family photos.

Bad: Documentation

Hailing from the technology space means that I should be restrained in criticizing other professions’ end-user documentation. The boat came with a nice Jeanneau-branded satchel full of dead trees; the quality of exposition and language is, well, mixed. Highlights are the books for the Yamaha engine and the boat itself. The low point is the Lowrance navigation electronics tome, obviously executed by manic pixies on acid. The information is more or less all there but requires deep digging and Zen calm to extract.

My favorite though is the anchor-winch system, which is written in impenetrably-nautical English. Fortunately it’s accompanied by a diagram with all the parts carefully named and numbered. Unfortunately, about half the nautical names studding the text do not appear in the picture.

To be fair, I managed to figure it out well enough to anchor us (in shallow water with nearly no wind) for firework-watching.

My niece capturing a water-color of Indian Arm

My niece Anne capturing a water-color impression of Indian Arm.

Good: Piloting

The driver’s seat is comfy, the steering and throttle are crisp and responsive, and the view forward and aft is excellent. Steering at speed is a little heavier and slower than our previous inboard-outboard, but it’s plenty good enough to dodge a floating log. I’d actually like a bigger steering wheel that’s closer to me, so there’s another boat-improvement project.

Good: Access

Getting from the cockpit around to the foredeck, and up and down the sides for washing and so on, is all dead easy. The cabin is a little off-center, leaving a walkway along one side; and both sides have intelligently-placed handholds to make things easy and safe.

Bad: Flat bottom

The draft is remarkably small and the bottom, compared to the last boat, is pretty flat. This means that when you hit big waves, for example a ferry wake that you stupidly failed to notice until you were right on top of it cruising at 40km/h, you tend to skip along from wave to wave, hitting each one with a jarring “slap” of the flat bottom. This can fling passengers about a bit in a seriously uncomfortable way. Protip: Be on the sharp lookout for incoming waves and slow the hell down.

I’m not a bossy skipper but we have imposed one rule: If you want to move around the cabin, say so and we’ll slow down while you do. This after I nearly put my niece in orbit when she was going to get her backpack and I slammed on the brakes because I thought I saw some peril out front.

Good: Home office

I’m doing WFB (work from boat) one afternoon most weeks now, and it’s just terrific. The aft passenger-side bench is reasonably ergonomic and the table’s at a sane height. I often make a cup of tea and stash a snack in the fridge. I have taken conference calls, drafted and reviewed documents, reviewed code, and once (cackling with glee) checked in code to the AWS production repository.

I haven’t convinced any colleagues to come down for an in-boat meeting yet; it’s just a matter of time. But I’m just not gonna install whiteboards.

Mixed: Online community

The biggest is the Owners’ Forum, which is OK but suffers from Jeanneau having so many products. There’s also a group on Facebook, obviously. I’ve picked up valuable tips in both places.

Bad: Missing pieces

There’s no automatic bilge pump, which I find shocking, but on the other hand I have to say it stays almost bone-dry down there, even with mixed hot & cold weather, bashing through pretty rough seas, several days of heavy rain, and regular thorough washing (the honeymoon is still on).

There’s no horn; our previous boat had one and while I only ever used it once or twice, I was glad of it.

There’s no built-in heater. Our journeys typically aren’t long enough to need one on the water, but this might be an issue in home-office mode. Multiple owners have installed diesel heaters, and I have a nice little AC space heater that I’ll try out when on shore power. Similarly, there’s no air conditioner, which is more of a problem than you might think up here at 49°30'N because the pilothouse has so much glass, it’s a greenhouse.

Jeanneau 795 tied up at Keats Island

There are only two cleats, fore and aft. When you’re tying up to a floating dock for a weekend in Howe Sound (see above), which after all is part of the Pacific, you really want one and ideally two spring lines along with the basic fore and aft. Several owners have figured out how to install an extra central cleat, and I’ll look to do that.

And your conclusion is?

Count the “Good”, “Bad”, and “Mixed” headlines above. The good stuff wins, by a wide margin. I’ve got no standing to say whether or not this is a winner or loser against the competition because I haven’t owned the competition. What I can say, a few months in, is that it meets our needs very well.


Here are the things I’ve purchased to improve the experience:

  1. SeaTeak 62634 Insulated Four-Drink Binocular Rack — I have two of these things velcro’ed down behind the sink. The binoc-shaped spaces also work for big coffee mugs with handles.

  2. Dell Ultra HD 4K 24-Inch Monitor P2415Q — just the right size for outboarding to my company MBPro, and comes with USB so I only need one plug to power everything. I need to install something to hang it up on the berth bulkhead when not in use, at the moment it’s lying face-down on the mattress, which is OK but takes space.

  3. 4.5" 12V Stepless Speed Car Fan — sold by different vendors in the US & Canada. Like I said, it can get toasty in the pilothouse but this guy takes care of it just by keeping the air moving.

  4. Rod Holder Mount Boat Flagpole — the 795 has two fishing-rod holders but no flagpole. Hey-presto! The Canadian flag looks great out there but we haven’t figured out which minor ensign to fly beneath it. Patti Smith fan club? Antifa emblems? Not sure.

  5. From Davis Instruments, Shockles LineSnubbers and LineGrabbers; nothing specific to this boat, just a coincidence that I discovered them recently. If you tie up where it might get rough, you need these.


My relationship with the previous boat was pretty prosaic. It got us back and forth to the cabin and was kind of charming with its wood trim, but it always needed fixing and there were important subsystems I never learned to understand. This is a whole different kettle of fish. I’m starting to develop sympathy with the oft-repeated Kenneth Grahame quote from The Wind in the Willows:

Believe me, my young friend, there is nothing — absolutely nothing — half so much worth doing as simply messing about in boats. Simply messing… about in boats — or with boats. In or out of ’em, it doesn’t matter. Nothing seems really to matter, that’s the charm of it. Whether you get away, or whether you don’t; whether you arrive at your destination or whether you reach somewhere else, or whether you never get anywhere at all, you’re always busy, and you never do anything in particular; and when you’ve done it there’s always something else to do, and you can do it if you like, but you’d much better not.

Google Camera RAW vs JPG 27 Jul 2019, 9:00 pm

I recently wrote about how to move the excellent photos from the Google Pixel phone Camera app into a desktop Lightroom workflow. I was pleased that it’s easy to tell the camera to generate DNG “RAW” files and include them in the process. But apparently, the camera’s JPGs are better and more useful than the DNGs. That’s weird.

Here’s a pair of pictures to illustrate. This morning, the cat found a sunny corner of the back porch and was squirming around out of pure joy.

Cat on the back porch (Google Pixel DNG)Cat on the back porch (Google Pixel JPG)

The DNG is above, the JPG below. Of course they’re both JPGs now in the blog, but both are straight out of the camera, resized and JPGized by Lightroom with no sharpening or anything. It’s not that dramatic here, but flipping back and forth in Lightroom, the difference isn’t subtle. The JPG has had some lens correction, the blown-out highlights have been recovered, there’s been a bit of sharpening (look at the cat’s belly hair and the broom bristles), and the color’s been tweaked — that watering-can is dead-on in the JPG but has extra yellow in the DNG.

In this CNET piece, Marc Levoy, who invented the term “computational photography”, says “The JPEGs from the Pixel camera may actually be more detailed than the DNGs in some cases” and yeah, no kidding. In fact — and this puzzles me — the JPG is 4032x3024 in pixels while the DNG is 4016x3008, which is to say it’s 112K bigger. But I don’t think that’s what Levoy meant.

Also: “Our philosophy with raw is that there should be zero compromise,” Levoy said. “We run Super Res Zoom and HDR+ on these raw files. There is an incredible amount of dynamic range.” That doesn’t match my experience, but then he was talking about the Pixel 3 and I still have a 2. Also, Stephen Shankland said: “I rather like Google’s computational DNGs from Pixel 3. They HDR-ize the raw input to create the DNG. It’s not perfect but I find it darned useful. (I also generally like the Pixel camera app’s JPEGs, though they can look overprocessed to my eye.)” So I look forward to giving Pixel 3 (or 4) DNGs a try.

Why do you want DNG anyhow?

Photographers like RAW versions of photos because they’re more editable. One of the most common editing modes is rescuing lost data from highlights or dark areas that look blown-out or dimmed-out in the original — you hear people saying that a good camera RAW is “deep”, and that’s certainly true of the files from the Fujifilm X-cameras.

Consider these three pictures. Once again, the first is the DNG, the second the JPG, and in the third, I decided to see if I could recover image in the dark area behind the hydrangea blossoms. (Not a thing I’d normally do on this shot, I like the dark framing.)

Hydrangea (Google Pixel DNG)Hydrangea (Google Pixel JPG)Hydrangea (Google Pixel JPG, edited)

I think the results speak for themselves. There’s a lot of useful data in this JPG.

Which raises one more question: By aggressively digging in with Lightroom, could I replicate what the Google camera software did, or maybe even improve on it? So I tried that, and there was progress but at no point did I think I was really replicating that tasty JPG, and I got bored trying.

So for now I think I’m going to turn off the DNG capture on the camera app. Sshhh, don’t tell any Real Photographers.

Google Pixel Photo Workflow 27 Jul 2019, 9:00 pm

I recently wrote about the excessive difficulty of moving the Google Pixel phone’s excellent pictures through a Lightroom workflow. It turns out that Lightroom has a solution; herewith details, upsides, downsides, and alternatives. Also, cautionary words for Adobe on Lightroom Classic

To start with, Adobe has a page telling you how to Sync Lightroom Classic with Lightroom ecosystem.

Sidebar: “Lightroom Classic”

If you’re not sure what “Lightroom Classic” means, they also have a page whose URL suggests it’s about Lightroom Classic vs Lightroom CC — but the “CC” designation has apparently been lost, so the cloud-centric version is just Lightroom. For those of us with cameras that aren’t phones and produce huge raw files, Classic is the place to be.

Now, anyone who’s software-savvy has to be nervous about using a product with “Classic” in its name because that usually means “we don’t care about this, won’t invest in it, and will probably discontinue it.” For the moment I’m not going to worry because I suspect the Lightroom customer based is overweighted with people carrying serious cameras that really need the disk-based workflow, and Adobe just can’t afford to blow us off.

Having said that, I owe thanks to someone with an address who wrote me an email beginning “You are working too hard.” and outlined the How-To. But later in our exchange, they said “I do recommend the CC version, I believe that's where most of the energy is being focused.” That makes me nervous. Hey Adobe, you got a huge percentage of the world’s serious photogs to sign up for a monthly subscription; you had better treat us and our big cameras and our monster DNG files nice.

How-To: Details

That how-to-sync page is accurate as far as it goes, but I got stuck for the longest time because it says “After signing in, click your user name that now appears at the upper-left corner and ensure that the Sync With Lightroom option is turned on.” Only my Lightroom screen doesn’t have my name on it. That’s because (like many other, I bet) I run in full-screen mode. So drop out of full-screen; or just push your mouse up to the top left corner and your name will appear. Hey Adobe, why in the freaking hell is that preference hidden there instead of placed under the “Lightroom Sync” tab in the, you know, Preferences? But I digress.

The other important thing they don’t tell you is that after you’ve taken the photos, you need to wake up the Lightroom app on your Pixel and it’ll auto-magically notice the new pix and sync ’em. I’m OK with this because it lets me control when the sync happens, normally when I’m in the warm glow of home WiFi.

I used to use the Lightroom camera app (which presumably does this itself) because it had better ergonomics than Google’s, but then Google’s got the computational-photography magic where it shoots 50 times a second and combines them to produce unnaturally great pictures.


Deleting these synced photos gets a little weird. If you do it when you’re in the “All Synced Photos” folder, you get a message about how they’re going to be deleted from Lightroom but retained in the Catalog. Near as I can tell, that’s just wrong, they vanish from your phone and and your desktop Lightroom. If you’ve moved them into a regular working directory you just get the normal Lightroom “Deleted Selected Master Photo” dialog, but it still takes care of cleaning up the online and on-phone versions.


If you go check out the comments on my last piece, there are a bunch of interesting-sounding suggestions of other ways to move stuff in general and pictures in particular between your phone and your computer. I’m not going to check them out because the Lightroom process described here works for me. If you’re interested, I’d pay particular attention to one of the Sync apps from MetaCtrl because they’re by Trun Duc Tran, one of the best developers I ever worked with.


This whole investigation got started because, as Stephen Shankland noted, when you do this auto-syncing you no longer go through Lightroom’s “Import” process, which allows you to rename, add metadata, apply develop presets, and so on. Not an issue for me but it might be for you.

Google Photos Breakage 23 Jul 2019, 9:00 pm

The camera systems in the Google Pixel phones are excellent, but Google makes it way too hard for the serious photographer to get a decent workflow going. Does someone out there know a better way? If not, let’s get together and yell at Google. [Update: I found a good way to do this.]

Vancouver’s old Post Office under renovation


It’s like this: Pictures you take with your Pixel migrate into Google cloud and may be found at You don’t have to do anything to arrange this, it just happens. There are slideshow and sharing tools and they’re very decent. It’s a great service, I might even pay for it if Google asked.

But I — like most serious photographers — want to pull the photos, in their highest-resolution form, into a processing workflow. Mine is Lightroom-based, a popular (perhaps even majority) choice. In a real camera, when I say “highest-resolution” I mean a “RAW” format, which tend to come in proprietary flavors such as RAF for Fujifilm and CRW for Canon. It’s OK because Lightroom can handle all of them, and then there’s sort of a standard, developed by Adobe, called “DNG”. A few enlightened camera builders like Pentax generate DNG natively and hey, it turns out the Pixel can too. You go into the camera app Settings’ “Advanced” tab and enable a RAW control on the camera screen.

I’ve just started doing this and at the moment, my feeling is that the Googlecam software does a damn fine job of JPG generation and I’m not sure the DNG will help me that much. But the central problem remains the same: How do I get these files, be they DNG or JPG, into Lightroom?

Wet poppy



That stands for Android File Transfer, a tired old (32-bit, so my Mac whines at me) program that gives you a Finder-like interface to the Android Linux filesystem once you’ve USB-connected it to your Mac, pulled down the Android notifications, and told it to allow AFT. The JPG files are in /DCIM/Camera and the DNGs are in /Pictures/Raw. I guess what they want you to do is sort the files by date, remember where you last left off with the copying, and then select-and-drag to a staging folder in the nearest Finder window, from which you can do Lightroom input. After which you can take out your stone axe and join in the tribe’s mammoth hunt.

BTW, the Pixel will connect to the Mac in PTP mode and Lightroom can even see it, but then can’t see the pictures.


Using the Android file-sharing dialog, you can pick the pictures from the Photos app and send ’em to DropBox, and then after they’ve traveled from the phone on your armchair off to Dropbox’s home on the Internet and back down to your Dropbox Mac folder — this is not fast — you can go get ’em with Lightroom.

Color study


Dropbox is what I actually do. It’s klunky and slow, but gets the job done and has never to date actually failed to work. But we hear troubling things about Dropbox’s direction.

Google Cloud

You can get a Backup and Sync app from Google that should make some of your Google cloud files sync to your Mac. I’ve had just no end of pain with this one. You have to go mousing around in the preferences for Docs or Drive or something, a real maze of twisty little passages there, and tell it to sync photos as well as other stuff, and then last time I tried it, the process had busted the EXIF so the pictures didn’t know what time they’d been taken. Feaugh.

The other night, I was chatting with Stephen Shankland on Twitter and he seemed to have found a path through the Gcloud maze; hey Stephen, wanna blog that?

Plan D?

Anyone reading this know a better way?

EventBridge 11 Jul 2019, 9:00 pm

The launch of Amazon EventBridge, a somewhat but not entirely new thing, has been well-covered by Jeff Barr; if you want to know what it is, go read Jeff. This piece is to provide a bit of background and context on EventBridge. I didn’t actually make any direct contributions, but was upstream from this work at the definition and early-planning stage.

CloudWatch what?

My first work at AWS was on the project that launched in January 2016 as CloudWatch Events. To us it felt like a small, simple, service — write rules to route notifications of things happening out there in AWS to Lambdas or other useful destinations. It wasn’t a big team or a big task and, when it came time to name it and find it a home, it was hard to believe it deserved top-level service billing.

EventBridge stickers

Coming soon to a laptop near you. Warning: Prepared by engineers without involvement by marketing professionals or lawyers; let’s hope they’re OK.

Since CloudWatch already offered alarming and logging, eventing seemed like a nice third leg of the tripod, so our work launched as a tab on the CloudWatch page, and we thought that was OK.

Customers apparently liked it, and over the years, CloudWatch Events accumulated a mind-boggling number of users and a lot of the things they were doing weren’t really CloudWatch-y at all. Also, the whole Event-Driven Architectures drumbeat kept growing louder and louder out there in the community.

Last year, we got the idea of helping third parties (mostly SaaS vendors) integrate with their customers on AWS, and quickly became convinced that eventing was the right way to do this — while I’m a fan of the Webhook concept, the reality has not been a smooth ride. Once we’d made that call, enhancing the CloudWatch Events APIs to meet partner needs was pretty straightforward once we’d thought through the security dimensions. Except for, this was getting waaaay outside CloudWatch territory.

Which Bridge?

So, we decided that this service deserved top-level billing and went looking for a new name. The best possible answer would be “AWS Events”, right? Wrong. Go look at and hey, re:Invent, re:Inforce, AWS Summits… you get the picture. Thus EventBridge, which isn’t terrible.

(By the way, all your CloudWatch Events stuff still works and none of the existing API names or semantics have changed.)

The Event Ecosystem

It’s getting pretty big. Inside AWS, Lambda and SNS are event-centric. If you check out our competitors, you’ll notice more services with “Event” in their names every year. The numbers of events flowing through our various accumulated event streams has a lot of digits.

I’m personally pretty convinced that, while you can hook everything together with APIs, there are a whole lot of scenarios where choosing events buys you so much robustness and flexibility that it’s really hard not to. Is it perfect? Of course not: There are lots of places where the API ecosystem is slicker.

If you want to a really good explanation of why event-driven stuff might be in your future, the AWS NYC Summit talk by Mike Deck has what you need. As I write this the day of the summit, it doesn’t seem to be online but I’ll refresh once it gets there; and I bet Mike will reprise at future AWS, uh, events.

There’ll be lots more chapters in this story.

Wealth Tax Stupidity 9 Jul 2019, 9:00 pm

Canada’s mainstream conservative biz paper The Financial Post recently published The NDP’s new tax-the-rich plan is terrible, even by their standards and it is stuffed with white-hot stupidity and bad arithmetic. Arguing against any given tax is sane — that’s what conservatives are for, innit? — but if they’re going to use math that would get you an “F” in Grade 8, they deserve a whack with the cluestick.

Here’s a Post out-take:

The proposal is for an annual one per cent tax on wealth over $20 million. This means that if an Ontario resident to whom this tax applies invests $100,000 in a 1.5 per cent GIC for one year (about the rate currently offered by big banks), he or she will earn a $1,500 nominal return on which will be owed $1,833 in tax — $833 in income tax, including the higher top marginal income tax rate, plus another $1,000 for the wealth tax.

The effective tax rate on the GIC return is 122 per cent, which is what the NDP now calls “tax fairness.”

Let’s enumerate some facts:

  1. Canada has a reasonably-typical developed-world tax regime; in the top income bracket, you pay a little over 50% in tax.

  2. Wealthy people do not put their money in 1.5% GICs, they pay investment professionals to manage it. Typically, the wealthier they are, the better returns they can get because they can hire better money managers. (There’s interesting data on this in Piketty.) Let’s say they’re going to realize 4% or so, because in fact most wealthy people do better than that. I’m a one-percenter but nowhere near the hypothetical twenty-millioner in the narrative, and I do better.

  3. A little arithmetic reveals that Mr $20M will pull in about $800K in investment income, $400K net after tax. Actually, he’ll do better because some investment income will be capital gains, taxed at half the rate, and he’ll probably do better than 4% too. But then there’s the 1% wealth tax, so subtract that for a worst-case net income of $200K if he refuses to spend any of his capital.

    Now, people with that kind of money usually have other wealth-generating activities (how else did they get there?), which is on top of the $200K. Until they retire. The conventional wisdom says that a retiree can extract 4-5% of their capital per annum, which for this person would be pushing a million, except for there are lots of tax dodges open to retirees, so while they’d still be stuck with the $200K wealth tax, they could keep more of that million.

So, the things that were wrong about that $100K-deposit story:

  1. It’s completely bogus to combine the wealth tax and the income tax and say it’s “on the GIC return”. I’d say “Data fail” but that’s too kind; the 122% figure is a filthy, stinking lie.

  2. And anyhow, the whole scenario is science fiction, the “$20 million” part just doesn’t go with the “$100K GIC” part. If a conservative advocating lower taxes has to resort to this kind of specious bullshit, you have to wonder if it’s because they can’t find better arguments.


I think a small wealth tax, maybe starting in the low-double-digit millions, is a fine idea. I think that people subject to it will in practice remain extremely affluent. Their fortune would erode slowly over the years — having trouble seeing what’s wrong with that — but there’ll be plenty left to give their kids’ careers a rocket boost.

It’s perfectly reasonable to disagree with me. But if you’re going to resort to numerical flummery I’m going to think just you’re a greedy asshole with no intellectual legs to stand on.

Reasons to Cycle 6 Jul 2019, 9:00 pm

Recently I enthused on the life impact of getting an e-bike. The enthusiasm remains and I two-wheel to work almost every day. Often my thoughts are of the form “What makes this so great is…” Here are some of those, but there’s a very specific assumption: that your home city has decent bikelane infrastructure. Vancouver’s is not world-class but also not terrible, and I’ll toss in a few pix from my commute for non-bike-commuters who might not have seen what that means.

On 18th Ave in Mount Pleasant, Vancouver

I have 2½ blocks of regular street between home
and the nearest bike route. But it’s pretty nice.

It’s free

No charge to use the public roads or bikepaths. No charge to park, anywhere (the mind reels). The bike’s not free and I bought panniers and a lock, but compared to basically any car it’s peanuts, maintenance too (especially maintenance, actually). I have an e-bike so there are a few pennies’ worth of charging power every other week.

Public transit here is $24/week for my route, car parking is $10/day and way up, plus gas if you’re still misguidedly driving a fossil car.

On Vancouver’s Yukon bikewayOn Vancouver’s Yukon bikeway

One technique for making a street bike-friendly is frequent interruptions so that for cars so it’s not a direct route to anywhere. Another is a roundabout at every other corner that makes them wake up and steer. Another is to have lots of bikes on it.

It’s fast

This thing ticks along at 25km/h on the level, 20 uphill, 35-40 downhill. If I take my car and the traffic’s not bad I can get there way faster, but the traffic’s never not bad if you want to work reasonably conventional hours. You could argue that the car offers comfy seats, weather protection, and music. But…

No waiting!

Humans are born to travel; being on the road’s one of our natural conditions. (If you’ve ever doubted this, go read Chatwin’s The Songlines.) But I have never in my whole life encountered anyone who wasn’t irritated during those times when they’re trying to get somewhere and have to wait. Urban driving is all about waiting: For the light, for the other cars trying to get on the bridge or make that turn, for the slowpoke who isn’t sure where they’re going, for the pedestrians drifting across the road looking at their phones.

On Vancouver’s Yukon bikeway

It’s not perfect. This is a combo crosswalk for pedestrians, who have a right-of-way over cars, and bikes, who don’t. But if you meet their eyes they usually stop and then you should give them a friendly wave so they feel virtuous.

When you’re commuting by bike, you only ever stop for traffic lights and on a well-designed bikepath there aren’t many. It’s all flow and motion, the wind in your face and the scenery hurtling backward. For my money, this is the biggest win. I’ve never claimed the virtue of patience but I don’t think I’m that unusual.

On Vancouver’s Yukon bikewayOn Vancouver’s Yukon bikeway

Above, probably the most dangerous part of my ride. Not because it’s shared with cars — the lanes are well-marked and there’s a stop sign at every corner. It’s because it’s a steep downhill and you can have way too much fun going fast, which in itself would be OK except that bikes have really terrible braking power, especially on wet streets, which we have a lot of in Vancouver. I skidded right through a stop sign the other day, terrifying myself, and now I take it slower than I used to.

It’s good for you

This is a little more complicated than you might think, because cyclists are several times more likely than motorists to be killed per kilometer traversed. I can testify to this; in 2000 I was hit by a car that lurched into forward motion as I was coasting through a crosswalk 18 inches in front of its bumper, severely broke my shoulder, had surgery and spent a week in hospital. But — did I mention it was complicated? — cyclists cover a lot fewer kilometers than motorists, and the death rates vary strongly with age and fitness, and the data doesn’t necessarily apply to cyclists on a modern well-designed bikepath network.

On Vancouver’s Yukon bikeway

The only part of the route which is an ordinary road. It also goes right by a big cop shop, so you have to think the drivers will be a little more careful than average.

Another danger is pollution and yeah, urban cycling, during the current fossil-car interregnum, does involve inhaling exhaust.

But then there are the health benefits of getting a half-hour or more of low-impact aerobic exercise every weekday, and they are not subtle, not in the slightest.

Researchers at regular intervals over the years have tried to balance out the pros and cons. Spoiler: The upside wins, big-time. Probably the best survey I ran across researching this was Bicycling: Health Risk or Benefit? by Teschke, Reynolds, Riese, Gouge, and Winters (of UBC and SFU, here in Vancouver!) published back in 2012 but more recent papers I ran across came out about the same, and this one is nicely condensed and presented. Four of the studies they survey, from 2009 through 2012, offer numerical estimates of the ratio of benefit to risk, and those estimates are: 15:1, 9:1, 19:1, and 96:1.

On Vancouver’s Yukon bikeway

I get to bicycle over the ocean twice a day, every day. The view never gets old.

On Vancouver’s Yukon bikeway

This is what the bikepaths downtown look like. There’s more stopping for red lights, but they have no-nonsense concrete barriers between you and the cars.

Mentally too

Some of those studies actually call out mental-health benefits such as decreased risk of depression, and that’s interesting. But at another level, I feel intuitively that a half-hour in which I’m living in the moment, not gathering wool, not ingesting media, watching like a hawk for dorky drivers and pokey pedestrians, banking around corners and dodging potholes, pedaling hard to beat a yellow light… well, the benefit doesn’t feel subtle.

When I walk into the lobby at work I feel more alive than the mole people emerging from the car-park elevator.

On Vancouver’s Yukon bikeway" />

The last lap, up Vancouver downtown’s Robson street. Which is comically vacant all the time except for a few minutes around 9AM and 5PM. Lots of times you could spread out your yoga mat for a few stretches in either lane with no worries. They really ought to make it a pedestrian/bike street.

It’s good for the planet

Well, yeah. Come on and give it a try. If you’re a little old and/or creaky, splash out for an e-bike. Call it an investment because it is, and in things that are important.

CL XXXVIII: Refactorings 30 Jun 2019, 9:00 pm

What with our jobs and our kids, Cottage Life time has been tough in recent years. But we still believe in the place and the project enough to put money into repairing our dock and replacing our boat. Which raises issues of work-life balance and money laundering. And as always, these pieces are vehicles for pretty pictures of Keats Island and Howe Sound.

Looking north from Keats Island

This is actually from an earlier visit in April. Behind the elegant hydrangea-blossom corpses, Howe Sound and Gambier Island.


What happened was, last winter’s windstorms got nasty, and one of them cost us the aluminum ramp connecting our dock and our float. We need a dock because Keats Island doesn’t have much by way of roads. We need to tie the boat to a float because there are more than five meters of tide in Howe Sound. We need a ramp (with hinges at the top and wheels on the bottom) connecting the two. The old ramp, a flimsy piece of indoor construction scaffolding with rails welded on, was no great loss; and we hadn’t put a penny into the system in years.

Separately, our old boat (see here and here), now 31 years in age, had reached the end of its useful life, with ballooning maintenance costs and failing subsystems. So, as of late May, we’re the owners of a Jeanneau 795, called a Merry Fisher in Europe and an NC 795 in the New World. Boats have been built under the Jeanneau name (that link’s to French Wikipedia) since 1957 and are now manufactured in Poland. Here’s ours:

Jeanneau NC 795 under Vancouver’s Burrard Bridge

That’s Vancouver’s Burrard Bridge and part of the city center behind it. We’re at the Burrard Civic Marina, which despite the great location, is Vancouver’s cheapest boat parking. It’s a city operation and definitely not a Yacht Club, as in no lounges nor daiquiris nor gala socials, nor really much in the way of amenities. But it’s competently run and location makes up for a lot.

It turns out that boat design has made advances in the last thirty years, and apparently the French are good at it. Which is to say, the new one has a lot going for it. It’s just slightly larger than the old boat end-to-end and immensely more spacious inside. It’s comfy, quiet (the old one was loud), has a lot of light inside, and given that the 795’s a popular choice in the North Sea and the famously-blustery Western Mediterranean, probably safe for our inshore-boating needs.

Funny money?

So, my boat broker called up and said “The Jeanneau dealer in Richmond has a boat you ought to look at. A 2017 and the price is good.” I liked the look and haggled a bit, the survey came up good and the test-drive (we say “sea trial”) was fine and eventually we did the deal.

Then I noticed: The boat had only 42 hours on the motor. All the cushions were encased in the original plastic. It had been kept in a boathouse, which usually signals serious money. The buyer (whom I never met) had a Chinese name.

At this point, local news connoisseurs are rolling their eyes and going “Oh yeah?” Over the last few years, it’s been revealed that Vancouver, my beloved hometown, has been one of the world headquarters for the laundering of, uh, “funds of questionable origin”. In quite a few cases, those questionable origins have been located in China, and by elaborate mechanics that I don’t fully understand, processing them through Vancouver real-estate transactions and casino gambling and luxury car sales have made them clean. You want details, go follow Sam Cooper.

I bet no regulations whatsoever get in the way of buying a nice brand-new yacht with a duffel bag full of $100 bills and, well, there’s a significant chance that I personally helped launder some money. Live and learn.

But hey, possibly the guy snapped up the boat and then got busy at work or his wife hated it or his kids were seasick or business took a bad turn and he needed the cash, could all be perfectly legit. These are times that cultivate suspicion.


That stands for “Working From Boat” and involves a lifestyle problem. Which is, I’ve been getting tired of going to work every day, and toying with thoughts of retirement. I really enjoy my job and like the people there, but there are days when the office palls. Working from home, in moderation, is perfectly OK at Amazon but isn’t really an option for me, because our house only has one office, occupied by the world headquarters of Textuality and its CEO Lauren Wood.

I’ve only had the new boat a few weeks and, well let me tell ya, the prospect of an afternoon or two a week WFB pushes the retirement option somewhat off the front burner. In fact, a majority of my job is talking with people. But there’s still time during which I’m reviewing docs or code, writing docs, or even (*gasp*) writing code. I think the boat is going to be just the ticket. The marina WiFi is only OK so I’m looking at alternatives.

On top of which, basic civic-marina moorage is cheaper than office rent.

What about “Cottage Life”?

Oh, right, the point of having a boat was access to our island retreat. A few days back, we got word that Hanson Land & Sea had our dock and ramp and float all reconnected and, since they’d probably like to be paid, it was pretty urgent that we take a look. Potential problem: My 89-year-old Mom was visiting. But the new boat (which had never been to the island at this point) is quiet and comfy, right? So we loaded up all three generations and took off.

Lunch with Mom overlooking the Pacific, and a slow walk in the forest. Ahhhh…

Forest on Keats IslandForest on Keats Island

Listen to the trees.

(By the way, the new ramp, and the work hooking everything up, seem hunky-dory, so I can recommend Hanson if you need work done in Howe Sound.)

And then, look what Lauren found on our deck. Sad, of course, but what a wondrous piece of work. [Update: Someone on social media argues that by this time of year, the nest’s work is done and since the birds will shed no tears at its fall, neither should we.]

Fallen birds-nest

If you’re in Vancouver, ping me and drop by sometime for a cup of tea at my (occasional) waterfront office.

Auntie Beth’s Present 29 Jun 2019, 9:00 pm

My Aunt Beth died a few weeks ago. Her real name was Bertha Marian White (née Scott), here’s her obituary. I was close to her when I was a kid; she was an awfully nice person, and I’m sad. But she’d been fading for years, and in the way of death these days, Beth the person we knew pre-departed the spark of life in her body. One reason we loved her is she always gave the best birthday presents, and she did that again one last time this month.

Here’s a 1986 picture of Beth’s and my branches of the family.

Beth White and Jean Bray and their children in 1986

I’m not going to Net-publish the names of living people unless I’m pretty sure they’re OK with that. Beth is in the blue dress; beside her in glasses my Mom Jean Bray. Beth’s living sons are behind her, either side; the elder is Bill White, who we’ll be hearing more from. The woman in the front married Beth’s youngest and is holding Beth’s first grandson, who is now online at Henry White Music. At the right side of the picture, with glasses, is Beth’s now-deceased husband Ralph White, one of twelve siblings of whom ten are still living.

I’m in the red sweater and my brothers are beside me in white and with a beard in the back row. The woman in black was married to my bearded brother.

That birthday present

Beth’s memorial was on my birthday. It was so great to get together with family that, although I’m only one province over, I seldom see. What a great birthday present; thanks Beth! (And Myra, who organized it.)

Beth was a big talker and a fabulous cook, eccentric in her beliefs and habits. But I was just a kid; it never dawned on me that her regularly staying up till two in the morning on one project or another was because she was running the place, probably — this was the Sixties — with only moderate support from her all-male family.

At the memorial, I learned there were a couple of seriously tough periods when the ends really had to be stretched to meet. I’d never noticed, I was just a kid. And she lost her son Dan, who was a year older than me and I was pretty close to. He was shot by a drunk with a hunting rifle in a parking lot outside a bar. He’d been a party animal and a star athlete and I’m pretty sure he would have gone far in this world.

Beth was never ever without a smile. Now I wonder what they cost her.

The memorial event

It was in Beth’s youngest’s big house in Balzac, Alberta. That branch of the family has had five kids, of whom several are now reproducing. In the first picture Henry White, whom we saw as a baby above, sings Abide With Me, his sister accompanying. Good voice! Henry, an active Christian, also gave us a reading from Scripture (1 Corinthians 4:7-15, James 4:13-15, and Romans 5:18) and a few observations on it. Also there were personal memories and biographical notes from several of Beth’s grandkids,

Aunt Beth’s grandchildren perform at her funeral

Then Bill White, Beth’s oldest son, got up and gave us an extended tour through Beth’s life, throwing color on the facts, finding humor and sorrow. It was a masterful piece of work.

Aunt Beth’s eldest son speaks at her funeral

Then we said the Lord’s Prayer (my kids wouldn’t even know the words) and sat down with tea and good things to eat.

I’ve grown into another space — multicultural, coastal, technical — but this is my birth tribe, comfortable in their skins on the Prairies. We didn’t talk much about pipeline politics or theology, but I enjoyed every minute with every one of them. Thanks everyone.

Go Creeping In 12 Jun 2019, 9:00 pm

I’ve seen the inside of the Google and Amazon tech stacks. There are common threads that run through them and also, I bet, through most BigTechCos. Here and there down the stack is a lot of C++ and vestigial remnants from earlier days, Perl or PHP or whatever. Out in front of humans, of course, JS. But in between, there are oceans and oceans of Java; to a remarkable degree, it runs the Internet. Except for, here and there, you find a small but steadily increasing proportion of Go.

Golang gopher

If you want to know what’s going on at Google, go follow Brad Fitzpatrick. If you want to know what’s going on at Amazon, I shouldn’t spill those beans without asking for permission, which I’ve never been good at. But I can write about what I’m hearing and seeing when I look around, both inside here and out there on the Internet.

I don’t know of any co-ordinated campaigns, here or anywhere else, aimed at walking away from Java or encouraging Go (or any other replacement) in a top-down way. I do notice good engineers just going ahead and standing up Go-based microservices.

There are a bunch of reasons for this, and lots of smart people have written wise words on the subject. But here are my perceptions.


I initially fell in love with Ruby because other people’s code was just easier to read than anything else I’d previously encountered. You needed to learn what how blocks work and what |foo| means, then it all just fell into place. Then when, based on that, you figured out Enumerable, you might have had heart palpitations.

Go takes it a step further. You need to get used to type declarations being backward and how interfaces work. Then when you learn about channels and goroutines, you might experience shortness of breath.

It’s amazing —amazing I say — how little generics are missed. To date, Go remains the small, simple language that fogies like me can remember Java being. I suppose that can’t last, but for now, I can pop open almost any .go file and if I can’t understand it pretty quick, the chances are very high that the problem is in the code not me.


The Go runtime is garbage-collected, but the GC design is consciously optimized to be predictable and not induce latency; here’s a nice deep-dive. There’s no free lunch, so that excellent latency probably carries a price in throughput. Which for a whole lot of online services is a good bargain.

Performance isn’t a simple subject. But there’s a perception among insiders that Go’s performance is good enough and its latency is low enough. Furthermore, that you can expect pretty similar numbers for P50 and P99.9 latencies. And a Go program starts up fast, which we in the Serverless tribe really like.


They make it easy and idiomatic to arrange that some parts of your computation be done in parallel with other parts. And unlike other concurrency frameworks I’ve fought with, you can pretty well just fling a (potentially huge) number of tasks at goroutines, and empirically, the runtime does a good job of keeping the cores busy and the work flowing through.

And (assuming a little care with buffer sizes) it’s very unlikely that you’ll get a deadlock or an annoying race-condition bug. I’ve never had one and I’ve seen plenty in certain other languages beginning with “J”.


The fact that Go generates statically linked binaries warms greybeards’ hearts, but I’ve noticed the young pups seem to like that too. And the first time I realized I could type GOOS=linux go build on my Mac and run the output as a Lambda function I grinned from ear to ear.

Speaking of Lambda, the Go runtime has pleased me every time I’ve tried it. Also I built a custom Lambda runtime in Go and that worked great on almost the first attempt, which impressed the hell out of me.

The future?

Nope, no language is, the future is obviously polyglot. But it’s a tool I’m turning to a whole lot, and I’m not the only one.

Micromobility 9 Jun 2019, 9:00 pm

This buzzword has been echoing round the corners of Net conversation, not loud yet but the voices are those that have seemed smart in the past. I joined in a few months back by acquiring a Super Commuter+ 7 e-bike from Trek Bikes. Count me among the converted. I concluded what will probably be the last episode of my Jaguar Diary with “It makes me happy… but a new car isn’t a life-changer”. Well, I’m here to tell you that an e-bike is. And I suspect this whole Micromobility thing has legs.

Trek Commuter+ 7 e-bike and Jaguar I-Pace electric car

Two electric vehicles.

I’ve biked to work intermittently since I started at AWS in late 2014. But I’m fickle and wimpy. My route home has a continuous sixteen-block uphill segment, and it really hurts if you’re not pretty fit. At my age, you lose fitness faster and regain it slower. So if I went on a road-trip or got a bad cold or we had heavy snow and I didn’t cycle for a few weeks, I was back in sixteen-blocks-of-pain territory.

My doctor and my wife both said they thought the e-bike would be a good idea, and then I kept reading things on the Net about dubious velo-heads being won over. Some of those discussions included the experience of inhabiting an older body that struggles for fitness. Everyone seems to think that the exercise benefits, while not up to those you get from real do-it-all-yourself-biking, are still significant.

What it’s like

It’s important to understand that you don’t sit there motionless and cruise along like on a scooter or motorbike. If you don’t pedal, you don’t go. If you pedal harder, you go faster. The power design is smoothly intuitive; you hardly ever actually feel the electric assist directly. But for any given amount of pedaling pressure, you go a lot faster than you would on an unassisted bike. Yeah, the uphills still hurt, but less; also the pain ends faster.

It’s got ten gears and four boost levels: Eco, Tour, Sport, and Turbo. I find myself leaving it on Eco, sometimes switching to Tour for those sixteen blocks, but using the gears a lot, maybe more than on a regular bike. There’s a nice little Bosch Purion computer, where by “computer” I mean a speedometer and boost control. The boost stops working at 35km/h, which is dead easy to hit on level ground or going downhill.

It comes with a 110V AC charger which I need to use every week or two. I have no idea how long it takes to charge, but the battery’s full in the morning. You can detach the battery and take it inside if that makes charging easier.

The write-ups talk about how you can cruise into work and arrive fresh as a daisy, no shower needed. I dunno, I get a little sweaty but then it’s geek-informal where I work, if a suit were involved a shower would be in order. On the long uphill road home, I get plenty overheated.

The Commuter+ 7 is a heavy thing with a bulging battery, a fat frame, and fatter tires, which make for a comfy ride and cushion pothole punishment. When you turn off the boost, it’s a klunker. Also, I have panniers on the back, I drop my computer into a sleeve and the sleeve into the pannier, and arrive at work sans backpack; an oddity in geekville.

Because of the panniers, I’ve given up taking the car on almost all local shopping trips. The bike gets there about as fast, I can park it right in front of any store, and with the panniers I can carry along quite a few groceries and still have room for some beers.

But how does it feel?

It feels wonderful! My commute, which is almost exactly 4km, takes me under twenty minutes from my front door to my desk at work, a bit more or less depending on whether I make or miss traffic lights. It helps a lot that Vancouver has pretty good (and getting better) bike-route infrastructure.

Vancouver bike map legendCentral Vancouver biking map

Extracted from the official City of Vancouver cycling map.

My commute is between near the bottom center and near the top center on that map. There are only a few blocks where cars and bikes are sharing street space as “equals”: the two between my home and where I get on the bikeway, and one block that happens to be right outside the main central-city police station. The result is I feel safe. Having said that, the one time I got hit by a car — badly, with ensuing hospital time and surgery, in 2000 — was when a stopped car suddenly lurched forward into a crosswalk; so you’re never 100% safe.

The big safety problem is the cool downhill parts of the route, where I (and my cycling-commuter peers) go like hell. That route also includes leafy residential hoods and a bridge over the ocean. It’s really pretty awesome and, as in many other ways, I’m a lucky guy.

(I do experience a certain amount of guilt while blowing by people who are obviously fitter than me just because I’m e-assisted and they’re self-powered.) (But I can learn to live with it.)

The economics

An electric bike isn’t cheap - this thing lists at $3,800 US. There are cheaper e-bike choices, but also way more expensive ones. Public transit would cost about a thousand a year and takes nearly twice as long to get there. My car is electric and thus (ignoring capital cost) close to free at these ranges, but then parking is $150/month or so if you sign up for the whole month, and $15/day and up a-la-carte. Bike parking is so far one of life’s few free offerings.

Then there are the health benefits from 40 minutes of moderate cardio workout per day, and the emotional win of spending no time either squashed into a packed train or sitting alone in a traffic-jammed auto. When you’re biking you’re moving, except for those damn red lights; we hatessss them, my precioussss.


This is the broader category of which e-bikes are a member, but also includes scooter variations and then the variegated two- and three-wheelers, mostly electric, I saw in Beijing.

One of the voices arguing that all this is A Big Deal is that of Horace Dediu, a long-time commentator on mobile tech and Apple, who invented the term Micromobility and organizes conferences on the subject.

Is it really a big deal? It seems to scratch humans’ built-in get-there-faster itch while paving a whole lot less of paradise. And, also as Horace says:

Horace Dediu on Micromobility

My mind is open. I have a hunch that e-bikes will loom large among micromobility choices; compared to stand-up options like scooters and Segways, they’re a little safer and a little faster. Also, they embody technologies that’ve been refined since the dawn of the bicycle in the early 1800s, and continue to evolve.

Anyhow, if you’re an urban traveler I strongly recommend trying one out.

On SQS 26 May 2019, 9:00 pm

In my position I probably shouldn’t have a favorite AWS product, just like you shouldn’t have a favorite child. I do have a fave service but fortunately I’m not an (even partial) parent; so let’s hope that’s OK. I’m talking about Amazon Simple Queue Service, which nobody ever calls by its full name.

I’d been thinking I should write on the subject, then saw a Twitter thread from Rick Branson (trust me, don’t follow that link) which begins Queues are bad, but software developers love them. You’d think they would magically fix any overload or failure problem. But they don’t, and bring with them a bunch of their own problems. After that I couldn’t not write about queueing in general and SQS specifically.

SQS banner

SQS is nearly perfect

The perfect Web Service I mean: There are no capacity reservations! You can make as many queues as you want to, you can send as many messages as you want to, you can pull them off fast or slow depending how many readers you have. You can even just ignore them; there are people who’ll dump a few million messages onto a queue and almost never retrieve them, except when something goes terribly wrong and they need to recover their state. Those messages will age out and vanish after a little while (14 days is currently the max); but before they go, they’re stored carefully and are very unlikely to go missing.

Also, you can’t see hosts so you don’t have to worry about picking, configuring, or patching them. Win!

There are a bunch of technologies we couldn’t run at all without SQS, ranging from to modern Serverless stuff.

The API is the simplest thing imaginable: Send Messages, Receive Messages, Delete Messages. I love things that do one thing simply, quickly, and well. I can’t give away details, but there are lots of digits in the number of messages/second SQS handles on busy days. I can’t give away architectures, but the way the front-end and back-end work together to store messages quickly and reliably is drop-dead cool.

Why not entirely perfect? Well, SQS launched in 2006. Most parts of the service have been re-implemented at least once, but some moss has grown over the years. I sit next to the SQS team and know the big picture reasonably well, and I think we can make SQS cheaper and simpler to operate.

When it launched it cost 10¢ per thousand messages; now it’s 40¢ per million API calls. “Per-message” can be a bit tricky to work out because sending, receiving, and deleting makes three calls per, but then SQS helps you batch and most high-volume apps do. Anyhow, it’s absurdly cheaper than back then, and I wonder whether, in a few years, that 40¢/million number will look as high as 10¢/thousand does today.

The opposition

So let’s go back to Mr Branson’s tweet-rant. He raises a bunch of objections to queues which I’ll try to summarize:

  1. They can mask downstream failures

  2. They don’t necessarily preserve ordering (SQS doesn’t).

  3. When they are ordered, you probably need to shard to lots of different streams and keep track of the shard readers.

  4. They’re hard to capacity plan; it’s easy to fill up RAM and disks.

  5. They don’t exert back-pressure against clients that are overrunning your system.

Here’s his conclusion.

Anti-queue closer from Rick Branson

While there are good queues, I agree with his sentiment. If you can build a straightforward monolithic app and never think about all this asynchronous crap, go for it! If your system is big enough that you need to refactor into microservices for sanity’s sake, but you can get away with synchronous call chains, you definitely should.

But if you have software components that need to be hooked together, and sometimes the upstream runs faster than the downstream can handle, or you need to scale components independently to manage load, or you need to make temporary outages survivable by stashing traffic-in-transit, well… a queue becomes “absolutely necessary”.

The proportion of services I work on where queues are absolutely necessary rounds to 100%. And if you look at our customers, lots of them manage to get away without queues (good for them!) but a really huge number totally depend on them. And I don’t think that’s because the customers are stupid.

Mr Branson’s charges are accurate descriptions of queuing semantics; but what he sees as shortcomings, people who use queues see as features. Yeah, they mask errors and don’t exert back-pressure. So, suppose you have a retail website named after a river in Brazil, and you have fulfillment centers that deliver the stuff the website sells. You really want to protect the website from fulfillment-center errors and throttling. You want to know about those errors and throttling, and a well-designed messaging system should make that easy. Yeah, it can be a pain in the butt to capacity-plan a queue — ask anyone who runs their own. That’s why your local public-cloud provider offers them as a managed service. Yeah, some applications need ordering, so there are queuing services that offer it. Yeah, ordering often implies sharding, and so your ordered-queue service should provide a library to help with that.

But wait, there’s more!

More kinds of queues, I mean. AWS has six different ones. Actually, that page hasn’t been updated since we launched Managed Streaming for Kafka, so I guess we have seven now.

We actually did a Twitch video lecture series to help people sort out which of these might hit their sweet spot.

With a whole bunch of heroic work, we might be able to cram together all these services into a smaller number of packages, but I’d be astonished if that were a cost-effective piece of engineering.

So with respect, I have to disagree with Mr Branson. I’d go so far as to say that if you’re building a moderately complex piece of software that needs to integrate heterogeneous microservices and deal with variable sometimes-high request loads, then if your design doesn’t have a queuing component, quite possibly you’re Doing It Wrong.

Page processed in 0.093 seconds.

Powered by SimplePie 1.5, Build 20180814171753. Run the SimplePie Compatibility Test. SimplePie is © 2004–2019, Ryan Parman and Geoffrey Sneddon, and licensed under the BSD License.