ongoing by Tim Bray

ongoing fragmented essay by Tim Bray

Jag Diary 11: A Year In 12 Jan 2020, 10:00 pm

The I-Pace insurance renewal showed up so I knew a year must have (so quickly!) gone by. I’d thought I should write something to draw a line under this diary, and then just now the car saved my life, so now’s the time. Not much new information here but maybe a couple of smiles.

Jaguar I-Pace

The insurance — two drivers, good records, no commuting — cost around two and a half thousand Canadian dollars. That year’s other car expenses put together (electricity, replacing a vandalized window) are a tiny fraction of the insurance. Electric cars, even with sports-car performance like the Jag’s, are stupidly cheap to run. I think that as electrics replace fossil vehicles slowly then quickly, the world will need fewer automotive mechanics. Paint and body and glass shops will be fine, but there’s not much to be done under the hood.

Riding the rain

Next time you’re driving, consider where the joules that move your car came from. For most of you, the story involves a whole lot of heavy machinery digging holes and pumping sticky black crap out of the planet and then turning it into auto fuel in a process that really fucks up the atmosphere near the refinery, and then requires you to pull into a loveless harshly-lit concrete enclave to stand beside a pump staring blankly while dozens of dollars flow out of your bank account into an oil-and-gas company’s, so you can turn travel hours into outflows of planet-killing CO2.

Our car is rain-driven. Well, it’s partly snow, but anyhow the clouds come in off the Pacific and crash into the mountain ranges and dump their sun-elevated droplets, frozen or not, and eventually the water flows down and into dammed-off valleys and through turbines, most built decades back, and eventually the electrons end up in the 240V charger behind our house where I plug the car in overnight every week or two and invest less than $10 to fill it up while I sleep.

I think I’m winning this one.

Jaguar I-Pace

Jag encounter

Late last summer in the long dusk of a bright day, we were heading home and I found myself queued up for a tricky west-to-south left turn at a big old intersection (for Vancouver cognoscenti, 12th to Main).

The twilight made all the cars and people and buildings look great. Traffic was heavy and the pause was considerable; while hanging out there, I noticed the first northbound car waiting for the light and for me to be gone was a beautiful old silver Jag sedan, I think an XJ, gleaming like a long low jewel in the slanting sun. I’m inclined to think that our Caesium-blue electric looked pretty glamorous too just then, from outside. I saw the dude in the other Jag was looking at mine too, then just as the traffic opened I met his eyes and then I curved round him while we shared a big Jaguar smile.

Snow champ

Did I say above “saved my life”? Quite likely. Sunday night we had a nasty snowstorm, which causes problems in Vancouver because there are a lot of hills and also a lot of people who rarely if ever drive in snow and are mostly perfectly OK folk but just don’t have a clue how to deal. We had an old friend over for dinner who came by bus; looking outside after, I said “I better take you home.”

Which may have been an error in judgment. On the hills, the sideways victims included not just the usual over-powered minivans but more than one city bus, so it wasn’t just a casual dusting.

Jaguar I-Pace

The Jag weighs over two metric tons (a third or so battery) and has big wide wheels and a snow-and-ice four-wheel mode so it laughs at this stuff. I was gentle with the juice and slow on the hills and yeah, I had to dodge one swiveling Acura, but it was OK.

After I dropped Gareth off downtown, I headed for a left turn onto a big one-way downtown-access road (for Vancouver cognoscenti, Citadel Parade to the Dunsmuir viaduct). As I approached, the light turned green so I was starting to aim left when this huge van hurtled along the road I was trying to turn onto. He’d clearly sped up to beat the yellow and then foolishly tried too late to stop; he must have been doing 60 km/h or more, and mostly sideways. I hit the brake hard and maybe the ABS implementation is a little rough but that big cat shuddered to a stop right freaking there in the snow, hardly rotating at all. OK, maybe I wouldn’t have died; but it would have been seriously ugly. Mea culpa; I should have pulled up to a stop at the line, not trusting the green light on such a night.

On the way home among the amateur-hour chaos in the snow the music shuffle switched over to Hildegard von Bingen and boy, did that ever hit the spot.

The Jaguar I-Pace is a good car. Mind you, pretty well all the modern electrics are; in the big picture, fossil propulsion is done for. But even given all that, it’s really awfully good; at keeping me alive and making me smile.

Horopito 31 Dec 2019, 10:00 pm

It’s a place near the center of New Zealand’s North Island; we spent the last Christmas of the decade there. It’s considered remote in NZ which I guess makes it doubly so in the wider world. Unless you’re planning southern-hemisphere skiing you’re unlikely to go near it, so I felt words and pictures worth sharing.

What happened was, Lauren’s brother and his wife who live in Auckland came to visit us and we took them to our cabin; they liked the country-retreat concept so much that they bought an acre in Horopito and built a house on it. This was finished and moved-into on December 18th and in a flash of insane courage, they invited their three children, two children’s spouses, and five grandchildren along. And us. Lauren and I have been together for more than twenty years and have spent every Christmas with my family, so it was about time hers got a turn.

So it was off to New Zealand for us. Now, as an environmentalist loudmouth should I feel bad about jetting between the hemispheres? I should, and I do. But there’s that twenty-years thing and also Lauren’s Dad is getting to a state of age and health where we may not see him again. We’re obviously going to have to slash the amount of mass jet travel; I would hope that the global carbon budget can sustain trips made mostly for love, but I don’t know. I may never pull that transcontinental trigger again.

Horopito was once a lumber boomtown, but the town’s pretty well gone. This post office was closed in 1970.

Horopito Post Office

It has been pointed out to me that this picture would be suitable for an indie album cover. Any band that wants to use it may do so but you have to send a few bucks to the little girl, whom I know.

Horopito isn’t famous for anything but it’s near some well-known things, most notably Mount Ruapehu, the North Island’s tallest at 2,797m (9,177 ft) . (Actually, Ruapehu is part of a complex that includes Tongariro and Ngauruhoe (which Peter Jackson had stand in for Mount Doom.)) There are any number of glamorous pix of Ruapehu and its posse, but here’s a look out the front of the family property during complicated weather — most NZ weather is complicated.

Mount Ruapeho in clouds

We took two trips up the mountain. Here are views from the Whakapapa side:

View from near WhakapapaView from near Whakapapa

And here’s another looking southwest.

View southwest from Ruapehu

New Zealand is full of nice scenery, whichever way you look.

Smash Palace!

That’s all you need to say to explain where Horopito is to any New Zealander with even one or two grey hairs. It’s officially Horopito Motors, “the largest and only vintage car dismantlers in Australasia”. It’s really impressive in the satellite view. “Smash Palace” refers to a 1981 NZ-made movie in which the wrecking yard features; I’ve actually seen it and it’s not terrible.

smash PalaceSmash Palace

Of course, the main point was getting the family together. In a Southern-hemisphere Christmas, the turkey and ham at dinner is served cold, and then after digesting a bit, people retire to the lawn to toss frisbees and rugby balls and generally run around:

Christmas festivities

One evening the sky was crystal clear and a few of us stargazed. I didn’t take the trouble to figure out how to unleash the Pixel 4 astrophotography mode, but I did point it at the southern skies. On that half of the globe, Orion is upside down…

Southern-hemisphere Orian

… and looks like a shopping cart.

Also, what I think is the Southern Cross, first time I’ve seen it. The Pixel didn’t capture the Milky Way, perfectly evident behind the Cross, to the naked eye.

Maybe the Southern Cross

If you have a large loving family to welcome you, I think Horopito is a fine place to visit, and the quality of light is special.

Clouds over Horopito

I missed my own ancestral family, meeting for a Prairie Christmas back home in Canada. (But appreciated the fact that I was thirty or so Celsius degrees warmer.) And I’m so thankful and heartwarmed by the hospitality and love, effortlessly given in acre-sized servings by Mary and Martin and the rest of the Wood clan.

Content-based Filtering 18 Dec 2019, 10:00 pm

The publish/subscribe pattern is central to data in motion — event-driven and messaging-based apps, I mean. I’m increasingly convinced that pub/sub software just isn’t complete without some sort of declarative filtering technology, so that you can subscribe to a huge shared torrent of data and only see the parts of it that you need to process. You could look at everything and write code to reject the data you don’t care about, but it’s nice to write a declarative rule and have the system take care of the filtering for you.

This piece is about data-filtering technology we’ve been cooking up at AWS, and that I’ve personally put a whole lot of work into. The proximate cause for publishing now is that while this feature has been around for a while in the old CloudWatch Events and in SNS, we’re just rolling out all the latest bells and whistles in EventBridge. I want to write about it because it’s different enough from other filtering technologies to be interesting.

EventBridge’s events are delivered in JSON, but this tech ought to apply to any nested JSON-like structured data. The syntax is called “Event Patterns”, and the idea is that the filters don’t look like SQL or really any other popular query language, they look like the events they’re filtering.

To make this concrete, let’s look at a typical event you might encounter on EventBridge:

  "version": "0",
  "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718",
  "detail-type": "EC2 Instance State-change Notification",
  "source": "aws.ec2",
  "account": "111122223333",
  "time": "2017-12-22T18:43:48Z",
  "region": "us-west-1",
  "resources": [
    "arn:aws:ec2:us-west-1:123456789012:instance/ i-1234567890abcdef0"
  "detail": {
    "instance-id": " i-1234567890abcdef0",
    "state": "terminated"

In the following sections, all the examples will match this event.

Event Patterns have the same structure as the Events they match

Suppose you wanted to subscribe only to events from EC2. Here’s the Event Pattern:

  "source": [ "aws.ec2" ]

The pattern simply quotes the fields you want to match and provides the values you are looking for.

The sample event above, like most events, has a nested structure. Suppose you want to process all instance-termination events:

  "source": [ "aws.ec2" ],
  "detail-type": [ "EC2 Instance State-change Notification" ],
  "detail": {
    "state": [ "terminated" ]

Only specify the fields you care about

In the example above, you only provide values for three fields: The top-level fields “source” and “detail-type”, and the “state” field inside the “detail” object field. EventBridge ignores all the other fields in the event while applying the filter.

Match values are always in arrays

Note that the value to match is in a JSON array, that is to say surrounded by “[” and “]”. This is so that you can provide multiple values. Suppose you were interested in events from EC2 or Fargate:

  "source": [ "aws.ec2", "aws.fargate" ]

If the value of the “source” field was an array, that would work too: The pattern would match if the intersection between the pattern array and the event array (both treated as sets) was non-empty.

Ands and Ors

The filter language is a bit surprising in that when you provide multiple possible matches, as immediately above, that’s an OR operation — you match if any one of them does. But when your pattern has multiple fields, that’s an AND because all of the fields have to match. Oddly, this seems to meet people’s needs quite well. And if you really need OR’ed fields, you can post two filters, one with each of the options.

You can match all the JSON data types

Consider the following Auto Scaling event:

  "version": "0",
  "id": "3e3c153a-8339-4e30-8c35-687ebef853fe",
  "detail-type": "EC2 Instance Launch Successful",
  "source": "aws.autoscaling",
  "account": "123456789012",
  "time": "2015-11-11T21:31:47Z",
  "region": "us-east-1",
  "resources": [
  "detail": {
    "eventVersion": "",
    "responseElements": null

You can match the “responseElements” field as follows:

  "detail": {
    "responseElements": [ null ]

This works for numbers too. Consider the following Macie event (truncated for brevity, and pardon the klunky line breaks, need to make my CSS smarter about code):

  "version": "0",
  "id": "3e355723-fca9-4de3-9fd7-154c289d6b59",
  "detail-type": "Macie Alert",
  "source": "aws.macie",
  "account": "123456789012",
  "time": "2017-04-24T22:28:49Z",
  "region": "us-east-1",
  "resources": [
  "detail": {
    "notification-type": "ALERT_CREATED",
    "name": "Scanning bucket policies",
    "tags": [
    "url": "",
    "alert-arn": "arn:aws:macie:us-east-1:123456789012:trigger/trigger_id/alert/alert_id",
    "risk-score": 80,
    "trigger": {
      "rule-arn": "arn:aws:macie:us-east-1:123456789012:trigger/trigger_id",
      "alert-type": "basic",
      "created-at": "2017-01-02 19:54:00.644000",
      "description": "Alerting on failed enumeration of large number of bucket policies",
      "risk": 8
    "created-at": "2017-04-18T00:21:12.059000",
    // truncated for brevity
    . . . 

Suppose your security policies require you to react when Macie reports anything with a risk score of 80 and a trigger risk of 8:

  "source": [ "aws.macie" ],
  "detail": {
    "risk-score": [ 80 ],
    "trigger": {
      "risk": [ 8 ]

Numbers work properly

While the pattern above works, it doesn’t work that well, because it only matches against the JSON exactly as stated. So for example, if the programmer generating that Macie event changed their code so that it emitted "risk-score": 80.0, the rule wouldn’t match.

Fortunately, EventBridge has numeric matching built-in. This would allow you to implement security policies much more flexibly and reliably. For example, here’s a pattern that matches a trigger risk value of 8 (even if it’s expressed as “8.000” or “8.0e0”, and any risk-score value over 50 but less than or equal to 100.

  "source": [ "aws.macie" ],
  "detail-type": [ "Macie Alert" ],
  "detail": {
    "risk-score": [ { "numeric": [ ">", 50, "<=", 100 ] } ],
    "trigger": {
      "risk": [ { "numeric": [ "=", 8 ] } ]

This kind of numeric matching is useful, but is limited to values between -1.0e9 and +1.0e9 inclusive, with 15 digits of precision, that is to say six digits to the right of the decimal point.

Note that match expressions go into arrays just like literal values. Match expressions and literals can be mixed up as much as you want.

Prefix matching

Suppose you want to process all the auto-scaling events from AWS’s European regions. There’s a match expression for that.

  "source": [ "aws.autoscaling" ],
  "region": [ { "prefix": "eu-" } ]

IP address matching

For one reason or another, it’s not that uncommon to encounter IP addresses in event fields. Since the CIDR notation was explicitly designed to match IP ranges, it works well as a filter syntax:

  "caller-ip": [ { "cidr": "" } ] 

It works with IPv6 too!

  "caller-ip": [ { "cidr": "2001:db8::/120" } ] 

(Confession: I’ve never seen one of these in the wild in our event ecosystem.)

(Confession: I don’t think the CIDR capability has quite finished deploying as of the publishing of this blog.)

Existence Matching

Suppose you wanted to make an ElasticSearch full-text index of a bunch of events. To do this, you might want to select all the events that have a description field:

  "detail": {
    "description": [ { "exists": true } ],

You could also use { "exists": false } to select events that don’t contain some particular field.

Anything-but matching

Sometimes you want to exclude rather than include a particular field value. Suppose you want to process all the events except those that are CloudTrail reports of API calls:

  "detail-type": [ { "anything-but": "AWS API Call via CloudTrail" } ]

The anything-but match expression can blacklist literal strings or also a list of values, but the list has to contain either all strings or all numbers. Suppose you wanted to see all the events except those that came from EC2 or S3:

  "source": [ { "anything-but": [ "aws.ec2", "aws.s3" ] } ]

The anything-but match expression can also use a nested match expression to exclude prefixes. For example, EventBridge’s main event bus has a huge number of events coming from all the AWS services, but you can also inject your own events using the PutEvents API. You can distinguish AWS’s events and process only your own because the “source” field in all the AWS events begins with the string “aws.”.

  "source": [ { "anything-but": { "prefix": "aws." } } ]

The future?

People seem to like this idiom a lot — EventBridge has a huge number of customers. Also, we’ve got a super-efficient implementation currently processing an immense number of events per second, that I’d like to open-source. We keep getting requests for more filtering features (wildcards or regexes are an obvious direction) and have managed to keep new stuff rolling out steadily.

There’s one problem: It’s not SQL-flavored, and a high proportion of software people sort of think in SQL when they want to select data. There have been attempts to extend SQL to be a good citizen of the world of non-relational data; the one I’m most familiar with, because it’s recent and from AWS, is PartiQL.

I’m biased in that I’ve never actually liked SQL, but I recognize that this is not exactly a majority opinion. Anyhow, it’s on my mind.

The Oil Patch Up Against the Wall 15 Dec 2019, 10:00 pm

I’m terribly worried about how my children — and humans in general — are going to deal with catastrophic global warming in the likely case that the average global temperature spike is somewhere in the 2°-4°C range. I want to highlight a couple of weekend stories on the biggest news story of the twenty-first century: The tragedy of the deniers, and likely consequences for the perpetrators.

Like most literate people whose livelihood doesn’t depend directly on the fossil-energy industry, I believe the evidence is overwhelming that anthropogenic atmospheric CO2 overload has a strong greenhouse effect whose results are already visible in receding ice and flurries of “hundred-year” natural disasters.

I’m also convinced that we can move the needle further and faster, with less economic dislocation and pain, than many others believe. The energy-economics picture has been changing so fast that unless you’re paying close attention and are open-minded about new energy sources, you might think it’s reasonable to doubt the plausibility of wholesale replacement of fossil-fuel-based power generation by renewables.

Current energy pricing trends

These are unsubsidized prices. Unfortunately, the graph doesn’t include recent trends in energy storage pricing, which are good if not quite this dramatic; see for example Power storage is the missing link in green-energy plans from The Economist.

Yes, it would require massive investment on a wartime-like scale, but Lord knows there’s plenty of surplus capital out there looking for a profitable home; just consider how effortlessly SoftBank has raised tens of billions to squander on lies and fantasies. It’s pretty obvious that “carbon disinvestment” has become a prudent mainstream financial strategy, and that the renewable-generation sector is the single biggest and best investment opportunity of the next few decades, with paybacks to be had like those harvested during the rise of the Internet.

If you’re interested in modern energy economics (and you should be) a good place to start is with Gregor Macdonald, who stays on top of the numbers and is particularly good on what’s happening in China, and with electric vehicles.

Tragic Alberta opera

Consider this CBC story: Alberta wants to flip the script in oilpatch’s favour — it won't be easy.

A few words of background: Alberta, Canada’s second province from the left, has the nation’s highest average income, no sales tax, and generally fabulous social services, all in large part based on its petroleum revenue. Until some point in my forties, I was still getting a few bucks a month in royalties from the natural-gas well on the old family farm. As a result, Upton Sinclair’s famous soundbite applies to Alberta in spades: “It is difficult to get a man to understand something, when his salary depends on his not understanding it.”

Bearing all this in mind, I found the language about Alberta’s “war room”, whose objective is to “shift unfavorable views of the oil and gas sector”, oddly evocative. Here we have a chorus composed of successful members of a rich society’s elite singing a chorus that nobody believes, outside of a small and shrinking fraction of society. They inveigh against outsiders who are impugning their industry: “these same organizations trying to misinform prospective investors about our environmental performance” and “the political agenda emanating from Europe, which is trying to stigmatize development of hydrocarbon energy”.

One feels that there’s material for an opera or formal Noh play in these people fighting what seems to them like the good fight, all they want to do is protect their home-towns’ livelihoods. And they’re not wrong: A whole lot of people all over the world, including me, are indeed stigmatizing the development of hydrocarbon energy. Where by “stigmatize” we mean “slam on the fucking brakes before we wreck the world.”

By the way, the “war room” has terrible Google juice despite its $30M budget, but I managed to turn it up: Welcome to the Canadian Energy Centre, an alternate reality where you fight climate change by pumping high-carbon tar-sands crude.

The experience of the crisis

To most people, it all feels so abstract; Bad things are going to happen, but to other people a long way away, and it’s not obvious what it’ll be like. Here, in Paolo Bacigalupi’s beautiful, terrible story A Full Life (interestingly, published in MIT Technology Review) is what it will feel like for a very ordinary young person in the heart of America. Seriously, if the climate crisis feels a little abstract to you, read this. It’s more important than my words that you’re reading now.

Up against the wall!

Which brings me to this little two-tweet sequence from Greta Thunberg:

Tweet from Greta Thunberg

Ms Thunberg says her message is 100% non-violent and I believe her. But when the water levels start rising; when a hundred million people start walking north, empty-handed and hungry, out of the Bengal lowlands; when Mar-a-Lago is smashed and it wasn’t even an official hurricane; when California’s inland crops fail; when the fires burn a third of Sydney; when Arizona real estate goes to zero; and especially, when some climate-change surprise nobody thought of wreaks deadly havoc in a place nobody expected, people are going to be put up against walls and not in Greta’s “Swenglish” sense, no not at all.

Alberta should really stop calling their operation a “war room”.

@bluesky 11 Dec 2019, 10:00 pm

At my day job they say “Think big!”, and @bluesky is that. I can’t stop thinking about it, and bloggers can’t help thinking in public. This work can simultaneously be probably-doomed and obviously-worthwhile.

I’m aligned with where Twitter is starting this conversation. First, speaking under my AWS-employee hat: One of our standard soundbites is that people and enterprises should strive to avoid “undifferentiated heavy lifting”, i.e. wrangling software infrastructure. The basic machinery behind Twitter needs to keep track of who said what and make it available to others. Let’s go further and grant (as I would) that Twitter exhibits the essential nouns and verbs of text-centric social media: Annotated linking, following, echoing, answering, liking, muting, blocking. Which all feels to me like undifferentiated heavy lifting.

Where’s the value?

Twitter’s @bluesky hypothesis is that the real business opportunity in social media is extracting from the planetary-scale tangle a fragment stream that meets the needs of individuals sufficiently well that their attention can be monetized, and does so without being a vector for hate, bigotry, and ignorance.

It’s easy to be dubious of this claim and still convinced the project is worth pursuing. I’m both: I persist in resisting Twitter’s pleas to replace my people-I-follow-in-order feed with their “top tweets” flavor. But self-evidently I’m weird; I have the privilege and education and space and time to obsess over certain aspects of life and their truths and falsehoods, and invest in careful curation of my feed. It’s not unreasonable for people who are starved for time and less privileged to pose a simple request to Twitter (or a competitor): “Please inform me about the world quickly, in a way that enriches my life, without leading me into pathological social dysfunction.”

The second starting point I share with Twitter: Social media as currently constituted does a terrible job. The platform works better for GamerGaters than for humanists; better for Donald Trump than for the Dalai Lama; better for incels than for working mothers. So, what is to be done? The value of the quest hardly seems open to question.

What is the Internet?

It isn’t a place or a thing, it’s a lot of computers which implement protocols written down in IETF RFCs (and to a lesser degree other organizations’ publications) that reflect inspiration and evolution and are known to achieve certain effects in practice. So in the @bluesky launch thread it’s unsurprising to find belief in protocols as foundational to the project. One of the two documents they quote is Protocols, Not Platforms: A Technological Approach to Free Speech by Mike Masnick.

I’ve read Masnick carefully but I’m mostly unconvinced. He seems still to dream the foundational Internet dream that if we can just put people in touch with each other things will be better. Plus there’s market fundamentalism: Let people choose the products that most please them and this will lead to good outcomes and to products which are relatively free of dysfunction.

Which I’d like to believe but don’t. The free market is obviously a useful macroeconomic tool, but only when its freedom is carefully circumscribed. Every market needs a supporting framework of contract law and of regulation: Against fraud, misrepresentation, theft, Ponzi schemes and of course to promote safety: Aviation safety, electrical safety, chemical safety, nutritional safety, and so on. Can a protocol support the messy political mechanisms behind our imperfect but essential legal and regulatory frameworks?

The other issue that seems insufficiently considered is organized hostile action. Empirically, we observe powerful well-funded parties launching concerted efforts to bend the path of societal conversation towards Trump or away from Jesus’ Good-Samaritan lesson; or to boost Brexit. If a protocol could be effective in resisting this sort of adversary, that would be great.

I’m aware of no conclusive evidence that this is either possible or impossible. So I’d be willing to give it a shot. But I’d also broaden the focus; if there is a social-networking technology out there that can ameliorate the current dysfunction, it seems just as likely to be found in a carefully-thought-through API framework, or in some of Stephen Wolfram’s early-stage proposals (also linked in the @bluesky thread) for a level of indirection in ML-based recommendation algorithms.

Block what?

Both @jack in his @bluesky thread and his CTO following up nodded in the direction of blockchain. Any regular ongoing reader can hear my snorts of derision from anywhere on the globe. I’m not going to relitigate the argument at this point; the absence of succesful applications of the technology, all these years into it, should have removed the need. But I think the appeal to blockchain is a symptom of a powerful instinct that whatever we do to fix social media, it has to be involve extreme non-optional transparency. (Blockchain does this but unfortunately fails to be useful for other unrelated reasons.)

I share this instinct and would go further: A powerful focus on transparency and truth is an absolutely necessary (but of course not sufficient) precondition for addressing social media’s sins.

Existence proof

There is a successful online community where truth is the common currency: Wikipedia. Yes, it’s flawed in multiple important ways, but its insistence on publishing only what’s backed by evidence, and its rough consensus on what evidence is acceptable, is at the end of the day heartwarming. Its great flaw is not inaccuracy but inattention; the community of editors is generally insufficiently diverse and specifically mostly-male, and this reflects into Wikipedia’s pages.

Wikipedia’s contract with the world is simple: Any material that is not verifiable will eventually be removed. So obviously the meaning of “verifiable” matters and is thus a source of controversy. Healthy, reasonable, sensible controversy. Which we observe to have an imperfect but highly usable result, relatively free of the currient pathologies of other civic discourse.

Am I saying that the @bluesky solution comes down to “remove anything not verifiable”? No, but I’d be inclined to suggest that “verifiable vs not” should be an important input to the @bluesky algorithms and protocols.


Suppose the @bluesky team ends up looking at a few big ideas — let’s ignore whether they’re protocols or algorithms or ML models or some other sort of thing. My first question is “how do you tell if they work?” At AWS, something relatively simple, like the performance of a throttling algorithm under stress, can be difficult to test without, you know, putting it under stress.

We’ve developed incremental nondestructive imperfect techniques for evaluating this kind of thing, and you’d need something like that to make useful decisions about @bluesky ideas. You’d need to run them at scale in the context of an automated harness, and you’d need to do it more or less all the time, forever.

The construction of this infrastructure is, I believe, apt to be one of the hardest parts of @bluesky.

Will it work?

I doubt it. Social-media dysfunction seems closely related to aspects of human nature at the individual and societal levels. A solution might end up being more or less equivalent in difficulty to World Peace or Curing Cancer.

Which is obviously not a reason to not try.

Strongly Typed Events 2 Dec 2019, 10:00 pm

Back in 2016, in Message Processing Styles, I was sort of gloomy and negative about the notion of automated mapping between messages on the wire and strongly-typed programming data structures. Since we just launched a Schema Registry, and it’s got my fingerprints on it, I guess I must have changed my mind.

Eventing lessons

I’ve been mixed up in EventBridge, formerly known as CloudWatch Events, since it was scratchings on a whiteboard. It has a huge number of customers, including but not limited to the hundreds of thousands that run Lambda functions, and the volume of events per second flowing through the main buses are keeping a sizeable engineering team busy. This has taught me a few things.

First of all, events are strongly subject to Hyrum's Law: With a sufficient number of users of an API, it does not matter what you promise in the contract: all observable behaviors of your system will be depended on by somebody. Which is to say, once you’ve started shipping an event, it’s really hard, which is to say usually impossible, to change anything about it.

Second: Writing code to map back and forth between bits-on-the-wire and program data structures is a very bad use of developer time. Particularly when the messages on the wire are, as noted, very stable in practice.

Thus, the new schema registry. I’m not crazy about the name, because…

Schemas are boring

Nobody has ever read a message schema for pleasure, and very few for instruction. Among other things, most messages are in JSON, and I have repeatedly griped about the opacity and complexity of JSON Schema. So, why am I happy about the launch of a Schema Registry? Because it lets us do two useful things: Search and autocomplete.

Let’s talk about Autocomplete first. When I’m calling an API, I don’t have to remember the names of the events or their arguments, because my IDE does that for me. As of now, this is true for events as well; the IDE knows the names and types of the fields and sub-fields. This alone makes a schema registry useful. Or, to be precise, the code bindings and serializers were generate from the schema.

The search side is pretty simple. The schema registry is just a DynamoDB thing, nothing fancy about it. But we’ve wired up an ElasticSearch index so you can type random words at it to figure out which events have a field named “drama llama” or whatever else you need to deal with today.


This is an absolutely necessary schema-registry feature that most people will never use. It turns out that writing schemas is a difficult and not terribly pleasant activity. Such activities should be automated, and the schema registry comes with a thing that looks at message streams and infers schemas for them. They told us we couldn’t call it an “Inferrer” because everyone thinks that means Machine Learning. So it’s called “schema discovery” and it’s not rocket science at all, people have been doing schema inference for years and there’s good open-source code out there.

So if you want to write a schema and jam it into the registry, go ahead. For most people, I think it’s going to be easier to send a large-enough sample of your messages and let the code do the work. At least it’ll get the commas in the right place. It turns out that if you don’t like the auto-generated schema, you can update it by hand; like I said, it’s just a simple database with versioning semantics.

Tricky bits

By which I mean, what could go wrong? Well, as I said above, events rarely change… except when they do. In particular, the JSON world tends to believe that you can always add a new field without breaking things. Which you can, until you’ve interposed strong types. This is a problem, but it has a good solution. When it comes to bits-on-the-wire protocols, there are essentially two philosophies: Must-Understand (receiving software should blow up if it sees anything unexpected in the data) and Must-Ignore (receiving software must tactfully ignore unexpected data in an incoming message). There are some classes of application where the content is so sensitive that Must-Understand is called for, but for the vast majority of Cloud-native apps, I’m pretty sure that Must-Ignore is a better choice.

Having said that, we probably need smooth support for both approaches. Let me make this concrete with an example. Suppose you’re a Java programmer writing a Lambda to process EC2 Instance State-change Notification events, and through the magic of the schema registry, you don’t have to parse the JSON, you just get handed an EC2InstanceStateChangeNotification object. So, what happens when EC2 decides to toss in a new field? There are three plausible options. First, throw an exception. Second, stick the extra data into some sort of Map<String, Object> structure. Third, just pretend the extra data wasn’t there. None of these are insane.

There’s another world out there where the bits-on-the-wire aren’t in JSON, they’re in a “binary” format like Avro or Protocol Buffers or whatever. In that world you really need schemas because unlike JSON, you just can’t process the data without one. In the specific (popular) case of Avro-on-Kafka, there’s a whole body of practice around “schema evolution”, where you can update schemas and automatically discover whether the change is backward-compatible for existing consumers. This sounds like something we should look at across the schemas space.

Tactical futures

Speaking of those binary formats, I absolutely do not believe that the current OpenAPI schema dialect is the be-all and end-all. Here’s a secret: The registry database has a SchemaType field and I’m absolutely sure that in future, it’s going to have more than one possible value.

Another to-do is supporting code bindings in languages other than the current Java, TypeScript, and Python. At the top of my list would be Go and C#, but I know there are members of other religions out there. And for the existing languages, we should make the integrations more native. For example, the Java bindings should be in Maven.

And of course, we need support in all the platform utilities: CloudFormation, SAM, CDK, Terraform,, and any others that snuck in while I wasn’t looking.

Big futures

So, I seem to have had a change of worldview, from “JSON blobs on the wire are OK” to “It’s good to provide data types.” Having done that, everywhere I look around cloud-native apps I see things that deal with JSON blobs on the wire. Including a whole lot of AWS services. I’m beginning to think that more or less anything that deals with messages or events should have the option of viewing them as strongly-typed objects.

Which is going to be a whole lot of work, and not happen instantly. But as it says in Chapter 64 of the Dao De Jing: 千 里之行,始於足下 — “A journey of a thousand leagues begins with a single step”.

Electric Boats 29 Nov 2019, 10:00 pm

I love boating, but I hate the fact that powerboats guzzle loads of fossil fuel. I assuage my guilt by noting that the distance traveled is small — a couple of hours for each return trip to the cabin, and there are sadly less than twenty of those per year. Then I got into a discussion on /r/boating about whether electric boats are practical, so herewith some scratchpad calculations on that subject.

I’ve ridden on an electric boat, on the Daintree River in Queensland, on a small alligator-watching tour. This thing was flat and had room for maybe fifteen tourists under a canopy, necessary shelter from the brutal tropical sun; on top of the canopy were solar panels, which the pilot told me weren’t quite enough to run the boat up and down the river all day, he had to plug it in at night. The motor was a 70HP electric and our progress along the river was whisper-quiet; I loved it.

I should preface this by saying that I’m not a hull designer nor a battery technologist nor a marine propulsion engineer, so the calculations here have little more precision than Enrico Fermi’s famous atomic-bomb calculation. But possibly useful I think.

Narrowing the question

There are two classes of recreational motorboat: Those that go fast by planing, and those that cruise at hull speed, which is much slower and smoother. Typically, small motorboats plane and larger ones cruise. I’m going to consider my Jeanneau NC 795 as an example of a planing boat, and a Nordic Tug 32 as an example of a cruiser, because there’s one parked next to me and it’s a beautiful thing.


My car has a 90 kWh battery, of a size and weight that could be accommodated in either boat quite straightforwardly. A well-designed electric such as a Tesla typically burns 20 kWh/100km but you can’t use all the kWh in a battery, so you can reasonably expect a range of about 400km.

The Jeanneau gets about 1.1 km per liter of fuel while planing (it does two or three times better while cruising along at hull speed). Reviewers say that at 7-8 knots the Nordic burns about a gallon per hour, which my arithmetic says is 3.785 km/L.

A typical gas car gets about 10L / 100km, so 10 km/L. So the Nordic Tug is about 38% as efficient as turning fuel into km as a car, and the Jeanneau is only about 11% as efficient. (And of course both go much slower, but that’s not today’s issue.)

If the same is true for electric “fuel”, the battery that can take a Tesla 400km could take the Nordic tug about 150km and the Jeanneau a mere 44km.


There are boats that get worse mileage than the Jeanneau, but they’re super-macho muscle boats or extravagant yachts the size of houses. So for recreational boats accessible to mere mortals, the Jeanneau, which is in a class called “Express Cruiser”, is kind of a worst-case, a comfy family carrier that can be made to go fast on the way to the cabin, but you pay for it.

So the tentative conclusion is that at the moment, batteries are not that attractive for express cruisers. But for tugboat-class craft designed for smoothness not speed, I’d say the time to start building them is now. Among other things, marine engine maintenance is a major pain in boaters’ butts, and electric engines need a whole lot less. On top of which they’re a lot smaller, and space is always at a premium out on the water.

Variations and Futures

The following are things likely to undermine the calculations above:

  1. The Jeanneau could fit in quite a bit bigger battery than most cars; packing it into the hull so that the boat still performs well would be an interesting marine engineering problem.

  2. The Nordic Tug could pretty easily fit in a battery two or three times that size and, at hull speed, I suspect it wouldn’t slow down much.

  3. The torque curves of electric and gas engines are vastly different; which is to say, electrics don’t really have one. You have to get the Jeanneau’s engine up north of 4K RPM to really zoom along, which I suspect is not exactly fuel-optimized performance.

  4. Related to the previous item, I wouldn’t be surprised if there were considerable gains you could pull out of the low-RPM/high-torque electric engine in terms of propeller design and perhaps hull shape.

  5. Battery energy density is improving monotonically, but slowly.

  6. Boats tend to spend their time out under the open sky and many have flat roofs you could put solar panels on to (slowly) recharge your batteries. In fact, many do already just for the 12V cabin batteries that run the fridge and lights.

  7. I expect installation of Level 2 chargers on the dockside would require some innovation for safe operation in an exposed environment full of salt water. I doubt that it’d be practical to offer 50-100kW DC fast-charging gear at the gas barge.

I’ve long lusted after tugboat-style craft, but they’re expensive, bigger (thus moorage is hard to find), and go slower. Given a plausible electric offering, I think I could learn to live with that.

Wolfe and Gika 23 Nov 2019, 10:00 pm

Chelsea Wolfe I mean, and her opening act Thursday night was Ioanna Gika. It was exceptionally enjoyable, partly because nothing about it was wrong. Lovely music, great staging, good venue, exceptional sound. This happens rarely enough that it’s worth calling out when it does.

Chelsea Wolfe

Ms Wolfe has appeared on this blog multiple times. She has two modes: Acoustic, where the songs are ethereal splashes of complex slow-moving beauty; and electric, banshee keening nestled into a torrent of roaring guitar drone. Thursday night was sort of acoustic; Chelsea had an acoustic guitar and then a guy on keyboards and guitars.

Chelsea WolfeChelsea Wolfe

I’ve seen her perform twice now, and both times it’s been very… dark. Thematically and then visually. After four hours or so in her presence, I’m sure I wouldn’t recognize her on the street. It was challenging fun to photograph.

The music was not quite like her acoustic records, in that the musical backdrop was electric and synthetic, and dark like the room. Her melodies don’t orbit in your head for weeks, but the orchestration and raw beauty of the sound palette are formidable, and then she is a wonderful singer, voice full of grace and power with an astonishing high register that she pours out with no effort and exactly when the song’s flow needs it.

An unexpected pleasure was when she left her platform, sat down beside her accompanist and, to a minimal slow chord rhythm, performed Joni Mitchell’s Woodstock, taking it pretty straight but digging deep into each note and soaring up into “we are golden…” — I got all emo.

Ioanna Gika

She wasn’t announced and I didn’t catch her name when she gave it, so I didn’t know who she was till I asked the Internet after the show. Like Chelsea, she performed with just a keyboard/guitar accompanist. Perhaps even the same one, there wasn’t enough light to tell. She had a minimal little device on a stand that could play keyboard, but what she mostly did with it was capture her own voice, load it up with reverb, then sing counterpoint against it.

Thalassa by Ioanna Gika

Her music was quite a bit more dynamic than Wolfe’s even if she didn’t bring that voodoo intensity. Listening to her, I thought of the Cocteau Twins, and of Enya, and heard prog-rock echoes too. I really, really liked it and bought her CD Thalassa from the merch table. It’s great, but I actually liked the concert performance better; more intense, more complex, riskier. And also because of the sound.

Audio wow

The show was at the Vogue Theatre in Vancouver; I can’t remember the last time I was there, but it’s a nice room. I have to say this was the best live sound I’ve heard since the last time I heard unamplified acoustic music in a decent room, and I’ve never heard better live electric sound in my life that I can remember. I dunno what kind of microphones and signal chain they had, but the voices were full of three-dimensional artifacts, hints of chest and breath, not an atom of overload ever. Both women built huge low synth notes into their arrangements, placed with musical intelligence; these sounded like they were coming from the center of the earth, dark but crystal-clear.

More than a half-century into the era of live electric music, really great sound is still the exception not the rule. The touring-tech profession should be ashamed. And kudos to whoever did sound for the Wolfe/Gika gig.

Bits On the Wire 17 Nov 2019, 10:00 pm

Exactly 100% of everything on the Internet involves exchanging messages which represent items of interest to humans. These items can be classified into three baskets: One for “media” (images, sound, video), one for text (HTML, PDF, XML), and one for “objects” (chat messages, payments, love poems, order statuses). This is a survey of how Object data is encoded for transmission over the Internet. Discussed: JSON, binary formats like Avro and Protobufs, and the trade-offs. Much of what people believe to be true is not.

History sidebar

The first ever cross-systems data interchange format was ASN.1, still used in some low-level Internet protocols and crypto frameworks. ASN.1 had a pretty good data-type story, but not much in the way of labeling. Unfortunately, this was in the days before Open Source, so the ASN.1 software I encountered was slow, buggy, and expensive.

Then XML came along in 1998. It had no data-typing at all but sensibly labeled nested document-like data structures. More important, it had lots of fast solid open-source software you could download and use for free, so everybody started using it for everything.

Then sometime after 2005, a lot of people noticed JSON. The “O” stands for “Object” and for shipping objects around the network, it was way slicker than XML. By 2010 or so, the virtuous wrath of the RESTafarians had swept away the pathetic remnants of the WS-* cabal. Most REST APIs are JSON, so the Internet’s wires filled up with media, text, and JSON.


I think there’s still more of it out there than anything else, if only because there are so many incumbent REST CRUD APIs that are humming along staying out of the way and getting shit done.

JSON pros:

  1. Readers and writers are implemented in every computer language known to humankind, and they tend to interoperate correctly and frictionlessly with each other, particularly if you follow the interoperability guidelines in RFC 8259, which all the software I use seems to.

  2. It does a pretty good job of modeling nested-record structures.

  3. It’s all-text, so humans can read it, which is super extra helpful.

  4. You can receive a JSON message you know nothing about and pick it apart successfully without knowing its schema, assuming it has one, which it probably doesn’t. So you can accomplish a task like “Pull out the item-count and item-price fields that are nested in the top-level order-detail field” with pretty good results given just a blob of raw JSON.

  5. You can reliably distinguish between numbers, strings, booleans, and null.

JSON cons:

  1. The type system is impoverished. There is no timestamp type, no way to know whether a number should be treated as an integer or float or Bignum, no way to signal when string values are really enums, and so on.

  2. Numbers are specially impoverished; in general you should assume that your repertoire is that of an IEEE double-precision float (but without NaN or ∞) which is adequate for most purposes, as long as you’re OK with an integer range of ±253 (which you probably should be).

  3. Since JSON is textual, there is a temptation to edit it by hand, and this is painful since it’s nearly impossible to get the commas in the right places. On top of which there are no comments.

  4. JSON’s textuality, and the fact that it carries its field labels along, no matter how deeply nested and often repeated, suggest that it is unnecessarily verbose, particularly when numeric values are represented in textual form. Also, the text needs to be converted into binary form to be loaded into objects (or structs, or dicts) for processing by code in memory.

  5. JSON doesn’t have a universally-accepted schema language. I have been publicly disappointed over “JSON Schema”, the leading contender in that space; it’s just not very good. For a long time, the popular Swagger (now OpenAPI) protocols for specifying APIs used a variant version of a years-old release of JSON Schema; those are stable and well-tooled.


Mainstream binary formats

I think that once you get past JSON, Apache Avro might be the largest non-text non-media consumer of network bandwidth. This is due to its being wired into Hadoop and, more recently, the surging volume of Kafka traffic. Confluent, the makers of Kafka, provide good Avro-specific tooling. Most people who use Avro seem to be reasonably happy with it.


Protobufs (short for “Protocol Buffers”) I think would be the next-biggest non-media eater of network bandwidth. It’s out of Google and is used in gRPC which, as an AWS employee, I occasionally get bitched at for not supporting. When I worked at Google I heard lots of whining about having to use Protobufs, and it’s fair to say that they are not universally loved.

Next in line would be Thrift, which is kind of abstract and includes its own RPC protocol and is out of Facebook and I’ve never been near it.

JSON vs binary

This is a super-interesting topic. It is frequently declaimed that only an idiot would use JSON for anything because it’s faster to translate back and forth between data types in memory with Avro/Protobufs/Thrift/Whatever (hereinafter “binary”) than it is with JSON, and because binary is hugely more compact. Also binary comes with schemas, unlike JSON. And furthermore, binary lets you use gRPC, which must be brilliant since it’s from Google, and so much faster because it’s compact and can stream. So, get with it!

Is binary more compact than JSON?

Yes, but it depends. In one respect, absolutely, because JSON carries all its field labels along with it.

Also, binary represents numbers as native hardware numbers, while JSON uses strings of decimal digits. Which must be faster, right? Except for your typical hardware number these days occupies 8 bytes if it’s a float, and I can write lots of interesting floats in less than 8 digits; or 4 bytes for integers, and I can… hold on, a smart binary encoder can switch between 1, 2, 4, and 8-byte representations. As for strings, they’re all just the same UTF-8 bytes either way. But binary should win big on enums, which can be represented as small numbers.

So let’s grant that binary is going to be more compact as long as your data isn’t mostly all strings, and the string values aren’t massively longer than the field labels. But maybe not as much as you thought.

Unless of course you compress. This changes the picture and there are a few more it-depends clauses, but compression, in those scenarios where you can afford it, probably reduces the difference dramatically. And if you really care about size enough that it affects your format choices, you should be seriously looking at compression, because there are lots of cases where you’ve got CPU to spare and are network-limited.

Is binary faster than JSON?

Yes, but it depends. Here’s an interesting benchmark from Auth0 showing that if you’re working in JavaScript, the fact that JSON is built-in to the platform makes Protobuf’s advantages mostly disappear; but in an equivalent Java app, protobuf wins big-time.

Whether or not your data is number- or string-heavy matters in this context too, because serializing or deserializing strings is just copying UTF-8bytes.

I mentioned gRPC above, and one aspect of speed heavily touted by the binary tribe is in protobufs-on-gRPC which, they say, is obviously much faster than JSON over HTTP. Except for HTTP is increasingly HTTP/2, with longer-lived connections and interleaved requests. And is soon going to be QUIC, with UDP and no streams at all. And I wonder how “obvious” the speed advantage of gRPC is going to be in that world?

I linked to that one benchmark just now but that path leads to a slippery slope; the Web is positively stuffed with serialization/deserialization benchmarks, many of them suffering from various combinations of bias and incompetence. Which raises a question:

Do speed and size matter?

Can I be seriously asking that question? Sure, because lots of times the size and processing speed of your serialization format just don’t matter in the slightest, because your app is bottlenecked on database, or on garbage collection, or on a matrix inversion or an FFT or whatever.

What you should do about this

Start with the simplest possible thing that could possibly work. Then benchmark using your data with your messaging patterns. In the possible but not terribly likely case that your message transmission and serialization is a limiting factor in what you’re trying to do, go shopping for a better data encoding.

The data format long tail

Amazon Ion has been around for years running big systems inside Amazon, and decloaked in 2015-16. It’s a JSON superset with a usefully-enriched type system that comes in fully interoperable binary and textual formats. It has a schema facility. I’ve never used Ion but people at Amazon whose opinion I respect swear by it. Among other things, it’s used heavily in QLDB, which is my personal favorite new AWS service of recent years.

CBOR is another binary format, also a superset of JSON. I am super-impressed with the encoding and tagging designs. It also has a schema facility called CDDL that I haven’t really looked at. CBOR has implementations in really a lot of different languages.

I know of one very busy data stream at AWS that’s running at a couple of million objects a second where you inject JSON and receive JSON, but the data in the pipe is CBOR because at that volume size really starts to matter. It helped that the service is implemented in Java and the popular Jackson library handles CBOR in a way that’s totally transparent to the developer.

I hadn’t really heard much about MessagePack until I was researching this piece. It’s yet another “efficient binary serialization format”. The thing that strikes me is that every single person who’s used it seems to have positive things to say, and I haven’t encountered a “why this sucks” rant of the form that it’s pretty easy to find for every other object encoding mentioned in this piece. Checking it out is on my to-do list.

While on the subject of efficient something something binary somethings, I should mention Cap’n Proto and FlatBuffers, both of which seem to be like Avro only more so, and make extravagant claims about how you can encode/decode in negative nanoseconds. Neither seems to have swept away the opposition yet, though.

 [Shouldn’t you mention YAML? —Ed.]
 [No, this piece is about data on the
network. —T.]

On Schemas

Binary really needs schemas to work, because unless you know what those bits all snuggled up together mean, you can’t un-snuggle them into your software’s data structures. This creates a problem because the sender and receiver need to use the same (or at least compatible) schemas, and, well, they’re in different places, aren’t they? Otherwise what’s the point of having messaging software?

Now there are some systems, for example Hadoop, where you deal with huge streams of records all of which are the same type. So you only have to communicate the schema once. A useful trick is to have the first record you send be the schema which then lets you reliably parse all the others.

Avro’s wire format on Kafka has a neat trick: The second through fifth byte encode a 4-byte integer that identifies the schema. The number has no meaning, the schema registry assigns them one-by-one as you add new schemas. So assuming both the sender and the receiver are using the same schema registry, everything should work out fine. One can imagine a world in which you might want to share schemas widely and give them globally-unique names. But those 32-bit numbers are deliciously compact and stylishly postmodern in their minimalism, no syntax to worry about.

Some factions of the developer population are disturbed and upset that a whole lot of JSON is processed by programmers who don’t trouble themselves much about schemas. Let me tell you a story about that.

Back in 2015, I was working on the AWS service that launched as CloudWatch Events and is now known as EventBridge. It carries events from a huge number of AWS services in a huger number of distinct types. When we were designing it, I was challenged “Shouldn’t we require schemas for all the event types?” I made the call that no, we shouldn’t, because we wanted to make it super-easy for AWS services to onboard, and in a lot of cases the events were generated by procedural code and never had a schema anyhow.

We’ve taken a lot of flak for that, but I think it was the right call, because we did onboard all those services and now there are a huge number of customers getting good value out of EventBridge. Having said that, I think it’d be a good idea at some future point to have schemas for those events to make developers’ lives easier.

Not that most developers actually care about schemas as such. But they would like autocomplete to work in their IDEs, and they’d like to make it easy to transmogrify a JSON blob into a nice programming-language object. And schemas make that possible.

But let’s not kid ourselves; schemas aren’t free. You have to coördinate between sender and receiver, and you have to worry what happens when someone wants to add a new field to a message type — but in raw JSON, you don’t have to worry, you just toss in the new field and things don’t break. Flexibility is a good thing.

Events, pub/sub, and inertia

Speaking of changes in message formats, here’s something I’ve learned in recent years while working on AWS eventing technology: It’s really hard to change them. Publish/subscribe is basic to event-driven software, and the whole point of pub/sub is that the event generator needn’t know, and doesn’t have to care, about who’s going to be catching and processing those events. This buys valuable decoupling between services; the bigger your apps get and the higher the traffic volume, the more valuable the decoupling becomes. But it also means that you really really can’t make any backward-incompatible changes in your event formats because you will for damn sure break downstream software you probably never knew existed. I speak from bitter experience here.

Now, if your messages are in JSON, you can probably get away with throwing in new fields. But almost certainly not if you’re using a binary encoding.

What this means in practice is that if you have a good reason to update your event format, you can go ahead and do it, but then you probably have to emit a stream of new-style events while you keep emitting the old-style events too, because if you cut them off, cue more downstream breakage.

The take-away is that if you’re going to start emitting events from a piece of software, put just as much care into it as you would as you do in specifying an API. Because event formats are a contract, too. And never forget Hyrum’s Law:

With a sufficient number of users of an API,
it does not matter what you promise in the contract:
all observable behaviors of your system
will be depended on by somebody.

Messages too!

The single true answer to all questions about data encoding formats

“It depends.”

Subscription Friction 3 Nov 2019, 10:00 pm

At Canadian Thanksgiving, friends joined us at our cottage for turkey and the fixings. The food (what we made and what they brought) all came out great and we had happy stomachs. I did a lot of the cooking and wanted to check recipes and was brought face to horrified face with the failure of publishing to work on the Internet. The solution seems plain to me and this won’t be the first time I’ve offered it. But something really needs to be done.

What happened was, I wanted to refresh my memory on turkey and gravy. These are dishes one makes regularly but rarely and the details don’t stick to my mind: What’s the meat-thermometer reading for a stuffed bird, and what order do you deploy the drippings, roux, and giblet broth in for gravy?

It turns out the part of the Web where recipes live is a dystopian hellscape. Even though the medium seems made for the message: Lists, pictures, and search are built-in! Not to mention voting and hyperlinks. Anyone who thinks they’ve got a really great gravy procedure can tell the world, and as more people agree, the recipe should become easier to find.

Except for what you find is an SEO battle, red in tooth and claw, where recipe sites stinking of burn-rate terror plead for engagement and as a matter of principle make it really hard to get to that damn ingredients-list and step-by-step. Newsletter! Notifications! Survey! Also-see! Multidimensional parallax-enhanced five-way scrolling Single-Page Applications!

In fact, it bothered me so much that a few days later, I posted a nice simple chicken-kebab recipe just for that good feeling when you use the Web the way it was designed. I took the pictures, I wrote the text, and I posted it all on my own domain, and anyone who follows a link will have the words and pictures in front of their eyes right away, and there won’t be any popups of any kind.

But it’s not just recipes, obviously. It’s a disease that afflicts the larger community of those who try to add value to the Net and to our lives by sharing their stories and pictures and movies and poems and dreams, in the process we generally call “publishing”.

It isn’t working.

“Become a subscriber today!”

My words today are interspersed with pictures of the signs and signals that obscure an increasingly large proportion of the Web’s surface: Imminent-closing-paywall-gate warnings.

Subscribe to Foreign Policy!

I think there is something that is glaringly, screamingly obvious that at this point needs to be said loudly and repeatedly:


I suspect that a high proportion of my readers know this is true just from the feeling in their gut, but as an exercise in rhetoric let me offer some of the reasons why.

  1. Subscription fatigue is setting in. I’ll make a spending decision, even a big one, quickly and without regrets when whatever it is feels like the right thing. But a subscription, another bill showing up on every credit-card statement… forever?

  2. A very high proportion of the world’s curious-minded well-educated people subscribe to some subset of The Economist, the New York Times, the Guardian, the Washington Post, and the New Yorker. Most of us feel we’re subscribed enough.

  3. There’s a huge class of publications who’ll put out a handful of articles every year that I want to read, but not remotely enough to justify a subscription.

  4. The offers are sleazy. Whenever I read an absurdly low-cost subscription offer, I know the number showing up my bill is going to be a lot higher pretty damn quick.

  5. Obviously it’s not just journalism. What madness makes huge companies think that people will sign up for Netflix and Hulu and HBO and Amazon Video and Apple TV+ and Disney? I’m sure I’ve forgotten a few, and even more sure that more are on the way.

Subscribe to the Globe & Mail!

This. Will. Not. Scale.

We need friction somewhere

Information may want to be free, but writers want to be paid. In a world where you can follow any link and read whatever it points at instantly and for free, writers are going to starve. It’s the writers I care about the most, but let’s not forget the designers and copy-editors and fact-checkers and accountants and so on.

People who write and gather and curate recipes want to be paid too.

How about advertising?

The Grand Theory of the Web long said that the money comes from advertising. After all, there’s always been lots of free information out there, and there still is if you’re willing to put up a TV antenna. The reality is that unless you’re Google or Facebook, advertising is deeply broken. My favorite exposition of why ads as they are today don’t work is by Josh Marshall in Data Lords: The Real Story of Big Data, Facebook and the Future of News. It’s important to note that Marshall isn’t arguing that Web advertising is inherently broken; just that the current Facebook/Google business model is so successful that they’re basically skimming all the profit off the top of the system, and in particular ripping the guts out of independent niche publishers.

Subscribe to the Atlantic!

If you think “I already knew that”, take a minute and read that “Data Lords” piece linked above; it’s not as obvious as you might think.

Marshall is the founder and publisher of Talking Points Memo, one of the best US left-wing political publications, and had the good fortune to recognize the dead-endness of advertising early on and, starting in 2012, made a hard pivot to subscriptions. They’ve been innovative and aggressive and executed well, and it looks like they have an excellent chance of staying in it for the long haul.

What we’re seeing now is that more or less every Web publication has, in the last couple of years, come to the same conclusion and is trying the same pivot. Only it’s obviously not going to work.

Because This. Will. Not. Scale.

Which way forward?

One helpful thing would be to fix advertising. It’s easy to hate the ad business and ad culture but at the end of the day they’re probably a necessary facilitator in a market economy. And hey, if they want to pay part of the cost of the publications I like to read, I’m down with that. I’m not smart enough to have designed a regulatory framework that would restore health to the market, but I don’t think it’s controversial to say that it’d be really great if someone did.

Subscribe to the Spectator!

But it seems to me that there’s a more obvious way; let me buy stories one-at-a-time without signing up for a monthly-charge-forever. The idea is this: When I follow a link to a juicy-sounding story in, say, the Minneapolis Star-Tribune or the The Melbourne Age, instead of the avalanche of subscription wheedling, I get a standardized little popup saying “Ten cents to read this”, with several options: Yes or no on this piece, and Always-yes or Always-no for this publication. Someone is operating the service that makes this happen and will do an aggregate charge to my credit card every month.

Suppose the Net lights up because some site has got their hands on Donald Trump’s tax returns. Maybe when I follow that link, the standard popup asks me for a buck instead of a dime.

Subscriptions would still be possible and still make sense if you were reading a lot of pieces from some publication, and could come with benefits — for example, Josh Marshall’s Talking Points Memo offers a super-subscription which subtracts all the ads and makes the pages wonderfully fast and lightweight.

Is that even possible?

It’s not exactly a new idea. Something like it was originally proposed by Ted Nelson in Literary Machines in 1981. The fact that nobody’s made it work so far might make a reasonable person pessimistic as to whether anyone ever will.

Subscribe to Vanity Fair!

Well, I’m here to tell you that on the technology side, we have the tools we need to build this. You could spend a bunch of time devising an Internet Standard protocol for subscription services and wrangle it through the IETF or W3C or somewhere, and that’s not a terrible idea, but I’d probably want to build the software first. With an Amazon-style two-pizza team and modern cloud infrastructure I’d have it on the air in 2020, no problem.

The one design constraint I’d impose would be that this thing would have to work at small scale, not just for the The Bigs. [Disclosure: Yes, I’m a blogger and I’d like us to be able to make a buck too.] But once again, I just don’t see it as hard.

Bootstrapping the business side would be tough because the publishing industry is led by people who not only are not focused on technology but suffer from unrealistic fantasies as to what it is and isn’t. It’d be dead easy for Amazon or Google to offer this as a service, but the publishing community would, rightly or wrongly, assume it to be another way to suck all the money out of the sector.

Maybe there’s a role here for a consortium or institute; or for some large far-sighted publisher with a stable of properties to build this for themselves but with careful attention to making attractive to the rest of the industry?

My advice would be to get working on it fast. Because ads are broken and burn rates are high and pivoting to subscriptions was a really great idea in 2014 but it’s too late for that now.

Easy Reliable Chicken Kebabs 13 Oct 2019, 9:00 pm

This involves a certain amount of chopping stuff up, also attention to hygiene, but requires no particular technical skill and has never ever failed to get rave reviews.

Infrastructure and ingredients

  1. A barbecue. Charcoal is said to produce better flavor but gas is immensely faster and easier.

  2. Skewers. Metal or wood, whatever.

  3. Chicken. I’m lazy, I buy it skinless & boneless from the supermarket. Breasts are easier to work with but thighs come out a little better. A kg will satiate four large hungry people.

  4. Lemons. You could use the stuff that comes in bottles but that would be lame. Juicing lemons is easy.

  5. Olive oil. If you think it makes a difference you can pay up for “Extra Virgin”. I do recommend looking for a respectable brand because I gather there is some seriously sketchy stuff upstream of the liquid labeled as “olive oil” on many store shelves.

  6. Garlic. You could be sincere and slice or mash it, or you could buy a jar of garlic paste or minced garlic at the aforementioned supermarket, as I do.

  7. Black pepper. Fresh-ground really does make a difference.

  8. Other things to go on the skewers. The only essential thing is onions (but shallots are better). Also popular: Small tomatoes, mushrooms, sweet peppers (Aussies: I mean capsicum).

Checken kebab construction

The chopping part is kind of boring, so I watched the ALCS.

Safety note: Several of these steps require getting intimate with uncooked chicken. Which can contain the sort of microorganism that causes major unhappiness. So a hot-soap-and-water hand-wash is in order whenever you move from the raw-chicken part to any other part. And then give the part of the counter where you did the raw-chicken stuff a good scrub.


First, you make the marinade. Mine is equal-ish parts olive oil and lemon juice, with several large dollops of garlic paste, and lashings of black pepper. If I have oregano (dried not fresh) handy I throw some in; if not I don’t worry about it. Mix vigorously. You need enough marinade to cover your chicken and a little bit goes a surprisingly long way. Say 3 juicy lemonsworth and the same or a little more olive oil.

Cut up your chicken. You want pieces of a size and shape that go comfortably on the skewer. I tend to favor smaller rather than larger to increase the surface-area ratio, because the surface is where the marinade soaks in.

Drop the chicken into the marinade and swish it around. Put it in the fridge and ignore it for a while. I think two or three hours is plenty, but purists do all this the day before. Important: Count the pieces as you cut them up.

Now we’re getting ready to cook. Here’s where you need to get quantitative. Figure out how many skewers you want to make (or maybe just how many you have) and look studiously blank while you sip a glass of preprandial wine and calculate how many pieces of chicken per skewer. Since each chicken chunk should have one or more non-meat items separating it from the next, you now have sufficient information to calculate how many pieces of (for example) onion, mushroom, pepper, and so on you need. The one I did today had three pieces of chicken, two pieces of onion, one piece of pepper, two pieces of mushroom, and two cherry-tomato halves (one red, one yellow) on each skewer. I had forty meat morsels, and I made twelve skewers for the four of us, so the veggie numbers were easy to work out.

Chicken kebabs ready to cook

Ready for the grill.

Stop here and, if you are using wooden skewers, drop them in a sink-full or pot-full of water to get somewhat fireproof.

Cut up all your vegetables and fungi. I like to grab a bunch of cereal bowls, one for each ingredient. I try to cut them roughly the same size as I did the chicken, so that when I put them on the barbecue you won’t have a big hunk of onion shielding the meat from the flame, or the other way around. If you have a compost bin, move it from wherever it is to where you’re slicing, to capture the waste.

OK, pull the meat out of the fridge. Pour the marinade down the drain. Grab a big plate to put the completed skewers on. Arrange the chicken and all the other skewer-fodder and the plate as seems best to you. I’m a soulless geek so I work out a repeatable skewer script (e.g. tomato, chicken, onion, mushroom, chicken, pepper, mushroom, chicken, onion, tomato) and stick with it.

At some point halfway through, wash your hands and go fire up the barbecue.

Once you’re done you’ll have a satisfying heap of skewers on the plate, and quite a bit of chicken pollution on your hands and the counter. If you miscalculated and there are bits of leftover vegetable or chicken stick the ones you can on skewers and throw the rest out, they’re unsanitary.

Slap ’em on the barbecue, wait till one side is done, turn ’em over, wait again, then enjoy. Tongs help. If you’ve kept the size of the chicken morsels down, then when one side is visibly seared I find it’s reliably cooked inside. Four or five minutes each side works for me, but that’s on my barbecue with my preparation habits. You do really need to make sure the chicken’s done.

Chicken kebabs ready to eat


Rice works well. I like to toss asparagus or broccoli on the part of the barbecue that isn’t full of skewers, but whatever. White wine, or (wouldn’t have said this even a couple of years ago) a good artisanal cider.


There won’t be any. The lemon-soaked garlic-loaded flame-kissed skewer payloads are ravishing. And it’s hard to get this one wrong.

[Disclosure: The attentive observer will notice that the pictures are of a different batch than the one the text describes.]

Records and Lenses 6 Oct 2019, 9:00 pm

Sunday included more fun than the recent average — at my age, chilling is often more attractive than partying. Sunday featured vinyl, vintage lenses, Southern guitar boogie, and a photo-assignment. With pictures! (Which may be a little puzzling, but stay with me.)

Record convention poster

What happened was, I noticed that there was a Fall 2019 Record Convention and on impulse took the e-bike crosstown to the glamorous (not really) Croation Cultural Centre. I strolled in the door of the big room with all the people and the nice lady said “You want the record convention or the camera convention? This is the cameras.” Who knew? So I went over to the other side and ended up buying five records: Willie Nelson’s Teatro (with Emmylou and Daniel Lanois), Howlin’ Wolf’s Big City Blues, Yellowman and Fathead’s Bad Boy Skanking, James Blood Ulmer’s Free Lancing, and Sons of Kemet’s Your Queen is a Reptile.

It was just a high-school-gym-sized room with a big square of tables facing outward facing the other big square against the walls facing inward, all the tables full of boxes full of LPs. I posted an appropriately lo-rez video on Instagram. A few of the vendors had organizing principles, ranging from genre to alphabet to price. Lots didn’t.

What a bunch of stoners. Except for a few fresh-faced hipsters and the odd grizzled geek like yours truly. Good clean fun.

Main Street by night


Since I’d come all the way, I coughed up $5 for the camera show, which was pretty well maximum steampunk. Old cameras! Some with bellows and bulbs, some the size of your head. I have enough cameras but you can never have too many lenses. There wasn’t much chance of anything Fujifilm-related (it’s from this millennium) but I have a Pentax K-mount adapter and as I wrote in that context, “there are an astonishing number of excellent K-mount lenses in every imaginable length and width on the market.” So I walked around the tables saying “K-mount?” to anyone showing lenses and lots had some.

Main Street by night

Protip: Nifty fifty

If you’re stingy and want to take lovely portraits it turns out that for basically any camera in the world you can get a K-mount adapter for your camera cheap, and then go to a show like this (or eBay or whatever) and pick up a classic Pentax 50mm F1.4 for almost no money and with a little practice and love, you’ll be making your friends and loved ones look better than they really do.

But I have a nifty fifty already. I’d got two thirds of the way around the camera trail and saw nothing interesting, but then there was this Mittel-Euro dude with an astonishingly small and slim 100mm F/2.8. He wanted $120 Canadian. I walked away and completed the orbit without turning up any gold, so I went back and offered him $100 and he took it. It turns out to be an M 100/2.8, i.e. probably 40 years old.

Fujifilm XT-30 with Pentax M

Isn’t it adorable?


Sunday evening I had tickets to see the North Mississippi Allstars. So I strapped the 100mm on the Fujifilm and thought I’d try for some electric-stage drama. Only the security guy took one look and said “gotta coat-check professional cameras.” Professional?! Uh…

Anyhow, the Allstars were just fine, post-Allmans southern boogie, more on the countrified side with a killer washboard solo (no, really) and really a lot of very solid guitar. Plus they played Down By The Riverside (“Ain’t gonna study war no more”) which always gets me choked up. I’d go if they came near you, assuming you like electric white-boy music.


It turns out that at Amazon Vancouver we’ve got a little photocritique social group, where someone deals out a weekly theme and we all post pix and suggest improvements to each other. This week’s theme is “long exposure”. So as soon as I got in the taxi home (For the Vancouver-savvy: up Main from the DTES to Riley Park), with a couple of drinks under my belt and a head full of electric guitar grace, I rolled down the backseat window, set the shutter to ⅛ second and the aperture to F/4, and took pictures of pretty well everything. Three made me smile.

Main Street by night

And I think to myself… what a wonderful world.

Discogs Pain 5 Oct 2019, 9:00 pm

As I continue slowly ingesting the 900 classical LPs I inherited, I’ve developed a relationship with Discogs. It’s a good place to track your collection (here’s mine, in-progress). This is the story of my first attempt to use it to buy music, a laughable failure. It’s by way of public service, just leaving a marker, a warning others to be careful about charging into the marketplace.

There’s a lot to like about Discogs. The screens are unfussy and fast and their coverage of everything ever recorded is just awesome. The LP collection I’m working my way through has plenty of obscurities by no-hit wonders, but I’ve been through a couple hundred records now and only hit one that it didn’t already have (for your amusement: Music from The Golden Age of Silent Movies played by Gaylord Carter at the Mighty Wurlitzer Theater Organ).

Also, they’ve apparently taken a mere $2.5M in venture investment, show the world a human face, and seem to genuinely care about old recordings in a tone that’s enthusiast rather than unicorn-wannabe.

Fast Freight

Back when I was doing Songs of the Day, I wrote about Fast Freight, a standout on a generally-pretty-corny 1958 mono album by the Kingston Trio that I inherited from my Dad. It’s a lovely performance which I also use as an audio showpiece; when a technical friend looks at my audio setup and snottily wonders why I still play LPs, I use this to show what the technology can do. Which usually works pretty well

The Kingston Trio

Only Dad’s copy’s getting kind of ratty, and it occurred to me that maybe I could use Discogs to replace it. Sure enough, there were plenty on sale, so I picked one on the basis that it was advertised as having “near-mint” surfaces and was offered by a seller with a good rep.

So here’s my sad story:

  1. I jumped through the Discogs hoops, not terrible at all, and placed the order.

  2. Almost instantly, the site came back saying the status was “Invoice sent”. I didn’t see any invoice on the screen or in my Discogs account or my email in-basket. I decided to wait, on the assumption it would show up.

  3. A few days later, Discogs popped up a notice saying that payment was due and unless I did so within a day or so my reputation would suffer.

  4. I fired off a barrage of messages to Discogs support and the seller saying, in effect “WTF, I want to pay, how do I pay?” Discogs was pretty prompt getting back, said “probably best to reach out to the seller.”

  5. The seller eventually got back to me and said if I wasn’t set up on Discogs’ internal payment system, he could take PayPal. I poked around and couldn’t figure out how to set up to use their payment system (still can’t). He asked me to send it marked “friends and family” so “PayPal wouldn’t hold it”.

  6. I sent him the money and PayPal did indeed put it on hold, and had deducted a big service fee. So he refunded it to me. This led to a hilarious, where by “hilarious” I mean “really irritating”, sequence of attempts to pay him, by splitting the payment up, by marking smaller amounts “Friends and family”, and I forget the rest. None of it worked, PayPal was determined not to let my money get through to him. This leads me to wonder how he’d got himself on PayPal’s hold-the-payments list.

  7. The seller also said that somewhere along the way I’d accidentally left negative feedback, could I please remove it? I couldn’t find any feedback anywhere.

  8. Finally, by routing PayPal to a different email address, he got the money and let me know he’d sent the record.

  9. A reasonable amount of time later, the record showed up. It was all scratched to ratshit and there was a big cut in the middle of Fast Freight so that the needle kept skipping back.

  10. My Discogs account still has a permanent notification that payment is outstanding for this order and my reputation is in danger.

I’m Doing It Wrong

That’s the only conclusion I can draw. I need to go and research how to buy things on Discogs without getting taken for a ride. I suspect it’s possible but it should be easier.

At the Climate Strike 28 Sep 2019, 9:00 pm

I went and so did lots of others, but many couldn’t so I thought I’d try to share the scale and the feel.

Help Save Our Winters

(Mounted on a hockey stick)

I’m pretty old and cynical and still, this felt like it mattered. Even though it was good-humored and consciously funny. Partly it was just the scale — one of our big north-south thoroughfares full of streaming strikers it seemed forever, but they say the peak of the parade only lasted an hour and a half. The police estimated a hundred thousand but they are prone to undercounting based on an institutional fear of too many people in one place going the same direction. Having said that, the cops were pretty exellent, see below.

Many spectated.

Spectators at the Climate Strike

The crowd was young, people whom they say don’t vote, and anyhow many were too young to vote. But their good-humored passion could not fail to lift hearts. Well, mine anyhow.

This Pipeline Shall Not Pass

Not that I was the only greybeard. If you’d subtracted the kids it still would have been a pretty big deal.

Don’t Care? Would Be A Lot Cooler If Ya Did

I didn’t get a picture of the kid with the sign saying “I’m skipping Fortnite to be here!”

Photographers at the Climate Strike

Once we got downtown, all the balconies and windows were full of people watching.

This Is Not A Single-Use Planet

The most popular chant was “What do we want?” “Climate justice!” “When do we want it?” “Now!” My voice is hoarse today. That chant is a common variation on a very old theme but it was the first time I’ve been part of it. After a little practice, the crowd gets into a more practiced and powerful rhythm; in particular the “Now!” becomes an explosion, elating to be part of and maybe a little scary for the intended audience. I hope so anyhow.

There Is No Planet “B”

There was this skinny little guy walking along, well under 10, whose chant was “Want to play in the snow? Stop global warming?” Pretty pedestrian, but he had a surprising volume and amazing endurance, he kept going and going and going and eventually the rhythm was fascinating and voices around him stilled.

Some People Just Wanna Watch The World Burn

I was shocked that Justin Trudeau, pipeline promoter, had the gall to appear at the Montréal strike. His smarmy-robot mantra about “The environment and the economy” isn’t fooling anyone any more. The best route to a sound twenty-first century economy requires creative destruction centered on the fossil-fuel sector. There’s plenty of capital and demand and opportunity to fill in behind, but first we have to step away from petroleum extraction, big-time.

I’m Not A Sign Guy

Towards the end of the route, the crowd started piling up and I couldn’t make any forward progress. Somewhere up ahead I could hear speeches, but the point had been made and my feet were killing me, so I took the train home. Hats off to Vancouver’s cops and TransLink’s helpers who kept the throngs moving through City Center station and damn did they ever have those trains shuttling through on the double-quick. I admire calm-faced competence when I see it.

A Very Happy Young Girl

Everybody was self-congratulatory about the nonviolence and good cheer, and unlike other recent events, there are no counter-demonstrators in evidence. The contras are all behind the walls of the glass towers our chants were echoing off of, and those guys are way too busy to pay attention.

When the waters start rising and the crops start failing, the glass will start breaking.

We Aren’t Witches

A few signs here and there were really obscene, a repeated theme of, uh, let’s phrase it as inappropriate sexual relations involving a female parent. 100% of the people carrying these were women.

We’re Coming For Your Money

And of course, there were a variety of other causes looking for listeners in what smells like the progressive mainstream. Communists, vegans, co-operative housing advocates, I even saw a sign for the IWW, which I thought as a sort of historically-significant exotic fringe thing when I was a hot-headed young leftist 40 years ago. Seems the Wobblies have staying power.

I’m An Edgy Teen

The cops had been smart, blocked access to the Cambie bridge and its feeders well in advance. Things went a little sideways downtown where a lot of people were trying to commute or go shopping and unfortunately got on the wrong cross-street and ended up motionless for a couple of hours while the strikers streamed past. I hope nobody’s life was seriously damaged, but a lot of lives are going to be seriously damaged if we don’t get our climate-change act together.

The buses were blocked on Granville Street.

Halted Buses at the Climate Strike

You’ve probably heard of airplane boneyards, places in the desert where they park hundreds of disused unwanted jet planes? When sanity sets in and it becomes just too expensive for people to waste energy the way internal-combustion cars do, there’ll be auto boneyards too.

On Sharding 25 Sep 2019, 9:00 pm

If you need to handle really a lot of traffic, there’s only one way to do it: sharding. Which is to say, splitting up the incoming requests among as many hosts (or Lambda functions, or message brokers, or data streams) as you need. Once you get this working you can handle an essentially unlimited request volume. Of course, you have to make choices on how you’re going to divide up the traffic among the shards. I’ve had intense exposure to the options since I came to work at AWS.

Random spray

This is the simplest thing imaginable. For each message, you make a UUID and use it as the Partition Key (if your downstream uses that for sharding), otherwise just pick a target shard using your favorite random number generator.

There’s a lot to like about it. Your front-end can be dumb and stateless and fast. The load will get dealt out evenly among the shards. Spoiler alert: I think this is a good choice and you should do it if you can possibly can.

A common variation on this theme involves auto-scaling. For example, if you have a fleet of hosts processing messages from an SQS queue, auto-scaling the fleet size based on the queue depth is a fairly common practice. Once again, admirably simple.

“Smart” sharding

The idea is that you do extra work to avoid problems, for example some shards getting overloaded and others being idle. Another kind of problem is one of your upstream sources sending “poison pill” messages that cause the receiving shard to lock up or otherwise misbehave

Load-sensitivity is one “smart” approach. The idea is that you keep track of the load on each shard, and selectively route traffic to the lightly-loaded ones and away from the busy ones. Simplest thing is, if you have some sort of load metric, always pick the shard with the lowest value.

I’ve seen this done, and work OK. But it isn’t that easy. You have to figure out a meaningful load metric (queue depth? how much traffic received recently? CPU load?) and communicate that from each shard to the front end.

If poison pills are your worry — this is not rare at all — the smartest approach is usually shuffle sharding. See Colm MacCárthaigh’s awesome thread on the subject, which you should go read if you haven’t already. He has pretty pictures (see below for one I stole) and math too!

Shuffle sharding by Colm MacCárthaigh


Also known as “session affinity”. Another term you sometimes hear is “sticky sessions”; for a discussion see this article over at Red Hat. The idea is that you route all the events from the same upstream source to the same shard, by using an account number or session identifier of some sort as a partition key.

Affinity is attractive: If all the clickstream clicks from the same user or state-change events from the same workflow or whatever go to the same host, you can hold the relevant state in that host’s memory, which means you can respond faster and hit your database less hard.

Ladies, enbies, gentlemen, and others: I’ve had really lousy luck with affinity in sharding. Lots of things can go wrong. The most obvious: When traffic isn’t evenly distributed among upstream sources.

inverse-square curve

One time I was working with a stream of AWS customer events, and I thought it’d be smart to deal them out to Kinesis shards by account number. That was bad — the messages per second per account rate was distributed across account numbers in a classic inverse-square curve like the one in the margin. To make this concrete, in one of the early tests I noticed that the top 10 account numbers accounted for half the traffic. Ouch. (I don’t think this was a rare situation, I think it’s probably common as dirt.)

The general lesson is that if you unfortunately send too many “hot” upstream sources to the same shard, it’s easy to overload it. The worst case of this is when you get a single “whale” customer whose traffic is too much for any single shard.

So in this situation I’m talking about, we switched to random-spray, and the whole system settled down and ran nice and smooth. Except for, processing each message required consulting a database keyed by account number and the database bills got kind of high.

So I got what I thought was a brilliant idea, hid in a corner, and wrote a “best-effort affinity” library that tried to cluster requests for each customer on as few shards as possible. It seemed to work and our database bills went down by a factor of six and I felt smart.

Since then, it’s turned into a nightmare. There’s all sorts of special-case code to deal with extra-big whales and sudden surges and other corner cases we didn’t think of. Now people roll their eyes when smoke starts coming out of the service and mutter “that affinity thing”. They’re usually too polite to say “Tim’s dumb affinity code”.

That’s not all, folks

Here’s another way things can go wrong: What happens when you have session affinity, and you have per-session state built up in some host, and then that host goes down? (Not necessarily a crash, maybe you just have to patch the software.)

Now, because you’re a good designer, you’ve been writing all your updates to a journal of some sort, so when your shard goes pear-shaped, you find another host for it and replay the journal and you’re back in business.

To accomplish this you need to:

  1. Reliably detect when a node fails. As opposed to being stuck in a long GC cycle, or waiting for a slow dependency.

  2. Find a new host to place the shard on.

  3. Find the right journal and replay it to reconstruct all the lost state.

Anyone can see this takes work. It can be done. It is being done inside a well-known AWS service I sit near. But the code isn’t simple at all, and on-call people sigh and mutter “shard migration” in exactly the same tone of voice they use for “Tim’s dumb affinity code.”

But the database!

So am I saying that storing state in shards is a no-no, that we should all go back to stateless random spray and pay the cost in time and compute to refresh our state on every damn message?

Well yeah, if you can. That fleet running my dumb affinity code? Turns out when we reduced the database bills by a factor of six we saved (blush) a laughably small amount of money.

What about latency, then? Well, if you were using a partition key for sharding, you can probably use it for retrieving state from a key/value store, like DynamoDB or Cassandra or Mongo. When things are set up well, you can expect steady-state retrieval latencies in the single-digit milliseconds. You might be able to use a caching accelerator like DynamoDB’s DAX and do a lot better.

In fact, if the sessions you might have tried to shard on have that inverse-square behavior I was talking about, where a small number of keys cover a high proportion of the traffic, that might actually help the caching work better.

But the cache is a distraction. The performance you’re going to get will depend on your record sizes and update patterns and anyhow you probabl don’t care about the mean or median as much as the P99.

Which is to say, run some tests. You might just find that you’re getting enough performance out of your database that you can random-spray across your shards, have a stateless front-end and auto-scaled back-end and sleep sound at night because your nice simple system pretty well takes care of itself.

The Cloud and Open Source 21 Sep 2019, 9:00 pm

This is eventually about the public cloud and Open Source, but — apologies in advance — takes an indirect and long-ish path.

In AWS engineering, we develop stuff and we operate stuff. I think the second is more important.

We have good hardware and software engineers, and infrastructure that feels pretty magic to me (faves: the racks and networking gear, the consensus manager underlying QLDB, and the voodoo that makes S3 go). But, like Bill Joy said, “Wherever you work, most of the smart people are somewhere else”, so I’m not gonna kid myself that we’re magically unique at programming.

But on the operations side, the picture is really unique. First of all, there are very few places in the world where you can get operational experience at this scale. Second, AWS doesn’t run on SRE culture; the same engineers who write the code live by the dashboards and alarms and metrics that try to reflect and protect the customers’ experience (not perfectly, but we make progress).

The obsessive focus on operational excellence isn’t subtle and it’s not a secret. There’s been a re:Invent presentation about how we run our ops meetings and we even open-sourced the AWS Ops Wheel.

But it’s not all meetings. We build and deploy a lot of technology with no direct connection to any feature or function or API that a customer will ever see. These are all about having the right dashboards, and being able to extract the key ratio from petabytes of logs, and predicting what might melt down before it even gets warm.

The asshole ratio

I’ve already written that at AWS, it’s lower than I experienced at other BigTech outfits. Here’s why this is relevant: There is plenty of evidence that you can be a white-hot flaming asswipe and still ship great software. But (going out on a limb) I don’t think you can be an asshole and be good at operations.

Because ops requires being humble in the face of the evidence, acknowledging fallibility, assuming that the problem is your problem even when quite likely it’s not, and always eager to investigate theories B, C, and D even when you’re pretty sure your current theory A is right-on. Since problems in complex services are almost never solved by a single individual’s efforts, you have to be good at working with people under pressure.

Those LPs

I have a hypothesis about that good ratio and it involves the Amazon Leadership Principles (we just say LPs). I’ve gotten flack from friends who think having such things is lame and corny. But in practice they turn out to be useful, and to explain how I’m going to take side-trip into modern clinical medicine.

There’s this guy Atul Gawande, a surgeon and writer whose work I’ve admired, mostly in The New Yorker, for years. I recommend pretty well anything he writes but in particular I recommend The Checklist Manifesto. Do me (and yourself) a favor, follow that link and read the Malcolm Gladwell review excerpt. From which:

“…the routine tasks of surgeons have now become so incredibly complicated that mistakes of one kind or another are virtually inevitable: it’s just too easy for an otherwise competent doctor to miss a step, or forget to ask a key question or, in the stress and pressure of the moment, to fail to plan properly for every eventuality.“ [Sounds just like updating a million-TPS Web Service. -Tim] “Gawande then visits with pilots and the people who build skyscrapers and comes back with a solution. Experts need checklists–literally–written guides that walk them through the key steps in any complex procedure.”

Well, one insanely-complex routine task that we do all the time is hiring. You know what the LPs are at hiring time? A checklist. Now even the typical all-day interview marathon isn’t gonna reliably dig into every LP, but we do an acceptable job of taking a close look at enough of them. I believe that’s very helpful in bringing down the asshole ratio.

Open Source

Which brings me to the touchy subject of the relationship between Cloud Providers and Open Source. We and our competitors have made a good business of infrastructure operations, keeping service-oriented software servicing; reliably, durably, 24/7/365. The core EC2 business is about operating Linux boxes and IP networking at extreme scale, efficiently enough that we can rent them out at an attractive price and still make a buck.

In recent Open-Source years, some very gifted people have created wonderful pieces of software — Kafka, ElasticSearch, Mongo — and taken a new course, launching VC-financed companies to monetize with service and support. Then sometimes they find themselves competing with multiple public-cloud providers.

I have a load of sympathy for the virtuoso engineers who created these wonderful pieces of work. But here’s the thing: I have at least as much for the customers who (let’s take Kafka for an example) just need reliable high-performance streaming. A direct quote: “I’ll cheerfully pay monthly to never worry about Zookeeper again.”

On the other hand, I have little sympathy with modern VC-driven business models.

It’s like this: The qualities that make people great at carving high-value software out of nothingness aren’t necessarily the ones that make them good at operations. This has two unfortunate effects: They don’t necessarily have the right skills to build and run a crack operations team, and they might not manage to get a job at an operations-obsessed company.

I have recent personal experience with failing to hire a senior committer to a well-known OSS project, and also with paying an “open-source company” for tech support when we were spinning up a service around a package we didn’t know very well. Both of these left me unhappy.

Jack and Jonathan

Let me tell you a story. Sometime around 2008, I and Jonathan Schwartz, then the CEO of Sun Microsystems, made a sales call on Jack Dorsey at Twitter. Sun had acquired MySQL and Twitter was using the hell out of it. We wanted them to start paying us for support; after all, they were existentially dependent on this technology and everyone knew that serious Enterprises would never use unsupported software.

Jack was nice, and listened to our pitch, but we didn’t get the business.

And while, as a career software guy, I entirely love open-source culture and technologies and methods, the hypothesis that Open Source in and of itself constitutes a business model is not well supported by the evidence.

Which way forward?

Google Cloud’s recent Open Source partnerships are interesting. I look at that list of companies and it’s not obvious to me that they’re going to offer better operational excellence than Google’s, but maybe I’m wrong. It’s an interesting and probably useful experiment.

At the end of the day I’m not that worried. Most of us who’ve open-sourced stuff love the creative process for its own sake; touching and improving other engineers’ lives. The skillset evidenced by having done so will probably help you get really good jobs. Yeah, you might not get to be a Bay Area Unicorn. But you probably weren’t going to anyhow.

CL XXXIX: Island Wildlife 8 Sep 2019, 9:00 pm

Our probably-last Cottage-Life weekend of 2019 featured cetacean encounters and rodent rage. But I didn’t manage to photograph any of that, so just the usual trees and sunsets.

Like this:

Howe Sound sunset


This morning I was alone at the breakfast table, considering the sea as one does, when I saw them and was yelling “whales!” at the family. There were five killer whales, one an adorable juvenile who put a little hop into every surface-to-breathe. They weren’t in a hurry, stayed for a while.

“Orcas” is more common but I say “killer whales”. Scientists prefer that too, although “orca” has etymological standing, per Wikipedia. A whale specialist told me “orca” was popularized by the marketing group at SeaWorld; they didn’t like having “killer” in their big stars’ names; family values, y’know.

Keats Island forest

Anyhow, our local killer whale population is in trouble, and that trouble is about to get worse since the Government of Canada has in its wisdom decided to bless the tar-sands pipeline that will run a tanker more or less every day through their home waters. So a random visit is a precious gift, one that might never be repeated.

Here’s today’s only picture with an animal: A feral kid scrambling up a alien-flavored tree construct.

Kid climbing tree and stump on Keats Island

Rodent rage

What happened was, I came up the ramp to the front of the cabin and there, where we have the barbecue and the recycling, was a big chunky raccoon with a beautiful silky coat, bursting with health and vigor, looking for leavings. I thought “Let’s put on a show here and disincent repeat visits” so I yelled “Hey, get outta here” and charged him. He skedaddled to the corner of the cabin and turned left, but had traction problems on the deck so I rounded the corner right behind him. He opened a lead on the dirt trail alongside the cabin and turned left again. I chased him down the path at the cabin’s back but it wasn’t close; he turned left yet again but was out of sight by the time I got to that third turn.

I thought I’d made my point and sauntered back up to the front door — and there was the raccoon again, finishing his foraging. I kind of lost it, shrieking “Gimme a **** break you **** sleazy ****!” and sprinting like a teenager.

Genuine rage makes all the difference. He headed straight sideways into the woods, never looking back. I thought he was running before, but this time he turned on his warp drive, like when they say “Engage!” on Star Trek.

He probably skulked back when we sailed for home this afternoon but we keep the place battened down in absentia. I hope we’ll be less likely to see him on future occasions when we’re in residence.

Portraits of Puppets 21 Aug 2019, 9:00 pm

If you happened to check out my Twitter feed on the weekend, you’ll know that I attended a pair of dueling rallies outside a train station in central Vancouver. On one side, a crowd in black supporting the Hong Kong protests; on the other a red-clad flag-festooned squad bringing Beijing’s message. I was dressed in black and took pictures of the other side.

Pro-Beijing demonstrators in expensive cars

The issue

It’s a no-brainer. Hong Kong isn’t perfect but it’s a civilization, with laws and with access to the world. China is a big hulking cut-off-from-the-world prison for the mind, built on systemic brutality and corruption. I admire the Hong Kongers’ courage and fear for their future. I can’t protect them from the PRC but at least I can show where I stand, and who knows, it might even make a difference if enough other people do too and the Beijing bastards decide that crushing HK might be bad for business.

People in red

I didn’t take pictures of the pro-HK side because you can bet the other side wouldn’t hesitate to use such things against them. It was probably superfluous since the Beijingers were loaded with cameras.

Pro-Beijing demonstrators

Now you’ve seen all the signs they had. It was all very uniform and organized on the Beijing side, everyone was waving the same thing. On the HK side there was an explosion of hand-lettered signs among a scattering of HK and Canadian flags. In the picture above, I particularly liked the worried-looking dude looking left through glasses, and got a nice picture of him when the sun came out.

Pro-Beijing demonstrator

He didn’t seem to be having much fun, but that’s probably a little misleading because there were definitely people on that side who were into it.

Pro-Beijing demonstratorPro-Beijing demonstratorPro-Beijing demonstrator

These two dudes were definitely full of that old school spirit, mind you one of them had his little camera rolling non-stop.

And you have to ask who these people were? I suspect they fell into three baskets. First, committed pro-Party people, maybe from the Consulate, maybe with less official standing, genuinely on the tyrants’ side — the rewards are good. Second, Chinese folk here in Vancouver who’ve stayed inside the Party-line bubble, there are media offerings to help. Third, people who don’t like the Party or (more likely) don’t like politics, who’ve had effortless-but-irresistable family or professional pressure applied.

Let’s just call them all puppets, because that’s how the people pulling the strings think about them. Here’s the puppeteers’ infrastructure:

Pro-Beijing demonstrators

Through the crowd, you can see the table where puppets can get their placards and posters and flags. I’d just love to know who organized that table and paid for the printing.

The shouting contest

That’s what the demonstrations turned out to be. The size of the red and black crowds was roughly equal — maybe a few more on the black side? — and the police did a good job of keeping space between them; it helped that nobody I saw apparently wanted to start a fight.

Disclosure: I thought how satisfying a sudden charge across the open space at the puppets would have been, but fortunately I’m grown-up enough to keep my fantasy life where it belongs.

In terms of faces and if you ignored the colors, a lot of the people on either side could have been transplanted to the other without anyone noticing. But the black side was a little older and more grizzled and a whole lot more spontaneous and cracked better jokes and the signs were better and by the way were on the side of freedom.

Coda, with hot cars

I was kind of in the middle of the black demo and noticed that every few minutes, there’d be a roar of approval from the puppet side, countered by a thunder of booing from ours. By watching where people were looking, I traced the source to the road going by. What was happening was that a few bright Beijing sparks were driving their expensive sports cars round and round the block waving PRC flags.

Pro-Beijing demonstrators in expensive carsPro-Beijing demonstrators in a Ferrari

Who’s the white dude driving the Ferrari, I want to know.

Which I think kind of underlines the key point. Like Orwell said, the object of power is power. A chief pleasure of power is showing it off, and driving around in Lambos and McLarens and Ferraris is a pretty satisfying way to do that. Particularly when you can soak up applause from the plebeians on your side and jeers from your enemies.

It’s pretty simple

The people of Hong Kong don’t want to be censored, tortured, imprisoned, and killed by those whose asshole kids are driving supercars around West Coast cities across the Pacific. I’m with them.

Talking Hong Kong Blues 18 Aug 2019, 9:00 pm

I’m imagining a discussion that might have taken place in Baidaihe at some point this month at the annual CCP summer offsite.

“Getting ugly in Hong Kong, and I’m not sure our Ms Lam is moving things in the right direction.”

“I hear from the people on the spot that what the good people want is just peace and quiet, this is just a bunch of teenage assholes making trouble.”

“Nobody wants to give their boss bad news. Haven’t you watched the BBC coverage? Maybe you’re hearing good things from your staff, but let’s suppose the gweilo TV is right? What are we going to do?”

“We’re doing one thing that’s working, going after the troublemakers’ bosses. We took down Hogg at Cathay Pacific. That’ll make every ambitious manager in HK go on the warpath to keep their employees in the office and off the damn streets. Hong Kong, it’s about three things: Money, money, and money.”

“Except for, the bad guys are getting 20% of the population out in the streets. That’ll include people who work for every fucking bank and real-estate developer and shipping company, are we gonna get every CEO in South China fired?”

“But the police say there were only 128,000 people out!”

“The HK police are idiots and in case you hadn’t notice, they’re losing in the streets.”

“I think they’re winning. There haven’t been any arrests or violence at the last three days of protests.”

“You think that’s good?! If the word starts going around that you can get away with large-scale activism as long as you keep it peaceful… do you like the idea of four million people out on the streets of Shanghai? Or a couple of million in Guangzhou?”

“What do you mean about the word getting around? The people of China are well-protected from dangerous foreign ideas, they’re not going to watching those shitty BBC liars.”

“Don’t you look at tourism figures? Fifty one million people from our side visited Hong Kong last year. They’ll all be talking to their friends and relations.”

“Yeah, well that’s maybe ten million people, a lot of them visiting every week on business. And, let’s be honest, they’re the same ones who travel overseas and already have lots of exposure to fake news from people who hate the Party. They probably all have VPNs already.”

“On top of which, those people are making good money and they owe it to us and they know it. They’ll bloody well watch what they say.”

“You guys, this is the same kind of thinking that got our 1989 leadership into trouble, letting those ‘innocent’ students stay in Tienanmen until they thought they owned it and we had to go in with the tanks and machine guns!”

“He’s right. We have the muscle all built up in Shenzen, they can be holding down Central and Tsim Sha Tsui in 72 hours and there’ll be no more of those fucking umbrellas. On top of which, the good people there will throw flowers at our guys and go back to making money in peace and quiet.”

“Suppose they don’t. Suppose there are a quarter million assholes dressed in black yelling ‘Gaa Yau!’ at each other and ‘Two Systems!’ at us and, flashing lasers and the real fringe throwing molotovs, and all with masks so we can’t ID many, and fading away into the MTR, and then another quarter million out the next day?”

“Brother, if it really comes down to them versus us, it’ll be us. Just like in 1989. It’s not just riot-control equipment waiting there in Shenzen. And any solo hero standing in front of a PLA tank this time is going to be ashes before he gets on CNN.”

“Screw CNN. It’ll be live on YouTube and Instagram and Twitter with a couple of billion people watching, and highlights of PLA tanks squishing Hong Kong patriots waiting for people who were asleep at the time.”

“So what? The people who matter need to do business with us, what do they care what kids watch on Instagram? Are they going to walk away from the chance?”

“Well, Google did.

Here’s another thing. Suppose they’re holding out in Mong Kok and every other skeezy neighborhood away from Central and there are people in all those buildings throwing shit at us from the 3rd through 20th floors, and they turn trucks sideways in those awful little streets, how are we going to get them out?”

“The PLA is not going to be stopped by a bunch of acne-faced cockroaches! Whose side are you on?”

“Western politics is weird. They eventually turned their backs on people from their own tribe in Rhodesia and South Africa in favor of a bunch of black people!

“We have our people getting our side of the story out in every Western capital; the right kind of students marching, shouting down the local HK troublemakers.”

“Give me a break, those clueless princelings haven’t the vaguest what they’re living among. I see their latest brilliant idea is to drive around in their Lambos and McLarens waving Chinese flags. Are you really really sure you want to make that bet?”

“Look, our economy is less about imports and exports every year. If the world doesn’t want us any more, then we don’t need them! We’ll just turn our backs and China will be China for Chinese, and it’ll be great.”

“Yeah, well I don’t want to miss aprés-ski in Zermatt or my place with that view in West Vancouver.”

“You might have to, because if those HK cockroaches prove they can tell us to fuck off and go on having a decent life and making money… you talk about bringing in muscle from Shenzen, what I worry about is people there starting to dress in black.”

“Yep, let’s just keep the PLA ready to roll, and hope it doesn’t have to.”

“Hope is not a strategy.”

Jeanneau 795 Review 6 Aug 2019, 9:00 pm

In Europe this boat is called the Merry Fisher 795 and in the New World, the NC 795. I’ve owned it for a few months and improved it a bit and taken it a few places and feel like sharing.

Why review?

At this point, regular readers are thinking WTF, boat review?!? I’ve only been on a handful, I’ve only owned one since 2012, I’m still occasionally baffled by nautical jargon, and my command of knots remains imperfect.

Here’s why. When you go shopping for a refrigerator or car or coffee-maker or TV or (especially) camera, there are loads of excellent detailed skeptical-voiced reviews you can read before you cough up the money. Boats (which cost more money than most of those things) are different. All the online reviews seem to be from dealers or magazine-writers on the comp, and are by and large paeans of praise.

Jeanneau 795

There are owners’ forums out there, but they tend to focus on specific problems and solutions. What’s especially missing is “I have one of these, here are the good parts and bad parts.”

I’m not completely unqualified. It’s been at my dock for a few months, I’ve installed improvements, I’ve piloted twenty-plus hours on it, motored through extreme beauty and nasty scary rough water, taken guests on pleasure cruises and a grouchy family on a tired commute, and used it as an office for a few afternoons.

So I’ll see if I can beg a few links from other 795 owners on the forums and get this a bit of GoogleJuice with the aim of better equipping other boat shoppers like me.


Jeanneau has been making boats since 1957. That link is to French Wikipedia; the English version is mostly empty and I should fix it up. The interesting, complicated story is nicely told in English by Malcolm Perrins on a Jeanneau-owners community site. The company, since its founding by Henry Jeanneau, has been sold multiple times to US and French companies and is now owned by Beneteau.

The dealer told me that the Jeanneau powerboats are built in Poland — this made him happy because for some obscure reason it leads to favorable import-tax treatment. The Jeanneau America site says “Built in America” and the first version of this piece doubted that, but a reader from Michigan wrote “We have the NC 895. It is built in Cadillac Michigan. They took the old Fourwinns plant.”

Our boat’s curtains are labeled “Made in France” and the appliances such as chargers and thrusters and fridges are Eurobuilt and their manuals have Italian or French as the first language, with English further back in the book. So I’m inclined to believe the France/Poland story in this case.

People who are buying a boat care a lot about dimensions because one of the hardest parts is finding a place that’s big enough and deep enough to park it. The 795 is 7.34m or or 24’4" long, and 2.99m or 9’9" wide, with a hull depth of a mere 0.56m or 1’10" — that’s with the outboard hoisted, which is how you normally park it.

The 795 comes with a Yamaha outboard, either 150 or 200hp, and lots of options. It’s got a modest-sized berth in the bow, a tiny but functional head (as in bathroom), and similarly tiny stove and fridge. What electronics you get apparently depends on the dealer.

Good: Engine

We have the Yamaha F200 and since it’s an outboard, there’s more room inside the boat. I’d never really been aware of this line of motors but now when I walk around any marina I see that somewhere between a third and a half of the powerboats are wearing them. So, right in the middle of the mainstream.

It’s got a very decent little electronic control screen on the dashboard and the docs are clear and comprehensive.

We set it at 4500RPM and it pushes the boat along at a little over 40km/h, depending on wind and waves. If you open it wide up on smooth water you can get up well over fifty clicks but the experience is not relaxing, or cheap either.

Good: Comfort

Not just good, excellent. The pilothouse has room for a driver and two more people in comfort, four if they’re not chunky or need extra personal space. (Protip: The aft bench is way more comfy.) The cockpit out back has forward-facing seating for three with a cushion to lean back on, and then a couple more benches but they’re less comfy. We’ve been out for a slow cruise on a warm night to watch fireworks with seven aboard and it was just fine.

Fireworks in English Bay, photographed from a Jeanneau 795

The pilothouse is really the best feature. It has a sliding “Alaska bulkhead” which means a glass door that closes, leaving the motor and its racket outside; inside, you can have a civilized conversation without shouting.

Good: Swimming platform

It’s just big enough and has a nice practical swimming ladder. We’ve used it every time we’ve been to the cabin. I shot that fireworks picture above sitting on the platform dangling my feet in the Pacific; very relaxing.

Bad: Living quarters

While they advertise two berths, realistically there’s just not enough space for more than one couple and they’d better be intimate. What with the tiny fridge and stove, I don’t think this is the boat for a lengthy family cruise up a wild coastline.

Good: Windshield

And I mean awesome. This puppy’s front glass is the size of a small European nation and when you’re sailing home with the sun behind you in a long Canadian sunset with the mountains filling the sky in front, well, there just aren’t words for that.

Vancouver through Jeanneau 795 windshield

Coming into a Vancouver from a weekend at the cottage; about two thirds of the windshield are shown. That’s the West End at the left and the Burrard Street Bridge behind the wiper. The little grey screen on the left is the Yamaha engine readout; some timing thing prevents the Pixel 2 from photographing it properly.

The wipers’ coverage isn’t that great, leaving swathes of uncleaned glass in dirty weather, but you can see the important stuff. And it comes with a windshield-washing squirter system just like your car’s, which turns out to be brilliant when you hit big waves and they splash up and want to leave sticky salt crystals where you’re trying to look out. You load it with windshield fluid from the gas station.

It’s worth mentioning the side windows too, which open and close easily and let loads of fresh air in at cruising speed without blasting your head off, and seem completely rain-proof too.

Good: Bow thruster

This is magic. We have a nice easy tie-up along the side of a dock, not crammed into a little slip, but it’s on the left as you come in and the boat wants to be tied up with its right side to the dock, so a 180° turn in tight quarters is called for. With the thruster and a light touch, it’s reasonably straightforward. The thruster is also useful as compensation for any dumb piloting errors around the dock — of course, these never happen when I’m at the wheel.

Good: It’s hackable

In the Jeanneau owners’ community I found an active boat-improvement culture; they’re always adding something or replacing something else. By dint of extensive research from primary sources, by which I mean watching YouTube videos, I have learned how to attach things to fiberglass (Protip: Get a countersink bit for your drill) and have so far improved ours by fastening the fire extinguisher to a handy bulkhead, equipping the head with a toilet-paper rod, and installing a garbage-bag holder. Call me Ishmael.

There are a variety of surfaces suitable for equipping with electronic upgrades or just decorations. We’ve decorated a couple with family photos.

Bad: Documentation

Hailing from the technology space means that I should be restrained in criticizing other professions’ end-user documentation. The boat came with a nice Jeanneau-branded satchel full of dead trees; the quality of exposition and language is, well, mixed. Highlights are the books for the Yamaha engine and the boat itself. The low point is the Lowrance navigation electronics tome, obviously executed by manic pixies on acid. The information is more or less all there but requires deep digging and Zen calm to extract.

My favorite though is the anchor-winch system, which is written in impenetrably-nautical English. Fortunately it’s accompanied by a diagram with all the parts carefully named and numbered. Unfortunately, about half the nautical names studding the text do not appear in the picture.

To be fair, I managed to figure it out well enough to anchor us (in shallow water with nearly no wind) for firework-watching.

My niece capturing a water-color of Indian Arm

My niece Anne capturing a water-color impression of Indian Arm.

Good: Piloting

The driver’s seat is comfy, the steering and throttle are crisp and responsive, and the view forward and aft is excellent. Steering at speed is a little heavier and slower than our previous inboard-outboard, but it’s plenty good enough to dodge a floating log. I’d actually like a bigger steering wheel that’s closer to me, so there’s another boat-improvement project.

Good: Access

Getting from the cockpit around to the foredeck, and up and down the sides for washing and so on, is all dead easy. The cabin is a little off-center, leaving a walkway along one side; and both sides have intelligently-placed handholds to make things easy and safe.

Bad: Flat bottom

The draft is remarkably small and the bottom, compared to the last boat, is pretty flat. This means that when you hit big waves, for example a ferry wake that you stupidly failed to notice until you were right on top of it cruising at 40km/h, you tend to skip along from wave to wave, hitting each one with a jarring “slap” of the flat bottom. This can fling passengers about a bit in a seriously uncomfortable way. Protip: Be on the sharp lookout for incoming waves and slow the hell down.

I’m not a bossy skipper but we have imposed one rule: If you want to move around the cabin, say so and we’ll slow down while you do. This after I nearly put my niece in orbit when she was going to get her backpack and I slammed on the brakes because I thought I saw some peril out front.

Good: Home office

I’m doing WFB (work from boat) one afternoon most weeks now, and it’s just terrific. The aft passenger-side bench is reasonably ergonomic and the table’s at a sane height. I often make a cup of tea and stash a snack in the fridge. I have taken conference calls, drafted and reviewed documents, reviewed code, and once (cackling with glee) checked in code to the AWS production repository.

I haven’t convinced any colleagues to come down for an in-boat meeting yet; it’s just a matter of time. But I’m just not gonna install whiteboards.

Mixed: Online community

The biggest is the Owners’ Forum, which is OK but suffers from Jeanneau having so many products. There’s also a group on Facebook, obviously. I’ve picked up valuable tips in both places.

Bad: Missing pieces

There’s no automatic bilge pump, which I find shocking, but on the other hand I have to say it stays almost bone-dry down there, even with mixed hot & cold weather, bashing through pretty rough seas, several days of heavy rain, and regular thorough washing (the honeymoon is still on).

There’s no horn; our previous boat had one and while I only ever used it once or twice, I was glad of it.

There’s no built-in heater. Our journeys typically aren’t long enough to need one on the water, but this might be an issue in home-office mode. Multiple owners have installed diesel heaters, and I have a nice little AC space heater that I’ll try out when on shore power. Similarly, there’s no air conditioner, which is more of a problem than you might think up here at 49°30'N because the pilothouse has so much glass, it’s a greenhouse.

Jeanneau 795 tied up at Keats Island

There are only two cleats, fore and aft. When you’re tying up to a floating dock for a weekend in Howe Sound (see above), which after all is part of the Pacific, you really want one and ideally two spring lines along with the basic fore and aft. Several owners have figured out how to install an extra central cleat, and I’ll look to do that.

And your conclusion is?

Count the “Good”, “Bad”, and “Mixed” headlines above. The good stuff wins, by a wide margin. I’ve got no standing to say whether or not this is a winner or loser against the competition because I haven’t owned the competition. What I can say, a few months in, is that it meets our needs very well.


Here are the things I’ve purchased to improve the experience:

  1. SeaTeak 62634 Insulated Four-Drink Binocular Rack — I have two of these things velcro’ed down behind the sink. The binoc-shaped spaces also work for big coffee mugs with handles.

  2. Dell Ultra HD 4K 24-Inch Monitor P2415Q — just the right size for outboarding to my company MBPro, and comes with USB so I only need one plug to power everything. I need to install something to hang it up on the berth bulkhead when not in use, at the moment it’s lying face-down on the mattress, which is OK but takes space.

  3. 4.5" 12V Stepless Speed Car Fan — sold by different vendors in the US & Canada. Like I said, it can get toasty in the pilothouse but this guy takes care of it just by keeping the air moving.

  4. Rod Holder Mount Boat Flagpole — the 795 has two fishing-rod holders but no flagpole. Hey-presto! The Canadian flag looks great out there but we haven’t figured out which minor ensign to fly beneath it. Patti Smith fan club? Antifa emblems? Not sure.

  5. From Davis Instruments, Shockles LineSnubbers and LineGrabbers; nothing specific to this boat, just a coincidence that I discovered them recently. If you tie up where it might get rough, you need these.


My relationship with the previous boat was pretty prosaic. It got us back and forth to the cabin and was kind of charming with its wood trim, but it always needed fixing and there were important subsystems I never learned to understand. This is a whole different kettle of fish. I’m starting to develop sympathy with the oft-repeated Kenneth Grahame quote from The Wind in the Willows:

Believe me, my young friend, there is nothing — absolutely nothing — half so much worth doing as simply messing about in boats. Simply messing… about in boats — or with boats. In or out of ’em, it doesn’t matter. Nothing seems really to matter, that’s the charm of it. Whether you get away, or whether you don’t; whether you arrive at your destination or whether you reach somewhere else, or whether you never get anywhere at all, you’re always busy, and you never do anything in particular; and when you’ve done it there’s always something else to do, and you can do it if you like, but you’d much better not.

Page processed in 2.937 seconds.

Powered by SimplePie 1.5, Build 20180814171753. Run the SimplePie Compatibility Test. SimplePie is © 2004–2020, Ryan Parman and Geoffrey Sneddon, and licensed under the BSD License.