ongoing by Tim Bray

ongoing fragmented essay by Tim Bray

Plague Journal, April 4 4 Apr 2020, 9:00 pm

I’m an optimist so I don’t put the year in the title. Once again: Writing is therapeutic. Open up whatever program you use to write stuff in, and see what comes out. Today’s adventure was the Socially Distanced Farmers’ Market.

Socially Distanced Farmers’ Market

They were organized as hell, the market subdivided into three Zones, each with its own line-up, social distance chalked on the sidewalk. The people density was unusually high in the neighborhood and a lot of people have started just walking down the middle of the street, screw the motorists. I find this cheering.

People are so open and friendly! Everyone has a smile for everyone and random conversations break out between strangers. This is usually a good thing, but in the Zone 1 lineup I found myself behind a conspiracy-theorist, talking about how the “higher ups” were getting ready to impose food rationing.

I think serious food shortages unlikely in the developed world, but parts of our agricultural industry depend for harvesting on poorly-paid abusively-treated migrant laborers. When they can’t come, we’ll see if agribiz lets the produce rot in the fields or raises wages enough to attract unemployed Canadians; and produce prices correspondingly.

Market menu

Potatoes. More potatoes. Damn, really a lot of potatoes. We have a stash but I bought fingerlings anyhow because they were cute. The first new harvest in mid-spring is rhubarb; I bet there’s some starting next week.

The shopping list comprised apples, rolled oats, salad greens, dill, chives, cilantro, and a red onion. I scored about 50% — by noon-ish when I got there, a lot of vendors’ shelves were looking bare. But also I got artisanal gin, handcrafted chocolate, and wildflower honey.

We’re having a virtual movie party with a friend this evening; Turtle Diary, I think.

Take time to be kind to each other; loved ones and strangers too.

MacOS Lore, Early 2020 4 Apr 2020, 9:00 pm

I’ve used a Mac for an extremely long time; long enough that this blog’s topic tag is Mac OS X, what macOS used to be called. Herewith a looking-back summary.

Previously on Mac Lore

Clicking on that “Mac OS X” link cost me a half hour rat-holing on my journey over the years. So you don’t have to, here’s a compendium of advice I give people about the Right Way To Do It; although some of the recommendations are not Mac-specific:

  1. My Dock

    Your screen is wider than it is high. So put the Dock on the side not the bottom.

  2. Dock size and magnification are a matter of taste, but don’t auto-hide it, because…

  3. Remove from the Dock every app that you don’t use regularly. That way, everything there is either running or likely to be, and it becomes a useful visual status check. To the right is a snapshot of mine as I write this.

  4. Go spend some quality time in the System Preferences for Trackpad. Definitely turn on “Tap to click” and “Secondary click”. Then use the accessibility preferences to enable double-tap-and-drag.

  5. While you’re in System Preferences, make sure your keyboard repeat rate is turned up to the max; few things are more boring than holding down the spacebar or whatever and watching the cursor inch across the screen.

  6. Command-space, which brings up Spotlight search, is your friend. It’s really pretty good. Not enough people know that you can highlight things in the result list and type command-I to get a nice little popup with useful information about what you just found.

  7. Use the tab trick in your favorite browser for one-click access to things you care about. (When I wrote that piece it didn’t work in Safari, but now it does.)

  8. Keep a couple of browsers around. Chrome and Safari are both great on Mac, Firefox is OK but recently I’ve found it slow. It’s common to use one for work stuff and the other for personal. Another option is to be logged into Google in only one of them and Google-invisible in the other. Speaking of which, Safari is starting to have a strong privacy story.

  9. Preview

    Your browser will open PDFs directly and want you to read them there. Don’t. Download ’em and open ’em up in the awesome Preview app. Particularly if they’re big or complicated; Preview laughs at 500-page graphically-complex documents and provides a superior read/search/navigate experience.

  10. Despite the fact that Preview is great, do not try to use it to fill in legal forms. It will look like it’s trying to work, but it won’t. For that purpose (and that purpose only) go get Adobe Acrobat Reader.

  11. Related to Preview: Let’s assume you’re a professional who sometimes needs to show off your work. So use the command-control-shift-4 gesture to grab a piece of your screen, shift over to Preview, hit command-N and it creates a new graphic with what you just captured. The only fly in the ointment is that when you save it, it’ll want to use PNG and you almost always want JPG, so you have to toggle that on the Save menu.

    This is how I captured the Dock image above.

  12. Keynote

    If you have to give a presentation, use Keynote; it and Preview are Apple’s two truly great Mac apps. Do not go near PowerPoint, it’s a travesty on Mac.

  13. Learn to use the control-key navigation tricks. They make editing text — any text in almost any app — dramatically faster.

  14. Turn off all the notifications you possibly can. You should own your time. If you have a reasonably active life there will always be new things to read in mail and Twitter and Slack and so on; so go read them when you come to a stopping point. The only notification I leave on is the desktop Signal app, because you have to know me pretty well to reach me there. And (at work) mentions on Amazon Chime.

  15. Inbox Zero is a great idea but unattainable by most of us. Instead, try the Low-stress Inbox technique.

  16. Use a password manager. Really, please use one.

I wrote the first of these in 2002. I wonder how many more are in my future?

Plague Journal, April 2 2 Apr 2020, 9:00 pm

Hey folks, one decent therapy for times like these when the world’s trying to drive you crazy is to tell your story; doesn’t matter if anyone’s listening. This is adapted from an email to the family that got kind of out of control. Write your own #PlagueJournal entries and I’ll read ’em.

I feel vaguely like I’m setting a bad example as I cycle furiously on empty-ish streets across town each day to the boat and back; the smallest office I’ve ever had, but the view is decent.

The weather remains obstinately wintry, temperature refusing to venture out of single digits, which is OK when the sun’s out which it mostly isn’t. Nonetheless everything that can grow flowers is already showing them or expanding the buds as fast as possible. Mom was supposed to come visit us about now and there would have been lots to look at.

Downtown, people gather on their balconies and cheer wildly for three minutes for the caregivers at 7PM when the hospital shifts change; we were driving through the neighborhood last week, unsuspecting, as it exploded. Not a dry cheek in the car. Locally we’ve revived and expanded a long-neglected neighborhood mailing list, seeking people who might need some help; plenty are offering but everyone seems to be making a go of it. Anyhow, this very evening we gathered in a socially-distanced way at the end of the block to bang pots and drums and tambourines and clap hands for three minutes, lots of smiles and none forced.

Some people are much more monastic in their isolation: go shopping once every other week, stay inside. We find the grocery stores are sanitizing and social-distancing effectively, so we shop more often. Also we’d really like some of the restaurants to survive this thing so we’re getting take-out a couple or three times a week. Plus we go for lots of walks - there’s a new sidewalk courtesy where you make space for each other, stepping into the (empty) roads or on people’s lawns if need be. Very Canadian.

We pick charities and send them money but so much of the population was already living so close to the edge, these wounds will take a long time to heal.

We’re actually keeping in better touch with our friends than in healthy times, via Zoom and Skype and so on. Unfortunately what we talk about mostly is the plague news. I’m kind of tired of talking about it. One of the best military blogs is entitled War is Boring — well, so is Covid-19.

I, a data-driven numbers guy, find the daily recitation of statistics maddening, although everyone in the province loves our chief medic Dr Bonnie Henry, who has a silken voice and a Stoic demeanor. They give numbers like “number of cases” which I think means “positive tests”, a number that is entirely useless because the testing is (quite reasonably) directed at the most vulnerable and critical demographics. I am beginning to zero in on the number of cases admitted to hospital every day, or rather the rate of change in the number admitted — at least there’s clarity in what that means — and in BC, the rate of change in daily admissions is zero-ish. Which is not exactly good but not catastrophic. Catastrophic is New York today and Florida & Louisiana & Alabama & Texas looking forward four weeks. I don’t want to think about India and Africa.

Alberta is doing a little better than us, Ontario worse but not terrible, Quebec worse. But not bad like America, so far.

At work, we are running hot. Everyone’s stuck at home and using the Internet, which means us. Over on the retail side, the order spikes are frightening given that a lot of our employees are staying home for excellent reasons and the ones who are coming in have to work at half-speed due to constant disinfecting and social distancing. I see headlines in progressive publications saying how we are cruelly ignoring the plague conditions; and internal emails about all the products that are going on four-or-more-week delivery because they have to run everything extra-slow to protect the staff so they can keep shipping groceries and cleaning products. I really honestly don’t know what to think.

Our side of the company just has to make sure there are enough incremental waves of computers available each new day to keep Zoom and Netflix and ambulance-dispatch apps on the air.

The boy and the girl are both at-school-online right now, Lauren and I at-work-online. If it weren’t for the boat we’d be in trouble since the guest room is under post-asbestos-remediation reconstruction and we’re packed in pretty tight already 9-to-5 without me being in the house.

I’d really rather not be living inside a historically-significant news story. But we all are, so the only choice is to make the best of it. The virus doesn’t care how brave you are, only how smart you are.

In closing: https://xkcd.com/2287/.

Facet: Push vs Pull 21 Mar 2020, 9:00 pm

If you want to process events, you can fetch them from the infrastructure or you can have the infrastructure hand them to you. Neither idea is crazy.

[This is part of the Event Facets series.]

When you make a request to fetch data, that’s called “pull”. Which is an off-by-one error; the “u” should be an “o” as in “poll”, because that’s how most network stuff works. I’ve heard old farts of my own vintage claim that on the network, polling is really all there is if you dig in deep enough. But we do use the term “push”, in which — maybe it’s just an illusion — the infrastructure reaches out and hands the event to you.

For example

I’ll use two AWS services that I work nearby.

First, SQS, which is pure pull. You make one call to receive a batch of messages, and then another to delete them, once processed. If they’re being pushed in the front end of the service faster than you’re taking them out, they build up inside, which is usually OK.

Pull

Pull!
Credit: Petar Milošević / CC BY-SA

Push

Push!
Credit: Rjhartley at English Wikipedia / CC BY-SA

In SNS, by contrast, events are dropped onto a “Topic” and then you subscribe things to the topic. You can subscribe SQS queues, Lambda functions, mobile notifications, and arbitrary HTTP endpoints. When someone drops messages onto the topic, SNS “pushes” them to all the subscribed parties.

On webhooks

“Webhook” is just another word for push. I remember when I first heard the idea in the early days of the Web, I thought it was complete magic: “I’ll just send you something, and the something will include the address where you HTTP POST back to me.” Asynchronous, loosely-coupled, what’s not to like?

There are a lot of webhooks out there. We talk to the EventBridge Partners who are injecting their customers’ events into AWS, and one thing we talk about is good ways to get events from customers back to them. The first answer is usually “we do Webhooks”, which means push.

On being pushy

On the face of it, push delivery sounds seductive. Particularly when you can back your webhook with something like a Lambda function. And sometimes it works great. But there are two big problems.

The first is scaling. If you’re putting up an endpoint and inviting whoever to push events to it, you’re implicitly making a commitment to have it available, secured, and ready to accept the traffic. It’s really easy to get a nasty surprise. Particularly if you’re building a public-facing cloud app and it gets popular. I can guarantee that we have plenty of services here in AWS that can accidentally crush your webhook like a bug with a traffic surge.

Then there’s administration. You want data pushed to you, but you want it done securely. That means whoever’s doing the pushing has to have credentials, and you have to trust them to manage them, and you have to arrange for rotation and invalidation as appropriate, and you really don’t want to be handling support calls of the form “I didn’t change anything and now my pushes are failing with HTTP 403!”

On pulling

Polling, on the face of it, is a little more work. If you care about latency, you’re going to have to have a host post a long-poll against the eventing infrastructure, which means you’re going to have to own and manage a host. (If you’re not latency-sensitive, just arrange to fire a Lambda every minute or so to pick up events, way easier.)

On the other hand, you need never have a scaling problem, because you control your own polling rate, so you can arrange to only pick up work that you have capacity to handle.

On the admin side, everybody already owns the problem of managing their cloud service provider credentials so presumably that’s not extra work.

I’m not saying push is never a good idea; just that grizzled old distributed-systems geeks like me have a sort of warm comfy feeling about polling, especially for the core bread-and-butter workloads.

A small case study

Ssshhh! I’m going to tell an AWS-internals secret.

I talk a lot with the SNS team. They have a front-end fleet, a metadata fleet, and then the fleet that actually does the deliveries to the subscriptions. It turns out that the software that delivers to HTTP-endpoint subscriptions is big, and maybe the most complex part of the whole service.

When I learned that I asked one of the engineers why and she told me “Because those are other people’s endpoints. They make a configuration mistake or forget to rotate credentials or send a flood of traffic that’s like eight times what they can handle. And we have to keep running in all those cases.”

Plague Advice 14 Mar 2020, 9:00 pm

A couple weeks back “Social Distancing” would’ve been a Big Thinker’s title in The Times, about the Downside Of Facebook. Now it’s a best practice if you care about flattening the COVID-19 curve and saving grandmothers. I’m a believer; recently I tweeted Cancel Everything and I meant it. But this shouldn’t mean that you can’t go outside; or shop; or photograph.

Today we went to the Riley Park Winter Farmers Market, eleven blocks away in cold spring sun.

At the Riley Park Farmers Market

It’s way less crowded than the supermarket.
Damn it was cold, 5°C at best. But I bought beets.

Plague Advice

Find a way to support your local merchants to the extent you can while staying safe. Don’t be the one who who didn’t know what you’d got till it was gone.

Here in BC we’re not in hard-lockdown where you have to stay inside and order your food. I and many others here put a lot of weight on words from our Provincial Health Officer Dr. Bonnie Henry (no Twitter of her own but she has a fan club). As of this writing, she says that for now, it’s OK to shop for food and basics, to eat out, and especially to do things that are outside. Farmers markets offer two out of three.

Just because that’s OK for us doesn’t mean it’s OK for you. BC seems to have got lucky with early containment; it helps that anybody with Those Symptoms can have a COVID-19 test for the asking.

I suppose the containment will break down at some point and Dr Henry will bring the hard-lockdown hammer. Because she knows her shit and speaks the truth, she’ll bring the populace along with her.

Bonnie Henry

How has Knowing One’s Shit and Speaking Truth become, so often, anything but the default expected behavior?

Plague Advice

Wherever you are, find your local Dr Henry equivalent and just Do whatever the fuck they say.

Back to Farmers Markets. There’s more space between you and the staff and the other shoppers. The aisles are wider. There’s no door or doorhandle. No shared shopping carts. No sharing atmosphere in an enclosed space.

Now I see I did one thing wrong: I paid with cash. These days every greyhaired ponytailed organic-herbs vendor has (in Canada at least) a little goober you can wave your credit card or phone at; no touching required. Cash is a notorious germ vector at the best of times, which these really totally aren’t.

Plague Advice

After you’ve been to the market, don’t touch your face till you get home, then wash the hell out of your hands. Where by “the market” I mean anywhere.

At the market there was a stringband, and also a dude in a wheelchair with an N95 facemask and electric guitar, soaring chords with snarly echo. I left donations in both their open instrument cases; I hope they were careful to wash their hands after handling the cash.

At the Riley Park Farmers Market

Plague Advice

Support performing artists. Because performances aren’t safe so performers don’t get paid. I have tickets to see Martin Barre, the Cowboy Junkies, and Billie Eilish; none of those performances will happen and I’m not asking for my money back. Go visit the Web sites of the musicians you like and find out how they sell music and merch and buy some already.

When I read the stories about the hard-lockdown locales, the policy seems to include no going outside. Sure, don’t enter crowded spaces, including public transit. And don’t crawl malls. But if there’s a green space you can walk to and it’s big enough that people can maintain a respectful distance, I don’t see the downside.

The upside is you’ll stay saner.

Plague Advice

Go (carefully) outside!

Facet: Broker vs Serverless 10 Mar 2020, 9:00 pm

Your event infrastructure might be a service in the cloud or might be an actual computer (or cluster) you connect to. Both choices are perfectly sensible. The trade-offs? It’s complicated.

[This is part of the Event Facets series.]

At AWS where I work, our mainstream home-grown services (Kinesis, SQS, SNS) are all serverless. I mean, there are servers, lots of ’em, but you can’t see ’em, there’s just an endpoint that you connect to, usually with HTTP, to produce and consume events. Which should be unsurprising, AWS is built on the proposition that everything should be a service. Here are two:

SQSSNS

But there’s another approach, usually referred to as a “message broker”. Brokers are software packages that push around events or messages, and they typically need to be installed and run on servers that you own or rent. Then your app connects to them to send and receive. Here are two:

ActiveMQRabbitMQ

The trade-offs, well, they’re complicated.

Serverless?

These services have one huge advantage in that you don’t have to think about scaling. You can pretty well fire as much traffic at them as you want at them, and they’ll find a way to soak it up. Load forecasting sucks and the penalties for being wrong are severe, so it’s best just to not do it. Also, with serverless you don’t have to own and monitor and patch and maintain the servers, which is nice.

Brokers?

First some background: When you work with a broker, you often don’t use HTTP. You nail up a TCP connection and just pump bytes back and forth. For this to work, you obviously need some sort of session and message framing protocol and boy, are there ever a lot of them, for example MQTT, STOMP, OpenWire, and AMQP (watch out for AMQP, it comes in different, incompatible, versions).

Often you don’t have to worry about that; you use some combination of a standardized API like JMS or a popular library like Apache Camel and the messaging Just Happens.

The fact that you’re using permanent connections mean that you should be able to expect lower latency, because you don’t have the well-known HTTP overheads in setting up and tearing down connections. On the other hand, it means your software has to deal with the situation when your nailed-up connection breaks, which 100% of network connections eventually do. (Once again, your library may take care of this for you.)

Except for…

Let’s start with that scaling advantage that services like SQS and SNS offer. It’s real. But… maybe you don’t care. If you load up one of the MQs on a big honkin’ EC2 instance with lots of CPU and memory and threads, you can push an astonishing number of messages/second through it. Like, maybe ten times the number you’ll ever need to send for the foreseeable future.

On the other hand, how about that latency advantage that brokers give you, because of the nailed-up connection? It’s real. But… on the HTTP side, we have HTTP long polling, which can reduce receive latency a lot. And of course an increasing share of HTTP is now HTTP/2, which multiplexes multiple requests on a single long-lived connection.

And that’s not all. The next step after HTTP/2 is QUIC, likely to be rebranded as HTTP/3. It provides HTTP request semantics using UDP, so there are no permanent connections at all, and in principle amazingly low latencies should be possible.

Reliability?

On the serverless side, the story is excellent. SNS and SQS use all the usual AWS availability-zone tricks to ensure that even if hosts crash (which by the way they do all the time), data keeps flowing.

Brokers have a decent story too, if not quite as rock-solid. ActiveMQ makes it easy to set up broker pairs backed by a single filesystem-based message store, where the backup takes over reasonably quickly on host failure. (Unless of course you’ve built up a huge number of un-received messages, in which case restarts can get very sketchy.) RabbitMQ runs in clusters, often of size three, and when you lose one you can add a new one back in and it’ll pick up state from the others. Once again, if you’ve got a huge backlog, you may experience pain.

In conclusion…

It’s complicated. But you already knew that.

Facet: Point-to-Point vs Pub/Sub 9 Mar 2020, 9:00 pm

When there’s an event in the cloud, how many different receivers can receive it? There are two plausible answers: Just one, or anyone.

[This is part of the Event Facets series.]

Postman delivers letter

Point-to-point

The idea is that any given message can only be read by one receiver. You might imagine that, as the old photo above suggests, the sender supplies an address, choosing the receiver. But that’s actually a pretty rare scenario. What’s much more common is the way that SQS works.

In SQS, when you send a message, you have to pick which queue you’re sending it on. Any number of receivers can be polling the queue, but only one will receive any given message. The API is clever; once you’ve received a message and processed it successfully, you’re supposed to delete it. There’s a time window after you’ve received it during which nobody else can see it. If the message isn’t deleted in time, it’ll pop back into availability. It’s called the “visibility timeout” and there are APIs for setting and adjusting it.

Think about a back-end system in one of Amazon.com’s warehouses, picking up order events from the Web site. You want enough event readers in the warehouse to keep it busy, and you want to have each order processed by only one of them; so this flavor of point-to-point is just the ticket.

It’s not a rare scenario at all; the Apache Artemis message broker calls it “anycast”.

Publish and Subscribe

Credit: Terje Skjerdal from Høvåg, Norway / CC BY

Pub/Sub

It’s short for “Publish and Subscribe”, which nobody ever says. This is probably the default mode for most of the software that has the word “event” in its name. The idea is, you publish your events onto a Bus or a Stream or a Topic or a Queue — there’s no standard terminology. Then receivers subscribe to the whatever-it’s-called and the events on it are in principle available to all of them.

There are all sorts of variations in how subscriptions work; push vs pull, filter vs firehose; they’re important enough to deserve their own entries in this series.

There are loads and loads of apps in every sector of business that are a good fit for pub/sub. I’d be amazed if there are readers in a business of any size that doesn’t have some running.

At AWS, we have multiple services that do pub/sub: Kinesis, Managed Kafka, MQ, and then my favorite, SNS, which is used by more or less everyone to do more or less anything. The way it works is, you publish messages to at Topic, and then you subscribe receivers to it. They can be SQS queues, Lambda functions, mobile-device messaging, and arbitrary HTTP endpoints. And you can have as many subscribers as you want, SNS will do the fan-out.

Facet: Deduping 8 Mar 2020, 9:00 pm

When you fire an event into the cloud, can you be sure it’ll only come out again once? It turns out that sometimes they come out more often than they go in. This may or may not be a problem in your application. If it is, there are techniques to help work around it.

“At-least once”

This is a phrase you’ll hear a lot when you hang around with eventing/messaging people, (and cloud people generally). Builders work so hard at making sure everything gets delivered that they can end up doing it more than once.

How can that happen?

Here’s a scenario: Suppose your software wants to retrieve and process events from a bus or topic or stream or whatever the service is called. And suppose you retrieve one, then something goes wrong before you can acknowledge it. For example, the host your code’s running on might have failed. Or your code just crashed (mine never has bugs, but they tell me it happens). Or your software is written in Java and unfortunately went into a 45-second stop-the-world garbage-collection stall.

When any of these happen, the eventing software will get the idea that you didn’t successfully receive the data, and since its primary purpose in life is to deliver reliably, it will try again. Which means you end up processing it twice.

On the way in…

events in

…on the way out.

events out, with dupes

Production note: For pictures of events in action, I’ve adopted this technique, created by Colm MacCárthaigh, of using emojois. It’d be a good all-purpose solution except for I’ve found the emjoi repertoire and representation highly nonportable between Mac and Windows.

Sometimes duplicates aren’t your fault. Suppose one data center gets cut off from all the others in an AWS region. Yes, we try really hard to arrange for multiple redundant connections, but shit happens. A little birdie told that one day a few years ago, a badly-built bridge in Beijing fell down and it had been carrying three different network providers’ fibres.

We call this situation a network partition. When it happens, both sides of the partition will try really hard not to lose any data. The details can get complicated, but duplicate messages are a common result.

Obviously this could be a problem for your app.

What to do?

There are situations where you can ignore the problem. Since duplicates are rare, in an analytics app or anything else that’s doing stats, they’re probably not a big deal. But there are lots of situations where they are.

If your app has a database that does transactions, you’re probably OK, because you can safely remember when you’ve seen each event’s unique ID, and just discard duplicates.

Another useful technique is idempotency. That is to say, structure your application such that API calls can be repeated without changing the result. An example is anything that can be expressed as a pure HTTP PUT request. You can set a field to a given value as many times as you like without doing any damage. Designing an app to work this way is tricky. But it’s an option that’s worth investigating, because having idempotent operations tends to produce apps that are robust in the face of all sorts of common failure scenarios.

Here’s one thing to note: Duplicates are rare, but when they do happen, they tend to come in clusters (think about that bridge in Beijing). I don’t know if that fact is useful in the context of your app, but just in case.

But there are some apps that just can’t live with dupes.

“Exactly once”

That‘s the terminology used for software that comes with built-in de-duping. One example would be SQS FIFO. Upon encountering this capability, you might ask yourself “why don’t I just use this for everything?” It turns out, just as with FIFO, de-duping isn’t free and in fact isn’t particularly cheap.

It can also get kind of complicated and there are more details than you might think. Consider this blog: How the Amazon SQS FIFO API Works. It dives deep on all the details, which ends up taking over two thousand words.

My advice would be to teach your software to live with duplicates, if at all possible. “At least once” systems are just part of the cloud landscape.

Eventing Facets 7 Mar 2020, 10:00 pm

What happened was, at re:Invent 2019 I gave a talk entitled Moving to Event-Driven Architecture, discussing a list of characteristics that distinguish eventing and messaging services. It was a lot of work pulling the material together and I’ve learned a couple of things since then; thus, welcome to the Eventing Facets blog series, of which this is the hub. It’s going to take a while to fill this out.

On-stage in Vegas

Eventing or messaging?

They’re just labels. If it’s publish-and-subscribe people usually say “event”; if there’s a designated receiver, “message”. But there’s a lot of messy ground in between; and in my experience, the software does more or less the same stuff whichever label’s on the cover. So I may simultaneously talk about eventing and use the word “message” to describe the byte packages being shuffled around.

Disclosure

I work for AWS and am a member of the “Serverless” group that includes Lambda, SQS, SNS, MQ, and EventBridge. I also support services that make large-scale use of Kinesis. You can expect this to have obvious effects on the perceptions and opinions offered herein. I’m also reasonably well-acquainted with Rabbit/MQ and Kafka, but without any hands-on.

Process

For each facet, I’ll introduce it here and write a blog piece or link to an existing one that covers the territory. In the process, I’ll build up a table relating how a list of well-known eventing/messaging services support it, or don’t; I premiered a version of that table in the talk (see the picture above) and more than one person has asked for a copy.

OK, here we go.

Facet: FIFO

When you inject events into the cloud, can you rely on them coming out in the same order? For details, see Facet: FIFO.

Facet: Deduplication

When you fire an event into the cloud, can you be sure it’ll only come out again once? See Facet: Deduping.

Facet: Subscribe or Receive

When there’s an event in the cloud, how many different receivers can receive it? Just one? See Facet: Point-to-Point vs. Pub/Sub.

Facet: Serverless or Broker

Does your eventing infrastructure look like a cloud service, just an endpoint in the cloud? Or is it an actual machine, or cluster of machines, that you own and operate and connect to? For details, see Facet: Broker vs Serverless.

Facet: Push or Pull

Do you have to poll the cloud for events, or can you get them pushed to you? For details, see Facet: Push vs Pull.

Facet: Filtering vs firehose

When you reach out into the cloud and use a queue or broker or bus to receive events, the simplest model is “I subscribe and you send me everything.” Often this is perfectly appropriate. But not always; I’ve seen software subscribed to high-volume sources that starts with really dumb code to look at the messages and decide whether they’re interesting, and ends up throwing most of them on the ground.

In recent years I’ve increasingly noticed the infrastructure software offering some sort of built-in filtering capability, so you can provide a query or pattern-match expression that says which events you want to see.

We built this into CloudWatch Events (now known as EventBridge) back in 2014, and people seemed to like it. Then a couple of years ago, SNS added filters to subscriptions and, since then, I’ve noticed that a steadily-increasing proportion of all subscriptions use them.

So I’m pretty convinced that some sort of filtering capability is a basic feature that eventing and messaging software really need. There’s no standard for how to do this. Some of the filtering expressions look like SQL and some look like regular expressions.

{
  "vendor": [ "vitamix", "instapot" ],
  "product_detail": {
    "price_usd": [
      { "numeric": ["<=", 150] }
    ]
  }
}

I don’t think this subject deserves its own article, especially since back in December I wrote Content-based Filtering, which describes on particular flavor of this technology that looks like the sample above, i.e. not like either SQL or regexes. I think the piece gives a pretty good feeling for the power and utility of event filtering generally.

Still to come

Here are the ones I know I’m going to write about, and maybe there’ll be more: Payload Flavors, Records vs. Documents, Uniform vs Heterogeneous Events, Replaying and Sidelining. Like I said, this will take a while.

Summarizing

Since this table includes references to software that I’ve never actually used, it’s perfectly possible that my research has been imperfect and there are errors. Please let me know if so and I’ll dig in deeper and correct if necessary.

ActiveMQ Artemis Event
Bridge
Kafka Kinesis RabbitMQ SNS SQS SQS
FIFO
FIFO?YesYesNoYesYesYesNoNoYes
Dedupe?*YesNoYesNoNoNoNoYes
P2P or
Pub/Sub
BothBothP/SP/SP/SBothP/SP2PP2P
Serverless
or Broker
BBSBBSSS
Push
or Pull
BothBothPushPullBothPushPullPull
Filtering?YesYesYes*NoNoYesNoNo

Notes

* It looks like the software can do this but either it isn’t straightforward or doesn’t work out of the box.

Kinesis is a cloud service but since you have to provision shards, it kind of acts like a broker too. On the other hand, adding and deleting shards is possible if a bit awkward.

Kinesis is pull-based, but there’s the Kinesis Client Library which gives a very push-like feeling.

Facet: FIFO 7 Mar 2020, 10:00 pm

When you inject events into the cloud, do you care whether they come out in the same order they went in? If you do, you’ll be asking for “FIFO” (rhymes with “Fly, foe!”) (stands for “First In First Out”). Some software has it, some doesn’t.

[This is part of the Event Facets series.]

If you think about it carefully, FIFO is only meaningful when you have a single-threaded sender and a single-threaded receiver; with more than one, who gets to say which message was sent (or received) first?

Caravan in the desert, Morocco.

Real-world FIFO semantics.
Credit: “Caravan in the desert. Morocco, Sahara.”
©Sergey Pesterev/Wikimedia Commons/CC BY-SA 4.0

In practice, this means that a FIFO event stream can really only run as fast a single computer can send messages. Assuming there’s any real business significance to the messages, this is pretty slow: hundreds of transactions/second is regarded as extremely fast for a FIFO sequence.

It turns out that in a high proportion of cases, what you have is a huge fast data stream that’s partitioned into many sub-streams, and the FIFO operates at the sub-stream level. Think of the clickstream coming off a big Web site. You might care about the order of events that are part of an individual session (there could be thousands or millions of sessions), but not whether one session’s events are before or after another’s.

This is common enough that you’ll hear people use phrases like “session ordering”.

That’s why I like the picture above. There are actually three strongly-ordered mini-caravans, which could arrive at the oasis in any old order while locally preserving strict within-caravan camel ordering.

If you look at actual real-world software, there’s usually good support for this big-river-of-ordered-sessions model. For example, both Kinesis and Kafka have session selection arguments and call them Partition Keys. Both expose their shards, which are selected by the partition keys and where the FIFO semantics apply. This makes them shine for huge-scale session-oriented pub/sub streaming.

SQS FIFO calls its session identifier Message Group IDs and hides the internal shards.

I always want FIFO, right?

Certainly, if you’re applying transactions to a bank account. But maybe not; Consider the case of Amazon.com. Every time you press “buy”, the Web site generates a message saying what you bought, how you paid for it, where, to ship it, and so on, and drops that into the cloud, where it’ll eventually be picked up by the systems that do acquisition and packaging and shipping and accounting and tax and compliance and so on. But if you think of it, since each of those orders is independent of all the others, the order doesn’t matter that much.

2019年のジャパンカップ スタート後の先行争い

Not FIFO.
Credit: “2019年のジャパンカップ スタート後の先行争い”,
by nakashi.

FIFO is hard

If your app is like Amazon.com, you should definitely use a service that doesn’t offer FIFO, because it’ll be cheaper and more elastic. It turns out that FIFO isn’t free — it’s not even cheap —  because it’s hard to build and operate.

It’s not hard to understand why. A typical big Web service is commonly sharded, which is to say there’s a front-end fleet of hosts handling a flow of requests from many sources, which it deals out to more hosts in the back-end fleet to process. Sharding lets you handle really any level of traffic.

The way it typically works is that incoming requests end up at some sort of “load balancer” to deal the traffic out. Obviously this means you can’t maintain global FIFO ordering; that’s sort of OK for the reasons described just above. If you want to maintain the more typical and useful “session FIFO” semantics, you need to route the messages associated with each session to the same shard. This practice is called “affinity” and, as I described here, it’s difficult.

On Soaking the Rich 19 Feb 2020, 10:00 pm

The government of BC, the Canadian province where I live, just released a new budget which, among other things, raises tax on high incomes. Here is an overview. The top marginal tax on incomes over C$220,000 goes from 16.8% to 20.5%. This is just provincial tax; what with the Feds, the total top marginal rate is now 53%. Not everyone is delighted. For example Garth Turner, finance/real-estate blogger, who emits a howl of right-wing grief. I’m comfortable speaking about this since I’m personally affected.

[These days, a Canuck buck is worth about $0.75 American.]

It’s worth noting that our province is led by the New Democratic Party (everyone says “NDP”) who are Social Democrats; by and large pretty moderate by world standards. On the American spectrum they’re red-toothed commies; not too far off what Bernie Sanders would like to see.

Numbers

Let’s do some math, OK? Garth Turner points out how awful this will be for people living in super-expensive Vancouver and making a paltry $250K. Hmm, the marginal increase is 3.7% on income over $220K. So those poor $250K people are going to be paying an extra $1110/year. Garth also excerpts a letter from an aggrieved doctor’s husband who says they’ll be paying an extra $1K a month; by my arithmetic the doctor’s making over $540K taxable, thus taking home over $250K, ignoring the tax dodges available to the self-employed, then there’s the husband’s income. Maybe I’m missing something, but neither of these calculations yield what feel to me like lifestyle-changing numbers.

Side-note: I don’t want to diss Garth too much because he’s a really excellent writer on personal finance in general and real-estate in particular; if you’re in Canada and have money to invest you should totally read him. Also he’s funny and runs cute dog pictures. He becomes less interesting as he veers into overly-predictable right-wing tax-grouch tropes. And, disclosure: I’m a customer.

Saving and spending

Here’s the thing: A high proportion of people subject to the tax hike are taking home more than they need to live on, so the effect of this move is that their savings (and wealth) will grow more slowly. Most such people retire with enough to live on, and are in their forties or older. So the tax hike won’t result in any short-term spending decreases to speak of, but starting say ten years from now, a demographic of well-off retirees will have a little less to pump into that future economy.

And those slightly-smaller savings, what about them? They join a world-wide glut of capital, surging around looking for a decent return. Plenty will look for it outside Canada, or in ventures where most of the profits go to money people.

On the other hand, the extra $200M or so a year this tax maneuver pulls in will all get spent by the government, the vast majority in paychecks to middle- and working-class civil servants and contractors, who will in turn spend almost all of it in the next twelve months right here in BC. Maybe I’m missing something, but this policy feels like an economic win/win. The only losers are one-percenters like me; modestly, a decade or two from now. Maybe I’ll only be able to vacation once a year.

Or is that kind of thinking dangerously socialist? Here’s some data from The Economist: Wage gains for low earners have helped sustain America’s economic expansion. It’s data-rich and convincing. Inequality not only sucks, it’s bloody inefficient and generally bad for the economy.

In fact, redistributionist policies might improve the economy enough that the return on my savings will make up for their slightly smaller size.

Conclusion

Raise the minimum wage and the marginal tax rates on people like me. Both as a left-winger and an investor, I approve.

Sally Leekie 14 Feb 2020, 10:00 pm

Happy Valentines! A day that celebrates love comes, in my mind, second only to the one that celebrates giving thanks. I didn’t do roses or chocolate, but I made dinner for a couple of people I love; one of the dishes was improvised and came out well, so this recipe is my valentine to the world.

The main ingredients are salmon and leeks; thus “Sally Leekie”.

Ingredients

  1. Salmon: We use wild frozen-at-sea Pacific Sockeye, usually obtainable in Vancouver. 750g fed three generously.

  2. Leeks: A couple of big ones.

  3. Garlic: Confession: For this dish, I used minced garlic out of a jar that I bought in a supermarket; a couple of heaping teaspoons-full.

  4. Seasoning: Oregano, Fennel, and black pepper, all served liberally.

  5. Oil: Now this is interesting. I started with olive oil because it was at the front of the cupboard, then thought “Leeks? Everyone knows you cook them in butter!” so I added some of that too. It came out nicely.

Sauteeing leeks

Sauteeing leeks.

Process

  1. I covered the bottom of a sautee pan with oil, tossed in the garlic and seasoning, and heated it to not-quite-bubbling-or-smoking for ten or fifteen minutes.

  2. While this was happening, I chopped the leeks and salmon. If you haven’t done leeks before, you have to cut them lengthwise then take them over to the sink and wash out the mud and gunk that tends to occur; leeks are just not hygenic vegetables. Salmon into bite-size chunks.

  3. I turned up the heat to the point that a few bubbles were occurring, and put the chopped leeks through in three batches. Watch out; they cook down tremendously, so you want to start with more than you need. Maybe five minutes a batch; until they’ve lost their curl but not their color. I used a bowl in the warming drawer under the oven to accumulate the leeks and keep them warm.

  4. Once the leeks were done I dropped the salmon into the garlicky spiced leek-flavored oil and sauteed that for maybe ten minutes, till all the sides were sealed and it was getting ready to eat.

  5. Finally, I took the leeks out of the warming drawer, tossed them in with the salmon, turned the heat up so there was a bit of smoke, and tossed the mixture together.

It looked neat, the salmon pink contrasting with the leek green. There were no leftovers and sincere compliments.

Why Google Did Android 9 Feb 2020, 10:00 pm

What happened was, in the late stages of my career at Sun Microsystems, as we were sliding into Oracle’s loathsome embrace, I had discovered Android. The programming language was Java, and not a dorky “ME” subset. My employer was saying nice things about it, and I’d long craved something I could both carry in my pocket and program. I discovered it was pretty easy to program and eventually published the Android Diary series in this space, which got pretty lively readership.

Thus, I shouldn’t have been surprised when, shortly after leaving Sun, I got outreach from Google’s Developer Relations org. I was receptive and almost immediately I found myself in Mountain View for the famous Google Interview Day. My first session was with Vic Gundotra, who was a major Google V.I.P. at the time. He opened by saying “I’ve been reading your blog and I think I know a lot about you. What would you like to know about us?”

That was easy. I asked “Why is Google doing Android? Are you serious or is it just a hobby?” (Because at Sun we’d had a lot of hobbies — sideline technologies that we couldn’t seem to give up — and that sucked and I didn’t want to work on one.)

Vic said something like (It’s ten years later and I’m paraphrasing) “The iPhone is really good. The way things are going, Apple’s going to have a monopoly on Internet-capable mobile devices. That means they’ll be the gatekeepers for everything, including advertising, saying who can and can’t, setting prices, taking a cut. That’s an existential threat to Google. Android doesn’t have to win, to win. It just has to get enough market so there’s a diverse and competitive mobile-advertising market.”

I don’t know about you, but I found that totally convincing. And I suppose a lot of industry insiders are thinking “Well of course everyone knew that!” I didn’t. I made it through the interviews and they offered me the job and I had four good years at Google.

I wonder if Vic was right about what would’ve happened if they hadn’t done Android?

Seasonturn 8 Feb 2020, 10:00 pm

It’s still February, winter obviously, and yet there was a bit of chilly sun today to greet 2020’s crocuses, the photo-introduction of which has become an annual ritual in this space.

Here are two shots from January 15th, just twenty-four days ago.

January snow, VancouverJanuary snow, Vancouver

The snow was considerable but didn’t last long. The rain’s been extreme even by Vancouver standards, just relentless; we’re all feeling a bit climatically bruised. There could be more snow, but I’m ignoring that. Because of this:

CrocusesCrocusesCrocusesCrocuses

These photos shot with the recently-acquired
steampunk Pentax M 100/2.8.

This year they’re up a little earlier and there are lots! They’re spreading a little wider, inhabiting parts of the front garden they haven’t previously. Fine by me.

Bye-bye Time Machine 4 Feb 2020, 10:00 pm

I recently switched my backup tool from Apple’s Time Machine to Arq Backup which for my needs is clearly better. And once I’d realized that, I wondered whose needs would be best served by Time Machine. To be honest I’m having a hard time with that.

Time Machine setup

I’d used Time Capsules and Airports for many years and then switched to a Synology DS416j with mirrored 6TB drives. It took a little jiggery-pokery to get the Macs and the Synology talking nicely, but then things seemed to work for a while. But not recently. At regular intervals Time Machine says something like “Time Machine has enthrophased the gnocchometric continuum, and you need to back up from scratch again.” Maybe the part before backup from scratch was actually “monophorically phosphorylated the interpretive-dance phlogiston”; can’t quite remember because, whatever it was, that’s how much sense it made.

Arq Backup

We have a pretty fast home network but also really big disks, so start from scratch meant multiple days waiting for that do-over backup to do its job. This is not a confidence-building experience.

Arq

On a parallel timeline, at some point I’d asked the world “What’s good Mac backup software?” and a lot of people said “Arq.” OK then, I signed up and paid and gave it a whirl. Because I’d somewhat lost faith that I had anything useful in terms of Time Machine backups, I went and bought a 2TB USB drive at the nearest drugstore and plugged that in. I have to say the onboarding user experience could be better; Arq’s terminology for what you’re backing up from and to is not as self-explanatory as one might like. But I’m pretty sure I figured out the right incantations.

Once running, the Arq user experience is vastly better than Time Machine’s. It tells you what it’s doing, and what it’s doing takes what seems like a reasonable amount of time. Time Machine never tells you anything actually useful about what it’s doing or (all too often) why it can’t, with the unifying theme being that whatever it does or fails to, the process takes hours.

But nobody cares about backup!

The only thing anyone cares about is restore.

I have verified that I can restore data from Time Machine; the only times I’ve done so have been when migrating from one Mac to another. It works fine but (like everything else about Time Machine) is painfully slow and opaque. I also have read enough testimonials from Arq users to be convinced that it’ll do the job when I ask.

Apple Time Machine

What is Time Machine for??

The hero moment is that Finder view replicated back over a timeline, so you can go find the version of any file at any date and bring it into the present day like, you know, a time machine. Like (I suspect) most people, I was dazzled with the UI the first time I saw it. But, in all those years the number of times I’ve used that restore mode is: Zero. As I said above, I’ve done a wholesale restore to a new Mac once or twice: for this purpose the fancy UI is irrelevant.

With a little thought, it becomes obvious why I don’t need the timeline.

What files on my computer do I care about?

  1. Photographs, which I load into Lightroom and (mostly) edit. Lightroom edits non-destructively by recording deltas against the RAW file. All I really care about is the most-improved version or the original RAW. If I want to try another treatment of the photo, I make a virtual copy. The net effect of all this is: All I ever care about is the most recent version of any of the files. So the timeline buys me nothing.

  2. Work documents, which we collaborate on and version like hell. We do this mostly with WorkDocs, which keeps all your doc versions somewhere in the cloud. All I really care about is the current working version that might be ahead of what’s in the cloud, so once again, only the latest. So the timeline buys me nothing.

  3. Code, which is hosted in Git, and all I care about is what’s in my own workspace, because it might be noticeably ahead of the last commit. So the timeline buys me nothing.

  4. Blog pieces, which live on both my MacBook and the tbray.org server. The process of getting them ready to push feels like monotonic improvement; I’ve never felt the slightest urge to dig up an old version. So the timeline buys me nothing.

That leaves one category where the timeline might be a winner: Deletion protection. Surely, if I delete some file then regret it, wouldn’t it be handy to scroll the timeline back to just before and pick it up? I guess this might be true for some people, but there’s this: I never delete anything. Seriously, if you’re going to erase data you should think hard first, and that thinking is a waste of precious time and brain bandwidth, because data is cheap to keep and the cost of being wrong is high. So the timeline buys me nothing.

[Now, there’s an exception. Things you really want to delete because they are evidence of sketchy sex kinks or premature antifascism or whatever. Protip: Put this kind of thing somewhere that isn’t backed up.]

Conclusion

Arq is great. A little bird told me it’s the product of a one-person operation and not a VC-funded aspirational Bay-Area Unicorn (the Internet needs more of those). Another little bird recently told me Arq got a serious look-over from a pretty elite OpSec group who said “LGTM” (the Internet needs more of those).

If you have a Mac and your data-handling practices are like mine, I advise sending Arq some money.

Bonus

In 2006 I wrote about Protecting Your Data, on mostly the same subject. Dedicated readers may enjoy tracking the evolution of my thinking on this subject.

Decarbonization 19 Jan 2020, 10:00 pm

We’re trying to decarbonize our family as much as we can. We’re not kidding ourselves that this will move any global-warming needles. But sharing the story might, a little bit.

Those who worry a lot about the climate emergency, and who wonder what they might do about it, are advised to have a look at To fix Climate Change, stop being a techie and start being a human by Paul Johnston (until recently a co-worker). It’s coldly realistic, noting that the Climate Emergency is not your fault and that you can’t fix it. Only radical large-scale global political action will, and his recommendation is that you find an organization promoting such change that meets your fancy (there are lots, Paul provides a helpful list) and join it.

Global GHG per capita, 2017

Such intensity of change is possible. It happened in the middle of the Twentieth century when faced with the threat of global Fascism; Governments unceremoniously fixed wages, told businesses what they could and couldn’t do, and sent millions of young men off to die.

It’s not just possible, it’s inevitable that this level of effort will happen again, when the threat level becomes impossible to ignore. We will doubtless have to conscript en masse to fight floods and fires and plagues; which however is better than attacking positions defended by machine guns. The important thing is that it happen sooner rather than later.

Thus evangelism is probably the most important human activity just now, which is why people like Greta Thunberg are today the most important people in the world.

A modest proposal: Decarb

“Decarbonize” has four syllables and “decarbonization” six. I propose we replace both the noun and the verb with “decarb” which has only two. Mind you, it’s used in the cannabis community as shorthand for decarboxylation, but I bet the world’s stoners would be happy to donate their two syllables to the cause of saving the planet. Anyhow, I’m gonna start using decarb if only because of the typing it saves.

So, why personal decarb, then?

Well, it feels good. And — more important — it sends a message. At some point, if everybody knows somebody who’s decarbing it will help bring the earth’s defenders’ message home, making it real and immediate, not just another titillation in their Facebook feed.

There’s a corollary. Decarbing is, by and large, not terribly unpleasant. The things we need to give up are I think not strongly linked to happiness, and some decarb life choices turn out to be pretty pleasing.

Caveats and cautions

It’s important that I acknowledge that I’m a hyperoverentitled well-off healthy straight white male. Decarbing is gonna be easier for me than it is for most, because life is easier for me. I’m willing to throw money at decarb problems before that becomes economically sensible because I have the money to throw; but I hope (and, on the evidence, believe) that pretty well every one of these directions will become increasingly economically plausible across a wider spectrum of incomes and lifestyles.

Because of my privileged position and because the target is moving, I’m not going to talk much about the costs of the decarb steps. And I’m not even sure it’d be helpful; these things depend a lot on where in the world you live and when you start moving forward.

Now for a survey of decarb opportunities.

Decarb: Get gas out of your car

Obviously fossil fuels are a big part of the problem, and automobiles make it worse because they are so appallingly inefficient at turning the available joules into kilometers of travel — most less than 35%. On the other side of the coin, electric vehicles are a win because they don’t care what time of day or night you charge them up, so they’re a good match for renewable energy sources.

Family progress report: Mixed.

For good news check out my Jaguar diary; I smile a little bit every time I cruise past a gas station. I’m here to tell you that automotive decarb isn’t only righteous, it’s fun.

On the bad-news side, we still have a 2007 Honda Odyssey van, an overpowered gas-eater. It was super-handy when our kids were young but now both of ’em can get around the city by themselves. I look forward to taking it off to SCRAP-IT.

Jaguar I-Pace

Decarb: Get yourself out of cars

Last time I checked, the carbon load presented by a car is only half fuel, more or less; the rest is manufacturing. So we need to build fewer cars which means finding other ways to get places. Public transit and micromobility are the obvious alternatives.

They’re good alternatives too, if you can manage them. If you haven’t tried a modern e-bike yet you really owe that to yourself; it’s a life-changer. I think e-bikes are better alternatives than scooters along almost every axis: Safety, comfort, speed, weather-imperviousness.

And transit is fine too, but there are lots of places where it’s really not a credible option.

Now, there are times when you need to use a car. But not necessarily to own one. It isn’t rocket science: If we share cars then we’ll manufacture fewer. There are taxis and car-shares like Car2Go and friends. (Uber et al are just taxi companies with apps; and money-losing ones at that. Their big advantage is that your ride is partly paid for by venture-capital investors who are going to lose their money, so it’s not sustainable.) The car-share picture varies from place to place: Car2Go seems to be exiting North America but here in Vancouver we have Evo and Modo and Zipcar.

Family progress report: Pretty good.

Since I got my e-bike, I just don’t want to drive to work any more and resent it when family logistics force me to.

Trek E-Bike

My wife works from home and my son takes a bus or a skateboard to college. My daughter still gets driven to school occasionally in some combinations of lousy weather and an extra-heavy load, but uses her bike and the bus and will do so increasingly as the years go by. I occasionally take the train but it’s slower than the bike and unpleasantly crowded at rush-hour; sometimes when I’ve done that I guiltily grab a car-share to come home. We use car-share a lot for going to concerts and so on because they avoid the parking hassle.

Now, due to the hyperoverentitledness noted above, we have the good fortune to live in a central part of the city and all these commute options are a half-hour or less. This part of decarb is way harder in the suburbs and that’s where many just have to live for purely economic reasons.

Decarb: Get fossil fuels out of the house

Houses run on some combination of electricity and fossil fuels, mostly natural gas but some heating oil. The biggest win in this space is what Amory Lovins memorably christened negawatts — energy saved just by eliminating wastage. In practical terms, this means insulating your house so it’s easier to heat and/or cool, and switching out incandescent lights for modern LEDs.

Ubiquitous LEDs are a new thing but they’re not the only new thing. For heating and cooling, heat pumps are increasingly attractive. Their performance advantage over traditional furnaces is complicated. Wikipedia says the proper measure is coefficient of performance (COP), the ratio of useful heat movement per work input. Obviously a traditional furnace can’t do better than 1.0; a heat pump’s COP is 3 or 4 at a moderate temperature like 10°C and decreases to 1.0 at around 0°C. So this is a win in a place like Vancouver but maybe not if you’re in Minnesota or Saskatchewan.

Heat pump technology also works for your hot-water tank.

Another new-ish tech, popular in parts of Europe for some time now, is induction cooking. It’s more efficient than gas cooking, which is in turn more efficient than a traditional electric cooktop. For those of you who abandoned electric for gas years ago because it was more responsive, think again: Induction reacts just as fast as gas and boils water way faster.

Note that all these technologies are electric, so your decarb advantage depends on how clean your local power is. Here in the Pacific Northwest where it’s mostly hydroelectric, it’s a no-brainer. But even if you’ve got relatively dirty power the efficiency advantage of the latest tech might put you ahead on the carbon count.

Also bear in mind that your local electricity will likely be getting cleaner. If you track large-scale energy economics, the cost of renewables, even without subsidies, even with the need for storage, has fallen to the point where it not only makes sense to use it for new demand, in some jurisdictions it makes sense to shut down high-carbon generating infrastructure in favor of newer better alternatives.

Family progress report: Getting there.

We had a twenty-year-old gas range which was starting to scare us from time to time, for example when the broiler came on with a boom that rattled the windows. So early this year we retired it in favor of a GE Café range, model CCHS900P2MS1. It makes the old gas stove feel like cave-man technology. Quieter, just as fast, insanely easier to clean, and safer. On the other hand, it requires cookware with significant ferrous content, which yours maybe doesn’t have; we had to replace a few of ours.

GE Café CCHS900P2MS1 Induction Range

Modern induction range with a traditional cast-iron frying pan, making pancakes.

Now, as for those negawatts: We live in a wooden Arts-and-Crafts style house built in 1912 and have successively updated the insulation and doors and windows here and there over the years to the point where it’s a whole lot more efficient than it was. Late last year we went after the last poorly-insulated corner and found ourselves spending several thousand dollars on asbestos remediation. Then when we had the basement all ripped to shreds we realized it was a good time to put in heat pumps to replace the 20-year-old gas furnace and hot-water tank. This is in progress; as I write, we’re getting by with space heaters and the fireplace; the new system turn-on is promised for tomorrow.

One consequence is that we can turn off the natural gas coming into the house, which makes us feel good on the decarb front. And then there’s the earthquake issue; Where we live we’re overdue for The Big One and when it comes, if your house doesn’t fall down on you, another bad outcome is a natural-gas leak blowing you to hell.

I’ll report back here on the heat-pump experience once we have some.

Decarb: Fly less

Yeah, the carbon load from air travel is bad. The first few electric airplanes are starting to appear; but no-one believes in electric long-haul flights any time soon.

Family progress report: bad, used to be worse.

For decades, as the public face of various technologies, I was an egregious sinner, always jetting off to this conference or that customer, always with the platinum frequent-flyer card, often upgraded to biz class.

Since I parted ways with Google in 2014 and retreated into back-room engineering, things have been better. But not perfect; we have family in New Zealand and Saskatchewan, and take to the air at least annually. I haven’t any notion what proportion of air-flight carbon has business upstream, what proportion vacations, and what proportion love. I hope the planet can afford the latter and a few vacations. If something’s gotta go, it’ll be biz travel.

Decarb: Eat less meat

The numbers for agriculture’s share of global carbon load are all over the place; I see figures from 9% to twice that. The UN Food and Agricultural Organization says global livestock alone accounts for 14.5% of all anthropogenic GHG emissions. So if the human population backed away from eating dead animals, it’d move the needle.

Family progress report: Lousy.

We eat more vegetarian food from year to year but the increase is slow; and often meat twice per day. Meat is tasty and nutritionally useful but what’s pernicious is that it’s easy; less chopping and other prep required. In a culture that’s chronically starved for time, this is going to be a heavy lift.

We have however cut way back on beef, which is the worst single part of the puzzle. I think it’s probably perfectly OK for a human to enjoy a steak or a burger; say, once per month.

Decarb: Random sins

You probably shouldn’t have a motorboat (which we do) and if you do you should use it sparingly (which we do).

You probably shouldn’t burn wood in a fireplace (which we do) and if you do you should use it sparingly (which we do).

[To be updated].

Orange Badge 14 Jan 2020, 10:00 pm

One of our admin people walked into my office, saying “Five-year anniversary, new badge, gimme your old one”. Which means I’ve been here five orbits of the sun, my longest job tenure since my seven-year stay last century at a company I co-founded.

Badge

Best and worst

Best things? The work is fun and the people are nice. Of those working here in Vancouver when I showed on up December 1st, 2014, more than 70% remain, which is impressive in high tech — pretty well any of them can walk down the street to Microsoft or a game company and get a job that week, maybe for more money.

Worst things? My progressive friends genuinely think Amazon embodies everything wrong with the civic structure of the twenty-first century world, and can’t understand why I’d stay.

Not gonna argue with them, but AWS also embodies most of what’s right. I’d rephrase it like this: Amazon is so highly competent at capitalism that we’ve become a proxy for the whole system; good, bad, and lots of everything in between.

I have worked for [pauses, counts] ten employers, private-sector and academic, and consulted for lots more. By a wide margin AWS is the best-managed (I can’t speak to the other parts of Amazon).

You can be a member of any political tribe (except back-to-the-land anarchist) and still believe that management is a task that someone needs to do. Thus, finding ways to do it better is of general interest. And yes, I believe that some of the techniques used here could fruitfully be applied across a wide swathe of the public and nonprofit sectors. Those techniques aren’t secret: Bar-raising, document-based decision-making, carefully-curated leadership principles, a couple more.

If I were dictator

I’d change the regulatory framework in ways that big-tech leadership (including Amazon’s) would probably dislike. But my changes would likely cause those leaders to be less disliked; by the public, I mean. In 2019 it’s hard to believe that just ten years ago the FAANG companies were more or less universally admired and seen as the nice, warm, fuzzy end of capitalism.

At some point, it’s going to be a real problem being management in a sector that’s widely feared and distrusted. But we in the tech tribe haven’t really internalized much about this yet.

Planning!

It’s a thing AWS is really good at. Our hiring, our product milestones, our region builds (OMG the region builds), our big public events, our promotions, our re-orgs, our new-office-space acquisition and occupation. Obviously nobody gets the shape of the local future right 100% of the time or even close, but I’m in awe of what my colleagues accomplish.

Others have noted that ideologists of the free market scoff at the notion of a “planned economy”, but that market’s biggest participants plan their own slices of the economy energetically, comprehensively, and all the time.

I think there are areas where you want to lay down a regulatory framework and get out of people’s way, and others where you want a firm notion about what a particular team will be doing in a particular month next year. There’s lots of room to disagree which is which.

As for me…

In late 2018 I wondered out loud about retirement. I still do, after all, this is the year that the Government of Canada will start sending me money every month just for being old.

I’ll for sure never have another “job”. After I’m done here I can’t see having a relationship with another entity that’s (a) economically motivated or (b) requires full-time-ish engagement.

Half the time I think that getting up every morning and going to work amounts to a failure of courage. Ch-ch-changes… Turn and face the strange sang David Bowie, and why shouldn’t I?

Maybe the new direction could be internal; there’s a lot of work to be done around the climate pledge, and a lot about the diversity problem we share with the industry.

But anyhow it’s a good gig. And I’ve invested a 40-year career in learning the things I need to know to do this.

Beats the hellouta me. Maybe I’ll surprise myself one of these days.

Jag Diary 11: A Year In 12 Jan 2020, 10:00 pm

The I-Pace insurance renewal showed up so I knew a year must have (so quickly!) gone by. I’d thought I should write something to draw a line under this diary, and then just now the car saved my life, so now’s the time. Not much new information here but maybe a couple of smiles.

Jaguar I-Pace

The insurance — two drivers, good records, no commuting — cost around two and a half thousand Canadian dollars. That year’s other car expenses put together (electricity, replacing a vandalized window) are a tiny fraction of the insurance. Electric cars, even with sports-car performance like the Jag’s, are stupidly cheap to run. I think that as electrics replace fossil vehicles slowly then quickly, the world will need fewer automotive mechanics. Paint and body and glass shops will be fine, but there’s not much to be done under the hood.

Riding the rain

Next time you’re driving, consider where the joules that move your car came from. For most of you, the story involves a whole lot of heavy machinery digging holes and pumping sticky black crap out of the planet and then turning it into auto fuel in a process that really fucks up the atmosphere near the refinery, and then requires you to pull into a loveless harshly-lit concrete enclave to stand beside a pump staring blankly while dozens of dollars flow out of your bank account into an oil-and-gas company’s, so you can turn travel hours into outflows of planet-killing CO2.

Our car is rain-driven. Well, it’s partly snow, but anyhow the clouds come in off the Pacific and crash into the mountain ranges and dump their sun-elevated droplets, frozen or not, and eventually the water flows down and into dammed-off valleys and through turbines, most built decades back, and eventually the electrons end up in the 240V charger behind our house where I plug the car in overnight every week or two and invest less than $10 to fill it up while I sleep.

I think I’m winning this one.

Jaguar I-Pace

Jag encounter

Late last summer in the long dusk of a bright day, we were heading home and I found myself queued up for a tricky west-to-south left turn at a big old intersection (for Vancouver cognoscenti, 12th to Main).

The twilight made all the cars and people and buildings look great. Traffic was heavy and the pause was considerable; while hanging out there, I noticed the first northbound car waiting for the light and for me to be gone was a beautiful old silver Jag sedan, I think an XJ, gleaming like a long low jewel in the slanting sun. I’m inclined to think that our Caesium-blue electric looked pretty glamorous too just then, from outside. I saw the dude in the other Jag was looking at mine too, then just as the traffic opened I met his eyes and then I curved round him while we shared a big Jaguar smile.

Snow champ

Did I say above “saved my life”? Quite likely. Sunday night we had a nasty snowstorm, which causes problems in Vancouver because there are a lot of hills and also a lot of people who rarely if ever drive in snow and are mostly perfectly OK folk but just don’t have a clue how to deal. We had an old friend over for dinner who came by bus; looking outside after, I said “I better take you home.”

Which may have been an error in judgment. On the hills, the sideways victims included not just the usual over-powered minivans but more than one city bus, so it wasn’t just a casual dusting.

Jaguar I-Pace

The Jag weighs over two metric tons (a third or so battery) and has big wide wheels and a snow-and-ice four-wheel mode so it laughs at this stuff. I was gentle with the juice and slow on the hills and yeah, I had to dodge one swiveling Acura, but it was OK.

After I dropped Gareth off downtown, I headed for a left turn onto a big one-way downtown-access road (for Vancouver cognoscenti, Citadel Parade to the Dunsmuir viaduct). As I approached, the light turned green so I was starting to aim left when this huge van hurtled along the road I was trying to turn onto. He’d clearly sped up to beat the yellow and then foolishly tried too late to stop; he must have been doing 60 km/h or more, and mostly sideways. I hit the brake hard and maybe the ABS implementation is a little rough but that big cat shuddered to a stop right freaking there in the snow, hardly rotating at all. OK, maybe I wouldn’t have died; but it would have been seriously ugly. Mea culpa; I should have pulled up to a stop at the line, not trusting the green light on such a night.

On the way home among the amateur-hour chaos in the snow the music shuffle switched over to Hildegard von Bingen and boy, did that ever hit the spot.

The Jaguar I-Pace is a good car. Mind you, pretty well all the modern electrics are; in the big picture, fossil propulsion is done for. But even given all that, it’s really awfully good; at keeping me alive and making me smile.

Horopito 31 Dec 2019, 10:00 pm

It’s a place near the center of New Zealand’s North Island; we spent the last Christmas of the decade there. It’s considered remote in NZ which I guess makes it doubly so in the wider world. Unless you’re planning southern-hemisphere skiing you’re unlikely to go near it, so I felt words and pictures worth sharing.

What happened was, Lauren’s brother and his wife who live in Auckland came to visit us and we took them to our cabin; they liked the country-retreat concept so much that they bought an acre in Horopito and built a house on it. This was finished and moved-into on December 18th and in a flash of insane courage, they invited their three children, two children’s spouses, and five grandchildren along. And us. Lauren and I have been together for more than twenty years and have spent every Christmas with my family, so it was about time hers got a turn.

So it was off to New Zealand for us. Now, as an environmentalist loudmouth should I feel bad about jetting between the hemispheres? I should, and I do. But there’s that twenty-years thing and also Lauren’s Dad is getting to a state of age and health where we may not see him again. We’re obviously going to have to slash the amount of mass jet travel; I would hope that the global carbon budget can sustain trips made mostly for love, but I don’t know. I may never pull that transcontinental trigger again.

Horopito was once a lumber boomtown, but the town’s pretty well gone. This post office was closed in 1970.

Horopito Post Office

It has been pointed out to me that this picture would be suitable for an indie album cover. Any band that wants to use it may do so but you have to send a few bucks to the little girl, whom I know.

Horopito isn’t famous for anything but it’s near some well-known things, most notably Mount Ruapehu, the North Island’s tallest at 2,797m (9,177 ft) . (Actually, Ruapehu is part of a complex that includes Tongariro and Ngauruhoe (which Peter Jackson had stand in for Mount Doom.)) There are any number of glamorous pix of Ruapehu and its posse, but here’s a look out the front of the family property during complicated weather — most NZ weather is complicated.

Mount Ruapeho in clouds

We took two trips up the mountain. Here are views from the Whakapapa side:

View from near WhakapapaView from near Whakapapa

And here’s another looking southwest.

View southwest from Ruapehu

New Zealand is full of nice scenery, whichever way you look.

Smash Palace!

That’s all you need to say to explain where Horopito is to any New Zealander with even one or two grey hairs. It’s officially Horopito Motors, “the largest and only vintage car dismantlers in Australasia”. It’s really impressive in the satellite view. “Smash Palace” refers to a 1981 NZ-made movie in which the wrecking yard features; I’ve actually seen it and it’s not terrible.

smash PalaceSmash Palace

Of course, the main point was getting the family together. In a Southern-hemisphere Christmas, the turkey and ham at dinner is served cold, and then after digesting a bit, people retire to the lawn to toss frisbees and rugby balls and generally run around:

Christmas festivities

One evening the sky was crystal clear and a few of us stargazed. I didn’t take the trouble to figure out how to unleash the Pixel 4 astrophotography mode, but I did point it at the southern skies. On that half of the globe, Orion is upside down…

Southern-hemisphere Orian

… and looks like a shopping cart.

Also, what I think is the Southern Cross, first time I’ve seen it. The Pixel didn’t capture the Milky Way, perfectly evident behind the Cross, to the naked eye.

Maybe the Southern Cross

If you have a large loving family to welcome you, I think Horopito is a fine place to visit, and the quality of light is special.

Clouds over Horopito

I missed my own ancestral family, meeting for a Prairie Christmas back home in Canada. (But appreciated the fact that I was thirty or so Celsius degrees warmer.) And I’m so thankful and heartwarmed by the hospitality and love, effortlessly given in acre-sized servings by Mary and Martin and the rest of the Wood clan.

Page processed in 2.734 seconds.

Powered by SimplePie 1.5, Build 20180814171753. Run the SimplePie Compatibility Test. SimplePie is © 2004–2020, Ryan Parman and Geoffrey Sneddon, and licensed under the BSD License.