ongoing by Tim Bray

ongoing fragmented essay by Tim Bray

Strongly Typed Events 2 Dec 2019, 10:00 pm

Back in 2016, in Message Processing Styles, I was sort of gloomy and negative about the notion of automated mapping between messages on the wire and strongly-typed programming data structures. Since we just launched a Schema Registry, and it’s got my fingerprints on it, I guess I must have changed my mind.

Eventing lessons

I’ve been mixed up in EventBridge, formerly known as CloudWatch Events, since it was scratchings on a whiteboard. It has a huge number of customers, including but not limited to the hundreds of thousands that run Lambda functions, and the volume of events per second flowing through the main buses are keeping a sizeable engineering team busy. This has taught me a few things.

First of all, events are strongly subject to Hyrum's Law: With a sufficient number of users of an API, it does not matter what you promise in the contract: all observable behaviors of your system will be depended on by somebody. Which is to say, once you’ve started shipping an event, it’s really hard, which is to say usually impossible, to change anything about it.

Second: Writing code to map back and forth between bits-on-the-wire and program data structures is a very bad use of developer time. Particularly when the messages on the wire are, as noted, very stable in practice.

Thus, the new schema registry. I’m not crazy about the name, because…

Schemas are boring

Nobody has ever read a message schema for pleasure, and very few for instruction. Among other things, most messages are in JSON, and I have repeatedly griped about the opacity and complexity of JSON Schema. So, why am I happy about the launch of a Schema Registry? Because it lets us do two useful things: Search and autocomplete.

Let’s talk about Autocomplete first. When I’m calling an API, I don’t have to remember the names of the events or their arguments, because my IDE does that for me. As of now, this is true for events as well; the IDE knows the names and types of the fields and sub-fields. This alone makes a schema registry useful. Or, to be precise, the code bindings and serializers were generate from the schema.

The search side is pretty simple. The schema registry is just a DynamoDB thing, nothing fancy about it. But we’ve wired up an ElasticSearch index so you can type random words at it to figure out which events have a field named “drama llama” or whatever else you need to deal with today.

Inference/Discovery

This is an absolutely necessary schema-registry feature that most people will never use. It turns out that writing schemas is a difficult and not terribly pleasant activity. Such activities should be automated, and the schema registry comes with a thing that looks at message streams and infers schemas for them. They told us we couldn’t call it an “Inferrer” because everyone thinks that means Machine Learning. So it’s called “schema discovery” and it’s not rocket science at all, people have been doing schema inference for years and there’s good open-source code out there.

So if you want to write a schema and jam it into the registry, go ahead. For most people, I think it’s going to be easier to send a large-enough sample of your messages and let the code do the work. At least it’ll get the commas in the right place. It turns out that if you don’t like the auto-generated schema, you can update it by hand; like I said, it’s just a simple database with versioning semantics.

Tricky bits

By which I mean, what could go wrong? Well, as I said above, events rarely change… except when they do. In particular, the JSON world tends to believe that you can always add a new field without breaking things. Which you can, until you’ve interposed strong types. This is a problem, but it has a good solution. When it comes to bits-on-the-wire protocols, there are essentially two philosophies: Must-Understand (receiving software should blow up if it sees anything unexpected in the data) and Must-Ignore (receiving software must tactfully ignore unexpected data in an incoming message). There are some classes of application where the content is so sensitive that Must-Understand is called for, but for the vast majority of Cloud-native apps, I’m pretty sure that Must-Ignore is a better choice.

Having said that, we probably need smooth support for both approaches. Let me make this concrete with an example. Suppose you’re a Java programmer writing a Lambda to process EC2 Instance State-change Notification events, and through the magic of the schema registry, you don’t have to parse the JSON, you just get handed an EC2InstanceStateChangeNotification object. So, what happens when EC2 decides to toss in a new field? There are three plausible options. First, throw an exception. Second, stick the extra data into some sort of Map<String, Object> structure. Third, just pretend the extra data wasn’t there. None of these are insane.

There’s another world out there where the bits-on-the-wire aren’t in JSON, they’re in a “binary” format like Avro or Protocol Buffers or whatever. In that world you really need schemas because unlike JSON, you just can’t process the data without one. In the specific (popular) case of Avro-on-Kafka, there’s a whole body of practice around “schema evolution”, where you can update schemas and automatically discover whether the change is backward-compatible for existing consumers. This sounds like something we should look at across the schemas space.

Tactical futures

Speaking of those binary formats, I absolutely do not believe that the current OpenAPI schema dialect is the be-all and end-all. Here’s a secret: The registry database has a SchemaType field and I’m absolutely sure that in future, it’s going to have more than one possible value.

Another to-do is supporting code bindings in languages other than the current Java, TypeScript, and Python. At the top of my list would be Go and C#, but I know there are members of other religions out there. And for the existing languages, we should make the integrations more native. For example, the Java bindings should be in Maven.

And of course, we need support in all the platform utilities: CloudFormation, SAM, CDK, Terraform, serverless.com, and any others that snuck in while I wasn’t looking.

Big futures

So, I seem to have had a change of worldview, from “JSON blobs on the wire are OK” to “It’s good to provide data types.” Having done that, everywhere I look around cloud-native apps I see things that deal with JSON blobs on the wire. Including a whole lot of AWS services. I’m beginning to think that more or less anything that deals with messages or events should have the option of viewing them as strongly-typed objects.

Which is going to be a whole lot of work, and not happen instantly. But as it says in Chapter 64 of the Dao De Jing: 千 里之行,始於足下 — “A journey of a thousand leagues begins with a single step”.

Electric Boats 29 Nov 2019, 10:00 pm

I love boating, but I hate the fact that powerboats guzzle loads of fossil fuel. I assuage my guilt by noting that the distance traveled is small — a couple of hours for each return trip to the cabin, and there are sadly less than twenty of those per year. Then I got into a discussion on /r/boating about whether electric boats are practical, so herewith some scratchpad calculations on that subject.

I’ve ridden on an electric boat, on the Daintree River in Queensland, on a small alligator-watching tour. This thing was flat and had room for maybe fifteen tourists under a canopy, necessary shelter from the brutal tropical sun; on top of the canopy were solar panels, which the pilot told me weren’t quite enough to run the boat up and down the river all day, he had to plug it in at night. The motor was a 70HP electric and our progress along the river was whisper-quiet; I loved it.

I should preface this by saying that I’m not a hull designer nor a battery technologist nor a marine propulsion engineer, so the calculations here have little more precision than Enrico Fermi’s famous atomic-bomb calculation. But possibly useful I think.

Narrowing the question

There are two classes of recreational motorboat: Those that go fast by planing, and those that cruise at hull speed, which is much slower and smoother. Typically, small motorboats plane and larger ones cruise. I’m going to consider my Jeanneau NC 795 as an example of a planing boat, and a Nordic Tug 32 as an example of a cruiser, because there’s one parked next to me and it’s a beautiful thing.

’Rithmetic

My car has a 90 kWh battery, of a size and weight that could be accommodated in either boat quite straightforwardly. A well-designed electric such as a Tesla typically burns 20 kWh/100km but you can’t use all the kWh in a battery, so you can reasonably expect a range of about 400km.

The Jeanneau gets about 1.1 km per liter of fuel while planing (it does two or three times better while cruising along at hull speed). Reviewers say that at 7-8 knots the Nordic burns about a gallon per hour, which my arithmetic says is 3.785 km/L.

A typical gas car gets about 10L / 100km, so 10 km/L. So the Nordic Tug is about 38% as efficient as turning fuel into km as a car, and the Jeanneau is only about 11% as efficient. (And of course both go much slower, but that’s not today’s issue.)

If the same is true for electric “fuel”, the battery that can take a Tesla 400km could take the Nordic tug about 150km and the Jeanneau a mere 44km.

Discussion

There are boats that get worse mileage than the Jeanneau, but they’re super-macho muscle boats or extravagant yachts the size of houses. So for recreational boats accessible to mere mortals, the Jeanneau, which is in a class called “Express Cruiser”, is kind of a worst-case, a comfy family carrier that can be made to go fast on the way to the cabin, but you pay for it.

So the tentative conclusion is that at the moment, batteries are not that attractive for express cruisers. But for tugboat-class craft designed for smoothness not speed, I’d say the time to start building them is now. Among other things, marine engine maintenance is a major pain in boaters’ butts, and electric engines need a whole lot less. On top of which they’re a lot smaller, and space is always at a premium out on the water.

Variations and Futures

The following are things likely to undermine the calculations above:

  1. The Jeanneau could fit in quite a bit bigger battery than most cars; packing it into the hull so that the boat still performs well would be an interesting marine engineering problem.

  2. The Nordic Tug could pretty easily fit in a battery two or three times that size and, at hull speed, I suspect it wouldn’t slow down much.

  3. The torque curves of electric and gas engines are vastly different; which is to say, electrics don’t really have one. You have to get the Jeanneau’s engine up north of 4K RPM to really zoom along, which I suspect is not exactly fuel-optimized performance.

  4. Related to the previous item, I wouldn’t be surprised if there were considerable gains you could pull out of the low-RPM/high-torque electric engine in terms of propeller design and perhaps hull shape.

  5. Battery energy density is improving monotonically, but slowly.

  6. Boats tend to spend their time out under the open sky and many have flat roofs you could put solar panels on to (slowly) recharge your batteries. In fact, many do already just for the 12V cabin batteries that run the fridge and lights.

  7. I expect installation of Level 2 chargers on the dockside would require some innovation for safe operation in an exposed environment full of salt water. I doubt that it’d be practical to offer 50-100kW DC fast-charging gear at the gas barge.

I’ve long lusted after tugboat-style craft, but they’re expensive, bigger (thus moorage is hard to find), and go slower. Given a plausible electric offering, I think I could learn to live with that.

Wolfe and Gika 23 Nov 2019, 10:00 pm

Chelsea Wolfe I mean, and her opening act Thursday night was Ioanna Gika. It was exceptionally enjoyable, partly because nothing about it was wrong. Lovely music, great staging, good venue, exceptional sound. This happens rarely enough that it’s worth calling out when it does.

Chelsea Wolfe

Ms Wolfe has appeared on this blog multiple times. She has two modes: Acoustic, where the songs are ethereal splashes of complex slow-moving beauty; and electric, banshee keening nestled into a torrent of roaring guitar drone. Thursday night was sort of acoustic; Chelsea had an acoustic guitar and then a guy on keyboards and guitars.

Chelsea WolfeChelsea Wolfe

I’ve seen her perform twice now, and both times it’s been very… dark. Thematically and then visually. After four hours or so in her presence, I’m sure I wouldn’t recognize her on the street. It was challenging fun to photograph.

The music was not quite like her acoustic records, in that the musical backdrop was electric and synthetic, and dark like the room. Her melodies don’t orbit in your head for weeks, but the orchestration and raw beauty of the sound palette are formidable, and then she is a wonderful singer, voice full of grace and power with an astonishing high register that she pours out with no effort and exactly when the song’s flow needs it.

An unexpected pleasure was when she left her platform, sat down beside her accompanist and, to a minimal slow chord rhythm, performed Joni Mitchell’s Woodstock, taking it pretty straight but digging deep into each note and soaring up into “we are golden…” — I got all emo.

Ioanna Gika

She wasn’t announced and I didn’t catch her name when she gave it, so I didn’t know who she was till I asked the Internet after the show. Like Chelsea, she performed with just a keyboard/guitar accompanist. Perhaps even the same one, there wasn’t enough light to tell. She had a minimal little device on a stand that could play keyboard, but what she mostly did with it was capture her own voice, load it up with reverb, then sing counterpoint against it.

Thalassa by Ioanna Gika

Her music was quite a bit more dynamic than Wolfe’s even if she didn’t bring that voodoo intensity. Listening to her, I thought of the Cocteau Twins, and of Enya, and heard prog-rock echoes too. I really, really liked it and bought her CD Thalassa from the merch table. It’s great, but I actually liked the concert performance better; more intense, more complex, riskier. And also because of the sound.

Audio wow

The show was at the Vogue Theatre in Vancouver; I can’t remember the last time I was there, but it’s a nice room. I have to say this was the best live sound I’ve heard since the last time I heard unamplified acoustic music in a decent room, and I’ve never heard better live electric sound in my life that I can remember. I dunno what kind of microphones and signal chain they had, but the voices were full of three-dimensional artifacts, hints of chest and breath, not an atom of overload ever. Both women built huge low synth notes into their arrangements, placed with musical intelligence; these sounded like they were coming from the center of the earth, dark but crystal-clear.

More than a half-century into the era of live electric music, really great sound is still the exception not the rule. The touring-tech profession should be ashamed. And kudos to whoever did sound for the Wolfe/Gika gig.

Bits On the Wire 17 Nov 2019, 10:00 pm

Exactly 100% of everything on the Internet involves exchanging messages which represent items of interest to humans. These items can be classified into three baskets: One for “media” (images, sound, video), one for text (HTML, PDF, XML), and one for “objects” (chat messages, payments, love poems, order statuses). This is a survey of how Object data is encoded for transmission over the Internet. Discussed: JSON, binary formats like Avro and Protobufs, and the trade-offs. Much of what people believe to be true is not.

History sidebar

The first ever cross-systems data interchange format was ASN.1, still used in some low-level Internet protocols and crypto frameworks. ASN.1 had a pretty good data-type story, but not much in the way of labeling. Unfortunately, this was in the days before Open Source, so the ASN.1 software I encountered was slow, buggy, and expensive.

Then XML came along in 1998. It had no data-typing at all but sensibly labeled nested document-like data structures. More important, it had lots of fast solid open-source software you could download and use for free, so everybody started using it for everything.

Then sometime after 2005, a lot of people noticed JSON. The “O” stands for “Object” and for shipping objects around the network, it was way slicker than XML. By 2010 or so, the virtuous wrath of the RESTafarians had swept away the pathetic remnants of the WS-* cabal. Most REST APIs are JSON, so the Internet’s wires filled up with media, text, and JSON.

On JSON

I think there’s still more of it out there than anything else, if only because there are so many incumbent REST CRUD APIs that are humming along staying out of the way and getting shit done.

JSON pros:

  1. Readers and writers are implemented in every computer language known to humankind, and they tend to interoperate correctly and frictionlessly with each other, particularly if you follow the interoperability guidelines in RFC 8259, which all the software I use seems to.

  2. It does a pretty good job of modeling nested-record structures.

  3. It’s all-text, so humans can read it, which is super extra helpful.

  4. You can receive a JSON message you know nothing about and pick it apart successfully without knowing its schema, assuming it has one, which it probably doesn’t. So you can accomplish a task like “Pull out the item-count and item-price fields that are nested in the top-level order-detail field” with pretty good results given just a blob of raw JSON.

  5. You can reliably distinguish between numbers, strings, booleans, and null.

JSON cons:

  1. The type system is impoverished. There is no timestamp type, no way to know whether a number should be treated as an integer or float or Bignum, no way to signal when string values are really enums, and so on.

  2. Numbers are specially impoverished; in general you should assume that your repertoire is that of an IEEE double-precision float (but without NaN or ∞) which is adequate for most purposes, as long as you’re OK with an integer range of ±253 (which you probably should be).

  3. Since JSON is textual, there is a temptation to edit it by hand, and this is painful since it’s nearly impossible to get the commas in the right places. On top of which there are no comments.

  4. JSON’s textuality, and the fact that it carries its field labels along, no matter how deeply nested and often repeated, suggest that it is unnecessarily verbose, particularly when numeric values are represented in textual form. Also, the text needs to be converted into binary form to be loaded into objects (or structs, or dicts) for processing by code in memory.

  5. JSON doesn’t have a universally-accepted schema language. I have been publicly disappointed over “JSON Schema”, the leading contender in that space; it’s just not very good. For a long time, the popular Swagger (now OpenAPI) protocols for specifying APIs used a variant version of a years-old release of JSON Schema; those are stable and well-tooled.

Avro

Mainstream binary formats

I think that once you get past JSON, Apache Avro might be the largest non-text non-media consumer of network bandwidth. This is due to its being wired into Hadoop and, more recently, the surging volume of Kafka traffic. Confluent, the makers of Kafka, provide good Avro-specific tooling. Most people who use Avro seem to be reasonably happy with it.

Protobufs

Protobufs (short for “Protocol Buffers”) I think would be the next-biggest non-media eater of network bandwidth. It’s out of Google and is used in gRPC which, as an AWS employee, I occasionally get bitched at for not supporting. When I worked at Google I heard lots of whining about having to use Protobufs, and it’s fair to say that they are not universally loved.

Next in line would be Thrift, which is kind of abstract and includes its own RPC protocol and is out of Facebook and I’ve never been near it.

JSON vs binary

This is a super-interesting topic. It is frequently declaimed that only an idiot would use JSON for anything because it’s faster to translate back and forth between data types in memory with Avro/Protobufs/Thrift/Whatever (hereinafter “binary”) than it is with JSON, and because binary is hugely more compact. Also binary comes with schemas, unlike JSON. And furthermore, binary lets you use gRPC, which must be brilliant since it’s from Google, and so much faster because it’s compact and can stream. So, get with it!

Is binary more compact than JSON?

Yes, but it depends. In one respect, absolutely, because JSON carries all its field labels along with it.

Also, binary represents numbers as native hardware numbers, while JSON uses strings of decimal digits. Which must be faster, right? Except for your typical hardware number these days occupies 8 bytes if it’s a float, and I can write lots of interesting floats in less than 8 digits; or 4 bytes for integers, and I can… hold on, a smart binary encoder can switch between 1, 2, 4, and 8-byte representations. As for strings, they’re all just the same UTF-8 bytes either way. But binary should win big on enums, which can be represented as small numbers.

So let’s grant that binary is going to be more compact as long as your data isn’t mostly all strings, and the string values aren’t massively longer than the field labels. But maybe not as much as you thought.

Unless of course you compress. This changes the picture and there are a few more it-depends clauses, but compression, in those scenarios where you can afford it, probably reduces the difference dramatically. And if you really care about size enough that it affects your format choices, you should be seriously looking at compression, because there are lots of cases where you’ve got CPU to spare and are network-limited.

Is binary faster than JSON?

Yes, but it depends. Here’s an interesting benchmark from Auth0 showing that if you’re working in JavaScript, the fact that JSON is built-in to the platform makes Protobuf’s advantages mostly disappear; but in an equivalent Java app, protobuf wins big-time.

Whether or not your data is number- or string-heavy matters in this context too, because serializing or deserializing strings is just copying UTF-8bytes.

I mentioned gRPC above, and one aspect of speed heavily touted by the binary tribe is in protobufs-on-gRPC which, they say, is obviously much faster than JSON over HTTP. Except for HTTP is increasingly HTTP/2, with longer-lived connections and interleaved requests. And is soon going to be QUIC, with UDP and no streams at all. And I wonder how “obvious” the speed advantage of gRPC is going to be in that world?

I linked to that one benchmark just now but that path leads to a slippery slope; the Web is positively stuffed with serialization/deserialization benchmarks, many of them suffering from various combinations of bias and incompetence. Which raises a question:

Do speed and size matter?

Can I be seriously asking that question? Sure, because lots of times the size and processing speed of your serialization format just don’t matter in the slightest, because your app is bottlenecked on database, or on garbage collection, or on a matrix inversion or an FFT or whatever.

What you should do about this

Start with the simplest possible thing that could possibly work. Then benchmark using your data with your messaging patterns. In the possible but not terribly likely case that your message transmission and serialization is a limiting factor in what you’re trying to do, go shopping for a better data encoding.

The data format long tail

Amazon Ion has been around for years running big systems inside Amazon, and decloaked in 2015-16. It’s a JSON superset with a usefully-enriched type system that comes in fully interoperable binary and textual formats. It has a schema facility. I’ve never used Ion but people at Amazon whose opinion I respect swear by it. Among other things, it’s used heavily in QLDB, which is my personal favorite new AWS service of recent years.

CBOR is another binary format, also a superset of JSON. I am super-impressed with the encoding and tagging designs. It also has a schema facility called CDDL that I haven’t really looked at. CBOR has implementations in really a lot of different languages.

I know of one very busy data stream at AWS that’s running at a couple of million objects a second where you inject JSON and receive JSON, but the data in the pipe is CBOR because at that volume size really starts to matter. It helped that the service is implemented in Java and the popular Jackson library handles CBOR in a way that’s totally transparent to the developer.

I hadn’t really heard much about MessagePack until I was researching this piece. It’s yet another “efficient binary serialization format”. The thing that strikes me is that every single person who’s used it seems to have positive things to say, and I haven’t encountered a “why this sucks” rant of the form that it’s pretty easy to find for every other object encoding mentioned in this piece. Checking it out is on my to-do list.

While on the subject of efficient something something binary somethings, I should mention Cap’n Proto and FlatBuffers, both of which seem to be like Avro only more so, and make extravagant claims about how you can encode/decode in negative nanoseconds. Neither seems to have swept away the opposition yet, though.

 [Shouldn’t you mention YAML? —Ed.]
 [No, this piece is about data on the
network. —T.]

On Schemas

Binary really needs schemas to work, because unless you know what those bits all snuggled up together mean, you can’t un-snuggle them into your software’s data structures. This creates a problem because the sender and receiver need to use the same (or at least compatible) schemas, and, well, they’re in different places, aren’t they? Otherwise what’s the point of having messaging software?

Now there are some systems, for example Hadoop, where you deal with huge streams of records all of which are the same type. So you only have to communicate the schema once. A useful trick is to have the first record you send be the schema which then lets you reliably parse all the others.

Avro’s wire format on Kafka has a neat trick: The second through fifth byte encode a 4-byte integer that identifies the schema. The number has no meaning, the schema registry assigns them one-by-one as you add new schemas. So assuming both the sender and the receiver are using the same schema registry, everything should work out fine. One can imagine a world in which you might want to share schemas widely and give them globally-unique names. But those 32-bit numbers are deliciously compact and stylishly postmodern in their minimalism, no syntax to worry about.

Some factions of the developer population are disturbed and upset that a whole lot of JSON is processed by programmers who don’t trouble themselves much about schemas. Let me tell you a story about that.

Back in 2015, I was working on the AWS service that launched as CloudWatch Events and is now known as EventBridge. It carries events from a huge number of AWS services in a huger number of distinct types. When we were designing it, I was challenged “Shouldn’t we require schemas for all the event types?” I made the call that no, we shouldn’t, because we wanted to make it super-easy for AWS services to onboard, and in a lot of cases the events were generated by procedural code and never had a schema anyhow.

We’ve taken a lot of flak for that, but I think it was the right call, because we did onboard all those services and now there are a huge number of customers getting good value out of EventBridge. Having said that, I think it’d be a good idea at some future point to have schemas for those events to make developers’ lives easier.

Not that most developers actually care about schemas as such. But they would like autocomplete to work in their IDEs, and they’d like to make it easy to transmogrify a JSON blob into a nice programming-language object. And schemas make that possible.

But let’s not kid ourselves; schemas aren’t free. You have to coördinate between sender and receiver, and you have to worry what happens when someone wants to add a new field to a message type — but in raw JSON, you don’t have to worry, you just toss in the new field and things don’t break. Flexibility is a good thing.

Events, pub/sub, and inertia

Speaking of changes in message formats, here’s something I’ve learned in recent years while working on AWS eventing technology: It’s really hard to change them. Publish/subscribe is basic to event-driven software, and the whole point of pub/sub is that the event generator needn’t know, and doesn’t have to care, about who’s going to be catching and processing those events. This buys valuable decoupling between services; the bigger your apps get and the higher the traffic volume, the more valuable the decoupling becomes. But it also means that you really really can’t make any backward-incompatible changes in your event formats because you will for damn sure break downstream software you probably never knew existed. I speak from bitter experience here.

Now, if your messages are in JSON, you can probably get away with throwing in new fields. But almost certainly not if you’re using a binary encoding.

What this means in practice is that if you have a good reason to update your event format, you can go ahead and do it, but then you probably have to emit a stream of new-style events while you keep emitting the old-style events too, because if you cut them off, cue more downstream breakage.

The take-away is that if you’re going to start emitting events from a piece of software, put just as much care into it as you would as you do in specifying an API. Because event formats are a contract, too. And never forget Hyrum’s Law:

With a sufficient number of users of an API,
it does not matter what you promise in the contract:
all observable behaviors of your system
will be depended on by somebody.

Messages too!

The single true answer to all questions about data encoding formats

“It depends.”

Subscription Friction 3 Nov 2019, 10:00 pm

At Canadian Thanksgiving, friends joined us at our cottage for turkey and the fixings. The food (what we made and what they brought) all came out great and we had happy stomachs. I did a lot of the cooking and wanted to check recipes and was brought face to horrified face with the failure of publishing to work on the Internet. The solution seems plain to me and this won’t be the first time I’ve offered it. But something really needs to be done.

What happened was, I wanted to refresh my memory on turkey and gravy. These are dishes one makes regularly but rarely and the details don’t stick to my mind: What’s the meat-thermometer reading for a stuffed bird, and what order do you deploy the drippings, roux, and giblet broth in for gravy?

It turns out the part of the Web where recipes live is a dystopian hellscape. Even though the medium seems made for the message: Lists, pictures, and search are built-in! Not to mention voting and hyperlinks. Anyone who thinks they’ve got a really great gravy procedure can tell the world, and as more people agree, the recipe should become easier to find.

Except for what you find is an SEO battle, red in tooth and claw, where recipe sites stinking of burn-rate terror plead for engagement and as a matter of principle make it really hard to get to that damn ingredients-list and step-by-step. Newsletter! Notifications! Survey! Also-see! Multidimensional parallax-enhanced five-way scrolling Single-Page Applications!

In fact, it bothered me so much that a few days later, I posted a nice simple chicken-kebab recipe just for that good feeling when you use the Web the way it was designed. I took the pictures, I wrote the text, and I posted it all on my own domain, and anyone who follows a link will have the words and pictures in front of their eyes right away, and there won’t be any popups of any kind.

But it’s not just recipes, obviously. It’s a disease that afflicts the larger community of those who try to add value to the Net and to our lives by sharing their stories and pictures and movies and poems and dreams, in the process we generally call “publishing”.

It isn’t working.

“Become a subscriber today!”

My words today are interspersed with pictures of the signs and signals that obscure an increasingly large proportion of the Web’s surface: Imminent-closing-paywall-gate warnings.

Subscribe to Foreign Policy!

I think there is something that is glaringly, screamingly obvious that at this point needs to be said loudly and repeatedly:

This.
Will.
Not.
Scale.

I suspect that a high proportion of my readers know this is true just from the feeling in their gut, but as an exercise in rhetoric let me offer some of the reasons why.

  1. Subscription fatigue is setting in. I’ll make a spending decision, even a big one, quickly and without regrets when whatever it is feels like the right thing. But a subscription, another bill showing up on every credit-card statement… forever?

  2. A very high proportion of the world’s curious-minded well-educated people subscribe to some subset of The Economist, the New York Times, the Guardian, the Washington Post, and the New Yorker. Most of us feel we’re subscribed enough.

  3. There’s a huge class of publications who’ll put out a handful of articles every year that I want to read, but not remotely enough to justify a subscription.

  4. The offers are sleazy. Whenever I read an absurdly low-cost subscription offer, I know the number showing up my bill is going to be a lot higher pretty damn quick.

  5. Obviously it’s not just journalism. What madness makes huge companies think that people will sign up for Netflix and Hulu and HBO and Amazon Video and Apple TV+ and Disney? I’m sure I’ve forgotten a few, and even more sure that more are on the way.

Subscribe to the Globe & Mail!

This. Will. Not. Scale.

We need friction somewhere

Information may want to be free, but writers want to be paid. In a world where you can follow any link and read whatever it points at instantly and for free, writers are going to starve. It’s the writers I care about the most, but let’s not forget the designers and copy-editors and fact-checkers and accountants and so on.

People who write and gather and curate recipes want to be paid too.

How about advertising?

The Grand Theory of the Web long said that the money comes from advertising. After all, there’s always been lots of free information out there, and there still is if you’re willing to put up a TV antenna. The reality is that unless you’re Google or Facebook, advertising is deeply broken. My favorite exposition of why ads as they are today don’t work is by Josh Marshall in Data Lords: The Real Story of Big Data, Facebook and the Future of News. It’s important to note that Marshall isn’t arguing that Web advertising is inherently broken; just that the current Facebook/Google business model is so successful that they’re basically skimming all the profit off the top of the system, and in particular ripping the guts out of independent niche publishers.

Subscribe to the Atlantic!

If you think “I already knew that”, take a minute and read that “Data Lords” piece linked above; it’s not as obvious as you might think.

Marshall is the founder and publisher of Talking Points Memo, one of the best US left-wing political publications, and had the good fortune to recognize the dead-endness of advertising early on and, starting in 2012, made a hard pivot to subscriptions. They’ve been innovative and aggressive and executed well, and it looks like they have an excellent chance of staying in it for the long haul.

What we’re seeing now is that more or less every Web publication has, in the last couple of years, come to the same conclusion and is trying the same pivot. Only it’s obviously not going to work.

Because This. Will. Not. Scale.

Which way forward?

One helpful thing would be to fix advertising. It’s easy to hate the ad business and ad culture but at the end of the day they’re probably a necessary facilitator in a market economy. And hey, if they want to pay part of the cost of the publications I like to read, I’m down with that. I’m not smart enough to have designed a regulatory framework that would restore health to the market, but I don’t think it’s controversial to say that it’d be really great if someone did.

Subscribe to the Spectator!

But it seems to me that there’s a more obvious way; let me buy stories one-at-a-time without signing up for a monthly-charge-forever. The idea is this: When I follow a link to a juicy-sounding story in, say, the Minneapolis Star-Tribune or the The Melbourne Age, instead of the avalanche of subscription wheedling, I get a standardized little popup saying “Ten cents to read this”, with several options: Yes or no on this piece, and Always-yes or Always-no for this publication. Someone is operating the service that makes this happen and will do an aggregate charge to my credit card every month.

Suppose the Net lights up because some site has got their hands on Donald Trump’s tax returns. Maybe when I follow that link, the standard popup asks me for a buck instead of a dime.

Subscriptions would still be possible and still make sense if you were reading a lot of pieces from some publication, and could come with benefits — for example, Josh Marshall’s Talking Points Memo offers a super-subscription which subtracts all the ads and makes the pages wonderfully fast and lightweight.

Is that even possible?

It’s not exactly a new idea. Something like it was originally proposed by Ted Nelson in Literary Machines in 1981. The fact that nobody’s made it work so far might make a reasonable person pessimistic as to whether anyone ever will.

Subscribe to Vanity Fair!

Well, I’m here to tell you that on the technology side, we have the tools we need to build this. You could spend a bunch of time devising an Internet Standard protocol for subscription services and wrangle it through the IETF or W3C or somewhere, and that’s not a terrible idea, but I’d probably want to build the software first. With an Amazon-style two-pizza team and modern cloud infrastructure I’d have it on the air in 2020, no problem.

The one design constraint I’d impose would be that this thing would have to work at small scale, not just for the The Bigs. [Disclosure: Yes, I’m a blogger and I’d like us to be able to make a buck too.] But once again, I just don’t see it as hard.

Bootstrapping the business side would be tough because the publishing industry is led by people who not only are not focused on technology but suffer from unrealistic fantasies as to what it is and isn’t. It’d be dead easy for Amazon or Google to offer this as a service, but the publishing community would, rightly or wrongly, assume it to be another way to suck all the money out of the sector.

Maybe there’s a role here for a consortium or institute; or for some large far-sighted publisher with a stable of properties to build this for themselves but with careful attention to making attractive to the rest of the industry?

My advice would be to get working on it fast. Because ads are broken and burn rates are high and pivoting to subscriptions was a really great idea in 2014 but it’s too late for that now.

Easy Reliable Chicken Kebabs 13 Oct 2019, 9:00 pm

This involves a certain amount of chopping stuff up, also attention to hygiene, but requires no particular technical skill and has never ever failed to get rave reviews.

Infrastructure and ingredients

  1. A barbecue. Charcoal is said to produce better flavor but gas is immensely faster and easier.

  2. Skewers. Metal or wood, whatever.

  3. Chicken. I’m lazy, I buy it skinless & boneless from the supermarket. Breasts are easier to work with but thighs come out a little better. A kg will satiate four large hungry people.

  4. Lemons. You could use the stuff that comes in bottles but that would be lame. Juicing lemons is easy.

  5. Olive oil. If you think it makes a difference you can pay up for “Extra Virgin”. I do recommend looking for a respectable brand because I gather there is some seriously sketchy stuff upstream of the liquid labeled as “olive oil” on many store shelves.

  6. Garlic. You could be sincere and slice or mash it, or you could buy a jar of garlic paste or minced garlic at the aforementioned supermarket, as I do.

  7. Black pepper. Fresh-ground really does make a difference.

  8. Other things to go on the skewers. The only essential thing is onions (but shallots are better). Also popular: Small tomatoes, mushrooms, sweet peppers (Aussies: I mean capsicum).

Checken kebab construction

The chopping part is kind of boring, so I watched the ALCS.

Safety note: Several of these steps require getting intimate with uncooked chicken. Which can contain the sort of microorganism that causes major unhappiness. So a hot-soap-and-water hand-wash is in order whenever you move from the raw-chicken part to any other part. And then give the part of the counter where you did the raw-chicken stuff a good scrub.

Procedure

First, you make the marinade. Mine is equal-ish parts olive oil and lemon juice, with several large dollops of garlic paste, and lashings of black pepper. If I have oregano (dried not fresh) handy I throw some in; if not I don’t worry about it. Mix vigorously. You need enough marinade to cover your chicken and a little bit goes a surprisingly long way. Say 3 juicy lemonsworth and the same or a little more olive oil.

Cut up your chicken. You want pieces of a size and shape that go comfortably on the skewer. I tend to favor smaller rather than larger to increase the surface-area ratio, because the surface is where the marinade soaks in.

Drop the chicken into the marinade and swish it around. Put it in the fridge and ignore it for a while. I think two or three hours is plenty, but purists do all this the day before. Important: Count the pieces as you cut them up.

Now we’re getting ready to cook. Here’s where you need to get quantitative. Figure out how many skewers you want to make (or maybe just how many you have) and look studiously blank while you sip a glass of preprandial wine and calculate how many pieces of chicken per skewer. Since each chicken chunk should have one or more non-meat items separating it from the next, you now have sufficient information to calculate how many pieces of (for example) onion, mushroom, pepper, and so on you need. The one I did today had three pieces of chicken, two pieces of onion, one piece of pepper, two pieces of mushroom, and two cherry-tomato halves (one red, one yellow) on each skewer. I had forty meat morsels, and I made twelve skewers for the four of us, so the veggie numbers were easy to work out.

Chicken kebabs ready to cook

Ready for the grill.

Stop here and, if you are using wooden skewers, drop them in a sink-full or pot-full of water to get somewhat fireproof.

Cut up all your vegetables and fungi. I like to grab a bunch of cereal bowls, one for each ingredient. I try to cut them roughly the same size as I did the chicken, so that when I put them on the barbecue you won’t have a big hunk of onion shielding the meat from the flame, or the other way around. If you have a compost bin, move it from wherever it is to where you’re slicing, to capture the waste.

OK, pull the meat out of the fridge. Pour the marinade down the drain. Grab a big plate to put the completed skewers on. Arrange the chicken and all the other skewer-fodder and the plate as seems best to you. I’m a soulless geek so I work out a repeatable skewer script (e.g. tomato, chicken, onion, mushroom, chicken, pepper, mushroom, chicken, onion, tomato) and stick with it.

At some point halfway through, wash your hands and go fire up the barbecue.

Once you’re done you’ll have a satisfying heap of skewers on the plate, and quite a bit of chicken pollution on your hands and the counter. If you miscalculated and there are bits of leftover vegetable or chicken stick the ones you can on skewers and throw the rest out, they’re unsanitary.

Slap ’em on the barbecue, wait till one side is done, turn ’em over, wait again, then enjoy. Tongs help. If you’ve kept the size of the chicken morsels down, then when one side is visibly seared I find it’s reliably cooked inside. Four or five minutes each side works for me, but that’s on my barbecue with my preparation habits. You do really need to make sure the chicken’s done.

Chicken kebabs ready to eat

Accompaniments

Rice works well. I like to toss asparagus or broccoli on the part of the barbecue that isn’t full of skewers, but whatever. White wine, or (wouldn’t have said this even a couple of years ago) a good artisanal cider.

Leftovers

There won’t be any. The lemon-soaked garlic-loaded flame-kissed skewer payloads are ravishing. And it’s hard to get this one wrong.

[Disclosure: The attentive observer will notice that the pictures are of a different batch than the one the text describes.]

Records and Lenses 6 Oct 2019, 9:00 pm

Sunday included more fun than the recent average — at my age, chilling is often more attractive than partying. Sunday featured vinyl, vintage lenses, Southern guitar boogie, and a photo-assignment. With pictures! (Which may be a little puzzling, but stay with me.)

Record convention poster

What happened was, I noticed that there was a Fall 2019 Record Convention and on impulse took the e-bike crosstown to the glamorous (not really) Croation Cultural Centre. I strolled in the door of the big room with all the people and the nice lady said “You want the record convention or the camera convention? This is the cameras.” Who knew? So I went over to the other side and ended up buying five records: Willie Nelson’s Teatro (with Emmylou and Daniel Lanois), Howlin’ Wolf’s Big City Blues, Yellowman and Fathead’s Bad Boy Skanking, James Blood Ulmer’s Free Lancing, and Sons of Kemet’s Your Queen is a Reptile.

It was just a high-school-gym-sized room with a big square of tables facing outward facing the other big square against the walls facing inward, all the tables full of boxes full of LPs. I posted an appropriately lo-rez video on Instagram. A few of the vendors had organizing principles, ranging from genre to alphabet to price. Lots didn’t.

What a bunch of stoners. Except for a few fresh-faced hipsters and the odd grizzled geek like yours truly. Good clean fun.

Main Street by night

Photopunk

Since I’d come all the way, I coughed up $5 for the camera show, which was pretty well maximum steampunk. Old cameras! Some with bellows and bulbs, some the size of your head. I have enough cameras but you can never have too many lenses. There wasn’t much chance of anything Fujifilm-related (it’s from this millennium) but I have a Pentax K-mount adapter and as I wrote in that context, “there are an astonishing number of excellent K-mount lenses in every imaginable length and width on the market.” So I walked around the tables saying “K-mount?” to anyone showing lenses and lots had some.

Main Street by night

Protip: Nifty fifty

If you’re stingy and want to take lovely portraits it turns out that for basically any camera in the world you can get a K-mount adapter for your camera cheap, and then go to a show like this (or eBay or whatever) and pick up a classic Pentax 50mm F1.4 for almost no money and with a little practice and love, you’ll be making your friends and loved ones look better than they really do.

But I have a nifty fifty already. I’d got two thirds of the way around the camera trail and saw nothing interesting, but then there was this Mittel-Euro dude with an astonishingly small and slim 100mm F/2.8. He wanted $120 Canadian. I walked away and completed the orbit without turning up any gold, so I went back and offered him $100 and he took it. It turns out to be an M 100/2.8, i.e. probably 40 years old.

Fujifilm XT-30 with Pentax M

Isn’t it adorable?

Allstars!

Sunday evening I had tickets to see the North Mississippi Allstars. So I strapped the 100mm on the Fujifilm and thought I’d try for some electric-stage drama. Only the security guy took one look and said “gotta coat-check professional cameras.” Professional?! Uh…

Anyhow, the Allstars were just fine, post-Allmans southern boogie, more on the countrified side with a killer washboard solo (no, really) and really a lot of very solid guitar. Plus they played Down By The Riverside (“Ain’t gonna study war no more”) which always gets me choked up. I’d go if they came near you, assuming you like electric white-boy music.

Photocrit

It turns out that at Amazon Vancouver we’ve got a little photocritique social group, where someone deals out a weekly theme and we all post pix and suggest improvements to each other. This week’s theme is “long exposure”. So as soon as I got in the taxi home (For the Vancouver-savvy: up Main from the DTES to Riley Park), with a couple of drinks under my belt and a head full of electric guitar grace, I rolled down the backseat window, set the shutter to ⅛ second and the aperture to F/4, and took pictures of pretty well everything. Three made me smile.

Main Street by night

And I think to myself… what a wonderful world.

Discogs Pain 5 Oct 2019, 9:00 pm

As I continue slowly ingesting the 900 classical LPs I inherited, I’ve developed a relationship with Discogs. It’s a good place to track your collection (here’s mine, in-progress). This is the story of my first attempt to use it to buy music, a laughable failure. It’s by way of public service, just leaving a marker, a warning others to be careful about charging into the marketplace.

There’s a lot to like about Discogs. The screens are unfussy and fast and their coverage of everything ever recorded is just awesome. The LP collection I’m working my way through has plenty of obscurities by no-hit wonders, but I’ve been through a couple hundred records now and only hit one that it didn’t already have (for your amusement: Music from The Golden Age of Silent Movies played by Gaylord Carter at the Mighty Wurlitzer Theater Organ).

Also, they’ve apparently taken a mere $2.5M in venture investment, show the world a human face, and seem to genuinely care about old recordings in a tone that’s enthusiast rather than unicorn-wannabe.

Fast Freight

Back when I was doing Songs of the Day, I wrote about Fast Freight, a standout on a generally-pretty-corny 1958 mono album by the Kingston Trio that I inherited from my Dad. It’s a lovely performance which I also use as an audio showpiece; when a technical friend looks at my audio setup and snottily wonders why I still play LPs, I use this to show what the technology can do. Which usually works pretty well

The Kingston Trio

Only Dad’s copy’s getting kind of ratty, and it occurred to me that maybe I could use Discogs to replace it. Sure enough, there were plenty on sale, so I picked one on the basis that it was advertised as having “near-mint” surfaces and was offered by a seller with a good rep.

So here’s my sad story:

  1. I jumped through the Discogs hoops, not terrible at all, and placed the order.

  2. Almost instantly, the site came back saying the status was “Invoice sent”. I didn’t see any invoice on the screen or in my Discogs account or my email in-basket. I decided to wait, on the assumption it would show up.

  3. A few days later, Discogs popped up a notice saying that payment was due and unless I did so within a day or so my reputation would suffer.

  4. I fired off a barrage of messages to Discogs support and the seller saying, in effect “WTF, I want to pay, how do I pay?” Discogs was pretty prompt getting back, said “probably best to reach out to the seller.”

  5. The seller eventually got back to me and said if I wasn’t set up on Discogs’ internal payment system, he could take PayPal. I poked around Discogs.com and couldn’t figure out how to set up to use their payment system (still can’t). He asked me to send it marked “friends and family” so “PayPal wouldn’t hold it”.

  6. I sent him the money and PayPal did indeed put it on hold, and had deducted a big service fee. So he refunded it to me. This led to a hilarious, where by “hilarious” I mean “really irritating”, sequence of attempts to pay him, by splitting the payment up, by marking smaller amounts “Friends and family”, and I forget the rest. None of it worked, PayPal was determined not to let my money get through to him. This leads me to wonder how he’d got himself on PayPal’s hold-the-payments list.

  7. The seller also said that somewhere along the way I’d accidentally left negative feedback, could I please remove it? I couldn’t find any feedback anywhere.

  8. Finally, by routing PayPal to a different email address, he got the money and let me know he’d sent the record.

  9. A reasonable amount of time later, the record showed up. It was all scratched to ratshit and there was a big cut in the middle of Fast Freight so that the needle kept skipping back.

  10. My Discogs account still has a permanent notification that payment is outstanding for this order and my reputation is in danger.

I’m Doing It Wrong

That’s the only conclusion I can draw. I need to go and research how to buy things on Discogs without getting taken for a ride. I suspect it’s possible but it should be easier.

At the Climate Strike 28 Sep 2019, 9:00 pm

I went and so did lots of others, but many couldn’t so I thought I’d try to share the scale and the feel.

Help Save Our Winters

(Mounted on a hockey stick)

I’m pretty old and cynical and still, this felt like it mattered. Even though it was good-humored and consciously funny. Partly it was just the scale — one of our big north-south thoroughfares full of streaming strikers it seemed forever, but they say the peak of the parade only lasted an hour and a half. The police estimated a hundred thousand but they are prone to undercounting based on an institutional fear of too many people in one place going the same direction. Having said that, the cops were pretty exellent, see below.

Many spectated.

Spectators at the Climate Strike

The crowd was young, people whom they say don’t vote, and anyhow many were too young to vote. But their good-humored passion could not fail to lift hearts. Well, mine anyhow.

This Pipeline Shall Not Pass

Not that I was the only greybeard. If you’d subtracted the kids it still would have been a pretty big deal.

Don’t Care? Would Be A Lot Cooler If Ya Did

I didn’t get a picture of the kid with the sign saying “I’m skipping Fortnite to be here!”

Photographers at the Climate Strike

Once we got downtown, all the balconies and windows were full of people watching.

This Is Not A Single-Use Planet

The most popular chant was “What do we want?” “Climate justice!” “When do we want it?” “Now!” My voice is hoarse today. That chant is a common variation on a very old theme but it was the first time I’ve been part of it. After a little practice, the crowd gets into a more practiced and powerful rhythm; in particular the “Now!” becomes an explosion, elating to be part of and maybe a little scary for the intended audience. I hope so anyhow.

There Is No Planet “B”

There was this skinny little guy walking along, well under 10, whose chant was “Want to play in the snow? Stop global warming?” Pretty pedestrian, but he had a surprising volume and amazing endurance, he kept going and going and going and eventually the rhythm was fascinating and voices around him stilled.

Some People Just Wanna Watch The World Burn

I was shocked that Justin Trudeau, pipeline promoter, had the gall to appear at the Montréal strike. His smarmy-robot mantra about “The environment and the economy” isn’t fooling anyone any more. The best route to a sound twenty-first century economy requires creative destruction centered on the fossil-fuel sector. There’s plenty of capital and demand and opportunity to fill in behind, but first we have to step away from petroleum extraction, big-time.

I’m Not A Sign Guy

Towards the end of the route, the crowd started piling up and I couldn’t make any forward progress. Somewhere up ahead I could hear speeches, but the point had been made and my feet were killing me, so I took the train home. Hats off to Vancouver’s cops and TransLink’s helpers who kept the throngs moving through City Center station and damn did they ever have those trains shuttling through on the double-quick. I admire calm-faced competence when I see it.

A Very Happy Young Girl

Everybody was self-congratulatory about the nonviolence and good cheer, and unlike other recent events, there are no counter-demonstrators in evidence. The contras are all behind the walls of the glass towers our chants were echoing off of, and those guys are way too busy to pay attention.

When the waters start rising and the crops start failing, the glass will start breaking.

We Aren’t Witches

A few signs here and there were really obscene, a repeated theme of, uh, let’s phrase it as inappropriate sexual relations involving a female parent. 100% of the people carrying these were women.

We’re Coming For Your Money

And of course, there were a variety of other causes looking for listeners in what smells like the progressive mainstream. Communists, vegans, co-operative housing advocates, I even saw a sign for the IWW, which I thought as a sort of historically-significant exotic fringe thing when I was a hot-headed young leftist 40 years ago. Seems the Wobblies have staying power.

I’m An Edgy Teen

The cops had been smart, blocked access to the Cambie bridge and its feeders well in advance. Things went a little sideways downtown where a lot of people were trying to commute or go shopping and unfortunately got on the wrong cross-street and ended up motionless for a couple of hours while the strikers streamed past. I hope nobody’s life was seriously damaged, but a lot of lives are going to be seriously damaged if we don’t get our climate-change act together.

The buses were blocked on Granville Street.

Halted Buses at the Climate Strike

You’ve probably heard of airplane boneyards, places in the desert where they park hundreds of disused unwanted jet planes? When sanity sets in and it becomes just too expensive for people to waste energy the way internal-combustion cars do, there’ll be auto boneyards too.

On Sharding 25 Sep 2019, 9:00 pm

If you need to handle really a lot of traffic, there’s only one way to do it: sharding. Which is to say, splitting up the incoming requests among as many hosts (or Lambda functions, or message brokers, or data streams) as you need. Once you get this working you can handle an essentially unlimited request volume. Of course, you have to make choices on how you’re going to divide up the traffic among the shards. I’ve had intense exposure to the options since I came to work at AWS.

Random spray

This is the simplest thing imaginable. For each message, you make a UUID and use it as the Partition Key (if your downstream uses that for sharding), otherwise just pick a target shard using your favorite random number generator.

There’s a lot to like about it. Your front-end can be dumb and stateless and fast. The load will get dealt out evenly among the shards. Spoiler alert: I think this is a good choice and you should do it if you can possibly can.

A common variation on this theme involves auto-scaling. For example, if you have a fleet of hosts processing messages from an SQS queue, auto-scaling the fleet size based on the queue depth is a fairly common practice. Once again, admirably simple.

“Smart” sharding

The idea is that you do extra work to avoid problems, for example some shards getting overloaded and others being idle. Another kind of problem is one of your upstream sources sending “poison pill” messages that cause the receiving shard to lock up or otherwise misbehave

Load-sensitivity is one “smart” approach. The idea is that you keep track of the load on each shard, and selectively route traffic to the lightly-loaded ones and away from the busy ones. Simplest thing is, if you have some sort of load metric, always pick the shard with the lowest value.

I’ve seen this done, and work OK. But it isn’t that easy. You have to figure out a meaningful load metric (queue depth? how much traffic received recently? CPU load?) and communicate that from each shard to the front end.

If poison pills are your worry — this is not rare at all — the smartest approach is usually shuffle sharding. See Colm MacCárthaigh’s awesome thread on the subject, which you should go read if you haven’t already. He has pretty pictures (see below for one I stole) and math too!

Shuffle sharding by Colm MacCárthaigh

Affinity

Also known as “session affinity”. Another term you sometimes hear is “sticky sessions”; for a discussion see this article over at Red Hat. The idea is that you route all the events from the same upstream source to the same shard, by using an account number or session identifier of some sort as a partition key.

Affinity is attractive: If all the clickstream clicks from the same user or state-change events from the same workflow or whatever go to the same host, you can hold the relevant state in that host’s memory, which means you can respond faster and hit your database less hard.

Ladies, enbies, gentlemen, and others: I’ve had really lousy luck with affinity in sharding. Lots of things can go wrong. The most obvious: When traffic isn’t evenly distributed among upstream sources.

inverse-square curve

One time I was working with a stream of AWS customer events, and I thought it’d be smart to deal them out to Kinesis shards by account number. That was bad — the messages per second per account rate was distributed across account numbers in a classic inverse-square curve like the one in the margin. To make this concrete, in one of the early tests I noticed that the top 10 account numbers accounted for half the traffic. Ouch. (I don’t think this was a rare situation, I think it’s probably common as dirt.)

The general lesson is that if you unfortunately send too many “hot” upstream sources to the same shard, it’s easy to overload it. The worst case of this is when you get a single “whale” customer whose traffic is too much for any single shard.

So in this situation I’m talking about, we switched to random-spray, and the whole system settled down and ran nice and smooth. Except for, processing each message required consulting a database keyed by account number and the database bills got kind of high.

So I got what I thought was a brilliant idea, hid in a corner, and wrote a “best-effort affinity” library that tried to cluster requests for each customer on as few shards as possible. It seemed to work and our database bills went down by a factor of six and I felt smart.

Since then, it’s turned into a nightmare. There’s all sorts of special-case code to deal with extra-big whales and sudden surges and other corner cases we didn’t think of. Now people roll their eyes when smoke starts coming out of the service and mutter “that affinity thing”. They’re usually too polite to say “Tim’s dumb affinity code”.

That’s not all, folks

Here’s another way things can go wrong: What happens when you have session affinity, and you have per-session state built up in some host, and then that host goes down? (Not necessarily a crash, maybe you just have to patch the software.)

Now, because you’re a good designer, you’ve been writing all your updates to a journal of some sort, so when your shard goes pear-shaped, you find another host for it and replay the journal and you’re back in business.

To accomplish this you need to:

  1. Reliably detect when a node fails. As opposed to being stuck in a long GC cycle, or waiting for a slow dependency.

  2. Find a new host to place the shard on.

  3. Find the right journal and replay it to reconstruct all the lost state.

Anyone can see this takes work. It can be done. It is being done inside a well-known AWS service I sit near. But the code isn’t simple at all, and on-call people sigh and mutter “shard migration” in exactly the same tone of voice they use for “Tim’s dumb affinity code.”

But the database!

So am I saying that storing state in shards is a no-no, that we should all go back to stateless random spray and pay the cost in time and compute to refresh our state on every damn message?

Well yeah, if you can. That fleet running my dumb affinity code? Turns out when we reduced the database bills by a factor of six we saved (blush) a laughably small amount of money.

What about latency, then? Well, if you were using a partition key for sharding, you can probably use it for retrieving state from a key/value store, like DynamoDB or Cassandra or Mongo. When things are set up well, you can expect steady-state retrieval latencies in the single-digit milliseconds. You might be able to use a caching accelerator like DynamoDB’s DAX and do a lot better.

In fact, if the sessions you might have tried to shard on have that inverse-square behavior I was talking about, where a small number of keys cover a high proportion of the traffic, that might actually help the caching work better.

But the cache is a distraction. The performance you’re going to get will depend on your record sizes and update patterns and anyhow you probabl don’t care about the mean or median as much as the P99.

Which is to say, run some tests. You might just find that you’re getting enough performance out of your database that you can random-spray across your shards, have a stateless front-end and auto-scaled back-end and sleep sound at night because your nice simple system pretty well takes care of itself.

The Cloud and Open Source 21 Sep 2019, 9:00 pm

This is eventually about the public cloud and Open Source, but — apologies in advance — takes an indirect and long-ish path.

In AWS engineering, we develop stuff and we operate stuff. I think the second is more important.

We have good hardware and software engineers, and infrastructure that feels pretty magic to me (faves: the racks and networking gear, the consensus manager underlying QLDB, and the voodoo that makes S3 go). But, like Bill Joy said, “Wherever you work, most of the smart people are somewhere else”, so I’m not gonna kid myself that we’re magically unique at programming.

But on the operations side, the picture is really unique. First of all, there are very few places in the world where you can get operational experience at this scale. Second, AWS doesn’t run on SRE culture; the same engineers who write the code live by the dashboards and alarms and metrics that try to reflect and protect the customers’ experience (not perfectly, but we make progress).

The obsessive focus on operational excellence isn’t subtle and it’s not a secret. There’s been a re:Invent presentation about how we run our ops meetings and we even open-sourced the AWS Ops Wheel.

But it’s not all meetings. We build and deploy a lot of technology with no direct connection to any feature or function or API that a customer will ever see. These are all about having the right dashboards, and being able to extract the key ratio from petabytes of logs, and predicting what might melt down before it even gets warm.

The asshole ratio

I’ve already written that at AWS, it’s lower than I experienced at other BigTech outfits. Here’s why this is relevant: There is plenty of evidence that you can be a white-hot flaming asswipe and still ship great software. But (going out on a limb) I don’t think you can be an asshole and be good at operations.

Because ops requires being humble in the face of the evidence, acknowledging fallibility, assuming that the problem is your problem even when quite likely it’s not, and always eager to investigate theories B, C, and D even when you’re pretty sure your current theory A is right-on. Since problems in complex services are almost never solved by a single individual’s efforts, you have to be good at working with people under pressure.

Those LPs

I have a hypothesis about that good ratio and it involves the Amazon Leadership Principles (we just say LPs). I’ve gotten flack from friends who think having such things is lame and corny. But in practice they turn out to be useful, and to explain how I’m going to take side-trip into modern clinical medicine.

There’s this guy Atul Gawande, a surgeon and writer whose work I’ve admired, mostly in The New Yorker, for years. I recommend pretty well anything he writes but in particular I recommend The Checklist Manifesto. Do me (and yourself) a favor, follow that link and read the Malcolm Gladwell review excerpt. From which:

“…the routine tasks of surgeons have now become so incredibly complicated that mistakes of one kind or another are virtually inevitable: it’s just too easy for an otherwise competent doctor to miss a step, or forget to ask a key question or, in the stress and pressure of the moment, to fail to plan properly for every eventuality.“ [Sounds just like updating a million-TPS Web Service. -Tim] “Gawande then visits with pilots and the people who build skyscrapers and comes back with a solution. Experts need checklists–literally–written guides that walk them through the key steps in any complex procedure.”

Well, one insanely-complex routine task that we do all the time is hiring. You know what the LPs are at hiring time? A checklist. Now even the typical all-day interview marathon isn’t gonna reliably dig into every LP, but we do an acceptable job of taking a close look at enough of them. I believe that’s very helpful in bringing down the asshole ratio.

Open Source

Which brings me to the touchy subject of the relationship between Cloud Providers and Open Source. We and our competitors have made a good business of infrastructure operations, keeping service-oriented software servicing; reliably, durably, 24/7/365. The core EC2 business is about operating Linux boxes and IP networking at extreme scale, efficiently enough that we can rent them out at an attractive price and still make a buck.

In recent Open-Source years, some very gifted people have created wonderful pieces of software — Kafka, ElasticSearch, Mongo — and taken a new course, launching VC-financed companies to monetize with service and support. Then sometimes they find themselves competing with multiple public-cloud providers.

I have a load of sympathy for the virtuoso engineers who created these wonderful pieces of work. But here’s the thing: I have at least as much for the customers who (let’s take Kafka for an example) just need reliable high-performance streaming. A direct quote: “I’ll cheerfully pay monthly to never worry about Zookeeper again.”

On the other hand, I have little sympathy with modern VC-driven business models.

It’s like this: The qualities that make people great at carving high-value software out of nothingness aren’t necessarily the ones that make them good at operations. This has two unfortunate effects: They don’t necessarily have the right skills to build and run a crack operations team, and they might not manage to get a job at an operations-obsessed company.

I have recent personal experience with failing to hire a senior committer to a well-known OSS project, and also with paying an “open-source company” for tech support when we were spinning up a service around a package we didn’t know very well. Both of these left me unhappy.

Jack and Jonathan

Let me tell you a story. Sometime around 2008, I and Jonathan Schwartz, then the CEO of Sun Microsystems, made a sales call on Jack Dorsey at Twitter. Sun had acquired MySQL and Twitter was using the hell out of it. We wanted them to start paying us for support; after all, they were existentially dependent on this technology and everyone knew that serious Enterprises would never use unsupported software.

Jack was nice, and listened to our pitch, but we didn’t get the business.

And while, as a career software guy, I entirely love open-source culture and technologies and methods, the hypothesis that Open Source in and of itself constitutes a business model is not well supported by the evidence.

Which way forward?

Google Cloud’s recent Open Source partnerships are interesting. I look at that list of companies and it’s not obvious to me that they’re going to offer better operational excellence than Google’s, but maybe I’m wrong. It’s an interesting and probably useful experiment.

At the end of the day I’m not that worried. Most of us who’ve open-sourced stuff love the creative process for its own sake; touching and improving other engineers’ lives. The skillset evidenced by having done so will probably help you get really good jobs. Yeah, you might not get to be a Bay Area Unicorn. But you probably weren’t going to anyhow.

CL XXXIX: Island Wildlife 8 Sep 2019, 9:00 pm

Our probably-last Cottage-Life weekend of 2019 featured cetacean encounters and rodent rage. But I didn’t manage to photograph any of that, so just the usual trees and sunsets.

Like this:

Howe Sound sunset

Killers!

This morning I was alone at the breakfast table, considering the sea as one does, when I saw them and was yelling “whales!” at the family. There were five killer whales, one an adorable juvenile who put a little hop into every surface-to-breathe. They weren’t in a hurry, stayed for a while.

“Orcas” is more common but I say “killer whales”. Scientists prefer that too, although “orca” has etymological standing, per Wikipedia. A whale specialist told me “orca” was popularized by the marketing group at SeaWorld; they didn’t like having “killer” in their big stars’ names; family values, y’know.

Keats Island forest

Anyhow, our local killer whale population is in trouble, and that trouble is about to get worse since the Government of Canada has in its wisdom decided to bless the tar-sands pipeline that will run a tanker more or less every day through their home waters. So a random visit is a precious gift, one that might never be repeated.

Here’s today’s only picture with an animal: A feral kid scrambling up a alien-flavored tree construct.

Kid climbing tree and stump on Keats Island

Rodent rage

What happened was, I came up the ramp to the front of the cabin and there, where we have the barbecue and the recycling, was a big chunky raccoon with a beautiful silky coat, bursting with health and vigor, looking for leavings. I thought “Let’s put on a show here and disincent repeat visits” so I yelled “Hey, get outta here” and charged him. He skedaddled to the corner of the cabin and turned left, but had traction problems on the deck so I rounded the corner right behind him. He opened a lead on the dirt trail alongside the cabin and turned left again. I chased him down the path at the cabin’s back but it wasn’t close; he turned left yet again but was out of sight by the time I got to that third turn.

I thought I’d made my point and sauntered back up to the front door — and there was the raccoon again, finishing his foraging. I kind of lost it, shrieking “Gimme a **** break you **** sleazy ****!” and sprinting like a teenager.

Genuine rage makes all the difference. He headed straight sideways into the woods, never looking back. I thought he was running before, but this time he turned on his warp drive, like when they say “Engage!” on Star Trek.

He probably skulked back when we sailed for home this afternoon but we keep the place battened down in absentia. I hope we’ll be less likely to see him on future occasions when we’re in residence.

Portraits of Puppets 21 Aug 2019, 9:00 pm

If you happened to check out my Twitter feed on the weekend, you’ll know that I attended a pair of dueling rallies outside a train station in central Vancouver. On one side, a crowd in black supporting the Hong Kong protests; on the other a red-clad flag-festooned squad bringing Beijing’s message. I was dressed in black and took pictures of the other side.

Pro-Beijing demonstrators in expensive cars

The issue

It’s a no-brainer. Hong Kong isn’t perfect but it’s a civilization, with laws and with access to the world. China is a big hulking cut-off-from-the-world prison for the mind, built on systemic brutality and corruption. I admire the Hong Kongers’ courage and fear for their future. I can’t protect them from the PRC but at least I can show where I stand, and who knows, it might even make a difference if enough other people do too and the Beijing bastards decide that crushing HK might be bad for business.

People in red

I didn’t take pictures of the pro-HK side because you can bet the other side wouldn’t hesitate to use such things against them. It was probably superfluous since the Beijingers were loaded with cameras.

Pro-Beijing demonstrators

Now you’ve seen all the signs they had. It was all very uniform and organized on the Beijing side, everyone was waving the same thing. On the HK side there was an explosion of hand-lettered signs among a scattering of HK and Canadian flags. In the picture above, I particularly liked the worried-looking dude looking left through glasses, and got a nice picture of him when the sun came out.

Pro-Beijing demonstrator

He didn’t seem to be having much fun, but that’s probably a little misleading because there were definitely people on that side who were into it.

Pro-Beijing demonstratorPro-Beijing demonstratorPro-Beijing demonstrator

These two dudes were definitely full of that old school spirit, mind you one of them had his little camera rolling non-stop.

And you have to ask who these people were? I suspect they fell into three baskets. First, committed pro-Party people, maybe from the Consulate, maybe with less official standing, genuinely on the tyrants’ side — the rewards are good. Second, Chinese folk here in Vancouver who’ve stayed inside the Party-line bubble, there are media offerings to help. Third, people who don’t like the Party or (more likely) don’t like politics, who’ve had effortless-but-irresistable family or professional pressure applied.

Let’s just call them all puppets, because that’s how the people pulling the strings think about them. Here’s the puppeteers’ infrastructure:

Pro-Beijing demonstrators

Through the crowd, you can see the table where puppets can get their placards and posters and flags. I’d just love to know who organized that table and paid for the printing.

The shouting contest

That’s what the demonstrations turned out to be. The size of the red and black crowds was roughly equal — maybe a few more on the black side? — and the police did a good job of keeping space between them; it helped that nobody I saw apparently wanted to start a fight.

Disclosure: I thought how satisfying a sudden charge across the open space at the puppets would have been, but fortunately I’m grown-up enough to keep my fantasy life where it belongs.

In terms of faces and if you ignored the colors, a lot of the people on either side could have been transplanted to the other without anyone noticing. But the black side was a little older and more grizzled and a whole lot more spontaneous and cracked better jokes and the signs were better and by the way were on the side of freedom.

Coda, with hot cars

I was kind of in the middle of the black demo and noticed that every few minutes, there’d be a roar of approval from the puppet side, countered by a thunder of booing from ours. By watching where people were looking, I traced the source to the road going by. What was happening was that a few bright Beijing sparks were driving their expensive sports cars round and round the block waving PRC flags.

Pro-Beijing demonstrators in expensive carsPro-Beijing demonstrators in a Ferrari

Who’s the white dude driving the Ferrari, I want to know.

Which I think kind of underlines the key point. Like Orwell said, the object of power is power. A chief pleasure of power is showing it off, and driving around in Lambos and McLarens and Ferraris is a pretty satisfying way to do that. Particularly when you can soak up applause from the plebeians on your side and jeers from your enemies.

It’s pretty simple

The people of Hong Kong don’t want to be censored, tortured, imprisoned, and killed by those whose asshole kids are driving supercars around West Coast cities across the Pacific. I’m with them.

Talking Hong Kong Blues 18 Aug 2019, 9:00 pm

I’m imagining a discussion that might have taken place in Baidaihe at some point this month at the annual CCP summer offsite.

“Getting ugly in Hong Kong, and I’m not sure our Ms Lam is moving things in the right direction.”

“I hear from the people on the spot that what the good people want is just peace and quiet, this is just a bunch of teenage assholes making trouble.”

“Nobody wants to give their boss bad news. Haven’t you watched the BBC coverage? Maybe you’re hearing good things from your staff, but let’s suppose the gweilo TV is right? What are we going to do?”

“We’re doing one thing that’s working, going after the troublemakers’ bosses. We took down Hogg at Cathay Pacific. That’ll make every ambitious manager in HK go on the warpath to keep their employees in the office and off the damn streets. Hong Kong, it’s about three things: Money, money, and money.”

“Except for, the bad guys are getting 20% of the population out in the streets. That’ll include people who work for every fucking bank and real-estate developer and shipping company, are we gonna get every CEO in South China fired?”

“But the police say there were only 128,000 people out!”

“The HK police are idiots and in case you hadn’t notice, they’re losing in the streets.”

“I think they’re winning. There haven’t been any arrests or violence at the last three days of protests.”

“You think that’s good?! If the word starts going around that you can get away with large-scale activism as long as you keep it peaceful… do you like the idea of four million people out on the streets of Shanghai? Or a couple of million in Guangzhou?”

“What do you mean about the word getting around? The people of China are well-protected from dangerous foreign ideas, they’re not going to watching those shitty BBC liars.”

“Don’t you look at tourism figures? Fifty one million people from our side visited Hong Kong last year. They’ll all be talking to their friends and relations.”

“Yeah, well that’s maybe ten million people, a lot of them visiting every week on business. And, let’s be honest, they’re the same ones who travel overseas and already have lots of exposure to fake news from people who hate the Party. They probably all have VPNs already.”

“On top of which, those people are making good money and they owe it to us and they know it. They’ll bloody well watch what they say.”

“You guys, this is the same kind of thinking that got our 1989 leadership into trouble, letting those ‘innocent’ students stay in Tienanmen until they thought they owned it and we had to go in with the tanks and machine guns!”

“He’s right. We have the muscle all built up in Shenzen, they can be holding down Central and Tsim Sha Tsui in 72 hours and there’ll be no more of those fucking umbrellas. On top of which, the good people there will throw flowers at our guys and go back to making money in peace and quiet.”

“Suppose they don’t. Suppose there are a quarter million assholes dressed in black yelling ‘Gaa Yau!’ at each other and ‘Two Systems!’ at us and, flashing lasers and the real fringe throwing molotovs, and all with masks so we can’t ID many, and fading away into the MTR, and then another quarter million out the next day?”

“Brother, if it really comes down to them versus us, it’ll be us. Just like in 1989. It’s not just riot-control equipment waiting there in Shenzen. And any solo hero standing in front of a PLA tank this time is going to be ashes before he gets on CNN.”

“Screw CNN. It’ll be live on YouTube and Instagram and Twitter with a couple of billion people watching, and highlights of PLA tanks squishing Hong Kong patriots waiting for people who were asleep at the time.”

“So what? The people who matter need to do business with us, what do they care what kids watch on Instagram? Are they going to walk away from the chance?”

“Well, Google did.

Here’s another thing. Suppose they’re holding out in Mong Kok and every other skeezy neighborhood away from Central and there are people in all those buildings throwing shit at us from the 3rd through 20th floors, and they turn trucks sideways in those awful little streets, how are we going to get them out?”

“The PLA is not going to be stopped by a bunch of acne-faced cockroaches! Whose side are you on?”

“Western politics is weird. They eventually turned their backs on people from their own tribe in Rhodesia and South Africa in favor of a bunch of black people!

“We have our people getting our side of the story out in every Western capital; the right kind of students marching, shouting down the local HK troublemakers.”

“Give me a break, those clueless princelings haven’t the vaguest what they’re living among. I see their latest brilliant idea is to drive around in their Lambos and McLarens waving Chinese flags. Are you really really sure you want to make that bet?”

“Look, our economy is less about imports and exports every year. If the world doesn’t want us any more, then we don’t need them! We’ll just turn our backs and China will be China for Chinese, and it’ll be great.”

“Yeah, well I don’t want to miss aprés-ski in Zermatt or my place with that view in West Vancouver.”

“You might have to, because if those HK cockroaches prove they can tell us to fuck off and go on having a decent life and making money… you talk about bringing in muscle from Shenzen, what I worry about is people there starting to dress in black.”

“Yep, let’s just keep the PLA ready to roll, and hope it doesn’t have to.”

“Hope is not a strategy.”

Jeanneau 795 Review 6 Aug 2019, 9:00 pm

In Europe this boat is called the Merry Fisher 795 and in the New World, the NC 795. I’ve owned it for a few months and improved it a bit and taken it a few places and feel like sharing.

Why review?

At this point, regular readers are thinking WTF, boat review?!? I’ve only been on a handful, I’ve only owned one since 2012, I’m still occasionally baffled by nautical jargon, and my command of knots remains imperfect.

Here’s why. When you go shopping for a refrigerator or car or coffee-maker or TV or (especially) camera, there are loads of excellent detailed skeptical-voiced reviews you can read before you cough up the money. Boats (which cost more money than most of those things) are different. All the online reviews seem to be from dealers or magazine-writers on the comp, and are by and large paeans of praise.

Jeanneau 795

There are owners’ forums out there, but they tend to focus on specific problems and solutions. What’s especially missing is “I have one of these, here are the good parts and bad parts.”

I’m not completely unqualified. It’s been at my dock for a few months, I’ve installed improvements, I’ve piloted twenty-plus hours on it, motored through extreme beauty and nasty scary rough water, taken guests on pleasure cruises and a grouchy family on a tired commute, and used it as an office for a few afternoons.

So I’ll see if I can beg a few links from other 795 owners on the forums and get this a bit of GoogleJuice with the aim of better equipping other boat shoppers like me.

Facts

Jeanneau has been making boats since 1957. That link is to French Wikipedia; the English version is mostly empty and I should fix it up. The interesting, complicated story is nicely told in English by Malcolm Perrins on a Jeanneau-owners community site. The company, since its founding by Henry Jeanneau, has been sold multiple times to US and French companies and is now owned by Beneteau.

The dealer told me that the Jeanneau powerboats are built in Poland — this made him happy because for some obscure reason it leads to favorable import-tax treatment. The Jeanneau America site says “Built in America” and the first version of this piece doubted that, but a reader from Michigan wrote “We have the NC 895. It is built in Cadillac Michigan. They took the old Fourwinns plant.”

Our boat’s curtains are labeled “Made in France” and the appliances such as chargers and thrusters and fridges are Eurobuilt and their manuals have Italian or French as the first language, with English further back in the book. So I’m inclined to believe the France/Poland story in this case.

People who are buying a boat care a lot about dimensions because one of the hardest parts is finding a place that’s big enough and deep enough to park it. The 795 is 7.34m or or 24’4" long, and 2.99m or 9’9" wide, with a hull depth of a mere 0.56m or 1’10" — that’s with the outboard hoisted, which is how you normally park it.

The 795 comes with a Yamaha outboard, either 150 or 200hp, and lots of options. It’s got a modest-sized berth in the bow, a tiny but functional head (as in bathroom), and similarly tiny stove and fridge. What electronics you get apparently depends on the dealer.

Good: Engine

We have the Yamaha F200 and since it’s an outboard, there’s more room inside the boat. I’d never really been aware of this line of motors but now when I walk around any marina I see that somewhere between a third and a half of the powerboats are wearing them. So, right in the middle of the mainstream.

It’s got a very decent little electronic control screen on the dashboard and the docs are clear and comprehensive.

We set it at 4500RPM and it pushes the boat along at a little over 40km/h, depending on wind and waves. If you open it wide up on smooth water you can get up well over fifty clicks but the experience is not relaxing, or cheap either.

Good: Comfort

Not just good, excellent. The pilothouse has room for a driver and two more people in comfort, four if they’re not chunky or need extra personal space. (Protip: The aft bench is way more comfy.) The cockpit out back has forward-facing seating for three with a cushion to lean back on, and then a couple more benches but they’re less comfy. We’ve been out for a slow cruise on a warm night to watch fireworks with seven aboard and it was just fine.

Fireworks in English Bay, photographed from a Jeanneau 795

The pilothouse is really the best feature. It has a sliding “Alaska bulkhead” which means a glass door that closes, leaving the motor and its racket outside; inside, you can have a civilized conversation without shouting.

Good: Swimming platform

It’s just big enough and has a nice practical swimming ladder. We’ve used it every time we’ve been to the cabin. I shot that fireworks picture above sitting on the platform dangling my feet in the Pacific; very relaxing.

Bad: Living quarters

While they advertise two berths, realistically there’s just not enough space for more than one couple and they’d better be intimate. What with the tiny fridge and stove, I don’t think this is the boat for a lengthy family cruise up a wild coastline.

Good: Windshield

And I mean awesome. This puppy’s front glass is the size of a small European nation and when you’re sailing home with the sun behind you in a long Canadian sunset with the mountains filling the sky in front, well, there just aren’t words for that.

Vancouver through Jeanneau 795 windshield

Coming into a Vancouver from a weekend at the cottage; about two thirds of the windshield are shown. That’s the West End at the left and the Burrard Street Bridge behind the wiper. The little grey screen on the left is the Yamaha engine readout; some timing thing prevents the Pixel 2 from photographing it properly.

The wipers’ coverage isn’t that great, leaving swathes of uncleaned glass in dirty weather, but you can see the important stuff. And it comes with a windshield-washing squirter system just like your car’s, which turns out to be brilliant when you hit big waves and they splash up and want to leave sticky salt crystals where you’re trying to look out. You load it with windshield fluid from the gas station.

It’s worth mentioning the side windows too, which open and close easily and let loads of fresh air in at cruising speed without blasting your head off, and seem completely rain-proof too.

Good: Bow thruster

This is magic. We have a nice easy tie-up along the side of a dock, not crammed into a little slip, but it’s on the left as you come in and the boat wants to be tied up with its right side to the dock, so a 180° turn in tight quarters is called for. With the thruster and a light touch, it’s reasonably straightforward. The thruster is also useful as compensation for any dumb piloting errors around the dock — of course, these never happen when I’m at the wheel.

Good: It’s hackable

In the Jeanneau owners’ community I found an active boat-improvement culture; they’re always adding something or replacing something else. By dint of extensive research from primary sources, by which I mean watching YouTube videos, I have learned how to attach things to fiberglass (Protip: Get a countersink bit for your drill) and have so far improved ours by fastening the fire extinguisher to a handy bulkhead, equipping the head with a toilet-paper rod, and installing a garbage-bag holder. Call me Ishmael.

There are a variety of surfaces suitable for equipping with electronic upgrades or just decorations. We’ve decorated a couple with family photos.

Bad: Documentation

Hailing from the technology space means that I should be restrained in criticizing other professions’ end-user documentation. The boat came with a nice Jeanneau-branded satchel full of dead trees; the quality of exposition and language is, well, mixed. Highlights are the books for the Yamaha engine and the boat itself. The low point is the Lowrance navigation electronics tome, obviously executed by manic pixies on acid. The information is more or less all there but requires deep digging and Zen calm to extract.

My favorite though is the anchor-winch system, which is written in impenetrably-nautical English. Fortunately it’s accompanied by a diagram with all the parts carefully named and numbered. Unfortunately, about half the nautical names studding the text do not appear in the picture.

To be fair, I managed to figure it out well enough to anchor us (in shallow water with nearly no wind) for firework-watching.

My niece capturing a water-color of Indian Arm

My niece Anne capturing a water-color impression of Indian Arm.

Good: Piloting

The driver’s seat is comfy, the steering and throttle are crisp and responsive, and the view forward and aft is excellent. Steering at speed is a little heavier and slower than our previous inboard-outboard, but it’s plenty good enough to dodge a floating log. I’d actually like a bigger steering wheel that’s closer to me, so there’s another boat-improvement project.

Good: Access

Getting from the cockpit around to the foredeck, and up and down the sides for washing and so on, is all dead easy. The cabin is a little off-center, leaving a walkway along one side; and both sides have intelligently-placed handholds to make things easy and safe.

Bad: Flat bottom

The draft is remarkably small and the bottom, compared to the last boat, is pretty flat. This means that when you hit big waves, for example a ferry wake that you stupidly failed to notice until you were right on top of it cruising at 40km/h, you tend to skip along from wave to wave, hitting each one with a jarring “slap” of the flat bottom. This can fling passengers about a bit in a seriously uncomfortable way. Protip: Be on the sharp lookout for incoming waves and slow the hell down.

I’m not a bossy skipper but we have imposed one rule: If you want to move around the cabin, say so and we’ll slow down while you do. This after I nearly put my niece in orbit when she was going to get her backpack and I slammed on the brakes because I thought I saw some peril out front.

Good: Home office

I’m doing WFB (work from boat) one afternoon most weeks now, and it’s just terrific. The aft passenger-side bench is reasonably ergonomic and the table’s at a sane height. I often make a cup of tea and stash a snack in the fridge. I have taken conference calls, drafted and reviewed documents, reviewed code, and once (cackling with glee) checked in code to the AWS production repository.

I haven’t convinced any colleagues to come down for an in-boat meeting yet; it’s just a matter of time. But I’m just not gonna install whiteboards.

Mixed: Online community

The biggest is the Owners’ Forum, which is OK but suffers from Jeanneau having so many products. There’s also a group on Facebook, obviously. I’ve picked up valuable tips in both places.

Bad: Missing pieces

There’s no automatic bilge pump, which I find shocking, but on the other hand I have to say it stays almost bone-dry down there, even with mixed hot & cold weather, bashing through pretty rough seas, several days of heavy rain, and regular thorough washing (the honeymoon is still on).

There’s no horn; our previous boat had one and while I only ever used it once or twice, I was glad of it.

There’s no built-in heater. Our journeys typically aren’t long enough to need one on the water, but this might be an issue in home-office mode. Multiple owners have installed diesel heaters, and I have a nice little AC space heater that I’ll try out when on shore power. Similarly, there’s no air conditioner, which is more of a problem than you might think up here at 49°30'N because the pilothouse has so much glass, it’s a greenhouse.

Jeanneau 795 tied up at Keats Island

There are only two cleats, fore and aft. When you’re tying up to a floating dock for a weekend in Howe Sound (see above), which after all is part of the Pacific, you really want one and ideally two spring lines along with the basic fore and aft. Several owners have figured out how to install an extra central cleat, and I’ll look to do that.

And your conclusion is?

Count the “Good”, “Bad”, and “Mixed” headlines above. The good stuff wins, by a wide margin. I’ve got no standing to say whether or not this is a winner or loser against the competition because I haven’t owned the competition. What I can say, a few months in, is that it meets our needs very well.

Accessorizing

Here are the things I’ve purchased to improve the experience:

  1. SeaTeak 62634 Insulated Four-Drink Binocular Rack — I have two of these things velcro’ed down behind the sink. The binoc-shaped spaces also work for big coffee mugs with handles.

  2. Dell Ultra HD 4K 24-Inch Monitor P2415Q — just the right size for outboarding to my company MBPro, and comes with USB so I only need one plug to power everything. I need to install something to hang it up on the berth bulkhead when not in use, at the moment it’s lying face-down on the mattress, which is OK but takes space.

  3. 4.5" 12V Stepless Speed Car Fan — sold by different vendors in the US & Canada. Like I said, it can get toasty in the pilothouse but this guy takes care of it just by keeping the air moving.

  4. Rod Holder Mount Boat Flagpole — the 795 has two fishing-rod holders but no flagpole. Hey-presto! The Canadian flag looks great out there but we haven’t figured out which minor ensign to fly beneath it. Patti Smith fan club? Antifa emblems? Not sure.

  5. From Davis Instruments, Shockles LineSnubbers and LineGrabbers; nothing specific to this boat, just a coincidence that I discovered them recently. If you tie up where it might get rough, you need these.

Summary

My relationship with the previous boat was pretty prosaic. It got us back and forth to the cabin and was kind of charming with its wood trim, but it always needed fixing and there were important subsystems I never learned to understand. This is a whole different kettle of fish. I’m starting to develop sympathy with the oft-repeated Kenneth Grahame quote from The Wind in the Willows:

Believe me, my young friend, there is nothing — absolutely nothing — half so much worth doing as simply messing about in boats. Simply messing… about in boats — or with boats. In or out of ’em, it doesn’t matter. Nothing seems really to matter, that’s the charm of it. Whether you get away, or whether you don’t; whether you arrive at your destination or whether you reach somewhere else, or whether you never get anywhere at all, you’re always busy, and you never do anything in particular; and when you’ve done it there’s always something else to do, and you can do it if you like, but you’d much better not.

Google Pixel Photo Workflow 27 Jul 2019, 9:00 pm

I recently wrote about the excessive difficulty of moving the Google Pixel phone’s excellent pictures through a Lightroom workflow. It turns out that Lightroom has a solution; herewith details, upsides, downsides, and alternatives. Also, cautionary words for Adobe on Lightroom Classic

To start with, Adobe has a page telling you how to Sync Lightroom Classic with Lightroom ecosystem.

Sidebar: “Lightroom Classic”

If you’re not sure what “Lightroom Classic” means, they also have a page whose URL suggests it’s about Lightroom Classic vs Lightroom CC — but the “CC” designation has apparently been lost, so the cloud-centric version is just Lightroom. For those of us with cameras that aren’t phones and produce huge raw files, Classic is the place to be.

Now, anyone who’s software-savvy has to be nervous about using a product with “Classic” in its name because that usually means “we don’t care about this, won’t invest in it, and will probably discontinue it.” For the moment I’m not going to worry because I suspect the Lightroom customer based is overweighted with people carrying serious cameras that really need the disk-based workflow, and Adobe just can’t afford to blow us off.

Having said that, I owe thanks to someone with an @adobe.com address who wrote me an email beginning “You are working too hard.” and outlined the How-To. But later in our exchange, they said “I do recommend the CC version, I believe that's where most of the energy is being focused.” That makes me nervous. Hey Adobe, you got a huge percentage of the world’s serious photogs to sign up for a monthly subscription; you had better treat us and our big cameras and our monster DNG files nice.

How-To: Details

That how-to-sync page is accurate as far as it goes, but I got stuck for the longest time because it says “After signing in, click your user name that now appears at the upper-left corner and ensure that the Sync With Lightroom option is turned on.” Only my Lightroom screen doesn’t have my name on it. That’s because (like many other, I bet) I run in full-screen mode. So drop out of full-screen; or just push your mouse up to the top left corner and your name will appear. Hey Adobe, why in the freaking hell is that preference hidden there instead of placed under the “Lightroom Sync” tab in the, you know, Preferences? But I digress.

The other important thing they don’t tell you is that after you’ve taken the photos, you need to wake up the Lightroom app on your Pixel and it’ll auto-magically notice the new pix and sync ’em. I’m OK with this because it lets me control when the sync happens, normally when I’m in the warm glow of home WiFi.

I used to use the Lightroom camera app (which presumably does this itself) because it had better ergonomics than Google’s, but then Google’s got the computational-photography magic where it shoots 50 times a second and combines them to produce unnaturally great pictures.

Deletion

Deleting these synced photos gets a little weird. If you do it when you’re in the “All Synced Photos” folder, you get a message about how they’re going to be deleted from Lightroom but retained in the Catalog. Near as I can tell, that’s just wrong, they vanish from your phone and lightroom.adobe.com and your desktop Lightroom. If you’ve moved them into a regular working directory you just get the normal Lightroom “Deleted Selected Master Photo” dialog, but it still takes care of cleaning up the online and on-phone versions.

Alternatives

If you go check out the comments on my last piece, there are a bunch of interesting-sounding suggestions of other ways to move stuff in general and pictures in particular between your phone and your computer. I’m not going to check them out because the Lightroom process described here works for me. If you’re interested, I’d pay particular attention to one of the Sync apps from MetaCtrl because they’re by Trun Duc Tran, one of the best developers I ever worked with.

Downsides

This whole investigation got started because, as Stephen Shankland noted, when you do this auto-syncing you no longer go through Lightroom’s “Import” process, which allows you to rename, add metadata, apply develop presets, and so on. Not an issue for me but it might be for you.

Google Camera RAW vs JPG 27 Jul 2019, 9:00 pm

I recently wrote about how to move the excellent photos from the Google Pixel phone Camera app into a desktop Lightroom workflow. I was pleased that it’s easy to tell the camera to generate DNG “RAW” files and include them in the process. But apparently, the camera’s JPGs are better and more useful than the DNGs. That’s weird.

Here’s a pair of pictures to illustrate. This morning, the cat found a sunny corner of the back porch and was squirming around out of pure joy.

Cat on the back porch (Google Pixel DNG)Cat on the back porch (Google Pixel JPG)

The DNG is above, the JPG below. Of course they’re both JPGs now in the blog, but both are straight out of the camera, resized and JPGized by Lightroom with no sharpening or anything. It’s not that dramatic here, but flipping back and forth in Lightroom, the difference isn’t subtle. The JPG has had some lens correction, the blown-out highlights have been recovered, there’s been a bit of sharpening (look at the cat’s belly hair and the broom bristles), and the color’s been tweaked — that watering-can is dead-on in the JPG but has extra yellow in the DNG.

In this CNET piece, Marc Levoy, who invented the term “computational photography”, says “The JPEGs from the Pixel camera may actually be more detailed than the DNGs in some cases” and yeah, no kidding. In fact — and this puzzles me — the JPG is 4032x3024 in pixels while the DNG is 4016x3008, which is to say it’s 112K bigger. But I don’t think that’s what Levoy meant.

Also: “Our philosophy with raw is that there should be zero compromise,” Levoy said. “We run Super Res Zoom and HDR+ on these raw files. There is an incredible amount of dynamic range.” That doesn’t match my experience, but then he was talking about the Pixel 3 and I still have a 2. Also, Stephen Shankland said: “I rather like Google’s computational DNGs from Pixel 3. They HDR-ize the raw input to create the DNG. It’s not perfect but I find it darned useful. (I also generally like the Pixel camera app’s JPEGs, though they can look overprocessed to my eye.)” So I look forward to giving Pixel 3 (or 4) DNGs a try.

Why do you want DNG anyhow?

Photographers like RAW versions of photos because they’re more editable. One of the most common editing modes is rescuing lost data from highlights or dark areas that look blown-out or dimmed-out in the original — you hear people saying that a good camera RAW is “deep”, and that’s certainly true of the files from the Fujifilm X-cameras.

Consider these three pictures. Once again, the first is the DNG, the second the JPG, and in the third, I decided to see if I could recover image in the dark area behind the hydrangea blossoms. (Not a thing I’d normally do on this shot, I like the dark framing.)

Hydrangea (Google Pixel DNG)Hydrangea (Google Pixel JPG)Hydrangea (Google Pixel JPG, edited)

I think the results speak for themselves. There’s a lot of useful data in this JPG.

Which raises one more question: By aggressively digging in with Lightroom, could I replicate what the Google camera software did, or maybe even improve on it? So I tried that, and there was progress but at no point did I think I was really replicating that tasty JPG, and I got bored trying.

So for now I think I’m going to turn off the DNG capture on the camera app. Sshhh, don’t tell any Real Photographers.

Google Photos Breakage 23 Jul 2019, 9:00 pm

The camera systems in the Google Pixel phones are excellent, but Google makes it way too hard for the serious photographer to get a decent workflow going. Does someone out there know a better way? If not, let’s get together and yell at Google. [Update: I found a good way to do this.]

Vancouver’s old Post Office under renovation

Pixel.

It’s like this: Pictures you take with your Pixel migrate into Google cloud and may be found at photos.google.com. You don’t have to do anything to arrange this, it just happens. There are slideshow and sharing tools and they’re very decent. It’s a great service, I might even pay for it if Google asked.

But I — like most serious photographers — want to pull the photos, in their highest-resolution form, into a processing workflow. Mine is Lightroom-based, a popular (perhaps even majority) choice. In a real camera, when I say “highest-resolution” I mean a “RAW” format, which tend to come in proprietary flavors such as RAF for Fujifilm and CRW for Canon. It’s OK because Lightroom can handle all of them, and then there’s sort of a standard, developed by Adobe, called “DNG”. A few enlightened camera builders like Pentax generate DNG natively and hey, it turns out the Pixel can too. You go into the camera app Settings’ “Advanced” tab and enable a RAW control on the camera screen.

I’ve just started doing this and at the moment, my feeling is that the Googlecam software does a damn fine job of JPG generation and I’m not sure the DNG will help me that much. But the central problem remains the same: How do I get these files, be they DNG or JPG, into Lightroom?

Wet poppy

Pixel.

AFT

That stands for Android File Transfer, a tired old (32-bit, so my Mac whines at me) program that gives you a Finder-like interface to the Android Linux filesystem once you’ve USB-connected it to your Mac, pulled down the Android notifications, and told it to allow AFT. The JPG files are in /DCIM/Camera and the DNGs are in /Pictures/Raw. I guess what they want you to do is sort the files by date, remember where you last left off with the copying, and then select-and-drag to a staging folder in the nearest Finder window, from which you can do Lightroom input. After which you can take out your stone axe and join in the tribe’s mammoth hunt.

BTW, the Pixel will connect to the Mac in PTP mode and Lightroom can even see it, but then can’t see the pictures.

Dropbox

Using the Android file-sharing dialog, you can pick the pictures from the Photos app and send ’em to DropBox, and then after they’ve traveled from the phone on your armchair off to Dropbox’s home on the Internet and back down to your Dropbox Mac folder — this is not fast — you can go get ’em with Lightroom.

Color study

Pixel.

Dropbox is what I actually do. It’s klunky and slow, but gets the job done and has never to date actually failed to work. But we hear troubling things about Dropbox’s direction.

Google Cloud

You can get a Backup and Sync app from Google that should make some of your Google cloud files sync to your Mac. I’ve had just no end of pain with this one. You have to go mousing around in the preferences for Docs or Drive or something, a real maze of twisty little passages there, and tell it to sync photos as well as other stuff, and then last time I tried it, the process had busted the EXIF so the pictures didn’t know what time they’d been taken. Feaugh.

The other night, I was chatting with Stephen Shankland on Twitter and he seemed to have found a path through the Gcloud maze; hey Stephen, wanna blog that?

Plan D?

Anyone reading this know a better way?

EventBridge 11 Jul 2019, 9:00 pm

The launch of Amazon EventBridge, a somewhat but not entirely new thing, has been well-covered by Jeff Barr; if you want to know what it is, go read Jeff. This piece is to provide a bit of background and context on EventBridge. I didn’t actually make any direct contributions, but was upstream from this work at the definition and early-planning stage.

CloudWatch what?

My first work at AWS was on the project that launched in January 2016 as CloudWatch Events. To us it felt like a small, simple, service — write rules to route notifications of things happening out there in AWS to Lambdas or other useful destinations. It wasn’t a big team or a big task and, when it came time to name it and find it a home, it was hard to believe it deserved top-level service billing.

EventBridge stickers

Coming soon to a laptop near you. Warning: Prepared by engineers without involvement by marketing professionals or lawyers; let’s hope they’re OK.

Since CloudWatch already offered alarming and logging, eventing seemed like a nice third leg of the tripod, so our work launched as a tab on the CloudWatch page, and we thought that was OK.

Customers apparently liked it, and over the years, CloudWatch Events accumulated a mind-boggling number of users and a lot of the things they were doing weren’t really CloudWatch-y at all. Also, the whole Event-Driven Architectures drumbeat kept growing louder and louder out there in the community.

Last year, we got the idea of helping third parties (mostly SaaS vendors) integrate with their customers on AWS, and quickly became convinced that eventing was the right way to do this — while I’m a fan of the Webhook concept, the reality has not been a smooth ride. Once we’d made that call, enhancing the CloudWatch Events APIs to meet partner needs was pretty straightforward once we’d thought through the security dimensions. Except for, this was getting waaaay outside CloudWatch territory.

Which Bridge?

So, we decided that this service deserved top-level billing and went looking for a new name. The best possible answer would be “AWS Events”, right? Wrong. Go look at aws.amazon.com/events and hey, re:Invent, re:Inforce, AWS Summits… you get the picture. Thus EventBridge, which isn’t terrible.

(By the way, all your CloudWatch Events stuff still works and none of the existing API names or semantics have changed.)

The Event Ecosystem

It’s getting pretty big. Inside AWS, Lambda and SNS are event-centric. If you check out our competitors, you’ll notice more services with “Event” in their names every year. The numbers of events flowing through our various accumulated event streams has a lot of digits.

I’m personally pretty convinced that, while you can hook everything together with APIs, there are a whole lot of scenarios where choosing events buys you so much robustness and flexibility that it’s really hard not to. Is it perfect? Of course not: There are lots of places where the API ecosystem is slicker.

If you want to a really good explanation of why event-driven stuff might be in your future, the AWS NYC Summit talk by Mike Deck has what you need. As I write this the day of the summit, it doesn’t seem to be online but I’ll refresh once it gets there; and I bet Mike will reprise at future AWS, uh, events.

There’ll be lots more chapters in this story.

Wealth Tax Stupidity 9 Jul 2019, 9:00 pm

Canada’s mainstream conservative biz paper The Financial Post recently published The NDP’s new tax-the-rich plan is terrible, even by their standards and it is stuffed with white-hot stupidity and bad arithmetic. Arguing against any given tax is sane — that’s what conservatives are for, innit? — but if they’re going to use math that would get you an “F” in Grade 8, they deserve a whack with the cluestick.

Here’s a Post out-take:

The proposal is for an annual one per cent tax on wealth over $20 million. This means that if an Ontario resident to whom this tax applies invests $100,000 in a 1.5 per cent GIC for one year (about the rate currently offered by big banks), he or she will earn a $1,500 nominal return on which will be owed $1,833 in tax — $833 in income tax, including the higher top marginal income tax rate, plus another $1,000 for the wealth tax.

The effective tax rate on the GIC return is 122 per cent, which is what the NDP now calls “tax fairness.”

Let’s enumerate some facts:

  1. Canada has a reasonably-typical developed-world tax regime; in the top income bracket, you pay a little over 50% in tax.

  2. Wealthy people do not put their money in 1.5% GICs, they pay investment professionals to manage it. Typically, the wealthier they are, the better returns they can get because they can hire better money managers. (There’s interesting data on this in Piketty.) Let’s say they’re going to realize 4% or so, because in fact most wealthy people do better than that. I’m a one-percenter but nowhere near the hypothetical twenty-millioner in the narrative, and I do better.

  3. A little arithmetic reveals that Mr $20M will pull in about $800K in investment income, $400K net after tax. Actually, he’ll do better because some investment income will be capital gains, taxed at half the rate, and he’ll probably do better than 4% too. But then there’s the 1% wealth tax, so subtract that for a worst-case net income of $200K if he refuses to spend any of his capital.

    Now, people with that kind of money usually have other wealth-generating activities (how else did they get there?), which is on top of the $200K. Until they retire. The conventional wisdom says that a retiree can extract 4-5% of their capital per annum, which for this person would be pushing a million, except for there are lots of tax dodges open to retirees, so while they’d still be stuck with the $200K wealth tax, they could keep more of that million.

So, the things that were wrong about that $100K-deposit story:

  1. It’s completely bogus to combine the wealth tax and the income tax and say it’s “on the GIC return”. I’d say “Data fail” but that’s too kind; the 122% figure is a filthy, stinking lie.

  2. And anyhow, the whole scenario is science fiction, the “$20 million” part just doesn’t go with the “$100K GIC” part. If a conservative advocating lower taxes has to resort to this kind of specious bullshit, you have to wonder if it’s because they can’t find better arguments.

Take-away

I think a small wealth tax, maybe starting in the low-double-digit millions, is a fine idea. I think that people subject to it will in practice remain extremely affluent. Their fortune would erode slowly over the years — having trouble seeing what’s wrong with that — but there’ll be plenty left to give their kids’ careers a rocket boost.

It’s perfectly reasonable to disagree with me. But if you’re going to resort to numerical flummery I’m going to think just you’re a greedy asshole with no intellectual legs to stand on.

Page processed in 0.71 seconds.

Powered by SimplePie 1.5, Build 20180814171753. Run the SimplePie Compatibility Test. SimplePie is © 2004–2019, Ryan Parman and Geoffrey Sneddon, and licensed under the BSD License.