A Staff Engineer's Guide to Inventing Work

Platform teams are engineering-led rather than product-led. There is almost never a product manager handing you a roadmap, no revenue line to follow, and no market to lose. This means that work does not exist unless an engineer invents it. The continuous struggle is to find ways to increase the value our platform provides to the users of the system. A big part of a staff engineer’s job in a platform team is to figure out what the team should build next - I like to call that “inventing work”. Thankfully, the signals that help us to invent work are already out there, and they arrive from four directions - from the systems, from the users, from your organization, and from the industry. What follows is a guide to reading each of them.

Defining a Platform is Hard

Engineering platforms are a vague concept. Software organisations across the board agree on the need to ‘platformise’ layers of their stack, but struggle to define the term. The question ‘what is a platform’ is met with a response ‘something similar to AWS, but at a higher layer of the company software stack’. I have previously argued why this is a false analogy.

I think we can agree that there is a dichotomy in engineering platforms: public platforms and internal (or private) platforms. AWS S3, Snowflake, and others are examples of public platforms, while internal platforms are engineering platforms built within a software organisation to serve internal users.

My approach here is to start with a reasonable definition of platforms in general and arrive at a reasonable definition for internal platforms. This was not as straightforward as it sounds. The minimum we can gain from such an exercise is the ability to identify internal platforms in the wild (“Is X an internal platform?” ). ¹

So, what is a platform?

AWS Is NOT Your Ideal

Let me start with an assertion. Every platform engineering team ¹ in every organisation aspires to be like AWS ².

Every platform team wants to be like AWS, because like AWS, they provide infrastructure abstractions to users. AWS provides infrastructure via the abstractions of VMs and disks and write-capacity-units, while platform teams provide infrastructure using higher abstractions which solve service definitions, database or message queue provisioning, and service right-sizing ³.

This similarity prompts leaders of platform engineering teams to model their teams as agnostic providers of universal, non-leaky (within SLO bounds), self-served abstractions for their engineering organisation. Platform teams structured as such detached units struggle to define cohesive roadmaps which provide increasing value to business. But how does your platform differ from AWS?

Converse Conway's Law

Melvin Conway in his 1968 paper How Do Committees Invent? postulated the now-famous Conway’s Law.

Organisations which design systems are constrained to produce designs which are copies of the communication structures of these organisations.

This homomorphism between organisational communication structures and systems designed by them, has become an adage in software management. It implies a one-way effect, though. But, does it work in the other direction?

Innovation Loops

The purpose of an engineering organization (at the risk of sounding frivolously reductionist) is to build business value. You can grow an organization’s delivered business value over time by: ¹

• training members: investing in people,
• improving process: investing in shaping behaviour and communication,
• staking technical leverage: investing in technology.

A cumulative side-effect of these approaches is to strengthen innovation loops.

Innovation Loops

Innovation loops are informal, intrapreneurial feedback loops in engineering teams which builds products & features to address user demand & pain. It is innovation which circumvents the software development cycle involving product & market research teams. In mature teams, innovation loops complement & reinforce the existing, evolutionary product development feedback cycle. I call product development evolutionary, in contrast to the more revolutionary (or reactive) trait of innovation loops.

Innovation loops are more prevalent in infrastructure teams than in product-focused teams. This could be partly explained by the availability of direct communication channels to users which infrastructure teams possess, and product-focused teams do not.

Hanlon's Razor: Some Comments

Do not attribute to malice that which can be explained by the less criminal motives of ignorance and lethargy.

An aphorism of utmost utility in my life is the Hanlon’s Razor. I find it a liberating rule of thumb to weigh a lot of unavoidably unpleasant experiences in daily life. In a less formal & more terse form that I prefer, it reads:

Stupid people abound; Malicious people, less so.

There is a neat wikipedia article on it which focuses on its origin, and also introduced me to an earlier form of the aphorism by Goethe.

Misunderstandings and lethargy perhaps produce more wrong in the world than deceit and malice do. At least the latter two are certainly rarer. Johann Wolfgang von Goethe, in The Sorrows of Young Werther

An Early Employee's Field Guide to Workplace Arguments

TL; DR Conflicts are common in an early-stage startup. This post lists a set of mental models an early employee can use to prevent, judge, diffuse and take leverage of conflicts.