Do you like LinkedIn’s endorsements feature?

I’m preparing for product management interviews. I’ll publish some of my case practice here on the blog.

The following is based on a practice question given by Lewis Lin in his book Decode and Conquer.

For this exercise, let’s assume that:
— I’m applying for a senior-level (equivalent to Google’s L6) product management role at a growth-stage startup like Snowflake.
— This is a first-round interview taking place over the phone without video.
— The interviewer is a UX designer. Random gender generator says it’s a “she”.

Interviewer: Now I’d like to give you a case and hear how you work through it.

Me: Sounds good — let’s do it.

What I’m thinking: I have two goals at the outset of any case.

First, I want to understand what type of question I’m getting. Most questions that show up in PM interviews can be classified as one of a small number of types. As soon as I know which type I’m dealing with, I’ll know a lot about how to approach it.

Second, I want to understand the context in which the question is arising. Who am I? Who are you? What’s happened to give rise to this question? If the context is undefined, the problem will be hard to grapple with. If the scenario is clear, I’ll be able to use my actual experience to ground my decisions.

As I listen to the question, this is what I’ll be thinking about.

Interviewer: This is a design critique question. It’s meant to give you a chance to show how you think about design and giving feedback. The question is: what do you think of LinkedIn’s endorsement feature?

What I’m thinking: What’s the question type? She said “design critique”, which is a term of art for designers — it’s a conversation in which a designer presents work in order to receive feedback for the purpose of improving the work. It’s also a common type of question for product managers. I’m pretty sure that’s what this is, but I’ll want to double-check.

What’s the scenario? She didn’t give much context, but knowing that it’s a design critique gives me enough to assume and confirm: rather than asking open-ended questions to get information, I’m going to invent a scenario and ask her if we can assume that that’s what’s going on. This tactic has two benefits: first, it’s fast and efficient. Second, it decreases the chance we’ll wind up in an area that’s unfamiliar to me.

Me: Great. Let me see if I understand the question. You said “design critique”, so I’m assuming that this is a feature someone on the team is working on. I’m imagining that we both work at LinkedIn — let’s say you’re a designer and I’m a PM. And you’re asking me for informal feedback on a feature that you’re working on. Is that a fair assumption?

Interviewer: Sure, let’s go with that.

What I’m thinking: Now that I know what kind of question I’m dealing with, I want to make sure I understand the feature we’re talking about. Even though I’m pretty sure I know what she means by “LinkedIn’s endorsement feature”, I’ll double check. This might turn up some useful information, and if I’ve made the wrong assumption, it could save me from a major confusion.

Me: Great. And let me check whether I understand the feature that we’re talking about. Is this the feature that lives on someone’s profile page and says things like “SEO — 18 people have endorsed Matt for this skill”?

Interviewer: Exactly — that’s the one.

What I’m thinking: Now I have all of the context I need to start answering the question. If I’m not immediately sure where to go next, this would be a good time to ask for a moment to think.

Me: Got it. And the question is: let’s do a design critique on that feature. Do you mind if I take a minute or so to gather my thoughts?

Interviewer: Absolutely — go ahead.

What I’m thinking: So how am I going to approach this? Since we’re dealing with a design problem, I’ll want to make sense of who it’s for and what their needs are. For that, I’ll use the SSUN framework. And since I’m giving feedback on an existing solution, I’ll use the Design Scorecard method to structure my critique. That’s going to be my approach: SSUN and Design Scorecard.

Me: Okay. That’s a huge feature — very central to the product. I’d like to do two things. First, since it’s such a central feature, I’d like to walk through an exercise to get a clear user and use case in mind. Then, I’d propose we make a scorecard with two or three design goals and see how it does against those goals. How does that sound?

Interviewer: Sounds good.

What I’m thinking: I’ll work through the SSUN framework starting with Stakeholders. For each section I’ll first brainstorm a number of options, and then I’ll select one. To keep things moving and to stay attuned to the interviewer, I’ll use the ‘assume and confirm’ tactic at each step.

Me: Okay. To make sense of users and needs, I like to use a framework called SSUN — it stands for Stakeholders, Segments, Use cases, and Needs.

Starting with Stakeholders, let’s brainstorm a few. We’ve got:

  • Users
  • LinkedIn people — employees on various teams, executives, board, etc
  • Other parties on the platform like advertisers

We could brainstorm more, but those seem like the big ones.

As far as which one to focus on here, we’re probably most interested in the users, so I’m going to set aside the others for now, and just focus in on the users. Does that sound good?

Interviewer: Yeah, that sounds good.

Me: Okay. Then on to Segments. Within the user stakeholder group, we can sub-divide into a few segments.

LinkedIn is a career marketplace, so the main user segments are going to be:

  • People who are trying to show off their skills, and
  • People who are trying to find people that have certain skills.

Let’s for now call them “job-seekers” and “employers”.

We could brainstorm more segments, but I think these are the main ones.

Between these, my first thought is that we should focus on the employer side. The reason is that if employers trust and use endorsements, then job-seekers have a strong reason to get and to give endorsements. But if employers are ignoring the feature, then job-seekers are probably going to ignore it too. So in that sense, employers are the linchpin.

Does that sound okay?

Interviewer: Yep, that sounds good.

Me: Great. So next is use cases.

Let’s brainstorm a couple. As an employer, I’ve personally used LinkedIn in two ways:

  • One is to search for people.
  • The other is to evaluate a candidate who’s applied.

Let’s call those “outbound” and “inbound”.

There are definitely more use cases that we could brainstorm, but those are big ones. Let’s go with those two for now.

Of those, the one that seems most important here is the one where the employer is trying to evaluate an inbound candidate.

Shall we focus on that one?

Interviewer: Why does that one seem most important?

Me: Well, I think that one gives us the cleanest view of the need we talked about a minute ago. We identified this relationship where if the employer trusts the feature and uses it, then job-seekers will too. The inbound use case will put that trust front and center. The outbound case would touch on that, but it would also bring in additional things related to the mechanics of search.

Interviewer: Makes sense. Let’s go with that.

Me: Great. Then, the last thing is needs. What goals does the user have in this situation? Let’s brainstorm.

  • The first thing that comes to mind is something like accuracy of the signal, or trust. Basically, if I’m the employer, I want to know if this candidate is going to be successful in this role. I’m looking for information I can rely on.
  • Another thing is speed. I’m looking at lots of candidates, so the faster I can get a signal, the better.

Again we could brainstorm more needs but that feels good for now.

I think the one we want to focus on here is the first one: credibility of the signal. As we said earlier, that one feels like the linchpin.

Does that sound good?

Interviewer: Yeah, that makes sense.

What I’m thinking: Now I want to package all that work up with a neat user story. That’ll help us to remember what we’re working with in the next stage.

Me: Great. So if we put all of that into a user story, we have something like, “As an employer evaluating a candidate for a role, I want credible signals about this person’s skills.”

We could obviously go down different branches of that tree to make stories for the other segments et cetera. But for now let’s just focus on that one.

What I’m thinking: Now I’ve completed the SSUN framework, so I have a clear idea of who we’re designing for and what their problem is. My next goal is to set up the context for a good, disambiguated conversation about design — one that might result in useful feedback, and that will give us lots of footholds for a well-structured discussion.

Me: With that user story in mind, let’s talk about what’s working and not working for this feature.

I’d propose we start by making a scorecard. It’s hard to talk about whether something is successful if you haven’t said what the goals are.

Sound good?

Interviewer: Sure, sounds good.

Me: So to make a scorecard, let’s agree on 2-3 design criteria, and then for each criterion we’ll give three responses:

  • A 1-5 rating (basically a Likert scale rating. This gives us an apples-to-apples comparable.)
  • One or two things that are working well.
  • And one or two things that aren’t working well.

We can pick any design goals we want, but I’ve found it useful to say that good design is useful, easy, and honest. How do those three sound?

Interviewer: Sure, sounds good.

Me: Great. If this was real life I’d suggest that we both make a scorecard and fill it out, and then discuss. But since I’m in the hot seat here I’ll just do one and talk aloud as I go.

Interviewer: Sounds good.

Me: Okay.

Is it useful. I’d give it a 2 out of 5 on this. What’s working well is that it’s super easy to use. What’s not working well is that I don’t trust the information — I think it’s too easy to game.

Next, is it easy. On this one I’d give it a 5 out of 5. What’s working well is that it’s structured and consistent — it’s really easy to pick up at a glance. If I had to stretch and name something that’s not working well, maybe I’d point out that it still requires me to make some kind of inference to figure out how much of an expert it’s saying this person is. What does it mean that 12 people endorsed him for that skill?

Last, is it honest. On this one I’d give it a 2 out of 5. This goes back to what we said before. What’s working well is that it leaves the endorsement up to real people — so it’s as honest as those people are. What’s not working as well is that it projects a level of confidence about these endorsements that I’m not sure is warranted. There’s too much incentive, for too little cost, to game the system.

So it looks like that’s a 9 out of 15. Overall I think this feature has a lot of potential — but the trust issue is the main holdup for me right now.

Interviewer: Awesome! Thanks. Now let’s move on to…

Design Scorecard: a framework for giving feedback

This is a framework to use when you’re looking at a proposed solution to a well-defined design problem, and your goal is to provide feedback so that the solution can be improved. A typical scenario would be a design critique meeting in which the designer in charge of a problem is showing recent work and asking for feedback.

Best practices

Giving effective design feedback is an art. This method will help you with three best practices:

  • Identify design goals. If you don’t have clear goals, it’s very hard to evaluate whether a design is successful. Successful at what?
  • Get clear on the goals before you begin to evaluate the solution. This will make the process feel more objective and the ensuing conversation more productive.
  • Name what is working in addition to what isn’t. On a practical level, this will help to ensure that good stuff doesn’t get forgotten in the next iteration. On an emotional level, this positive reinforcement is like wind in the sail for the people doing the hard work of improving the feature.

First, select your design criteria.

Two or three is typically a good number. Less than two, and you’ll tend to lump everything into one bucket. More than three, and the process will tend to become unwieldy.

Industrial designer Dieter Rams has a famous list of ten design principles that you might choose from. According to Rams, good design is:

  1. Innovative
  2. Useful
  3. Aesthetic
  4. Makes a product understandable
  5. Unobtrusive
  6. Honest
  7. Long-lasting
  8. Thorough down to the last detail
  9. Environmentally friendly
  10. As little design as possible.

Second, evaluate the design.

For each of your agreed-upon criteria, give three responses:

  • A numerical rating on a 1-5 Likert Scale. This forces you to quantify your feedback and gives designers an apples-to-apples comparison.
  • One or two things that are working. This gives the designer some positive reinforcement and ideas for what to build on.
  • One or two things that could be improved. Since the goal is to improve, this is the meat of the critique.

Visualized, your score card will look like this:

CriterionLikert ratingWhat’s workingWhat could be improved
Useful2Gives me an at-a-glance picture of this person’s skillsI’m not sure that I can trust the information
Easy5Fits really easily with pre-existing mental models: one person endorsing another… and many people endorsing one person.I can see how many people endorsed, but I still have to make an inference about what that means. Can we give a takeaway metric?

How to frame a product design problem: the SSUN framework

I’m preparing for product management interviews. Like all PMs I love frameworks, and a fun thing about interview prep is that I get to quickly try out lots of frameworks and even invent new ones where what’s out there isn’t working for me.

A popular framework for product design cases is Lewis Lin’s CIRCLES Method. In my experience it’s been helpful as a starting point, but for a few reasons I’ve found it hard to use.

So I’ve been tinkering. I’m attracted to the idea of more modular frameworks that can be recombined as needed for a variety of cases. SSUN has a narrower scope than CIRCLES, and I’ve been finding it quite useful for its intended scope.

Here’s the idea.

When you’re facing a product design problem (either in an interview or on the job) you need to separate the problem from solutions. First get clear on exactly what the problem is — who you’re solving for, what their needs are, who else is affected and what their needs are, and who you’re ignoring (for now). Get that clear in mind before attempting to create or evaluate solutions. .

SSUN is designed to systematically make sense of problem space. It stands for Stakeholders, Segments, Use cases, Needs.

The way to approach it is from left to right.

If you do the whole thing you’ll form a tree that looks something like this:

But in an interview, you won’t have time to flesh out the whole tree. You’ll need to focus. The way to do that is to make each of the four steps a two-stage process: first brainstorm a list (diverge), then prioritize and select one (converge).

You’ll wind up with a tree that looks more like this:

“Need 1” is both well-defined enough and narrow enough to tackle in the space of a single interview question.

As an example, let’s say you’re faced with a product design prompt like this:

How would you improve LinkedIn’s endorsements feature?

After you’ve confirmed with the interviewer that you know what the endorsements feature is, you can apply the SSUN framework to get clear on who you’re designing for and what their goal is. That might look like this:

Now you’re clear on the problem, and you can move into solutions.

CHUS method > CIRCLES method

Lewis Lin’s CIRCLES method is a popular framework for tackling the type of product design questions that commonly show up in product management interviews. I found the framework helpful as a starting point, but cumbersome in practice. As a result, I created a simplified reformulation that I find superior. 

It’s called CHUS — an acronym for Context, Humans, Uses, Solutions. 

Let’s first review what a product design question is, then look at the CIRCLES method and its shortcomings. Finally, I’ll show you the CHUS method. 

Product design questions

Product design questions are meant to produce a signal on your ability to take a big ambiguous problem and turn it into a great product or feature. 

Facebook calls these “product sense” questions. They’re looking to see structured thinking going from macro — can you identify and justify user segments, key paint points, and product goals — to micro — what solutions would you build to address those pain points? 

Typical questions will look like this: 

  • Improve product X.
    Eg: Dropbox recently asked, “How would you improve Slack?”
  • Design a new product that does X.
    Eg, Facebook recently asked, “Design a product to help consumers find a doctor.” 
  • Design a product for group X.
    Eg, Google recently asked, “Design a refrigerator for the blind.” 

I actually enjoy practicing these kinds of questions because this really is the kind of thinking that you have to do a lot of as a PM — taking some broad, poorly constrained provocation, making sense of it and figuring out what to do about it.

Now let’s look at Lin’s framework for tackling these kinds of questions. 


Lin’s CIRCLES method has seven steps — one for each letter of the word CIRCLES. They are: 

  1. Comprehend the situation
  2. Identify the customer
  3. Report customer needs
  4. Cut, through prioritization
  5. List solutions
  6. Evaluate tradeoffs
  7. Summarize recommendation

What I like about the framework is that these are indeed seven good things to think about, in that order. It works well as a starting point. 

There are a few things that I don’t like about it.

The first is that it has seven steps, which is quite a lot to hold in mind at once.  It’s hard to quickly recall them all and outline a response in my head.

The second problem is that while the word “CIRCLES” is easy to remember, the letters of the acronym don’t point to words that are useful to remember. “I” stands for “Identify” — identify what? The customer? Pain points? Solutions? Risks?  The issue is that Lin formed the acronym around the verbs, but the verbs are practically interchangeable, like the action words on a resume (“delivered”, “achieved”). I suspect that interchangeability is actually why he did it: he chose to prioritize a nice-sounding acronym over useful pointers.

The third problem is that the framework is ontologically inelegant. What’s going on as you work through a question is that again and again you’re going wide to explore a space of possibilities and then narrowing down to choose a specific possibility to explore. First you do this around customer segments, then around needs, then around solutions. In CIRCLES, sometimes those diverge/converge steps are called out explicitly, and sometimes not. The Customer has one step, Needs and Solutions each are given two. This inelegance makes the framework harder to adapt on the fly — kind of like the way a piece of procedurally-written software is harder to adapt and extend than a piece of object-oriented software. 

I find it cumbersome to work with CIRCLES so I changed it up and simplified it. Now let’s look at the CHUS framework.

CHUS Method

Here we have four steps: 

  1. Context 
  2. Humans
  3. Uses
  4. Solutions

At each step, we’ll do some version of the classic design diamond: diverge by creating choices, then converge making choices. 

As you work through the case, it’s critical that you manage complexity. You’ll only have time to explore a tiny piece of the problem. Repeating the diamond will allow you to show that you know how to structure a complete exploration, while maintaining tractable complexity.

Here’s what it looks like.

At the context step, diverge by asking questions. Where’s this coming from in the org? Who am I in the example? Who’s it for? Why is it needed? Then converge by synthesizing: “Okay, we’re a national garage door lift manufacturer who sees an opportunity to gain market share by getting into the smart home space.” 

At the humans step, diverge by listing all the stakeholders you can think of. Customers, manufacturers, insurers, company employees and directors, law enforcement, etc. Within the customers group, list segments. Converge by prioritizing: in almost every case the primary stakeholder that you’ll focus on is the customer. Select a segment and offer rationale. 

At the uses step, diverge by listing needs, wants, and pain points for the user. Explore the normal solutions and use cases. Converge by selecting a specific use case. As a ___, I want ___ so that ___. 

At the solutions step, diverge by listing possible solutions that address the user’s goals and pain points. Converge by prioritizing solutions and identifying the best one. 

The benefits of CHUS over CIRCLES are simplicity and elegance. Yes, you have to learn two frameworks: the CHUS sequence, and the diverge-converge method. But separating these objects out lets you see each one individually more clearly, and lets you modulate its use as appropriate to the question. 

Additional benefit — the CHUS acronym, if pronounced as “choose”, points to something that you have to do a lot of in these kinds of questions: choose where you’re going to focus. You can’t cover the whole problem space in 20 minutes. You have to again and again choose which branch of the tree you’re going to explore. 

The optimal amount of planning

On the wall in an Uber office, painted in bold, confident letters is a General George Patton quote:

“A good plan violently executed now is better than a perfect plan executed next week.”

In war and startups, I bet Patton is right much of the time. Chaos in the field is high — predictions lose value the farther they attempt to reach into the future. And the action moves fast — wait a week and the enemy will be elsewhere. But there are also cases in which it’s better to perfect the plan for another week. Launching the Apollo 11 rocket with humans aboard? I’ll take the perfect plan next week. Performing an organ transplant? Let’s get it right.

If we mess with the parameters in Patton’s thought experiment, things start to get fuzzy. What if it’s a choice between a good plan today and a perfect plan in 5 days? In 1 day? In 1 hour? What if it’s a choice between a bad plan today and a perfect plan next week? How bad does today’s plan have to be before it’s worth it to wait on next week’s perfect plan?

Product managers have to make this choice regularly. Invest more in planning, or go ahead with what you’ve got? The wisdom in the zeitgeist pushes more strongly towards action than towards planning — we encourage one another to “fail fast”, to “break things”, to “ship it”. And that wisdom is responding to something real: it feels easier, safer, to sit in the armchair a bit longer, to polish the stone a bit more, before getting out in the world and putting things to the test — so we benefit the extra nudge to go.

But in some cases we shouldn’t move fast. In some cases we should make the plan better. We should improve our models more. We should think, analyze, write, discuss, whiteboard, challenge, criticize, and strategize. Some cases call for more planning.

How do we know which is which? This is the million-dollar question. I don’t have an algorithmic solution, but here are some heuristics.

First, simply recognizing that there’s an optimal amount of strategy — which means that in a given case there can be too much or too little — will get us asking the right questions. The “fuck it ship it” mantra will not get us all we need.

Second, consider whether the decision or action is reversible. The easier it is to change your mind later, the more you should lean towards acting rather than planning. The more irreversible, the more you’ll want to perfect your plan. The Apollo 11 crew, once launched, cannot be un-launched.

Third, consider the stakes. What are the implications if you act based on your current plan, and your current plan turns out to be bad? If it won’t matter too much, then go ahead and act. If it’s make or break, lean more towards planning. A bad moon landing plan probably means the crew dies.

Fourth, consider the opportunity cost. If you were to spend the next unit of time acting rather than planning, what value could you capture? If not much, then go ahead and plan. If a lot, then act.

Fifth, consider the chaos. If the field you’re working within is highly unpredictable, then detailed long-term plains will likely go out the window pretty quickly. Get out in the field and learn as you go.

Finally, consider the fact that planning, in an important sense, makes you smarter. When you’re planning, you’re ramifying your mental models of the the world. The more sophisticated and well-articulated your models, the more capable you’ll be of collecting the right data and making the right interpretations as you go along. This will have long-term, difficult-to-account-for benefits. As another legendary WWII general said, “Plans are worthless, but planning is everything.”

Concepts related to the 80/20 principle

The 80/20 rule or “Pareto Principle” says that for many phenomena, a majority (eg 80%) of the effects come from a minority (eg 20%) of the causes. For example, if you’re a lawyer, the top 20% of your clients probably generate around 80% of your revenue. Also, the worst 20% of your clients probably generate around 80% of your headaches.

What’s powerful about the concept is that if we can successfully distinguish the super-potent causes from the less-potent causes, we can prioritize our efforts and get a lot more bang for our buck.

There are a number of related concepts that I find super useful. Here are some.

  • Compounding returns — interest on interest for exponential growth
  • Matthew effect or law of accumulated advantage — the rich get richer
  • Leverage — any influence which is compounded or used to gain an advantage
  • Force multiplication — a factor that increases the effect size of a cause
  • The 1 Percent Rule — “over time the majority of the rewards in a given field will accumulate to the people, teams, and organizations that maintain a 1 percent advantage over the alternatives.”
  • Power law or “heavy tailed” distributions — “a relationship between two things in which a change in one thing can lead to a large change in the other, regardless of the initial quantities”
  • Return on investment — the ratio of benefit out to cost in

Peter Thiel’s contrarian thought exercise

Mr. Thiel shows, again and again, how he likes to “flip around” issues to see if conventional wisdom is wrong, a technique he calls Pyrrhonian skepticism.

“Maybe I do always have this background program running where I’m trying to think of, ‘O.K., what’s the opposite of what you’re saying?’ and then I’ll try that,” he says. “It works surprisingly often.” He has even wondered if his most famous investment, Facebook, contributes to herd mentality.

When I remark that President Obama had eight years without any ethical shadiness, Mr. Thiel flips it, noting: “But there’s a point where no corruption can be a bad thing. It can mean that things are too boring.”

When I ask if he is concerned about conflicts of interest, either for himself or the Trump children, who sat in on the tech meeting, he flips that one, too: “I don’t want to dismiss ethical concerns here, but I worry that ‘conflict of interest’ gets overly weaponized in our politics. I think in many cases, when there’s a conflict of interest, it’s an indication that someone understands something way better than if there’s no conflict of interest. If there’s no conflict of interest, it’s often because you’re just not interested.”

When I ask if Mr. Trump is “casting” cabinet members based on looks, Mr. Thiel challenges me: “You’re assuming that Trump thinks they matter too much. And maybe everyone else thinks they matter too little. Do you want America’s leading diplomat to look like a diplomat? Do you want the secretary of defense to look like a tough general, so maybe we don’t have to go on offense and we can stay on defense? I don’t know.”

Maureen Dowd, NYT interview with Peter Thiel

Ray Dalio: good synthesis requires successful navigation of levels

For Ray Dalio, making good decisions requires maintaining a true and rich picture of the realities that will affect your decision. To do that, you have to be able to synthesize an enormous amount of information. And to do that, you have to be able to successfully navigate what Dalio calls “levels”. 

Reality exists at different levels and each of them gives you different but valuable perspectives. It’s important to keep all of them in mind as you synthesize and make decisions, and to know how to navigate between them. 

Let’s say you’re looking at your hometown on Google Maps. Zoom in close enough to see the buildings and you won’t be able to see the region surrounding your town, which can tell you important things. Maybe your town sits next to a body of water. Zoom in too close and you won’t be able to tell if the shoreline is along a river, a lake, or an ocean. You need to know which level is appropriate to your decision.

To synthesize and communicate well, we learn to keep track of the high-level narrative. Dalio has a nice diagram:

Sometimes we need to go into the lower level details — but only when necessary, and we return to the high-level thread when we’ve accomplished what we need to at lower levels. Here’s what that might look like:

But sometimes things go awry. For example, we might get lost in the weeds:

Or, we might lose the thread entirely:

To avoid these pitfalls, Dalio recommends these four steps:

1. Remember that multiple levels exist for all subjects.

2. Be aware on what level you’re examining a given subject.

3. Consciously navigate levels rather than see subjects as undifferentiated piles of facts that can be browsed randomly.

4. Diagram the flow of your thought processes using the outline template shown on the previous page.

From his book Principles: Life and Work (p. 250).

Strategy is compression

Strategy is the practice of intentionally improving one’s world-model and acting in the world according to the dictates of the model.

A world model is a mental map of the world. It includes a representation of the way things currently are, a representation of the ends that would be good to get to, and a representation of the available means for getting from here to there. 

Human action is always mediated by a world model. Even when we’re acting ‘on intuition’ and not thinking explicitly about our map of the world, the brain is maintaining a model and evaluating possible actions within it. 

Sometimes our world-models are very robust and accurate. Sometimes they are sketchy. We can think of a “good” world-model as similar to a good compression: it drops the information that won’t turn out to matter, and focuses attention on the small bits of information that will matter. 

The better our world models, the more effective our actions will be at attaining the ends they’re designed for. 

Being strategic and acting on intuition are different, but they’re not opposites. To be strategic is to invest time and energy into improving your world-model. It happens ‘offline’. It’s preparatory. It’s done in moments of quiet reflection and abstract conversation. To act on intuition is to trust the ideas and feelings generated that your nervous system generates on the basis of your world-model.

To be successful, we want both: we want reflective planning time in which we strengthen our world-models, and we want live action time in which we go forward with the best mental models we have at the time. This difference — between investing to improve one’s model and going out and acting in the world — explains why Eisenhower’s famous quote resonates with us: plans are useless, but planning is essential. 

Hacking the test

The most damaging thing you learned in school wasn’t something you learned in any specific class. It was learning to get good grades.

Paul Graham

The thesis of Paul Graham’s most recent essay is that school trains us not to learn or to be smart, but to hack tests. His framing is novel (to my eyes) and powerful. 

What does it mean to hack tests? It means focusing on succeeding at the evaluation mechanism rather than at the thing the mechanism was intended to evaluate. In school, it means cramming before the test, even though you won’t remember anything later. It means reading only the required material, because the rest won’t show up on the test. It’s possible — you may have done it — to get good grades in school while learning very little. 

By the time we get out of school, we’ve had years and years of exposure to this kind of training. This training makes us dumb. Most of our actual goals in life don’t reward test hackers. Graham gives an illustration from his own experience advising startup founders. You might hack parts of the process, like getting investors. But if your company isn’t a good investment, it won’t succeed, and you’ll have wasted your investors’ money and your life

Eliezer Yudkowsky wrote about this in 2007 as “lost purposes”. He gave as a memorable example Soviet factories who were evaluated by the quantity of goods they produced, and who as a result produced useless goods (like tiny shoes) in large quantities. The system had lost track of the purpose of producing the goods. 

Graham’s framing led to insight in a number of areas for me. Some places test hacking shows up for me: 

  • A high school guidance counselor once called me the worst underachiever she’d seen in her career. Test hacking resonates as a description of the thing I disliked and mistrusted about school. It’s not a test of how smart I am. It’s not a test of how much I’ve learned. It’s a test of how much of a chump I am. How far I’m willing to go to win your approval, when you long ago lost sight of what you were supposed to be doing. I was not a rebel without a cause, as I’ve often described myself retrospectively. My cause was noble. I was too much a philosopher to participate in the bullshit. 
  • There’s a kind of illness I have observed in people who work at big tech companies for a few years. It might be explained by the fact that they’re forced to spend a huge part of their energy hacking tests.
  • In product management, people often fetishize some part of the process while having lost sight of the big picture. “Fail fast” is taken literally. A/B tests are performed one after another without an adequate understanding of the statistical inferences that can be drawn or the strategic landscape that’s being explored. 
  • In consciousness. Yudkowsky said that lost purposes only show up in organizations, and that these behaviors seen in an individual would be the mark of insanity. But if we take a multi-agent view of the human mind, it makes perfect sense that the mind-system would be full of test hacking. I think this is fundamental to understanding how our minds operate. 
  • My passion for deep conceptual understanding is the cousin of my deep disgust for test hacking. Insofar as it matters that you actually succeed, then you need to actually understand and not bullshit yourself.