Clippings from The Effective Enginner

Foreword by Bret Taylor (CEO of Quip)

  • Engineering was much more about leverage than programming ability
    1. Focus on high leverage activities, avoid bureaucracy
    2. Ask "Why do we need to do it at all?", rather than "How should we do this?"
    3. Work smart, not hard

Introduction

  • Work smarter, not harder
  • Working more hours isn't the most effective way to increase output
    • Working too many hours leads to decreased productivity and burnout.
    • Output may even turn out to be negative when it's necessary to repair the mistakes made by overworked and fatigued engineers.
  • To be effective engineers, we need to be able to identify which activities produce more impact with smaller time investments.
  • What Makes an Effective Engineer?
    • This book examines and describes what it means to be an effective engineer, and it distills the key lessons that I’ve learned. From author's experience of:
      1. worked as a software engineer at many technology companies, including Microsoft, Google, Ooyala, Quora, and Quip.
      2. participated in various aspects of growing an engineering team.
      3. created onboarding and mentoring programs and have trained dozens of new engineering hires
      4. shelves of productivity, team-building, personal psychology, business, and self-help books.
    • What makes an effective engineer?
      1. Produce results
      2. Get things done efficiently
      3. Know how to choose which results to deliver (Focus on value and impact)
    • An effective engineer, therefore, is defined by the rate at which he or she produces value per unit of time worked (Leverage).
  • What You'll Learn from Reading This Book
    • Technical knowledge, however, represents only a fraction of the skills that engineers need to learn in order to be effective.
    • More important for effectiveness, but often overlooked by engineers, are the meta-skills.
    • The Effective Engineer teaches you these meta-skills.
    • A useful framework, called leverage, for analyzing the effectiveness of different activities.
      1. actionable tools to increase your impact
      2. insights into the common engineering pitfalls

Part 1: Adopt the Right Mindsets

Focus on High-Leverage Activities

  • Quora's Onboarding/Mentoring Program
    • Mentoring program
      • Each new engineer was paired up with a mentor for two to three months, and the mentor was responsible for the new hire's success.
      • Mentoring activities included:
        • reviewing code
        • outlining technical skills to learn
        • and pair programming
        • discussing engineering tradeoffs
        • explaining how to prioritize better
        • offering guidance on how to work well with different team members.
      • Mentors also planned a sequence of starter tasks and projects to increase the new hires' mastery of our systems.
    • Onboarding program
      • a series of 10 recurring tech talks
      • 10 codelabs, documents that:
        • explained why we designed core abstractions
        • described how to use them
        • walked through the relevant code
        • provided exercises to validate understanding.
      • Through tech talks and codelabs, we gave our hires an overview of the codebase and the engineering focus areas, and taught them how to use our development and debugging tools.
  • Use Leverage as Your Yardstick for Effectiveness
    • The key to prioritizing different activities is assessing their leverage
    • Definition
      • Leverage is defined by a simple equation: the impact produced per time invested \((Impact Produced)/(Time Invested)\)
      • Leverage is the return on investment (ROI) for the effort that's put in
      • Effective engineers try to increase the numerator (impact produced) in that equation while keeping the denominator (time invested) small
    • Why?
      • Leverage is critical because time is your most limited resource
        • Unlike other resources, time cannot be stored, extended, or replaced
        • There's more work to be done than time available, and you'll need to start prioritizing
      • 80/20 principle / Pareto principle
        • That 20% comprises the high-leverage activities
        • Effective engineers use this simple equation as their central, guiding metric for determining how and where to spend their time.
    • that 1% time investment can have an outsized influence on the productivity and effectiveness of the other 99% of work hours.
      • Mentoring is a prime example of a high-ROI activity.
      • Pointing out a useful UNIX command could save minutes or hours on basic tasks.
      • Walking through debugging tools could drastically reduce development time for every new feature.
      • Reviewing code early and thoroughly to catch common errors could remove the need to re-address similar issues later and prevent bad habits from forming.
      • Teaching a new hire how to prioritize different projects to complete or skills to learn could easily increase his productivity.
      • Planning good starter projects that teach core abstractions and fundamental concepts could improve her software designs and reduce future maintenance requirements.
      • Since the success of our startup depended more on the success of the entire team than on what any one engineer could accomplish, investing in programs to ramp up new engineers as quickly and as seamlessly as possible was one of the highest-leverage activities that we could have done.
  • Increase Your Leverage in Three Ways
    • High Output Management: your overall leverage can only be increased in three ways:
      1. By reducing the time it takes to complete a certain activity
      2. By increasing the output of a particular activity
      3. By shifting to higher-leverage activities
    • Translated into three questions we can ask ourselves about any activity we're working on:
      1. How can I complete this activity in a shorter amount of time?
      2. How can I increase the value produced by this activity?
      3. Is there something else that I could spend my time on that would produce more value?
    • For example,
      1. to increase the meeting's leverage:
        1. Defaulting to a half-hour meeting instead of a one-hour meeting to get the same amount done in less time
        2. Preparing an agenda and a set of goals for the meeting and circulating them to attendees beforehand so that the meeting is focused and more productive.
        3. If an in-person discussion isn't actually necessary, replacing the meeting with a email discussion and spending the time building an important feature instead.
      2. to increase the leverage of your development time:
        1. Automating parts of the development or testing process that have thus far been done manually, so that you can iterate more quickly.
        2. Prioritizing tasks based on how critical they are for launch so that you maximize the value of what you finally ship.
        3. Talking with the customer support team to gain insights into the customers' biggest pain points, and using that knowledge to understand whether there's another feature you could be working on that would produce even more value with less development effort.
      3. to increase the leverage for performance engineering
        1. Learning to effectively use a profiling tool so that you can reduce the time it takes to identify each bottleneck.
        2. Measuring both the performance and visit frequency of each web page so that you can address the bottlenecks that affect the most traffic first, thereby increasing the impact of each fix you make.
        3. Working with product teams to design performant software from the outset, so that application speed is prioritized as a feature during product development rather than treated as a bug to be fixed.
  • Direct Energy Toward Leverage Points, Not Just Easy Wins
    • As you make your way through this book, constantly keep one lesson in mind: focus on high-leverage activities
    • Don't confuse high-leverage activities with easy wins
    • Leverage is used not just by effective engineers, but also by the world's most successful people.
  • Key Takeaways
    Use leverage to measure your engineering effectiveness.
    Focus on what generates the highest return on investment for your time spent.
    Systematically increase the leverage of your time.
    Find ways to get an activity done more quickly, to increase the impact of an activity, or to shift to activities with higher leverage.
    Focus your effort on leverage points.
    Time is your most limited asset. Identify the habits that produce disproportionately high impact for the time you invest.

Optimize for Learning

  • I left when I realized that google was no longer the optimal place for me to learn
  • In this chapter:
    1. Why optimizing for learning is a high-leverage activity
    2. How adopting a growth mindset is a prerequisite for improving our abilities
    3. The compounding effects of learning
    4. Why it's important to invest in your rate of learning
    5. Six key elements about a work environment that you should consider when looking for jobs or switching teams
    6. Actionable tips for taking advantage of learning opportunities on the job
    7. Some strategies for how you can always be learning

Adopt a Growth Mindset

  • Mindset
    • How we view our own effectiveness impacts how much effort we invest in improving it
    • Two mindsets
      1. Fixed mindset (human qualities are carved in stone)
      2. Growth mindset (human can cultivate and grieve their intelligence and skills through effort)
        • Those with growth mindsets are willing to take steps to better themselves
  • How do we build a growth mindset
    • Own your story
      • It's not about apologizing for where your resume doesn't line up
      • But rather telling your story
        • Who you are
        • What skills you've built
        • What you're excited about doing next and why
  • What adopting a growth mindset looks like
    • Accepting responsibility for each aspect of a situation that you can change
    • Rather than blaming failures and shortcomings on things outside your control
    • Taking control of your own story
    • Optimizing for experience where you effortlessly succeed
    • Investing in your rate of learning

Invest in Your Rate of Learning

  • Three lessons from compound interest
    1. Compounding leads to an exponential growth curve
    2. The earlier compounding starts, the sooner you hit the region of rapid growth and the faster you can reap its benefits
    3. Even small deltas in the interest rate can make massive difference in the long run
  • Learning, like interest, also compounds
    1. Learning follows an exponential growth curve
      • Knowledge gives you a foundation, enabling you to gain more knowledge even faster
    2. The earlier that you optimize for learning, the more time your learning has to compound
    3. Due to compounding, even small deltas in your own learning rate make a big difference over the long run
  • We tend to drastically underestimate the impact of small changes on our growth rate
  • How do we avoid complacency and instead shift ourselves toward a growth mindset?
    • Treat yourself like a startup The Startup of You
      • Startups initially prioritize learning over profitability to increase their chances of success
      • Setting yourself up for long-term success requires thinking of yourself as a startup or product in beta, a work-in-progress that needs to be invested-in and iterated in every single day
    • Continuous iteration Delivering Happiness
      • "Think about what it means to improve just 1% per day and build upon that every single day. (37x at the end of the year, not 365%/3.65x better)
  • How we can invest your time in activities with the highest learning rate

Seek Work Environments Conductive to Learning

  1. Fast growth
    • At fast growing teams and companies, the number of problems to solve exceeds available resources, providing ample opportunities to make a big impact and to increase your responsibilities
    • The growth also makes it easier to attract strong talent and build a strong team, which feeds back to generate even more growth
    • A lack of growth leads to stagnation and politics
  2. Training
  3. Openness
    • A growing organization isn't going to figure out the most effective product idea, engineering design, or organizational process on its first attempt.
    • If it can continuously learn and adapt from past mistakes, then it stands a much better chance of success
    • That's more likely to happen if employees challenge each others' decisions and incorporate feedback into future iterations
  4. Pace
    • A work environment that iterates quickly provides a faster feedback cycle and enables you to learn at a faster rate.
    • Things that slow down iteration speed
      • Lengthy release cycles
      • Formalized produce approvals
      • Indecisive leadership
    • Things that accelerate progress
      • Automation Tools
      • Lightweight approval processes
      • A willingness to experiment
    • Smaller teams and companies tend to have fewer bureaucratic barriers to getting things done than larger ones
    • Do push yourself, but also find a place that's sustainable for you in the long run (don't burn out)
  5. People
    • Surrounding yourself with people who are smarter, more talented, and more creative than you means surrounding yourself with potential teachers and mentors
    • Meet with potential team members ahead of time before starting a new position
  6. Autonomy
    • The freedom to choose what to work on and how to do it drives our ability to learn
    • As long as we have the support that we need to use that freedom effectively

Dedicate Time on the Job to Develop New Skills

  • Borrow the "20% time" lesson from Google
    • It's more effective to take it in one/ or two-hour chunks each day rather than in one full day each week
      • Because you can then make a daily habit out of improving your skills
    • What should you do?
      1. Develop a deeper understanding of areas you're already working on and tools that you already use
      2. Gain experience in "adjacent disciplines"
        • Disciplines related to your core role and where increased familiarity can make you more self-sufficient and effective
        • Product Management
          • User Research
          • Product Management
          • Backend Engineering
        • Infrastructure engineer
          • Maxine learning
          • Database internals
          • Web Development
        • Growth engineer
          • Data Science
          • Marketing
          • Behavioral psychology
        • Adjacent Disciplines will not only be useful
        • But you'll also be more likely to retain the information because you'll be actively practicing ot
  • 10 suggestions to take advantage of the resources available to you at work
    1. Study code for core abstractions written by the best engineers at your company.
    2. Write more code.
    3. Go through any technical, educational material available internally.
    4. Master the programming languages that you use.
      • Make sure that at least one of your languages is a scripting language (e.g., Python or Ruby) that you can use as your Swiss army knife for quick tasks.
    5. Send your code reviews to the harshest critics.
      • Optimize for getting good, thoughtful feedback rather than for lowering the barrier to getting your work checked in.
      • Discuss software designs with your company’s best designers in order to avoid writing great code for a design that doesn’t work well.
    6. Enroll in classes on areas where you want to improve.
    7. Participate in design discussions of projects you’re interested in.
      • Don’t wait for an invitation.
      • Ask project leads if they’d mind you being a silent observer or even a participant in a design meeting.
      • If mailing lists are open internally, add yourself or read through key conversations in the archives.
    8. Work on a diversity of projects.
      • Interleaving different projects can teach you what problems are common across projects and what might just be artifacts of your current one.
      • The interleaved practice of different skills is more effective than repeated, massed practice of a single skill at preparing people to tackle unfamiliar problems
    9. Make sure you’re on a team with at least a few senior engineers whom you can learn from.
    10. Jump fearlessly into code you don’t know.
      • engineering success was highly correlated with “having no fear in jumping into code they didn’t know.”
      • Fear of failure often holds us back, causing us to give up before we even try.
      • “in the practice of digging into things you don’t know, you get better at coding.”

Always Be Learning

  • We should always be asking:
    • How can I improve?
    • How could I have done this better?
    • What should I learn next to best prepare me for the future?
  • Other skills
    • might not translate directly into engineering benefits,
    • but the practice of adopting a growth mindset toward them still makes us better learners and more willing to stretch beyond our comfort zone.
    • research in positive psychology shows that continual learning is inextricably linked with increased happiness
  • 10 starting points to help inspire a habit of learning outside of the workplace
    1. Learn new programming languages and frameworks.
    2. Invest in skills that are in high demand
    3. Read books
      • Using books to discover how the world works
      • Books offer a way for you to learn from the lessons and mistakes of others (you can re-apply that knowledge without having to start from scratch.)
    4. Join a discussion group
    5. Attend talks, conferences, and meetups
    6. Build and maintain a strong network of relationships
      • the more people you meet, the more you’ll find serendipitous opportunities
      • “Lucky people dramatically increase the possibility of a lucky chance encounter by meeting a large number of people in their daily lives. The more people they meet, the greater opportunity they have of running into someone who could have a positive effect on their lives.” - The Luck Factor
    7. Follow bloggers who teach
    8. Write to teach
      • When you write to teach other people, you gain a deeper understanding of ideas you’re already familiar with and pinpoint the details that you didn’t fully understand.
      • The Feynman Technique: The Best Way to Learn Anything
      • Writing also provides an opportunity for mindful reflection on what you’ve learned.
      • Presenting at conferences can have similar benefits.
    9. Tinker on side projects
      • Creativity stems from combining existing and often disparate ideas in new ways.
      • Projects in seemingly orthogonal areas like drawing and writing can have benefits that flow over to help you be a better engineer.
    10. Pursue what you love
      • Spend time on what you’re passionate about instead,
      • Let that passion fuel your motivation to learn and to grow.
  • Embracing a growth mindset in which you’re motivated to learn about the things that excite you.

Key Takeaways

  1. Own your story
  2. Don't shortchange your learning rate
  3. Find work environments that can sustain your growth
  4. Capitalize on opportunities at work to improve your technical skills
  5. Locate learning opportunities outside of the workplace

Prioritize Regularly

  • In any engineering discipline (and in life), there will always be more tasks to do than you have time for.
  • Regular prioritization is a high-leverage activity
    • Because it determines the leverage of the rear of your time.
  • In this chapter:
    1. The strategies used to prioritize effectively
    2. Why it's important to track all to-dos in a single and easily accessible list
      • Make pairwise comparisons between what we're doing and what we could be doing instead
      • In order to iteratively shift our time toward higher-leverage activities
    3. Two simple heuristics to help us identify what's high-leverage
      1. Focusing on what directly produces value
      2. Focusing on the important and non-urgent
    4. Execute on high-priority tasks (after identifying them) (How you can accomplish your priorities)
      1. Protecting your maker's schedule
      2. Limiting the amount of work you have in progress
      3. Creating if-then plans to help fight procrastination
    5. A sample implementation of a prioritization workflow, using all the strategies above

Track To-Dos in a Single, Easily Accessible List

  • No matter how much of an expert you might be, a well-designed checklist can significantly improve your outcomes.
    • The Checklist Manifesto
  • The first step in effective prioritization is listing every task that you might need to do.
    • Getting Things Done: The Art of Stress-Free Productivity: David Allen: 9780142000281: Amazon.com: Books
    • Because the human brain is optimized for processing and not for storage.
      • The Magical Number Seven, Plus or Minus Two - Wikipedia
        • The average brain can actively hold only 7 +/- 2 items in its working memory.
        • Once there are more than 7 items, people fail to repeat digits and words back in the correct order over half the time.
      • Research shows that expending effort on remembering things
        1. reduces our attention
        2. impairs our decision-making abilities
        3. hurts our physical performance
  • To-do lists should have two major properties
    1. a canonical representation of our work
    2. easily accessible
      • allows you to quickly identify a task you can complete if you unexpectedly have a block of free time.
      • if you think up a new task, you can add it directly to your list even when you're out and about, rather than investing mental energy in trying to remember it.
  • A single master list is better than an assortment of sticky notes, sheets of paper, and emails, because these scattered alternatives are easily misplaced and make it harder for your brain to trust that they're comprehensive.
  • Your to-do list can take many forms.
    • Notebook
    • Task management software
    • Mobile app
    • Dropbox text file
  • When you consistently offload to-dos to your list,
    1. you reduce the number of tasks you need to remember down to just one: to check your single, master list
    2. you free your mind to focus on a higher-leverage activity: actually prioritizing your work.
  • Estimating both the time required and the value produced by each task is incredibly hard.
    1. we had limited data
    2. every task we tackled inspired new tasks that we'd then insert into our to-do list -> we never got to most of the backlogged items on our ranked list
  • Ask yourself on a recurring basis: Is there something else I could be doing that's higher-leverage?
    • The goal isn't to establish a total ordering of all your priorities, since any ordering you make will be based on imperfect information;
    • instead, it's to continuously shift your top priorities toward the ones with the highest leverage, given the information you have.

Focus on What Directly Produces Value

  • The time and effort expended do not necessarily correlate with the value produced.
  • Yishan Wong's answer to What are some ways to 'work smart' rather than just working hard in college? - Quora
    • Activity is not necessarily production
    • Many work activities do not directly contribute towards useful output
  • Focus on tasks that
    1. directly bring a product closer to launch
    2. directly increase the number of users, customers or sales
    3. directly impact the core business metric your team is responsible for.
  • When you get the important things right, the small things often don't matter.
  • Make sure that the effort you invest is proportional to its expected impact.
  • Prioritize the ones that produce the most value with the least amount of effort.
    • Once you do this a few times, it becomes easier to recognize which tasks are the most valuable.
  • Learn to say no.
    • Don't treat every invitation to do something as an obligation.
    • Explain how the meeting, bug, or project will detract from your other tasks, and discuss whether it should have higher priority.
  • Don't try to get everything done.
    • Focus on what matters
    • And what matters is what produces value.

Focus on the Important and Non-Urgent

  • Partitioning of activities based on urgency and importance

      Urgent Not Urgent
    Important Crises Planning and prevention
      Pressing Issues Building relationships
      Deadlines New opportunities
        Personal development
      (Quadrant 1) (Quadrant 2)
    Not Important Interruptions Surfing the web
      Most meetings Busy work
      Most emails and calls Time wasters
      (Quadrant 3) (Quadrant 4)
  • Carve out time to invest in skills development
    • Their productivity might slow down at first,
    • but with time, the new tools and workflows that they learn will increase their effectiveness and easily compensate for the initial loss.
  • Label everything on his to-do list from 1 through 4, based on which quadrant the activity fell under.
  • Be wary if you're spending too much time on Quadrant 1's important and urgent activities.
    • Examples
      • Frequent pager duty alerts might indicate a need for automated recovery procedures.
      • High-priority bugs might be a symptom of low test coverage.
      • Constant deadlines might be caused by poor project estimation and planning.
    • Investing in Quadrant 2 solutions can reduce urgent tasks and their associated stress.
  • The act of prioritization is itself a Quadrant 2 activity, one whose importance often gets overlooked because it's rarely urgent.

Protect Your Maker's Schedule

  • Engineers need longer and more contiguous blocks of time to be productive than many other professionals.
  • Flow requires focused attention; interruptions break flow.
  • Maker's Schedule, Manager's Schedule
  • Employees take an average of 10 to 15 minutes (or event 23 minutes) to return to focused activity after handling email and instant messaging interruptions
  • When possible, preserve larger blocks of focused time in your schedule.
    1. Schedule necessary meetings back-to-back or at the beginning or end of your work day, rather than scattering them throughout the day.
    2. If people ask you for help while you're in the middle of a focused activity, tell them that you'd be happy to do it before or after your breaks or during smaller chunks of your free time.
    3. Block off hours on your calendar (maybe even with a fake meeting) or schedule days like "No Meeting Wednesdays" to help consolidate chunks of time.
    4. Learn to say no to unimportant activities, such as meetings that don't require your attendance and other low-priority commitments that might fragment your schedule.

Limit the Amount of Work in Progress

  • When we fragment our attention too much, however, we end up reducing our overall productivity and hindering our ability to make substantive progress on any one thing.
  • Your Brain at Work
    • the brain's working memory is like a stage and its thoughts are like the actors.
    • The stage has a limited amount of space (for 7 ± 2 actors), but in order to make decisions, the prefrontal cortex needs to bring all the relevant actors onto the stage at once
    • When we work on too many things simultaneously, we spend most of our brain's mental energy moving actors on and off the stage rather than paying attention to their performance.
  • Personal Kanban
    • there is a limit to how many things we can work on at once.
    • Constant context switching hinders deep engagement in any one activity and reduces our overall chance of success.
  • Limiting work in progress means prioritizing and serializing different projects so that I can maintain strong momentum.
  • The same principle applies to how teams tackle projects as well.
    • When a small group of people fragment their efforts across too many tasks, they stop sharing the same context for design discussions or code reviews.
    • Competing priorities divide the team, and momentum on any one activity slows down.
  • The number of projects that you can work on simultaneously varies from person to person.
    • Use trial and error to figure out how many projects you can work on before quality and momentum drop
    • resist the urge to work on too many projects at once.

First Procrastination with If-Then Plans

  • Many people do not have sufficient motivation to summon the activation energy required to start a difficult task.
  • The if-then plan from the book Succeed
    • in which we identify ahead of time a situation where we plan to do a certain task.
    • Examples
      1. "if it's after my 3pm meeting, then I'll investigate this long-standing bug,"
      2. "if it's right after dinner, then I'll watch a lecture on Android development."
    • "planning creates a link between the situation or cue (the if) and the behavior that you should follow (the then)."
    • When the cue triggers, the then behavior "follows automatically without any conscious intent." (subconscious followup)
      • Procrastination primarily stems from a reluctance to expend the initial activation energy on a task.
      • This reluctance leads us to rationalize why it might be better to do something easier or more enjoyable, even if it has lower leverage.
      • When we're in the moment, the short-term value that we get from procrastinating can often dominate our decision-making process.
      • But when we make if-then plans and decide what to do ahead of time, we're more likely to consider the long-term benefits associated with a task.
    • The concept of if-then planning also can help fill in the small gaps in our maker’s schedule.
      • Save a list of short tasks that I need to get done and that don’t require a big chunk of uninterrupted time, and I use them to fill in the blank.
        • finishing a code review,
        • writing interview feedback,
        • responding to emails,
        • investigating a small bug,
        • writing an isolated unit test.

Make a Routine of Prioritization

  • Revisit priorities
  • As time passes, our current projects may no longer represent the best use of our time
  • Being retrospective and making a habit of revisiting your priorities
  • there’s no “best” prioritization workflow
    • The suggestions are endless
    • Take general principles
    • Iteratively adapt your own system until you find something that works well for you
    • If you don’t have a system, then reading productivity books or using another engineer’s system can help provide a starting point.
    • The actual mechanics of how you review your priorities matter less than adopting the habit of doing it.
  • A sample
    1. Have sections for
      1. This Week
      2. Today
      3. Doing
    2. Estimate and annotate each task with the number of 25-minute pomodoros
    3. Do morning prioritization
      • Identify a small number of important tasks that I wanted to accomplish during the rest of the day.
      • Whenever I didn’t get something done that I had previously prioritized, the routine gave me an opportunity to review why: had I worked on something else more important, misprioritized, or simply procrastinated?
    4. If it’s in the morning, then I’ll pick tasks requiring more mental effort and creativity.
    5. Timing what I’m doing might seem like overkill, but I’ve found that it increases my awareness of how much time I spend on a given task and makes me more accountable when distractions arise.
    6. Track how many 25-minute sessions I spend on a task, mainly to validate and learn about whether my initial estimates were accurate.
    7. On a Sunday afternoon or Monday morning, I review the priorities I’ve accomplished, examine the ones I had planned to finish but didn’t so I can understand why, and scope out what I hope to accomplish in the following week
  • when you have certain personal or professional goals that you want to achieve, you’ll find that prioritization has very high leverage

Key Takeaways

  • Write down and review to-dos.
  • Work on what directly leads to value.
  • Work on the important and non-urgent.
  • Reduce context switches.
  • Make if-then plans to combat procrastination.
  • Make prioritization a habit.

Part 2: Execute, Execute, Execute

Invest in Iteration Speed

  • Why is CD such a powerful tool?
    • Fundamentally, it allows engineers to make and deploy small, incremental changes rather than the larger, batches changes typical at other companies
    • That shift in approach eliminates a significant amount of overhead associated with traditional releases processes, making it easier to reason about changes and enabling engineers to iterate much more quickly

Move fast to learn fast

  • The faster that you can iterate, the more that you can learn about what works and what doesn't work.
    • With each iteration, you get a better sense of which changes you will point you in the right direction, making your future efforts much more effective.
    • If you never break anything, you're probably not moving fast enough
  • Moving fast doesn't necessarily mean moving recklessly
  • Options for increasing iteration speed
    1. CD
    2. investing in time-saving tools
    3. Improving your debugging loops
    4. Mastering your programming workflows
    5. Removing any bottlenecks that you identify

Invest in time-saving tools

  • There are only a finite number of work hours in the day
    • So increasing your effort as a way to increase your impact fails to scale
    • Tools ate the multipliers that allow you to scale your impact beyond the confines of the work day
      1. Time-saving Tools pay off large dividends
      2. Faster tools get used more often
      3. Faster tools can enable new development workflows that previously weren't possible
        • CD
        • Compilation Speed
        • REPL
        • CI
  • The time-saving properties of tools also scale with team adoption
    • Therefore, it's not sufficient to find or build a time-saving tool.
    • To maximize its benefits, you also need to increase its adoption across your team
    • The best way to accomplish that is by proving that the tool actually saves time
    • Sometimes, the time-saving tool that you built might be objectively superior to the existing one, but the switching costs discourage other engineers from actually changing their workflow and learning your tool
      • In these situations, it's worth investing the additional effort to lower the switching cost and to find a smoother way to integrate the tool into existing workflows
    • One side benefit of proving to people that your tool saves time is that it also earns you leeway with your manager and your peers to explore more ideas in the future
      • It can be difficult to convince others that an idea that you believe in is actually worth doing
      • So start small

Shorten your debugging and validation loops

  • Much of our engineering time is spent either debugging issues or validating that what we're building behaves as expected
  • Why we didn't think to do it earlier
    • When you're fully engaged with a bug you're testing or a new feature you're building, the last thing you want to do is to add more work
    • When you're already using a workflow that works, albeit with a few extra steps, it's easy to get complacent and not expend the mental cycles on devising a shorter one
  • Be mindful of what steps they're taking to reproduce issues and reflecting on which steps they might be able to short circuit

Master your programming environment

  • Given how much time we spend in our programming environments, the more efficient we can become, the more effective we will be as engineers
  • Minor improvements easily compound over time
  • Mastery is a process, not an event, and as you get more comfortable, the time savings will start to build
  • The key is to be mindful of which of your common, everyday actions slow you down, and then figure out how to perform those actions more efficiently
  • Chances are, others have already built the tools we need to accelerate our most common workflows
  • Some ways you can get started
    1. Get proficient with your favorite text editor or IDE
    2. Learn at least one productive, high-level programming language
      • Each Minute spent writing boilerplate code for a less productive language is a minute not spent tackling the meatier aspects of a program
    3. Get familiar with shell commands
    4. Prefer the keyboard over the mouse
    5. Automate your manual workflows
    6. Test our ideas on an interactive interpreter
    7. Make it fast and easy to run just the unit tests associated with your current changes
  • It lets us shift our limited time from the mechanics of programming to more important problems

Don't ignore your non-engineering bottlenecks

  • The best strategy for optimizing your iteration speed is the same as for optimizing the performance of any system
    1. Identify the biggest bottlenecks
    2. Figure out how to eliminate them
  • One common type of bottleneck is dependency on other prople
    • Oftentimes the cause is a misalignment if priorities rather than negative intention
    • The sooner you acknowledge that you need to personally address this bottleneck, the more likely you'll be able to either adapt your goals or establish consensus on the functionality's priority
    • Communication is critical for making progress on people-related bottlenecks
      • projects fail from under-communicating, not over- communicating
  • Another common type of bottleneck is obtaining approval from a key decision maker, typically an executive at the company
    • Prioritize building prototypes, collecting early data, conducting user studies, or whatever else is necessary to get preliminary project approval
    • Explicitly ask the decision makers what they care about the most, do that you can make sure to get those details right
    • Don't defer approvals until the end
  • The review processes that accompany any project launch,
    • whether they be
      • Verification by the quality assurance team
      • A scalability or reliability review by the performance team
      • An audit by the security team

Key takeaways

  • The faster you can iterate, the more you can learn
  • Invest In tooling
  • Optimize your debugging workflow
  • Master the fundamentals of your craft
  • Take a holistic view of your iteration loop

Measure What You Want to Improve

  • Without a reliable way of measuring user happiness, it would have been hard to determine whether a given change (to the search results page) actually resulted in forward progress
    • One way to access user happiness is to directly ask users about their experiences
      • Increasing user delight is a valuable and laudable notion
      • Delight it difficult to quantify
    • It's much better to use a metric (based on the behavior of Google's 100+ million monthly active users)
      • In the plex
      • A well-chosen metric can tackle a wide range of problems
  • In this chapter
    • Why metrics are important tools for effective engineers, to not only measure progress but also to drive it
    • How choosing which key metrics to use and not use can completely change what work gets prioritized
    • The importance of instrument of our systems to increase our understanding of what's happening
    • How internalizing some useful numbers can help shortcut many decisions
    • Why we need to be skeptical of data integrity
    • How we can defined ourselves against bad data

Use Metrics to Drive Progress

  • "If you can't measure it, you can't improve it" The Effective Executive: The Definitive Guide to Getting the Right Things Done (Harperbusiness Essentials)
  • Good metrics accomplish a number of goals
    1. They help you focus on the right thins
    2. When visualized over time, good metrics help guard against future regressions
    3. Good metrics can drive forward progress
      • A technique called performance ratcheting used in Box
        • Use metrics to set a threshold (performance ratchet)
        • Any new change that would push latency or other key indicators past the ratchet can't get deployed until it's optimized, or until some other feature is improved by a counterbalancing amount.
        • Moreover, every time the performance team makes a system-level improvement, they lower the ratchet further.
    4. A good metric lets you measure your effectiveness over time and compare the leverage of what you're doing against other activities you could be doing instead
  • Quantifying goals with a metric isn't always easy
    • Just because it's hard to measure a goal doesn't mean that it's not worthwhile to do so
  • Given the benefits of good metrics, it's worth asking yourself:
    1. Is there some way to measure the progress of what I'm doing?
    2. If a task I'm working on doesn't move a core metric, is it worth doing? Or is there a missing key metric?

Pick the Right Metric to Incentivize the Behavior You Want

  • Selecting what you measure is as important as measuring itself
    • The choice of which metric to measure dramatically impacts and influences the type of work that we do
    • There can be more than one way to measure progress toward any given goal
    • The magnitude of the goal for a metric also matters
    • What you don't measure is important as well
      • "average handle time" for customer service
  • Picking the right metric applies to your personal goals as well as your professional ones
  • When deciding which metrics to use, choose ones that
    1. maximize impact
      • economic denominator from Good to Great

        If you could pick one and only one ratio—profit per x …—to systematically increase over time, what x would have the greatest and most sustainable impact on your economic engine?

        • In the context of engineering, the core metric should be the one that, when systematically increased over time, leads you and the rest of your team to make the greatest and most sustainable impact.
        • Why having a single, unifying metric?
          1. it enables you to compare the output of disparate projects
          2. it helps your team decide how to handle externalities
    2. are actionable
      • One whose movements can be causally explained by the team's efforts
      • In contrast, vanity metrics (from The Lean Startup)
        • track gross numbers
        • Increases in vanity metrics may imply forward product progress
        • But they don't necessarily reflect the actual quality of the team's work
        • Examples
          • page views per month
          • total registered users
          • total paying customers
      • actionable metrics
        • Examples
          • signup conversion rate
          • the percentage of registered users that are active weekly over time
        • Through A/B testing, we can trace the movement of actionable metrics directly back to product changes or to feature launches
    3. are responsive yet robust
      responsive

      updates quickly enough to give feedback about whether a given change was positive or negative

      so that your team can learn where to apply future efforts

      robust
      external factors outside of the team's control don't lead to significant noise
      (no term)
      Responsiveness needs to be balanced with robustness.

Instrument Everything to Understand What's Going On

  • When it comes to day-to-day operations, however, you should be less discriminatory: measure and instrument as much as possible.
    Core metrics
    a high-level, big-picture activity
    Day-to-day operation metrics
    Gaining insight into what's going on with systems that we've built
  • When it comes to diagnosing problems, instrumentation is critical.
    • Only after Twitter engineers started monitoring and instrumenting their systems were they able to identify problems and build the much more reliable service that over 240 million people now use every month.
    • When we don't have visibility into our software, all we can do is guess at what's wrong.
      • That's a main reason why the 2013 HealthCare.gov launch was such an abysmal failure.
  • In a similar way, effectively optimizing a core metric requires systematically measuring a slew of other supporting metrics
  • Adopting a mindset of instrumentation means ensuring we have a set of dashboards that surface key health metrics and that enable us to drill down to the relevant data.
  • However, many of the questions we want to answer tend to be exploratory
    • Since we often don't know everything that we want to measure ahead of time
    • Therefore, we need to build flexible tools and abstractions that make it easy to track additional metrics
  • The more quickly that teams can identify the root cause of certain behaviors, the more rapidly they can address issues and make progress
  • You don't have to be a large engineering team operating at scale to start instrumenting your systems

Internalize Useful Numbers

  • The knowledge of useful numbers provides a valuable shortcut for knowing where to invest effort to maximize gains
  • Ensuring you have access to a few useful numbers to approximate your progress and benchmark your performance is a high-leverage investment: they provide the benefits of metrics at a much lower cost.
    • The numbers that matter to you will vary based on your focus area and your product.
      • Latency numbers every programmer should know
  • Why?
    1. Knowing useful numbers like these enables you, with a few back-of-the-envelope calculations, to quickly estimate the performance properties of a design without actually having to build it.
    2. Internalizing useful numbers can also help you spot anomalies in data measurements.
    3. Knowledge of useful numbers can clarify both the areas and scope for improvement.
  • How to get these numbers:
    • To obtain performance-related numbers, you can write small benchmarks to gather data you need.
    • Other numbers may require more research, like talking with teams (possibly at other companies) that have worked in similar focus areas, digging through your own historical data, or measuring parts of the data yourself.

Be Skeptical about Data Integrity

  • "All data can be abused...People interpret data the way they want to interpret it."
    • Sometimes, we pick easy-to-measure or slightly irrelevant metrics, and use them to tell a false narrative about what’s happening.
    • Other times, we confuse correlation with causality.
  • How to protect ourselves against data abuse
    • Our best defense is skepticism

      Bad math students
      they get to the end of the problem, and they’re just done.
      Good math students
      get to the end of the problem, look at their answer, and say, 'Does that roughly make sense?'
    • When it comes to metrics,
      1. compare the numbers with your intuition to see if they align.
      2. Try to arrive at the same data from a different direction and see if the metrics still make sense.
      3. If a metric implies some other property, try to measure the other property to make sure the conclusions are consistent.
  • Untrustworthy data that gets incorporated into decision-making processes provides negative leverage.
    • Unfortunately, it’s all too common for engineers to underinvest in data integrity, for a few reasons:
      1. Since engineers often work against tight deadlines, metrics—whose importance only shows up after launch—can get deprioritized.
      2. When building a new product or feature, it’s much easier to test and validate their interactions than to verify whether some seemingly plausible metric (like page views) is actually accurate.
      3. Engineers reason that because their metrics-related code was well unit-tested, the metrics themselves also should be accurate, even though there could be system-level errors or incorrect assumptions.
    • The net result is that metrics-related code tends to be less robust than code for other features.
  • Given the importance of metrics, investing the effort to ensure that your data is accurate is high-leverage.
  • Some strategies that you can use to increase confidence in your data integrity:
    1. Log data liberally, in case it turns out to be useful later on.
    2. Build tools to iterate on data accuracy sooner.
    3. Write end-to-end integration tests to validate your entire analytics pipeline.
    4. Examine collected data sooner.
    5. Cross-validate data accuracy by computing the same metric in multiple ways.
    6. When a number does look off, dig in to it early.

Key Takeaways

  1. Measure your progress
  2. Carefully choose your top-level metric
  3. Instrument your system
  4. Know your numbers
  5. Prioritize data integrity

Validate Your Ideas Early and Often

  • Feedback helps the team prioritize what to build next.
  • You can't get everything right
  • In this chapter
    1. How validating our ideas both early and often helps us get the right things done
    2. The importance of finding low-effort and iterative ways to validate that we're on the right track and to reduce wasted effort
    3. How to use A/B testing to continuously validate our product changes with quantitative data
    4. How much impact that kind of (A/B) testing can have
    5. A common anti-pattern (the one-person team) that sometimes hinders our ability to get feedback, and ways of dealing with that situation
    6. How the theme of building feedback and validation loops applies to every decision we make

Find Low-Effort Ways to Validate Your Work

  • Iterative approaches lead to fewer costly errors and give us opportunities between each iteration to collect data and correct our course.
    1. lead to fewer costly errors
      • The shorter each iteration cycle, the more quickly we can learn from our mistakes.
      • The longer the iteration cycle, the more likely it is that incorrect assumptions and errors will compound.
    2. give us opportunities between each iteration to collect data and correct our course.
      • Oftentimes when we build products and set goals, we embark on paths that aren’t clear-cut.
      • Demystifying the riskiest areas first lets you proactively update your plan and avoid nasty surprises that might invalidate your efforts later.

        What’s the scariest part of this project? That’s the part with the most unknowns and the most risk. Do that part first.

  • Continually ask ourselves: Can I expend a small fraction of the total effort to collect some data and validate that what I’m doing will work?
  • Minimum Viable Product

    that version of a new product which allows a team to collect the maximum amount of validated learning about customers with the least effort - The Lean Startup

    • that definition resembles our definition of leverage
    • building an MVP requires being creative
      • Dropbox's MVP is a short 4-minute video
    • the principle of validating our work with small efforts holds true for many engineering projects.
      • The cost of building a quick prototype doesn’t amount to much in the scheme of the larger project, but the data that it produces can save you a significant amount of pain and effort if it surfaces problems early, or convinces you that the larger migration wouldn’t be worthwhile.
      • The strategy of faking the full implementation of an idea to validate whether it will work is extremely powerful.
  • Invest a small amount of work to gather data to validate your project assumptions and goals.

Continuously Validate Product Changes with A/B Testing

  • Why
    1. Help you decide which variation to launch.
    2. Tell you how much better that variation actually is.
    3. Encourage an iterative approach to product development, in which teams validate their theories and iterate toward changes that work
  • How
    • When deciding what to A/B test, time is your limiting resource
    • Hone into differences that are high-leverage and practically significant, the ones that actually matter for your particular scale.
    • Initially, it’s tricky to determine what’s practically significant, but as you run more experiments, you’ll be able to prioritize better and determine which tests might give large payoffs.
  • Performed correctly, A/B testing enables us to validate our product ideas and transform an otherwise-baffling black box of user behavior data into understandable and actionable knowledge.

Beware the One-Person Team

  • The one-person team

    An iconic Silicon Valley story features the engineer who designs and builds an ambitious system all on his own. Expecting to launch his project soon, he sends a large code review to a teammate—only to learn about a major design flaw he’d missed, and to be informed that he should have built his system in a completely different way.

  • While there isn't anything inherently wrong with working on a one-person project, it does introduce additional risks that, if not addressed, can reduce your chance of success.
    1. it adds friction to the process of getting feedback
    2. The lows of a project are more demoralizing when you're working alone.
      • A single stall can grind the project to a halt, causing deadlines to slip
    3. The highs can be less motivating when you’re working alone.
      • Knowing that your teammates are depending on you increases your sense of accountability
      • The desire to help your team succeed can override the dips in motivation that everyone occasionally feels.
  • Some strategies to set up the necessary feedback channels to increase the chances of our projects succeeding.
    1. Be open and receptive to feedback
    2. Commit code early and often
    3. Request code reviews from thorough critics
    4. Ask to bounce ideas off your teammates
    5. Design the interface or API of a new system first
    6. Send out a design document before devoting your energy to your code
    7. If possible, structure ongoing projects so that there is some shared context with your teammates
    8. Solicit buy-in for controversial features before investing too much time

Build Feedback Loops for Your Decisions

  • Sometimes, validation is difficult.
    • There might not be many data points, or there may only be qualitative data available.
    • The more senior of an engineer you become (and particularly if you enter into management), the tougher and the more nebulous your decisions become.
  • Regardless of whether we engineers are writing code, creating a product, or managing teams, the methodology of how to make decisions remains the same. -> at its core, the willingness to run experiments demonstrates the scientific method at work.
  • Validation means
    1. formulating a hypothesis about what might work,
    2. designing an experiment to test it,
    3. understanding what good and bad outcomes look like,
    4. running the experiment, and learning from the results.

Key Takeaways

  • Approach a problem iteratively to reduce wasted effort
  • Reduce the risk of large implementations by using small validations
  • Use A/B testing to continuously validate your product hypotheses
  • When working on a solo project, find ways of soliciting regular feedback
  • Adopt a willingness to validate your decisions

Improve Your Project Estimation Skills

  • Businesses need accurate estimates to make long-term plans for their products.
  • We'll always operate under imperfect information. Therefore, successful project planning requires
    1. increasing the accuracy of our project estimates
    2. increasing our ability to adapt to changing requirements
  • In this chapter
    1. Tools to take charge of your own project plans and push back against unrealistic schedules
    2. Ways of decomposing project estimates to increase accuracy.
    3. How we can do a better job of budgeting for the unknown.
    4. How to clearly define a project's scope and establish measurable milestones,
    5. How to reduce risk as early as possible so that we can adapt sooner
    6. Why we need to be careful not to use overtime to sprint toward a deadline if we find ourselves falling behind: we may actually still be in the middle of a marathon.

Use Accurate Estimates to Drive Project Planning

  • Software Estimation by Steve McConnell
  • Project schedules often slip because we allow the target to alter the estimate.
  • A more productive approach is to use the estimates to inform project planning, rather than the other way around.
  • Understanding your business priorities fosters more productive conversations, letting you devise a better project plan.
    1. hold the date constant and deliver what is possible
    2. hold the feature set constant and push back the date until all the features can be delivered
  • Some concrete strategies to produce accurate estimates that provide us the flexibility we need
    • Decompose the project into granular tasks.
      • If a task will take more than two days, decompose it further.
      • A long estimate is a hiding place for nasty surprises.
        • Treat it (a long estimate) as a warning that you haven't thought through the task thoroughly enough to understand what's involved
      • The more granular a task's breakdown, the less likely that an unconsidered subtask will sneak up later.
    • Estimate based on how long tasks will take, not on how long you or someone else wants them to take.
    • Think of estimates as probability distributions, not best-case scenarios.
    • Let the person doing the actual task make the estimate
    • Beware of anchoring bias
      • Avoid committing to an initial number before actually outlining the tasks involved, as a low estimate can set an initial anchor that makes it hard to establish a more accurate estimate later on.
    • Use multiple approaches to estimate the same task
    • Beware the mythical man-month
    • Validate estimates against historical data
    • Use timeboxing to constrain tasks that can grow in scope
    • Allow others to challenge estimates
  • A little measurement can go a long way.
    • a simple spreadsheet for team members to track how many hours they estimated a task would take and how long it actually took.

Budget for the Unknown

  • What caused the schedule to slip excessively were all the unknown projects and issues that we hadn't estimated or accounted for at all.
  • When setting schedules, build in buffer time for the unexpected interruptions.
  • Explicitly track the time spent on tasks not initially part of the project plan, in order to build awareness

Define Specific Project Goals and Measurable Milestones

  • What frequently causes a project to slip is a fuzzy understanding of what constitutes success
  • Articulate a clear goal for the project based on a clear problem.
  • Setting a project goal produces two concrete benefits
    1. A well-defined goal provides an important filter for separating the must-haves from the nice-to-haves in the task list.
      • The more specific the goal, the more it can help us discriminate between features.
    2. It builds clarity and alignment across key stakeholders.
      • Building alignment also helps team members be more accountable for local tradeoffs that might hurt global goals.
      • Building alignment helps ensure that team members internalize those costs and make consistent tradeoffs.
  • Even more effective than defining specific goals is outlining measurable milestones to achieve them.
    • Each milestone was a very clear point where we had introduced some value that we didn't have before
      • The measurable quality enabled her team to scrutinize every single task and ask, "Is this task a prerequisite for this milestone?"
    • Milestones act as checkpoints for evaluating the progress of a project and as channels for communicating the team’s progress to the rest of the organization.

Reduce Risk Early

  • Effectively executing on a project means minimizing the risk that a deadline might slip and surfacing unexpected issues as early as possible.
  • Tackling the riskiest areas first helps us identify any estimation errors associated with them.
  • The goal from the beginning should be to maximize learning and minimize risk, so that we can adjust our project plan if necessary.
  • In addition to the specific risks associated with a project, a risk common to all large projects comes during system integration, which almost always takes longer than planned.
    • Code complexity grows as a function of the number of interactions between lines of code more than the actual number of lines, so we get surprised when subsystems interact in complex ways.
    • Moreover, it’s extremely hard to decompose integration into granular tasks during the early estimation phases of a project, when you’re not sure what the end state will look like.
    • How can we reduce integration risk?
      1. Build end-to-end scaffolding and do system testing earlier.
        1. it forces you to think more about the necessary glue between different pieces and how they interact, which can help refine the integration estimates and reduce project risk.
        2. if something breaks the end-to-end system during development, you can identify and fix it along the way, while dealing with much less code complexity, rather than scrambling to tackle it at the end.
        3. it amortizes the cost of integration throughout the development process, which helps build a stronger awareness of how much integration work is actually left.

Approach Rewrite Projects with Extreme Caution

  • Rewrite projects are particularly troublesome for a few reasons
    • They share the same project planning and estimation difficulties as other software projects.
    • Because we tend to be familiar with the original version, we typically underestimate rewrite projects more drastically than we would an undertaking in a new area.
    • It is easy and tempting to bundle additional improvements into a rewrite.
    • When a rewrite is ongoing, any new features or improvements must either be added to the rewritten version or they must be duplicated across the existing version and the new version. The cost of either option grows with the timeline of the project.
  • Chapter 5. The Second-System Effect
  • Engineers who successfully rewrite systems tend to do so by converting a large rewrite project into a series of smaller projects.
  • Rewriting a system incrementally is a high-leverage activity
    1. additional flexibility at each step to shift to other work that might be higher-leverage
    2. Dramatically reduce risk (worth to increase the overall amount of work)
    3. Significantly reduce the time pressure of the project
  • When doing an incremental rewrite is not possible, break the rewrite down into separate, targeted phases

Don't Sprint in the Middle of a Marathon

  • Working more hours doesn't necessarily mean hitting the launch date
    1. Hourly productivity decreases with additional hours worked
    2. You're probably more behind the schedule than you think
    3. Additional hours can burn out team members
    4. Working extra hours can hurt team dynamics
    5. Communication overhead increases as the deadline looms
    6. The sprint toward the deadline incentivizes technical debt
  • Unless there is a realistic plan for actually hitting the launch date by working extra hours, the best strategy in the long run is either to redefine the launch to encompass what the team can deliver by the target date, or to postpone the deadline to something more realistic.
  • When the decision has to be made, increase the probability that overtime will actually accomplish your goals by:
    • Making sure everyone understands the primary causes for why the timeline has slipped thus far.
    • Developing a realistic and revised version of the project plan and timeline.
    • Being ready to abandon the sprint if you slip even further from the revised timeline
  • The only way to get better (at project estimation and project planning) is by practicing these concepts, especially on smaller projects where the cost of poor estimations is lower

Key Takeaways

  • Incorporate estimates into the project plan
  • Allow buffer room for the unknown in the schedule
  • Define measurable milestones
  • Do the riskiest tasks first
  • Know the limits of overtime

Part 3: Build Long-Term Value

Balance Quality with Pragmatism

  • Ultimately, software quality boils down to a matter of tradeoffs, and there’s no one universal rule for how to do things.
  • writes early Facebook engineer Evan Priestley.
    • You must move quickly to build quality software
    • You must build quality software to move quickly
  • In this chapter,
    1. several strategies for building a high-quality codebase and consider the tradeoffs involved
    2. both the benefits and the costs of code reviews,
    3. some ways that teams can review code without unduly compromising iteration speed.
    4. how building the right abstraction can manage complexity and amplify engineering output
    5. how generalizing code too soon can slow us down.
    6. how extensive and automated testing makes fast iteration speed possible,
    7. why some tests have higher leverage than others.
    8. when it makes sense to accumulate technical debt and when we should repay it.

Establish a Sustainable Code Review Process

  • the benefits of code reviews
    • Catching bugs or design shortcomings early.
    • Increasing accountability for code changes.
    • Positive modeling of how to write good code.
    • Sharing working knowledge of the codebase.
    • Increasing long-term agility.
  • engineers who don’t do them often cite their concern about their impact on iteration speed.
  • deciding on code reviews doesn’t need to be a binary choice,
    • Rather, think of code reviews as being on a continuum.
    • They can be structured in different ways to reduce their overhead while still maintaining their benefits.
    • Experiment to find the right balance of code reviews that works for you and your team.

Manage Complexity through Abstraction

  • the right abstraction can dramatically amplify an engineer’s output. - Software Abstractions by Daniel Jackson
    1. It reduces the complexity of the original problem into easier-to-understand primitives.
    2. It reduces future application maintenance and makes it easier to apply future improvements.
    3. It solves the hard problems once and enables the solutions to be used multiple times.
  • building an abstraction for a problem comes with tradeoffs.
    1. takes more time than building one specific to a given problem.
      • That’s more likely to happen with software the team relies heavily on than with peripheral parts of the codebase
      • so focus more energy on making the core abstractions great.
    2. it’s possible to overinvest in them up front
    3. poor abstraction
      • When we’re looking for the right tool for the job and we find it easier to build something from scratch rather than incorporate an existing abstraction intended for our use case, that’s a signal that the abstraction might be ill-designed.
      • Create an abstraction too early, before you have a firm handle on the general problem you’re solving, and the resulting design can be overfitted to the available use cases.
      • Bad abstractions aren’t just wasted effort; they’re also liabilities that slow down future development.
  • what makes an abstraction good?
    • easy to learn
    • easy to use even without documentation
    • hard to misuse
    • sufficiently powerful to satisfy requirements
    • easy to extend
    • appropriate to the audience
  • Moreover, good abstractions disentwine complex concepts into simpler ones. (Simple made easy)
  • Study other people’s abstractions to learn how to build good ones yourself.
    1. Find popular abstractions in your codebase at work or from repositories on GitHub.
    2. Look through the open source projects at technology companies like Google, Facebook, LinkedIn, and Twitter.
    3. Study the interfaces of popular APIs developed by Parse, Stripe, Dropbox, Facebook, and Amazon Web Services, and figure out what makes it so easy for developers to build on top of their platforms.

Automate Testing

  • Unit test coverage and some degree of integration test coverage provide a scalable way of managing a growing codebase with a large team without constantly breaking the build or the product.
  • Automated testing doesn't just reduce bugs; it provides other benefits as well.
    • Decreases repetitive work that we'd otherwise need to do by hand.
    • Leads engineers to be much more accountable for the quality of their own work.
    • Allows engineers to make changes with significantly higher confidence
      • Automated tests mitigate against a culture in which people are fearful of modifying and improving a piece of code just because it might break.
    • Helps to efficiently identify who's accountable for the broken change
    • Executable documentation of what cases the original author considered and how to invoke the code
      • Average familiarity with a codebase decreases as both the code and the team grow
  • Dogmatically requiring test coverage for all code is unlikely to be the best use of your time
  • It can be difficult to foster a culture of automated testing
    • The inflection point came when a simple unit test visibly started to save time.
    • When the time savings became obvious, people looked for other strategic tests to help them iterate faster.
  • Writing the first test is often the hardest.
    • Focus on high-leverage tests
    • Once you have a few good tests, testing patterns, and libraries in place, the effort required to write future tests drops

Repay Technical Debt

  • Technical debt often is hard to quantify
    • Start small and approach the problem incrementally
  • Not all technical debt is worth repaying.
    • Repay the debt with the highest leverage
    • Code in highly-trafficked parts of the codebase that takes the least time to fix up.

Key Takeaways

  • Establish a culture of reviewing code
  • Invest in good software abstractions to simplify difficult problems
  • Scale code quality with automated testing
  • Manage your technical debt

Minimize Operational Burden

  • How Instagram supported 40 million users with only 13 employees
    • Whenever they could, the Instagram team picked proven and solid technologies instead of shiny or sexy new ones.

      Every single, additional [technology] you add, is guaranteed mathematically over time to go wrong, and at some point, you'll have consumed your entire team with operations.

  • Development costs didn't stop accruing at launch time; in fact, they were just starting to accumulate.
  • Shaving time off of recurring costs frees you to focus on what matters most.
  • In this chapter
    1. Strategies for minimizing operational burden
    2. Do the simple thing first -> why we should embrace operational simplicity.
    3. How building systems to fail fast makes them easier to maintain
    4. The importance of relentlessly automating mechanical tasks
    5. How making automation idempotent reduces recurring costs.
    6. Why we should practice and develop our ability to recover quickly.

Embrace Operational Simplicity

  • Simple solutions impose a lower operational burden because they're easier to understand, maintain, and modify
  • When you first start off trying to solve a problem, the first solutions you come up with are very complex, and most people stop there.
  • When engineering teams don't focus on doing the simple thing first, they either end up being less effective over time because their energy is spent on a high upkeep cost, or they reach a point where the operational burden gets so high that they're forced to simplify their architecture.
  • Having too complex of an architecture imposes a maintenance cost in a few ways:
    1. Engineering expertise gets splintered across multiple systems
    2. Increased complexity introduces more potential single points of failure.
    3. New engineers face a steeper learning curve when learning and understanding the new systems
    4. Effort towards improving abstractions, libraries, and tools gets diluted across the different systems
  • When system complexity grows faster than the engineering team's ability to maintain the system, productivity and progress suffer.
  • Remember: do the simple thing first.
    • Always ask, "What's the simplest solution that can get the job done while also reducing our future operational burden?"
    • Revisit sources of complexity, and find opportunities to trim them away.

Build Systems to Fail Fast

  • Many engineers associate robustness and reliability with an absence of crashes.
    • These techniques cause software to fail slowly.
    • The software may continue to run after an error, but this is often in exchange for less decipherable bugs further down the road.
    • The more directly we can link the feedback to a source, the more quickly that we can reproduce the problem and address the issue.
  • By failing fast, we can more quickly and effectively surface and address issues.
  • The more complex the system, the more time that fail-fast techniques can save.
  • Failing fast doesn't necessarily mean crashing your programs for users.
    1. use fail-fast techniques to surface issues immediately and as close to the actual source of error as possible;
    2. complement them with a global exception handler that reports the error to engineers while failing gracefully to the end user
  • Building systems to fail fast can be a very high-leverage activity. It helps reduce the time you spend maintaining and debugging software by surfacing problematic issues sooner and more directly.

Relentlessly Automate Mechanical Tasks

  • Engineers automate less frequently than they should, for a few reasons:
    • They don't have the time right now.
    • They suffer from the tragedy of the commons, in which individuals act rationally according to their own self-interest but contrary to the group's best long-term interests.
    • They lack familiarity with automation tools
    • They underestimate the future frequency of the task
    • They don't internalize the time savings over a long time horizon
  • Every time you do something that a machine can do, ask yourself whether it's worthwhile to automate it.
  • Two types of automation
    1. Automating mechanics
      • Straightforward
      • Testable
    2. Automating decision-making
      • Much more challenging
      • They rarely get tested well
      • By definition they run in unusual circumstances
  • Automation can produce diminishing returns as you move from automating mechanics to automating decision-making.
    • Given your finite time, focus first on automating mechanics.
    • Only after you've picked all the low-hanging fruit should you try to address the much harder problem of automating smart decisions.

Make Batch Processes Idempotent

  • When idempotence isn’t possible, structuring a batch process so that it’s at least retryable or reentrant can still help.
  • Idempotence also offers another benefit that many effective engineers take advantage of: the ability to run infrequent processes at a more frequent rate than strictly necessary, to expose problems sooner.
    • Convert infrequent workflows into more common ones by scheduling dry runs every day or week;

Hone Your Ability to Respond and Recover Quickly

  • Script for success and shift our decision-making away from high-stakes and high-pressure situations and into more controlled environments.
    • From The Score Takes Care of Itself
  • Netflix (Chaos Monkey), Google (Disaster Recovery Testing), and Dropbox (simulates additional load) all assume that the unexpected and the undesired will happen.
    • They practice their failure scenarios to strengthen their ability to recover quickly.
    • They believe that it’s better to proactively plan and script for those scenarios when things are calm, rather than scramble for solutions during circumstances outside of their control.
  • The best that we can do is to "script for success," practice failure scenarios, and work on our ability to recover quickly.

Key Takeaways

  • Do the simple thing first
  • Fail fast to pinpoint the source of errors
  • Automate mechanics over decision-making
  • Aim for idempotence and reentrancy
  • Plan and practice failure modes

Invest in Your Team's Growth

  • A succinct description of the different positions
    Staff engineer
    making the team better
    Principle engineer
    making the company better
    Distinguished engineer
    improving the industry
  • The secret to your own career success is to "focus primarily on making everyone around you succeed
  • Your career success depends largely on your company and team'a success, and the success of your company or team depends on more than just your individual contributiobs
  • You'll accomplish much more if those around you are aligned with you rather than against you, and you can do that by investing in their success
  • In this chapter
    1. techniques to invest in different phases of your team's growth
    2. why strong engineering companies make hiring a top priority
    3. what your role should be in the hiring process
    4. why designing a good onboarding process for new members of your team is a high-leverage activity and how to do it.
    5. how, once you've assembled your team, sharing ownership of code makes your team stronger
    6. how using post-mortems to build collective wisdom leads to long-term value.
    7. a discussion of how to build a great engineering culture.

Make Hiring Everyone's Responsibility

  • Hiring is an extremely high-leverage activity.
    • The smaller the company—and the more likely that the person you interview will be an immediate co-worker—the greater the leverage of those interviews.
    • If those 40 hours resulted in even just one additional hire, the 2,000+ hours of output that he or she would contribute per year would more than justify the cost.
  • How do we design an effective interview process?
    • A good interview process achieves two goals
      1. It screens for the type of people likely to do well on the team
      2. It gets candidates excited about the team, the mission, and the culture
        • Ideally, even if a candidate goes home without an offer, they still leave with a good impression of the team and refer their friends to interview with the company
        • One of your primary levers as an interviewer is therefore making the interview experience both fun and rigorous
    • As an interviewer, your goal is to optimize for questions with high signal-to-noise ratios
      • Questions that reveal a large amount of useful information (signal) about the candidate per minute spent, with little irrelevant or useless data (noise)
        • Good, well-executed questions let you confidently differentiate among candidates of varying abilities;
        • Bad, poorly managed questions leave you unsure whether to hire the candidate.
      • The types of questions that generate the most signal depend on the qualities most correlated with success on your team.
        • An increasing number of companies have shifted toward interviews that include a hands-on programming component.
      • Ample literature exists to help you get started on designing questions.
        • Cracking the Code Interview
  • How to continuously iterate on improving your interview process.
    • Take time with your team to identify which qualities in a potential teammate you care about the most
    • Periodically meet to discuss how effective the current recruiting and interview processes are at finding new hires who succeed on the team.
    • Design interview problems with multiple layers of difficulty that you can tailor to the candidate's ability by adding or removing variables and constraints
    • Control the interview pace to maintain a high signal-to-noise ratio.
      • Don't let interviewees ramble, get stumped, or get sidetracked for too long.
      • Either guide the interviewee along with hints, or wrap up and move on to a different question.
    • Scan for red flags by rapidly firing short-answer questions to probe a wide surface area.
    • Periodically shadow or pair with another team member during interviews
    • Don't be afraid to use unconventional interview approaches if they help you identify the signals that your team cares about.
  • It's worth the effort: the additional output from adding a strong engineer to your team far exceeds the output of many other investments that you could make.

Design a Good Onboarding Process

  • As the team grows and the surface area of new things to explore increases, it becomes harder and harder for a recent hire to figure out what to learn first without any guidance.
  • A quality onboarding process is a powerful leverage point for increasing team effectiveness
    1. First impressions matter
    2. Training a new engineer for an hour or two a day during her first month generates much more organizational impact than spending those same hours working on the product.
    3. The initial time investment to create onboarding resources continues to pay dividends with each additional team member.
  • Why?
    • Investing in your team's success means that you are more likely to succeed as well.
    • Effectively ramping up new team members ultimately gives you more flexibility to choose higher-leverage activities.
      • A stronger and larger team means
        1. easier code reviews
        2. more people available to fix bugs
        3. increased resources for on-call rotations and support
        4. greater opportunities to tackle more ambitious projects.
    • Onboarding is a win-win situation;
      1. the new hires receive valuable training,
      2. the mentors get more things done.
    • A poor onboarding experience reduces a team's effectiveness.
      1. Productive output gets lost when a recent hire takes longer to ramp up.
      2. Code quality suffers if new team members use abstractions or tools incorrectly, or if they aren't familiar with team conventions or expectations.
      3. Insufficient training means it's harder to accurately identify low performers
      4. Good engineers undergo unnecessary stress and may even get weeded out because of weak guidance.
  • How?
    1. Identify the goals that your team wants to achieve
      • Goals for Quora's onboarding program

        1. Ramp up new engineers as quickly as possible
        2. Impart the team's culture and values
        3. Expose new engineers to the breadth of fundamentals needed to succeed
          • A key part of a good onboarding program is ensuring that everyone starts on a consistent, solid foundation.
        4. Socially integrate new engineers onto the team
      • Your own goals may vary
      • What's important is understanding what you want to achieve so that you can focus your efforts appropriately
    2. Construct a set of mechanisms to accomplish these goals
      • Four main pillars for Quora's onboarding program
        1. Codelabs
          • A codelab is a document that explains why a core abstraction was designed and how it's used, walks through relevant parts of its code internals, and supplies programming exercises to validate understanding.
          • The codelabs clarified what abstractions were important to master early on and recommended a particular order for learning them.
          • They enabled new engineers to ramp up more quickly and make product changes sooner.
        2. Onboarding talks
          • Cover the things we believed were the most important for new hires to learn.
          • Together, the onboarding talks and codelabs helped ensure that new hires learned the fundamentals.
        3. Mentorship
          • Mentors checked in daily with their mentees during the first week, and then met for weekly 1:1s.
          • All of this helped establish the shared goal of ramping up new hires as quickly as possible, and set the expectation that they shouldn’t hesitate to seek guidance.
        4. Starter tasks
          • Complete a starter task by the end of the first week
          • This aggressive target conveyed the value of getting things done and moving fast.
          • It also meant that the team needed to remove enough onboarding friction for new hires to build momentum quickly.
          • Mentors were responsible for identifying starter tasks of increasing complexity for their mentees.
      • Building a good onboarding program is an iterative process.
      • To get a sense of what worked well and what could use some improvement.
        • Think about your own experience
        • Survey others on the team

Share Ownership of Code

  • Why
    • There’s a common misconception that being the sole engineer responsible for a project increases your value.
      • When you’re the bottleneck for a project, you lose your flexibility to work on other things.
    • From a company’s perspective, sharing ownership increases the bus factor to more than one.
      • Shared ownership also means that everyone participates in maintenance duties and one person doesn’t carry the entire burden.
      • Shared ownership eliminates isolated silos of information and enables an engineer to step in for another teammate, so that everyone can focus on whatever produces the most impact.
    • This gives senior engineers more free time to work on other projects and junior engineers an opportunity to ramp up on the infrastructure and the codebase.
  • How
    • To increase shared ownership, reduce the friction that other team members might encounter while browsing, understanding, and modifying code that you write or tools that you build.
    • Some strategies
      • Avoid one-person teams.
      • Review each other’s code and software designs.
      • Rotate different types of tasks and responsibilities across the team.
      • Keep code readable and code quality high.
      • Present tech talks on software decisions and architecture.
      • Document your software, either through high-level design documents or in code-level comments.
      • Document the complex workflows or non-obvious workarounds necessary for you to get things done.
      • Invest time in teaching and mentoring other team members.

Build Collective Wisdom through Post-Mortems

  • Write down
    1. What happened
    2. How it happened
    3. Why it happened
    4. What we can do to prevent it from happening in the future
  • Dedicate the same healthy retrospection to projects and launches
    1. How effectively did your feature actually accomplish your team’s goals?
    2. Was that actually the best use of your team’s time?
  • Some friction to doing this better
    1. If your team hasn’t defined a clear goal or metric for a launch, it’s difficult to assess its success.
    2. If your team doesn’t want to publicly declare months of work to be a failure, it’s tempting to close discussions.
    3. If your team is overwhelmed with new projects, it’s hard to make time for reflection.
  • The goal is not to levy blame but to maximize collective wisdom.
  • Use methodologies like "Five Whys" to understand the root cause of operational issues
  • Holding an honest conversation about a project can be uncomfortable.
    1. It requires acknowledging that months of effort may have resulted in failure, and viewing the failure as an opportunity for growth.
    2. It requires aligning behind a common goal of improving the product or team, and not focusing on where to assign blame.
    3. It requires being open and receptive to feedback, with the goal of building collective wisdom around what went wrong and what could’ve been done better

Build a Great Engineering Culture

  • great engineering cultures:
    1. Optimize for iteration speed.
    2. Push relentlessly towards automation.
    3. Build the right software abstractions.
    4. Focus on high code quality by using code reviews.
    5. Maintain a respectful work environment.
    6. Build shared ownership of code.
    7. Invest in automated testing.
    8. Allot experimentation time, either through 20% time or hackathons.
    9. Foster a culture of learning and continuous improvement.
    10. Hire the best.
  • It begins with the values of the initial team members, and it’s a continual work-in-progress that every engineer helps to shape.

Key Takeaways

  • Help the people around you be successful
  • Make hiring a priority
  • Invest in onboarding and mentoring
  • Build shared ownership of code
  • Debrief and document collective wisdom
  • Create a great engineering culture

Epilogue

  • Time is our most finite asset, and leverage—the value we produce per unit time—allows us to direct our time toward what matters most.
  • most of the advice in this book applies beyond engineering.
  • when it comes to achieving our work and life goals, leverage is a powerful framework for helping us focus on the right things.