Two Categories of Tests: High Value vs. Typical Tests

Before we begin, You can join The Software Essentialist for an additional 15% off before we ship the Pattern-First phase of craftship. Use the code TESTING-MASTERY before August 19th to join us in the course & community.

By now, you’re probably aware there’s a lot of different types of tests.

Sure, we have unit, snapshot, component, so on & so forth.

But speak to anyone and you’ll hear that categorizing the types of tests you write in the field can get kinda hairy.

Some developers say “don’t worry about the types of tests, just write tests”.

Others say “write mostly integration tests” or “write e2e tests”.

Getting the definitions down is a good place to start to answer “how TDD works in the real world”.

What we'll cover

In this letter, we’re going to:

• ✨ Learn what the main types of tests are & why we need different ones anyway
• ✨ Learn why test intent matters (& why I first break tests into high value customer-oriented vs. typical developer type tests)
• ✨ Understand the little known high value (acceptance) tests, how they work & why they’re extremely powerful

What are the different types of test categories?

The main test types that everyone is familiar with are unit, integration, and e2e.

This is the scope at which most developers debate back and forth about tests, but I see it a bit differently.

I believe it’s not enough to look at them like this — we have unit, integration, and e2e, of course, yes — but when we change our intent behind the test, something interesting happens.

For example, we can use unit tests, which we’ll learn are tests that run really fast — and which are typically just used to test something not all that life-changing or substantial like simple methods or utility classes...

OR we can use them to test very valuable things, like features.

For this reason, I group tests based on intent.

"Are we writing tests for the customer or for the developer?"

And the answer to this tells us if our tests are typical tests or if they are high value tests.

Category #1: Typical (developer-oriented) tests

Typical tests: These are your e2e, unit & integration tests — generally, every type of test can fit into one of these 3 broad types of tests. You’re probably somewhat familiar with these in at least some capacity.

Developer-oriented tests verify technical details

What do I mean when I refer to developer-oriented tests?

Well, these are tests that are purely focused on ensuring that the internals or some technical aspect of our systems work properly.

For example, tests like these are developer-oriented tests.

// Testing a date formatting utility
const formatDate = require('./formatDate');

test('formatDate', () => {
  // Test case 1: Simple date formatting
  const testDate = new Date(2021, 2, 12);
  const expectedResult = 'March 12, 2021';
  expect(formatDate(testDate)).toBe(expectedResult);

  // Test case 2: Formatting with time
  const testDateWithTime = new Date(2021, 2, 12, 16, 30);
  const expectedResultWithTime = 'March 12, 2021, at 4:30 PM';
  expect(formatDate(testDateWithTime)).toBe(expectedResultWithTime);

  // Test case 3: Handling invalid input
  const invalidDate = 'not-a-date';
  const expectedErrorMessage = 'Invalid date provided';
  expect(() => formatDate(invalidDate))
	  .toThrowError(expectedErrorMessage);
});

// Testing that we can connect to the database*
const models = require('./models');

describe('Database connection', () => {
  it('should connect to the database', async () => {
    try {
      await models.sequelize.authenticate();
      expect(true).toBe(true);
    } catch (error) {
      expect(error).toBeNull();
    }
  });
});

// Verifying we can fetch data from an API (not a great test)
const jobsAPI = require('./jobsClient');

describe('Jobs API client', () => {
  it('fetches data from the jobs API', async () => {
    const response = await api.fetchJobs();
    expect(response.status).toBe(200);
    expect(response.data).toHaveProperty('items');
  });
});

If you guessed that:

• the first one was a unit test
• the second one was an integration test
• and the third one was also an integration test

… then you’d be right.

And if you guessed wrong, that’s ok — we’ll learn about them next.

But most importantly, what’s the intent behind all of these tests?

Who are we really supporting by writing these tests?

Ultimately, these are tests which are meant to validate technical things — utilities, clients, classes, methods, functions, etc — and that makes them developer-oriented tests.

Coming back to Behaviour-Driven Design and the Abstraction Prism, developer-oriented tests are more-low level.

Yes, we still need them, but they’re low level.

And because they’re low level, they’re not as inherently valuable as the tests which verify valuable outcomes.

What do I mean by a valuable outcome?

That’s it. The outcome is the valuable thing — not the code. It’s the acceptance criteria. The features. The user stories. And these come from the customer.

There’s a fair amount of distance between these typical tests and these valuable abstraction layers.

What’s the closest type of test to the user stories and acceptance criteria?

Acceptance tests. In reality, customer-oriented tests are acceptance tests.

Category #2: High value (acceptance) tests

High value (acceptance) tests: These are tests which verify valuable customer/business outcomes. Commonly implemented as E2E tests, because they’re a subset of your typical tests, all typical can be written as high value acceptance tests. Done properly, a unit test can be a high value unit test. Done properly, an integration test can be a high value infrastructure test or an integration test.

Customer-oriented tests verify features/acceptance criteria

If developer-oriented tests support the developer, then acceptance tests are customer-oriented tests which support the customer.

How? By giving them what they want. The features.

For example, if the customer needed to synchronize their Notion Tasks to their Google Calendar, we could express the requirement with an acceptance test specification like so:

# useCases/syncTasksToCalendar/SyncTasksToCalendar.feature

Feature: Sync Notion tasks to Google Calendar

  Scenario: Sync new tasks
    Given there are tasks in my tasks database
    And they don't exist in my calendar
    When I sync my tasks database to my calendar
    Then I should see them in my calendar

And the test code would look something like this:

// useCases/syncTasksToCalendar/SyncTasksToCalendar.ts

import { defineFeature, loadFeature } from 'jest-cucumber';
import path from 'path'
import { SyncTasksToCalendar } from './SyncTasksToCalendar'
import { TasksDatabaseBuilder, TasksDatabaseSpy } from '../../testUtils/tasksDatabaseBuilder'
import faker from 'faker'
import { SyncServiceSpy } from '../../testUtils/syncServiceSpy'
import { ICalendarRepo } from '../../services/calendar/calendarRepo';
import { CalendarRepoBuilder } from '../../testUtils/calendarRepoBuilder'
import { FakeClock } from '../../testUtils/fakeClock'
import { DateUtil } from '../../../../shared/utils/DateUtil';

const feature = loadFeature(path.join(__dirname, './SyncTasksToCalendar.feature'));

defineFeature(feature, test => {
  letcalendarRepo: ICalendarRepo;
  lettasksDatabaseSpy: TasksDatabaseSpy;
  letsyncTasksToCalendar: SyncTasksToCalendar;
  letsyncServiceSpy: SyncServiceSpy;
  let fakeClock = new FakeClock(DateUtil.createDate(2021, 8, 11));

  beforeEach(() => {
    syncServiceSpy = new SyncServiceSpy();
  })

  test('Sync new tasks', ({ given, and, when, then }) => {

    given('there are tasks in my tasks database', () => {
      tasksDatabaseSpy = new TasksDatabaseBuilder(faker, fakeClock)
        .withAFullWeekOfTasks()
        .build();
    });

    and('they dont exist in my calendar', () => {
      calendarRepo = new CalendarRepoBuilder(faker)
        .withEmptyCalendar()
        .build();
    });

    // Act
    when('I sync my tasks database to my calendar', async () => {
      syncTasksToCalendar = new SyncTasksToCalendar(
        tasksDatabaseSpy, calendarRepo, syncServiceSpy
      )
      await syncTasksToCalendar.execute();
    });

    // Assert
    then('I should see them in my calendar', () => {
      expect(syncServiceSpy.getSyncPlan().creates.length)
        .toEqual(tasksDatabaseSpy.countTasks())
      expect(syncServiceSpy.getSyncPlan().updates.length).toEqual(0);
      expect(syncServiceSpy.getSyncPlan().deletes.length).toEqual(0);
    });
  });
});

Don’t worry about the various things in here you might not yet get like Builders, Spies and whatnot.

Just focus on the intent.

Hopefully it’s clear that if this works, it’s going to be a much, much more valuable sort of test to the customer than a test against a text util class, right?

Acceptance tests as the shared, single source of truth for what to implement on any side of the stack

“But Khalil, aren’t acceptance tests are supposed to verify the system from the perspective of the user? Does that mean we have to write them End to End?”

Not necessarily.

You can write acceptance tests at the:

• e2e scope
• unit testing scope
• integration testing scope

AND you can use the same test and execute it on the frontend, backend, desktop, mobile, wherever.

Check out this beautiful image again. You’ll need to enlarge it to see it properly.

Allow me to explain.

First, it starts by building the acceptance testing rig — which is the common denominator.

The acceptance test rig

The acceptance test rig is a necessary component we need to write our high value tests of any sort.

Whenever we cross architectural boundaries, we need 4 layers to set up our work in cohesive way.

Those 4 layers are the following:

• The Acceptance Test Layer — this is where we write our acceptance tests
• The Executable Specification Layer — this is your test code (ie: jest)
• The Domain Specific Language Layer — this is where you focus on expressing what, not how
• The Protocol Driver Layer — this is where we express the how to implement the what (ie: when we cross architectural boundaries, we often need this final layer to translate the domain language layer instructions into HTTP calls, GraphQL calls, console instructions, or browser clicks or button presses)

These 4 layers work together to spread out the process involved in translating an English looking acceptance test into real-life interactions.

Acceptance tests are the single source of truth

Regardless of the scope, the single source of truth is the acceptance test for all scopes.

Yet again, this is the power of contracts.

If a customer asks for a new feature, well, both the frontend and the backend can treat that acceptance test as a contract and implement the desired behaviour — albeit in different ways, with different tasks, but they implement it.

Why is it so important to know how to write high value tests?

"We are what we repeatedly do. Excellence, then, is not an act, but a habit." - Aristotle

I used to pride myself in the ability to make something work once, but that doesn't impress me at all anymore.

Building products and writing code in such a way that you can continue to stack value on top of value...

Past a certain threshold, it is HARD.

And if you know my story, you know I had to learn the hard way.

As I've covered in "Why You Have Spaghetti Code", value creation is a zig-zag act of divergence-convergence.

It's a game of vector dynamics if you will.

You set a target, not entirely sure how you'll get there, and then you bridge the gap with code. That's what we're doing with tests. Your tests are the vector conditions. Your code is the bridge.

Developers that know how to do this, either at the start of a feature, or after the fact (ie: producing value for a company by cleaning up technical dept in a slow, consistent manner)...

I think these devs are those that stand the most likely chance of not only standing out to get the best jobs and opportunities, but to actually thrive in them as well.

So yeah, all the easy stuff is... easy 😂.

Anyone can write code.

But solving business problems?

That's a metaphysical game. It's a whole different level, man.

How to get started?

In my opinion, the best type of testing you should practice are the tests against features.

But you kinda need to master the basics - the physical mechanics of TDD first.

In The Software Essentialist, we go through a ton of exercises to really drill this skillset down to prepare you for the hard stuff.

So start with basic TDD exercises first.

Once you feel like you're comfortable there, move onto the more complicated high value tests that we use to refactor legacy code and contractualize the complex test states that comes with the territory.

There's so much more, but just take the first step and start figuring it out.

Because the paradox of mastering testing, is that once you master testing, you master vector dynamics.

And if you master vector dynamics, you're mastering goal achievement.

And if you master goal achievement, well then nothing can stop you, really.

Summary

In summary, there are a number of different types of tests we can use (on any side of the stack).

I generalize them as the typical tests and the High Value (Acceptance) Tests.

We write tests primarily for the customer or for the developer, and differentiate tests based on the concerns/abstraction layers they validate.

—

Power & love to you, my friend.

And as always,

To Mastery.