We successfully failed our first user test

A Post in The Burndown Journey.
Follow along as we build a new project management tool in the open.

Matthew Gordon
Matthew Gordon - April 27, 2022

We did our first user test of Burndown yesterday with Expected Behavior's CTO, Jason. It was his first time seeing or using the software since we started working on it. Burndown is not complete enough to use internally, but his goal is to start using it to maintain a basic roadmap (a prioritized list of projects we're planning).

After logging in to a blank slate, he started making a project and immediately paused. You know the kind of pause where the user is presented with something that makes them uncertain or unexpected? That kind of pause. Here's the screen he was looking at:

an unstyled form for creating projects that includes the field Hours Per Day

I know it's not styled yet and leaves plenty to be desired from a UX perspective, but try to look past that for a moment to the meaning of the elements. He paused at "Hours Per Day", looked at it for 10-15 seconds, then filled in a number.

To explain what that field means, we need to talk about the Burndown graph. It currently looks like

alt text

As you schedule projects, the Burndown graph helps you keep a heads up on your capacity to actually do the work you have planned. The max line represents the capacity of your team to do work every day. If scheduled work exceeds capacity, everyone goes to frown town. The Burndown graph does a lot more than that, but I'll save that for future posts. For now, consider it a really simple example where somehow this team never works on more than one project at the same time.

The "Hours Per Day" from the form above controls the maximum number of hours of work Burndown will schedule for that project for a specific day. You see the problem. If the team has the capacity to work 20 hours a day, why have a bunch of 12 hour days instead of 20 hour days?

Setting aside that he wasn't entirely sure what the field would do, there were a few problems

  1. He didn't really know the capacity of the individual people on any given team
  2. He knew people had vacation that overlapped these projects, but had no way to represent the impact that would have on the project.
  3. He knew some people were assigned to more than one active project, but had no way to represent that.

We didn't really find out the depths of Jason's feelings on the topic until late in the interview, when he volunteered "Well, you obviously can't trust the dates". Your big picture project management tool can't really help you get the important work done first if it you don't think any of the delivery dates are meaningful, so I'd say that's a bit of a problem. Still, it was the helpful feedback we needed to really dig into the problem. It made us ask the question "What more information would we need to have to make Jason believe in the dates?"

After much discussion, we decided that having accurate delivery dates requires an in-app understanding of

  • Company Holidays - Obviously no one is going to work on those days.
  • Vacations - By definition, the person on vacation will not be doing work.
  • Work Days - What days does each person work? Expected Behavior has a very flexible work policy. For example, I've chosen to work Monday through Thursday and take Fridays off. Scheduling work for me on Fridays would be obviously misleading.
  • Capacity for each work day - On the days you work, how many hours do you have available? For example, I consistently have a lot of meetings on Monday so my project time is reduced.
  • Project Allocation - If someone is assigned to more than one active project, how much of their time are they devoting to each project?

You might be thinking those are a bit obvious, and you'd be right. We suspected we'd need some or all of those things, but we take a test driven approach to product development. It was much cheaper and faster to build the simple thing and do a user test than it would be to just assume we need everything. In this case, we do need the things we expected we'd need.

Supporting these changes will require some new features, including

  • assigning specific users to specific projects
  • letting users define when they work and how much they work on those days
  • a method for dividing a person's time between projects

This has some happy consequence.

First, there's no need to ask for "hours per day" when creating a project any more. Instead, we'll ask which team members are on the project. We can work out the hours per day from the availability of the assigned people, which should be far less work for everyone involved.

Second, Behaviornauts can stop tracking vacation in a Google calendar. We'll have somewhere better to record and share our vacation plans.

Now we have list of table-stakes features and some strong justifcation for building them. We'll make these changes over the next few weeks, then do another test to see what's next.

Next Post
« Learnings from 15 customer development interviews

Previous Post
Build Log 001 »

Follow along as we build Burndown. We’ll talk about design, development, and our project management philosophy.