Just how assured is your quality?

Bit of a rant this, I’m afraid. I daresay it won’t be the last, either.

Today, I managed to get myself into a conversation about games QA (Quality Assurance: games testing, for those of you who don’t already know). I should know better than to let myself get onto that subject, since it’s a bit of a personal bugbear. QA is, by and large, seen as an unfortunate necessity – a stinking pit of geeky gamers who tarnish the hip image of the games companies who proudly proclaim that the industry now makes more money than Hollywood. QA should, so the prevailing wisdom holds, be shoved away somewhere dark and out of the way where they can’t bother any of the ‘real’ industry professionals. It’s not a profession, you see: it’s an unskilled joke of a job which a trained monkey could do, so there’s no point in hiring competent employees and paying them a decent wage. Illiterate students will do fine. Just throw enough man-hours at the game, and it’ll sort itself out nicely.

It’s utter rot, of course. To do QA well takes intelligence, skill and dedication. Most QA departments are fairly shoddy affairs really, but it’s the standard, so their shortcomings aren’t noticed too much. If the standard of QA increased, I’m convinced that there’d be a trickle-up effect (that’s almost certainly not the right term, but you know what I mean), and the end result would be a better game and, eventually, a better industry. The conversation touched on this (okay, I yelled about it for a while), and then moved on to the question “Okay, Michael. If you’re so knowledgeable on the subject, what makes a good QA department?“

Well, I hadn’t really thought my line of argument through well enough to give a thorough accounting of what I think would make a really top class QA department, so I mumbled some vague stuff about higher calibre staff and better working conditions and then sort of trailed off…

I’ve had a while to think about it now though, and here’s my wish list: what I’d like to see in a QA department.

1. Hire staff permanently!
Many QA teams are hired on an essentially casual basis – what is known as a ‘zero-hour contract’. This means that the employee isn’t guaranteed any work at all and either waits around in the morning to find out whether they’re working that day or gets told on Friday whether they’ll be employed the next week. The other standard practice is to hire on a fixed term contract, but it’s still essentially casual work. This means that the employee doesn’t get any benefits – no sick pay (you don’t work, you don’t get paid) or medical cover or pension. Holiday pay is usually accrued as you work, rather than the employee being given a certain number of days at the beginning of the year.

This structure is, in theory, good for the company. If your company is working on three games, but two of them are causing problems and you can’t get testable builds for a week, you just tell two thirds of the QA staff not to come in. If you realise you need more testers, you can just grab some hobos off the street and plonk them down in front of a console, and hey presto! You’ve got more testers!

In practice, it leads to a very high turnover of staff, as testers stick it out for a couple of months before realising their job is shit and then they move on somewhere else. They’re replaced easily enough, but continuity of testing is very important, and with a high turnover of staff, you don’t get continuity. This leads to huge amounts of duplicate bugs, mis-labelled bugs (collision bugs labelled as graphics bugs, for example, which have a cumulative time-wasting effect as the bugs are passed around from programmer to programmer until they find the right person to fix it) and just bad bugs (people need a couple of weeks to get up to speed with how to write a bug, so for the first couple of weeks their bugs are usually pretty poor).

Hire permanent staff and they stick around. You get continuity of testing and you get good testers entering good bugs.

2. Pay a salary (with benefits) rather than an hourly wage.
This is an extension of point 1 really, as I don’t think it’s possible to hire someone permanently but pay them per hour. I may well be wrong though, I’ve got no idea what the law is about this stuff.

Salaries sound more expensive, but in my experience they’re probably not. If you’re paying per hour (let’s say £6), when your testers work overtime, that’s probably going to go up to £9 per hour, as time-and-a-half is pretty standard. If your testers are working lots of overtime (and that’s pretty common in the games industry), you’ll end up shelling out a fair chunk of change. If your testers are salaried and get benefits – pension, medical etc. you can get away with not paying them overtime. You can’t demand they work 15 hour days for months at a time, but if you’ve scheduled properly, they won’t have to. And in any case, working overtime is expected in the games industry. They’ll be reasonably willing to work some evenings and weekends, and because they’re salaried, they won’t expect to be paid for it (unless you do try to make them work 15 hour days as standard).

3. If you can, give your staff perks.
Free copies of the games should go without saying. Merchandise is always appreciated though – mouse mats, t-shirts, stickers and so on. If you’re producing any of that stuff as marketing or promotional items, give it to your staff as well. They’ll appreciate it and a happy worker is a better worker, as someone probably said. Days out and meals are also excellent motivators. If you’re working on a movie tie-in, take your staff to see the film.

4. Don’t hire idiots!
You’d think this would go without saying, but it doesn’t. There’s a piece of conventional wisdom that says anyone can QA games. As Freakonomics tells us, conventional wisdom is often wrong. This one’s half-wrong. Anyone can QA games, but not anyone can do it well. There’s a true story I know about, concerning an imbecile who should never have been hired. Because ‘it’s just QA’, no-one saw a problem with employing the idiot. As an example of his incompetence, the team were testing a 360 game using wireless controllers. He entered a bug saying that if you didn’t touch the controller for about 15 minutes, the game crashed. In actual fact, what happened was that the controller had turned itself off. The game hadn’t crashed, but because the controller had turned off, it had paused, so if you were brainless, you could have interpreted that as a crash. That’s obviously an extreme case, but it does illustrate my point.

Another example to share with you is my first ever interview for a testing job. It was approximately two minutes long, and the questions were as follows: “Do you play games?” “Do you play for about 20 hours a week?” “When can you start?” Now, I’m an excellent tester, but if that’s your criteria for a QA technician – that they play around 20 hours of games a week – you’re going to get some dreadful people passing the interview.

If you pay a decent salary, you’ll be able to attract decent staff and they’ll do a better job. In theory this should have a cumulative beneficial effect – if there’s a staff member who isn’t quite up to the standard of the others, they’ll help get him (or her) to the standard you want, and your QA department will become (and remain) a very good one. The games industry, for all its size and bombast about astronomic revenues, is surprisingly small, and reputations spread. If you’ve got a good reputation, you’ll attract good staff and the cycle will continue.

5. Treat QA as a profession.
It’s a profession in most other industries – it even has its own qualifications (the ISO 9000 set of standards), so why not treat it like a real job in the games industry?

I know that most people don’t join a QA department with the intention of testing games for the rest of their lives (they, like me, want to actually make the games, either in a design or or production or an art capacity). But it’s a pretty safe bet that they’re going to be in QA for at least a couple of years before they move into the wider world of the games industry, so you’ve got time to train your staff up into an excellent QA team.

(As a quick addendum, I’m of the opinion that a good grounding in QA should be compulsory for just about everyone in the games industry. It teaches you a hell of a lot about what works and what doesn’t, especially when it comes to games design. Certainly more than you could learn by going straight into design from university.)

Most QA departments of my knowledge (both first- and second-hand) have a decidedly un-professional outlook. A new starter will be shown to their desk, given a copy of the game and a controller and told to get on with it. Maybe they’ll be given a day or two to play the game first, to get a feel for it, but in general they’ll just be left to figure the job out for themselves. This is a really bad idea, in my opinion.

What ought to happen is that they’re shown how the bug database and any proprietory software (debug software or whatever) works, then they’re trained in how to write a bug. Show them two or three bugs of each type, show them how the bugs are written out (descriptions, repro steps, bugs written in third person, terminology) and then let them start on the game. Tell them to call a QA Lead over when they’ve found a bug, and then show them how to search the database properly. If the bug isn’t in the database, the Lead should then sit with them as they write out the bug, helping them to do it properly. Repeat this for the first five (or even ten) bugs, and it’ll go a long way towards ensuring they’re a decent QA technician. I guess what I’m saying here is ‘train your staff’. Sounds obvious, but in the games industry it’s the exception rather than the rule.

6. Introduce standards and stick to them.
There needs to be a consistent way of writing bugs, and the QA Leads need to check each bug to make sure it’s being followed (this is almost never done, even to such basic standards as spelling and grammar checks). I know of a QA Department (and this is, in my own experience, pretty standard), in which there  wasn’t a consistent writing style for ages. The publisher wanted one, so they sent a ‘how to write a bug’ document, listing the terminology they wanted the developer to use and so on. The document was printed out and handed round. And that was it. 75% of the testers ignored it, and the Leads (who had to approve each bug) didn’t enforce it. Every couple of weeks, the publisher would complain that the QA department wasn’t following their guidelines, so the Leads would print out the document again, hand it round again, and then ignore it. It might sound nit-picky to insist on a standard way of writing bugs (what does it matter if a bug is written in first or third person?) but without it, the department gets an unprofessional reputation, and sloppiness leads to more sloppiness.

8. Minimum Bug Counts:
Never, never, never introduce a minimum bug count. It’s good to be able to say to your publisher “Look: we’re working incredibly hard. We’ve logged 3,500 bugs this week!” but bug numbers are only a small part of QA’ing a title. What if a bug takes a couple of hours to get 100% reproduction steps for? What if that bug is in multiplayer, and takes four testers to recreate? If your testers have to log, say, ten bugs a day, and they find a crash bug that they can tell is going to take a while to narrow down repro steps for (and you can usually tell), at best they’re going to throw together some wildly inaccurate repro steps. At worst, they’re going to pretend they haven’t found it, because in the time it takes to get repro steps for the one crash bug, they could have logged three collision bugs. Or if the tester finds a global bug (one that effects all levels, for example), they may well log the same issue once per level, because that way they’re meeting their bug count quota.

You (as the developer) get to say to the publisher that your QA Department are logging loads of bugs, which might sound good, but your game might still crash if more than four players detonate a grenade at once. Your testers aren’t going to care about finding that sort of important bug, because all they’ll care about is filling their bug quota.

I can’t stress this enough: minimum bug counts will make good QA Technicians into bad ones. A friend described it like this (and I can’t phrase it any better myself): “if a QA Department introduces minimum bug counts, it’s the beginning of the end. Any tester with half a brain should immediately start looking for a new job.”

7. General thoughts:
Hot-desking is a bad idea. Give people their own desk. It’ll make them feel more secure and they’ll be happier. Happier workers do better work. (It also cuts down on time wasted updating builds if you’re working on more than one game.)

For the love of God, give your staff decent chairs. They’ll be sat in them for at least 40 hours a week. £10 chairs from PC World aren’t going to cut it here. Your staff are spending most of their lives working for you. Please make them comfortable (or at least stop them from injuring themselves!)

Keep the office tidy. Again, it sounds nit-picky, but I’ve seen QA Departments where kit is just strewn around. At best it leads to not being able to find the multi-tap when you’re looking for it. At worst it leads to theft, or at least things going permanently missing. It also leads to sloppy work, and that leads to a bad reputation for your QA Department.

Treat your QA staff with respect! This is an extreme case, but there’s a company where the QA staff are treated like criminals: no bags, coats, or electronic equipment of any kind is allowed in the QA office. Staff are not allowed to be in the office during break times, or before 8:45 am. This is clearly because the company don’t trust the QA staff not to steal things or take photos of the games and so on. That would be bad enough in itself, but it only applies to QA Technicians. The QA Leads, and anyone outside the QA department are allowed to bring whatever they want into the office, and to come and go as they please. Treat your staff like criminals and they’ll behave like criminals. The quality of work will fall and again your reputation will suffer.

In conclusion: most QA departments do some good things and some bad things. If only they did better things, the industry would be a much better place.