This forum has been too quiet for too long.

So it's discussion time.

How does your team handle Beta testing?
Every team deals with it, noone likes their work to be leaked before it is ready for consumption.

Do you keep it in team? Do you allow in enthusiastic followers of your mod? Do you randomly pick?

I usually end up setting up a list of milestones which, once completed, can be separately tested and then tested as a whole within the rest of the mod.

For each of these tests I write down a list of specific situations that I think could potentially break what was just implemented.  These situations could be based on almost anything (network latency - get a high and low pinger, etc) so I usually try to get a group of about 4 or 5 people who's systems are different enough to exhibit slightly different behavior.  Once the group is selected (out of a very close knit group of friends who have all been involved in modding and commercial development), the testing can begin.

First I do a one on one test with each of the testers.  I begin with doing a demonstration of the new feature in a few different ways.  After the initial demonstration, I get their impressions on the new features and ask how they think I could improve them while still in the game.  Then we proceed to try to destroy the features completely by trying the most off the wall bullshit during deathmatch.  Along the way we take a ton of screenshots so, for example, if any visual effects are acting up I can gauge where and why it happened.  

After the in-game testing is complete, we proceed to IRC (or to dinner) and I explain to them how the features work and what their intended effects are on gameplay as a whole.  We then discuss their impressions in a bit more detail.  By this time theyre familiar enough with the new features that they can provide valuable input and even insight that I wouldnt have seen on my own.

After the individual testing 'phase' has concluded, I determine if I think a larger scale test is necessary.  We usually start off with just 3 or 4 players, and then might move up to the entire testing team and the dev team at once.  At this point we've already ironed out most of the bugs for this milestone so this test is mainly there for us to play, enjoy ourselves, and relieve any stress we might have.  

After the group test, we'll jump into IRC and I'll begin talking about a few of the features I have planned for the next milestone.  The point of this is to see who might still be interested in participating in the tests AND who might be able to help out during the actual development phase before the next testing phase.

As a side note, I just completed reading the Masters of Doom book that came out a few days back.  Being more of a tech / lead programmer, I wanted to underscore the relationship the two Johns had during the beginning of their work together.  Carmack was the one developing the new tech and Romero would be with him, side by side, pushing the tech AS it was being implemented.  

I cant tell you enough how valuable a team member like this can be.   Not only can they inspire new features / ideas for the current project... they can produce ideas that dictate the direction your team follows in future projects.  I feel almost blessed to have a ton of people like this that I can rely upon during our teams development.

In short, for the tl;dr crew...
- Assemble TRUSTED testers based on their potential ability to break a wide spectrum of features.
- Read and appreciate the Masters of Doom.

well sounds interesting and good

but it also sounds as if you have phisical contact with your betatesters, wich we havnt....

We have the world's biggest mod QA team to test. sadly only 15-20% of them are actually useful but it provides those who want to test with constant games... the idea is to test internally with highly trusted players of the community, because you are partly making the mod for them so they should have @ least  a 5% say in things.. More offen then not they know more about playing the game then you do and can do things you never thought of.. a Team of 20 is a good size to test with, I think more then 50 would be kind of wasted since most would be just playing the game then actually testing the game..

oh well just some thoughts.

I completely agree. Getting the best players of a game/mod to test it is a huge plus. You can test your maps a hundred of times, and it might be that one of the top players finds a special trick jump in 5 minutes that you did not foresee.

If I recall it correctly in RTCW there was one guy who managed to do an airstrike jump, which shot him right into enemy base without having to blow up any barrier.