Guild Wars 2

So it seems as if quite a lot of the time Anet releases new patches it requires 1-2 extra patches to get it all sort of working correctly. Im wondering exactly how they test.

Little Background: I work in the development of model based test automation solutions, mostly utilized in telecom and safety critical systems (cars, planes and trains) . There is a lot of great testing technology available and im just curious why more companies like Anet do not adopt more advanced testing procedures.

Time + Budget.

They can’t push a patch back two weeks because it didn’t test well, players will complain.

They don’t have the budget to hire more staff for testing, the best software, etc. They’d have to charge more for the game/sub and players will complain.

So they do what they can and fix the rest once it goes live and they can tell where the problem areas are. Players still complain.

No matter what they do, players will complain. For the most part, they won’t quit the game, just yell a lot. So the company can save money and players will complain, or they can spend money and players will still complain.

The option to spend more money doesn’t MAKE more money for the company, and thus is not an attractive option.

Even well tested software have critical bugs. There are very few instances where software is release w/o any bugs. It is the nature of the beast. Generally, the testing environment even PST is not the same as the live servers.

tolunart.2095:

Time + Budget.

“Testing is expensive, and people give us money anyway, lets not do it!”

Sounds like EA.

I would argue that well executed test automation saves money in the long term. Although stat up costs are prohibitive for rapid release cycles like gaming firms.

Swizzle.7982:

tolunart.2095:

Time + Budget.

“Testing is expensive, and people give us money anyway, lets not do it!”

Sounds like EA.

Nope, just being practical.

What if I gave you a choice between:

A: I’ll give you $10 and punch you in the face.

B: You’ll give me $10 and I’ll punch you in the face.

You’re going to get punched no matter what. Do you want to get $10, or lose $10?

When the devs test, they test the way the game as intended. And there can be so many of them.

A few devs vs us (the population of a few thousand). You will see why we are the bigger sample and can find bugs more efficiently. Some of us like to break the game by flying over unintended areas or encounter unforseen situations, etc…

A test server =/= an actual server. There are tools in test servers that do not occur in playable servers.

DHK.8406:

I would argue that well executed test automation saves money in the long term. Although stat up costs are prohibitive for rapid release cycles like gaming firms.

Long term in an MMO doesn’t matter. Long term customers will stick with the game no matter what happens. Short term customers will move on no matter what happens. So they crunch the numbers and do whatever makes the most sense.

I played Rift for two years before getting bored and letting my sub expire. There were bugs present at launch that still hadn’t been fixed. Not game breaking bugs, just annoyances that weren’t enough to make anyone quit, and thus were never fixed because they were irrelevant.

IMHO, you need a tester server asap.

So saving money, and having a better product doesnt matter? That matters to every industry.

Because testing works SO well, lol.

http://i.imgur.com/H1K1q.jpg

DHK.8406:

So saving money, and having a better product doesnt matter? That matters to every industry.

Saving money, yes, having a better product, no…a better SELLING product is the important thing.

Saving money matters. It matters more than having a better product. Look at Bic pens – they suck, leak and so on, but they are cheap to make and they sell billions of them. There were a cheap alternative to fountain pens and make tons of cash. But they are a terrible product, quality-wise. People use them and throw them away, they don’t expect more from them.

rizzo.1079:

DHK.8406:

So saving money, and having a better product doesnt matter? That matters to every industry.

Saving money, yes, having a better product, no…a better SELLING product is the important thing.

There are products in every industry that focus on the high-quality, high-price market, and products that are focused on the cheap and disposable market. The $100 truffle and Kobe beef hamburger vs the Big Mac.

MMO publishers know the market they’re going after, and it isn’t the market that will pay $100 a month to play a bug-free game.

Even if they had little time and little budget, couldn’t they have at least made sure people could get into the dungeon? Isn’t that like the main thing that should be tested? Its like trying to buy a car, but u realize the car has no wheels.

I really don’t mind minor glitches like clipping etc, but having
the most important aspect of this retribution story (the dungeon)
not accessible tells me that they have not even tested at all.

Either that or someone at anet made a very bad mistake and uploaded
a wrong patch or something.

Sorry Anet, im already not impressed with this flame and frost quest, mainly because of having you brag it as having EVEN MOAR CONTENT THEN AN ENTIRE EXPANSION, and this bug is just unacceptable.

You have a good MMO on your hands. Don’t squander the opportunity you have by releasing these minor rubbish. (SAB is good tho, kudos to you)

tolunart.2095:

rizzo.1079:

DHK.8406:

So saving money, and having a better product doesnt matter? That matters to every industry.

Saving money, yes, having a better product, no…a better SELLING product is the important thing.

There are products in every industry that focus on the high-quality, high-price market, and products that are focused on the cheap and disposable market. The $100 truffle and Kobe beef hamburger vs the Big Mac.

MMO publishers know the market they’re going after, and it isn’t the market that will pay $100 a month to play a bug-free game.

Well I was just talking in general. Of course there are niche products like $100million yacht manufacturers, but in general a better selling product is preferable to a better quality product in the modern business environment.

rizzo.1079:

tolunart.2095:

rizzo.1079:

DHK.8406:

So saving money, and having a better product doesnt matter? That matters to every industry.

Saving money, yes, having a better product, no…a better SELLING product is the important thing.

There are products in every industry that focus on the high-quality, high-price market, and products that are focused on the cheap and disposable market. The $100 truffle and Kobe beef hamburger vs the Big Mac.

MMO publishers know the market they’re going after, and it isn’t the market that will pay $100 a month to play a bug-free game.

Well I was just talking in general. Of course there are niche products like $100million yacht manufacturers, but in general a better selling product is preferable to a better quality product in the modern business environment.

I know, I was agreeing with you, believe it or not. It was an extension of my previous post, about how selling an inferior product in vast quantities can be more successful than selling a small number of high quality products. Many companies are built on this concept.

MMOs are one such product, companies make money by getting a few dollars from hundreds of thousands or even millions of people, instead of making a lot of money from a few deep pockets.

Some F2P games go against this, relying on the deep pockets to provide a game when the majority spend nothing, but even then these customers do not demand a 100% bug-free environment, only one that scratches their particular itch, like Farmville. So bug testing is an afterthought in most of these games, it’s get it out the door on time, fix it later.

Computer software for the mass market in general has been doing this since Windows 95 was released. Customers for a specialized niche can afford to be more demanding, as there are not so many people out there buying to support a crappy product.

DHK.8406:

There is a lot of great testing technology available and im just curious why more companies like Anet do not adopt more advanced testing procedures.

As a dev myself, I’m curious as to how one would go about automating tests on systems like MMO combat… seems like there would be millions if not billions of possible test cases.

Most of the true bugs we see in the combat system are unintended interactions, how do you test for those?

GrandmaFunk.3052:

DHK.8406:

There is a lot of great testing technology available and im just curious why more companies like Anet do not adopt more advanced testing procedures.

As a dev myself, I’m curious as to how one would go about automating tests on systems like MMO combat… seems like there would be millions if not billions of possible test cases.

Most of the true bugs we see in the combat system are unintended interactions, how do you test for those?

That’s the irony of the process, you can’t test it because you don’t know about it… a single toon is made up of dozens of pieces that have millions of possible configurations, interacting with other toons that also have millions of possible configurations… there will always be unforseen circumstances.

I learned a little programming when I was younger, just simple BASIC and TurboPascal stuff that has long ago become obsolete. But I remember how much of a pain it was to search for typos and bugs in a simple database program, I can imagine how complex it is to debug something like an MMO, where hundreds of programs run simultaneously on thousands of different computers.

Coupled with the minimal benefits of doing so (guaranteed successful ad campaign: “Play World Of SwordMagic, we fix our bugs BEFORE patch day!”) it’s no wonder that so many bugs don’t show up until the software goes live, or that so many bugs cannot be duplicated in testing.

I dont want to get to theoretical on you But…

Yes, no system can be tested completely. In any type of test you have to come up with a balance between time to execute the tests and level of coverage. The other thing is that you dont test to catch bugs, you test to ensure the system meets its specification. So testing is not designed to remove all bugs, it is designed to ensure that the bugs are hard enough to trigger that you meet your requirements.

That said in order to test a system you have to go through the process of formalizing the system. This means breaking down each element into a stimulation and an expected response. When you do this you will find that you can break even very complex systems with millions of parts into much simpler representations . You are essentially abstracting a series of things into a stimulation or response. That stimulation could involve tens, hundreds or thousands of other things in the background. However you are abstracting it at a much higher level.

In doing so you can catch a great many obvious bugs really quickly, and you can also do rapid regression testing to ensure that you don’t break anything. That was previously working. You can also incorporate automated stress testing, as well as fuzzing to get really decent test coverage in a very limited time scale.

I would also be the first to admit that is really not something that anyone could implement after the fact, it has to be built into your development process. I just wish more companies realized that their is both cost and quality benefit to doing this.

JonathanSharp.7094:

We did test, but internal servers and external servers are not the same. Our internal tests were fine. This is why we beta!

That, in my opinion as computer science student, is a good reason.
The live servers are most likely running on a cluster, not just one single machine. Now you have a bunch of servers running there, each doing it’s stuff. A few for dungeons, a few for pvp… etc.
This makes sense as the system has to handle thousands of concurrent users. When you are playing, you are basically playing on on machine in the cluster. For your computer it looks like one server (perhaps not even that), but you are constantly switching machines.
Now according to Wikipedia ArenaNet has >300 employees: http://en.wikipedia.org/wiki/ArenaNet

From an economical point of view, you wouldn’t run a servercluster for this number. One machine can probably run this.

So when they test it, it might work, but once it goes on live, something unexpected happens when the client goes from one server to another, preventing you from joining a dungeon, a pvp match etc. etc.

DHK.8406:

I would also be the first to admit that is really not something that anyone could implement after the fact, it has to be built into your development process. I just wish more companies realized that their is both cost and quality benefit to doing this.

I think most companies realize that it does not benefit them to do so. But, that’s just my opinion… it’s the same complaint with every MMO, so either NO ONE at these companies is smart enough to realize this, or they already crunched the numbers and generally came to the same conclusion I have.

I dont think it has to do with smarts, I think it has more to do with the fact that its always been done one way and even if that way is outdated and has been improved significantly people wont give it up until they are forced.

Safety critical systems gave up the sort of testing that is done on rapid release cycle projects decades ago. Because of accidents and deaths and the bad press that comes with that. Because of that they are spending less money and manpower on testing all while tester to a much much higher standard.

Eh, sometime in the last ten years I’m sure someone said “hey, there’s an underserved market here, every MMO launches with buggy software that never quite gets fixed. My company could develop a bug-free MMO and it would be bigger than WoW!”

And yet such a product has never appeared. I think there is a reason for it. If you believe otherwise, go get rich.

Snowmoon.1758:

Even if they had little time and little budget, couldn’t they have at least made sure people could get into the dungeon? Isn’t that like the main thing that should be tested? Its like trying to buy a car, but u realize the car has no wheels.

I really don’t mind minor glitches like clipping etc, but having
the most important aspect of this retribution story (the dungeon)
not accessible tells me that they have not even tested at all.

Either that or someone at anet made a very bad mistake and uploaded
a wrong patch or something.

Sorry Anet, im already not impressed with this flame and frost quest, mainly because of having you brag it as having EVEN MOAR CONTENT THEN AN ENTIRE EXPANSION, and this bug is just unacceptable.

You have a good MMO on your hands. Don’t squander the opportunity you have by releasing these minor rubbish. (SAB is good tho, kudos to you)

There are so many ways this can happen. For example, it is ENTIRELY possible something could work perfectly on the test server and not work on the life server after uploading. It does happen.

Test servers are not live servers. The dungeon was a lot of fun. Do you really think no one on the test server tested it? How did they get into it on the test server? Probably the same way you’re supposed to.

There are other possibilities too. Like too many chefs spoil the broth. There are a lot of areas in this game that interact that aren’t obvious to each programmer working in a different area. It’s just not possible. Not only are MMOs massively complex, but also, the content development cycle keeps speeding up.

If you think that bugs like this can’t happen in any game, or any program, you have a lot to learn about programming.

tolunart.2095:

Time + Budget.

They can’t push a patch back two weeks because it didn’t test well, players will complain.

They don’t have the budget to hire more staff for testing, the best software, etc. They’d have to charge more for the game/sub and players will complain.

So they do what they can and fix the rest once it goes live and they can tell where the problem areas are. Players still complain.

No matter what they do, players will complain. For the most part, they won’t quit the game, just yell a lot. So the company can save money and players will complain, or they can spend money and players will still complain.

The option to spend more money doesn’t MAKE more money for the company, and thus is not an attractive option.

Not exactly true, plenty of F2P non sub games out there that use a PTR in their setup with both new content and balance to do REAL testing, been going on now for 17 yrs in the mmo dev community, pride is the only reason they haven’t adopted other mmo’s tried and true methods.

just look at DR plenty of other ways of keeping gold farmers out without harming players still using DR a year later.

Just look at how pets die instantly, STILL even after the update, they won’t adopt the 90% immunity to AOE rule because Blizz WoW developed it (i believe but it’s the most well known.)

tigirius.9014:

just look at DR plenty of other ways of keeping gold farmers out without harming players still using DR a year later.

I think it was said, or implied, it’s not just the gold farmers. It’s the people who just sit in one area and don’t move around. They would like people to get out and into other areas. . . rather than all congregating where the loot is best or the hunting is comfortable.

Sure, you could do it other ways. I can think of three off the top of my head. However, just because they’re my ideas doesn’t make them better :P

Just look at how pets die instantly, STILL even after the update, they won’t adopt the 90% immunity to AOE rule because Blizz WoW developed it (i believe but it’s the most well known.)

I think it’s because then pets would be way too good for certain aspects of play. If I knew my pet could take AoEs like they were nothing, I really would be letting it do most of my work.

I’m surprised the element of investment hasn’t been brought up.

As in:

You need to spend money to make money, or in this case, they need to invest in better testing solutions to save money from getting devs to spend excessive amounts of time being reactive on patch days, instead of pre-emptive spotting issues before it goes live.
So far, it’s only resulted in spending more time re-patching, the patch that broke the last patch that was patched and not working on moving on past that and going head-strong into squashing major bugs or long-running bugs that still exist.

When we’re talking about slightly bigger companies (Anet’s been stated as 300 strong), time and money are only something that needs to be better managed. Time and money are only a real issue if you’re someone like an indie team of 3-4 people, making a game that doesn’t have a constant flow of cash and you’ve been working from your own pocket up until release, or any issues along those lines.

Besides, money doesn’t seem to be an issue, because they’re actively hiring.

Sure but how much money will they have to spend to save them money. So the devs work harder on patch days. But the devs are already being paid. They don’t get paid more to work harder.

At any rate, I’m not so sure pride is the main reason Anet doesn’t have a public test server. I think they that create content VERY quickly, and I think that they don’t want to let the cat out of the bag before that content is released.

It’s very very hard to do that if everything is running on a public test server. That’s really the down side. There are no surprises in the game to anyone.

I work in automation too and the answer is pretty simple….

No lives depend on bugs in an MMO., while they’re far more critical in planes/cars … No seriously, no one ever died of an MMO bug. Crying, yes. Dying, no.

As long as people don’t die, delivering functional content has priority over making bugfree content.

marnick.4305:

I work in automation too and the answer is pretty simple….

No lives depend on bugs in an MMO., while they’re far more critical in planes/cars … No seriously, no one ever died of an MMO bug. Crying, yes. Dying, no.

As long as people don’t die, delivering functional content has priority over making bugfree content.

Are you giving a free-pass to all devs – and practically all forms of entertainment – that you can release anything at a lesser quality…because no lives depend on it.
I understand where you’re coming from but thats a pretty naive statement.

I work in the media side of things, and that would not fly. Hence why we get do multiple revisions before stuff goes live.

DHK.8406:

So it seems as if quite a lot of the time Anet releases new patches it requires 1-2 extra patches to get it all sort of working correctly. Im wondering exactly how they test.

Little Background: I work in the development of model based test automation solutions, mostly utilized in telecom and safety critical systems (cars, planes and trains) . There is a lot of great testing technology available and im just curious why more companies like Anet do not adopt more advanced testing procedures.

So I know I’m a bit late to the party here, but I mean if you truly wanted to help Anet with their testing, apparently they have a few open QA positions. :P

15 years in software testing here; American Express, FedEx, WorldCom before they went kaput, and a private banking-software company that handled stuff for the Fed, Citibank, Wachovia, the list goes on.

In the last company, our customers depended on our software to process and balance checks & ATMs. If there was a software bug because of a new release and they missed their deadline, it cost them millions of dollars in interest.

I was the only tester for two of the programs. My test cycle was around three weeks. Depending on the number of bugs fixed, it could take me the entire cycle to just go over the intended fixes. Forget about regression testing or integration testing. I’d smoke test the software to make sure the process worked from beginning to end without any customer customization (of which there was a lot), then I’d hit the “tickets” that were addressed by the devs. When the launch date hit, we launched regardless of whether I’d finished the cycle or not. Very, very rarely would I be allowed the authority to stop a build or delay it. When I got REALLY frustrated, I’d refuse to call it “certified”… and it launched anyways.

All of this is to say that this was with software that would cost customers millions of dollars if it was broken. An MMO does not cost people millions of dollars if it’s broken. I was a professional software test engineer, and gaming companies tend to hire entry-level people to test (ANet may be different, I dunno, but when I applied to gaming companies as a professional making over 60k they giggled at me and hired college students). Different industry, different priorities.

I don’t see it changing anytime soon. As long as the game doesn’t crash, explode, wipe hard drives, or prevent people from playing, I consider it a decent test cycle. Especially if they can fix the stuff in a few hours. I never, ever expect a patch for any MMO to go live without at least one hotfix.

My two dollars of contribution.

LowestTruth.2635:

15 years in software testing here; American Express, FedEx, WorldCom before they went kaput, and a private banking-software company that handled stuff for the Fed, Citibank, Wachovia, the list goes on.

In the last company, our customers depended on our software to process and balance checks & ATMs. If there was a software bug because of a new release and they missed their deadline, it cost them millions of dollars in interest.

I was the only tester for two of the programs. My test cycle was around three weeks. Depending on the number of bugs fixed, it could take me the entire cycle to just go over the intended fixes. Forget about regression testing or integration testing. I’d smoke test the software to make sure the process worked from beginning to end without any customer customization (of which there was a lot), then I’d hit the “tickets” that were addressed by the devs. When the launch date hit, we launched regardless of whether I’d finished the cycle or not. Very, very rarely would I be allowed the authority to stop a build or delay it. When I got REALLY frustrated, I’d refuse to call it “certified”… and it launched anyways.

All of this is to say that this was with software that would cost customers millions of dollars if it was broken. An MMO does not cost people millions of dollars if it’s broken. I was a professional software test engineer, and gaming companies tend to hire entry-level people to test (ANet may be different, I dunno, but when I applied to gaming companies as a professional making over 60k they giggled at me and hired college students). Different industry, different priorities.

I don’t see it changing anytime soon. As long as the game doesn’t crash, explode, wipe hard drives, or prevent people from playing, I consider it a decent test cycle. Especially if they can fix the stuff in a few hours. I never, ever expect a patch for any MMO to go live without at least one hotfix.

My two dollars of contribution.

To me…your situation felt like it was simply bad management.

nethykins.7986:

To me…your situation felt like it was simply bad management.

Four companies, two contracts, about six, seven? different bosses, not to mention interfacing with dozens of QA co-horts from other banks and vendors. I’m quite comfortable in stating it was SOP across two industries.

Well we as players are testing every thing that how mmorpg need to be. New content is put out we play it and tell Anet what wrong and what right about it and they fix it accordingly.

nethykins.7986:

To me…your situation felt like it was simply bad management.

Sure, this is like when everyone and their uncle can watch a football/hockey/basketball/etc game and point out all the flaws and pontificate about how it’s all so obvious and easy.

GrandmaFunk.3052:

nethykins.7986:

To me…your situation felt like it was simply bad management.

Sure, this is like when everyone and their uncle can watch a football/hockey/basketball/etc game and point out all the flaws and pontificate about how it’s all so obvious and easy.

Well, Opinions are funny like that…I can’t state the situation he went through was not true, I can only state opinions of what I thought it felt like from my own experiences. and reading his recollection (Hence why I said “to me” and “felt like”)

nethykins.7986:

To me…your situation felt like it was simply bad management.

It is and it isn’t, most industries refuse to adopt good testing practices until they are forced to. In this instance the small chance that a bug costs $$$ is not worth the certain $$$ that delaying release would cost.

I work in a highly specialized field of test automation. We develop tools that we sell to client mostly in telecom and embedded systems.

An MMO is really just a series of embedded systems but even though It would save them money and give them a better product, until an industry is forced… usually people stick with what they know even when it is expensive and ineffective.

Maybe they have several builds developing in parallel and, let’s face it, configuration control is hard and occassionally—those simple undocumented fixes you do every day, well, one falls through the cracks of interfacing a new block of code with the “next” build?

Chopps.5047:

Maybe they have several builds developing in parallel and, let’s face it, configuration control is hard and occassionally—those simple undocumented fixes you do every day, well, one falls through the cracks of interfacing a new block of code with the “next” build?

Well there was the time in GW1 they admitted their test environment didn’t catch something due to how it was handling code differently than the live environment. I’m sure someone can dig that up, but that got at least a quiet “what the heck?” from players who heard about it.

At least it’s not like recent companies who launched games using online connections which crashed hard on release day, right? The worst we had were horribly crammed overflows

Doesn’t ANET have an ALPHA to test builds on?

Im Mudbone.1437:

Doesn’t ANET have an ALPHA to test builds on?

Of course there’s an alpha server. What people are often arguing for is a public test server like some other games have.

Vayne.8563:

Im Mudbone.1437:

Doesn’t ANET have an ALPHA to test builds on?

Of course there’s an alpha server. What people are often arguing for is a public test server like some other games have.

Yup…mainly because issues like The Great “Precieved Loot thread” that outlined a bug, – which the devs constantly stood firmly that nothing was wrong until looking at it closer – would be found and rectified before it went live.

Or for catching really simple things that supposedly “slip the net” but are common actions anyone does in game…like the jumping bug that occurred after one patch, or some classes not being able to attack underwater after a patch…etc…etc…etc.

ok so having never played an rpg game with a public test server… how is progression handled?

In my mind I assume that you connect to it with your current set-up, but that no progress made on the test server is saved.. ie you don’t level, you don’t keep any loot, etc..

given that players are constantly whining about not getting enough rewards, insufficient loot, not enough incentives to play… who would actually spend their game time in a context where they get none of the things they already feel like they don’t get enough of?

GrandmaFunk.3052:

ok so having never played an rpg game with a public test server… how is progression handled?

In my mind I assume that you connect to it with your current set-up, but that no progress made on the test server is saved.. ie you don’t level, you don’t keep any loot, etc..

given that players are constantly whining about not getting enough rewards, insufficient loot, not enough incentives to play… who would actually spend their game time in a context where they get none of the things they already feel like they don’t get enough of?

To the best of my knowledge you copy/create a character on the test server that is raised to max level and given whatever items are necessary to run the content that is being tested. This character is separate from your normal ones and cannot be transferred off the server.

The purpose of the test server is to test upcoming content for bugs and gather feedback, not to progress your personal agenda. It’s like a permanent beta test, you’re supposed to do it because you want to help make the game better, not because you’re collecting loot. As a bonus you get to see new content before other people do.

If you have to ask “what do I get out of this?” then you shouldn’t be there.

Which goes back to my point, who would be there?

would the size of the player base willing to do this be significantly larger than Anet’s QA team?

also, having a public test server means leaking the content of game updates much earlier than currently… how do you stop it from having an impact on the TP?

Rift had a public test server. When I was on it which wasn’t that often (I did alpha testing for Rift prior to launch), it was always relatively dead. Because nothing you did on that server furthered your actual character on the main server. Most people would rather work on their characters and make some kind of progress in something, even if just gold or karma acquisition than play on a test server, for the benefit of the game.

All they’d really need to do is launch a PTR and stick all balance/spec changes in there for a month before actually pushing them out. Then they’d need to listen to feedback from the PTR players.

But feedback is only as good as the test players. I’ve seen some wildly bad ideas posted by test players in my day.

Back when I did alpha testing for Rift, I naturally had access to the alpha forums. They were not all that different from these forums, except they tended to be more respectful. lol

The point is, people who played the alpha content often disagreed about how things should work or be changed. It’s not always so obvious. Multiple opinions are often mutually exclusive, and the company is back to listening to one group or another.

I mean Rift had some crazy bad decisions even with the public test server. It didn’t seem to help at all.