Guild Wars 2

Hello ANet,

The past 3 days have been really rough on you, hasn’t it?

I myself have gone the full circle of disappointment, rage, pain, withdrawal, and acceptance, not only with your game, but more relevantly with the quality of service you are providing. This is my attempt to have closure with the horrible nightmare of the past 36 hours – by giving you a point by point constructive feedback based on one of the most successful and widely used service management paradigms.

To give you a background on why I think I am qualified to give this opinion, I work for one of the top three (in terms of IT services, Server and Enterprise hardware, personal and consumer PC’s and laptops) multinational companies in the world. I will not name my company, but let’s just say that if you ever used a printer, chances are, it was made by the company I work for.

One of the things that are ingrained in every single employee of my company are the concepts of ITIL. Having said this, I am ITIL Foundations and ITIL RCV (Release, Control, and Validation) certified – but really, you don’t need to be certified to realize this because a lot of the points I will raise below are common sense.

You’re not a “young” software company by any means, so I believe that there will be those among you (probably not the developers per se, but your project managers and leads) who have heard of ITIL.

If you need a comprehensive overview, the article on Wikipedia is excellent, and I will be using this document as my primary guide in providing feedback and suggestions as this is very accessible to the public: http://en.wikipedia.org/wiki/Information_Technology_Infrastructure_Library

While I will do a rundown of every major ITIL process, I will comment only to the items which are (painfully) obvious to us, consumers of the service you are providing. I will rate the game and your service on the following scale:

N/D = No Data/Not Evident to the Public
1 = Poor
2 = Mediocre
3 = Acceptable
4 = Good
5 = Excellent

1. Service Strategy – Service Strategy relies largely upon a market-driven approach. Key topics covered include service value definition, business-case development, service assets, market analysis, and service provider types

List of covered processes:
a. Strategy Management – ⁴
b. Service Portfolio Management – [N/D]
c. Financial management for IT services – [N/D]
d. Demand Management – ⁴
e. Business relationship management – ³

My Overall Rating: ⁴

Comments: You did a good job in this area. In a landscape quickly filling up with WoW clones, you took advantage of the fact that your primary competitor’s age is showing, and while the service and game they provide is top class, you identified the need for a fresh take on established norms. This in turn was translated into an equivalently tempting value proposition to the market and have subverted long time fans and subscribers from WoW into your game (me included).

2. Service Design – Service design addresses how a planned service solution interacts with the larger business and technical environments, service management systems required to support the service, processes which interact with the service, technology, and architecture required to support the service, and the supply chain required to support the planned service.

List of covered processes:
a. Design coordination – [N/D]
b. Service Catalogue – ²
c. Service level Management – ²
d. Availability Management – ⁴
e. Capacity Management – ²
f. IT Service Continuity Management (ITSCM) – [N/D]
g. Information Security Management System – ⁴
h. Supplier Management – [N/D]

My Overall Rating: ²

Comments: Let’s kick off with the good stuff. You clearly did well in terms of planning for availability. Your servers are hardly down given that the game is still in the first half of its first year. The game is very reliable during normal operations (i.e. this process does not take into account abnormal circumstances like the recent lost shores event). Another good area is ISMS. You continually communicated with your customers with regards to account security. You took proactive steps to ensure that hacking is kept minimally. You alerted users who are using e-mail accounts which have been externally compromised. You provided services (e-mail authentication upon changing of login IP addresses, mobile authentication) that contributed to keeping your customer’s information secure.

One of the key areas for improvement is Service Level Management. While you, as the service provider, does not have explicit or contractual obligations to your customers apart from what has been specified in the EULA, there is an implicit expectation from consumers that a certain level or quality of service should be provided. Any barriers that decrease this level of quality should be acted upon quickly, and service levels should be restored as soon as possible, or in the situation where this is not possible or realistic, proper communication channels are created aand expectations are set with your customers.

As evidenced not only by the lack of clear communication during the recent event, but also by the lack of transparency in acknowledging various issues with the software and service and providing a clear direction from your team moving forward, I have to give you a mediocre score in this area. You are not the worst of the worst since the game is still up and running and functioning nominally, but there is certainly a lot of room to improve upon.

Another key area for improvement is capacity management. This is different from availability management in the sense that while your service is consistently online and available, the service drastically drops in quality in correlation to changes in concurrent connections. Capacity planning is an analysis based on expected demand vs available resources. From a service-oriented point of view, this does not cover only technical areas (e.g. “horrible lag”) but encompass other areas as well. As a customer, I have the impression that your company underestimated your subscriber base and staffed resources that could not meet the demands of a much larger volume of users. This is evidenced by the very slow progress in fixing class bugs and issues, the staffing of forum moderators who are probably consistently overwhelmed by the number of forum posts they have to monitor, and the inability to provide timelines for project completion (which is another interrelated symptom of poor project management).

Finally, and while not as important operationally – but equally if not more important strategically, is your Service Catalogue. Your service offerings are extremely basic. I know that you have already announced the introduction of more services in the future, but the lack of a communicated timeline for when these services will become available (e.g. Name Change, Paid Server Transfers, etc) is hurting customer impression of continuous improvement in this game. Also, I find it strange that while your revenue stream is partially (or perhaps largely) dependent on the cash shop, the items you sell there are not interesting enough for me to want to buy them. I would like to see cash shop weapon skins that can be used in combat, cash shop armor skins that can be used in combat, mini pets that do not eat up the slots in my bag; these are things that will not make my in-game character more powerful per se, but will make me want to spend real life money on these items. As it stands, the only valuable items I can buy for gems are inventory upgrades.

3. Service Transition – Service transition relates to the delivery of services required by a business into live/operational use, and often encompasses the “project” side of IT rather than “BAU” (business as usual). This also covers topics such as managing changes to the “BAU” environment.

List of ITIL processes in Service Transition (ST):
a. Transition planning and support – ³
b. Change management – ¹
c. Service asset and configuration management – [N/D]
d. Release and deployment management – ⁴
e. Service validation and testing – ¹
f. Change evaluation – ¹
g. Knowledge management – ²

My Overall Rating: ²

Comments: Let’s focus first on what has been done right – you have great release and deployment management. There are very few, almost rare reports that updates to the game data stored locally incurred negative impact. The deployment process as it is visible to the customer is painless – we start the application, and the game updates itself. There is no need to download files elsewhere, and the updates do not corrupt our game and make it unplayable (something which has often happened with other MMO’s – even WoW).

Now let’s focus on the (pink) elephant in the room – the underlying cause of the myriad issues that so many people in the pages of these forums have been crying about day in and day out, but have been minimally acknowledge or addressed.

Change Management. Change management ensure that changes you perform on the application OR on the service do not negatively impact the quality of service being provided on the production environment. In companies practicing ITIL, a change manager is a role performed by a person, and that person ensures that any form of modification in the client/application code, server code, infrastructure, or even in the service being provided is approved thoroughly by all stakeholders, have passed the necessary quality control gates, and is fit to be moved to Production.

As a person who has been in the IT industry for several years, working both with SME’s and giant multinational clients, let me just say that frankly, sirs and madammes, you have one of the worst change management systems in place, and it’s painful not only for us – your customers, but it’s obviously painful for you as well. You have bugs that you have no idea how to resolve, probably buried in tons of code changes you do not have proper control of, and I feel that even though your content designers and developers are doing their best, they are not doing it efficiently. If this were not the case, there would not be kilometric bug lists in class forums that, up till now, have not yet been significantly shortened or addressed. Another obvious symptom is that your bug fixes keep creating further bugs (case in point, Mesmer berserker damage nerf due to a bug created by a recent bug fix).

By practicing correct and appropriate change management processes, your organization is forced to track changes through the application lifecyle. Your technical leaders gain visibility on what proposed changes can harm working status quo in production, your developers and testers (if you have any, because it seems that you don’t – but I’ll get to that later) are forced to perform systems integration testing, no negative impact testing, and other quality control measures that will ensure that the change you will bring to Production will work, and will not create more problems.

Testing. This area encompasses QA, UAT, and other forms of QC and testing. Let me begin with words of good will as such – “I know you are trying your best.”

Let me then follow this up with the following words, “But there are times that you have to use your brain, not your brawn.”

Frankly, ANet, I do not know where or how to begin discussing this with you. I am coming here from a state of extreme apallment. WoW has 10 million++ subscribers, and this is one area that they are doing very well in so there’s really no excuse. I don’t know if this is a command decision from some higher up in you company to push changes with the full knowledge that they contain very obvious, embarassing bugs, or if there is simply a lack in compliance to industry standard quality control processes (I’m really hoping it’s the latter, the lesser of two evils, because if it is the former, then it becomes a question of morals and ethics of people leading and making decisions in ANet).

Let me give you two very obvious instances of what I deem as shocking violations of any form of quality control or testing.

i) On my engineer, there is a trait called Deployable Turrets (http://wiki.guildwars2.com/wiki/Deployable_Turrets). The expectation set by this trait is that ALL of your turrets can be deployed using the ground targeting mechanic. I really don’t know what kind of testing you do, but a person playing an engineer figures out that something is really really wrong with this trait the very instant he gets it. The trait only works for two out of the four turrents Engineers have. There’s no complicated situation or circumstances or set of events that have to transpire in sequence for this bug to occur. It’s right there, in your face, and it’s not working. The fact that you were able to make this work for two of the turrets, but not all of them, actually raises more questions – what kind of testing did you perform? why has this not been addressed after more than 5 months? do your devs even play the classes they make? and so on.

ii) on my necromancer, there is a trait called Greater Marks (http://wiki.guildwars2.com/wiki/Greater_Marks). Again, this is a no-brainer. Thankfully, the effects of the marks are increased, but again, very obviously, there is a bug in the targeting reticle.

I have cited these two instances because these are cases where you can’t miss the bug even if you purposely wanted to. They are very very obvious, and should have been caught and fixed in testing, and as a customer, the thought that these in your face bugs still remain after such a long period of time makes me apprehensive about the level of commitment your team really has in bug fixing.

It would have been more understandable if these were subtle bugs wherein you need to do several steps to reproduce it, or if they were not immediately apparent to the player unless you go through the combat log and check it.

Service Validation. Take note, while this is one line item along with Testing in the list above, service validation is actually a different concept from testing and I will discuss this separately. Service validation means analysing, across your service lifecycle, whether your service (as a whole, not just your software or servers or databases) can support the design and service goals created during Service Design phase.

Let me point out evidence of how you’re not doing so well in this area. Since a few months ago, players have been reporting that there are certain steps in certain personal story questlines that are impossibly hard to do alone. Jeffrey Vaughn, content designer, responded to the community by asking us, the players, to cite which specific instances this happened. The surprise here is that there is an obvious, specific and consistent set of circumstances where encounters need to be toned down in difficulty – and a content designer should not have to crowd source this information. A simple scan through the event scripts, or database player activity logs, or something should already provide a clue which encounters need to be toned down. A properly designed service should provide tools for the teams operating and supporting the service (e.g. Content Designers) to appropriately correct, fix, or modify application behavior that revolve around a specific criteria.

How is this related to Service Validation? The situation I illustrated above simply shows that your content designers are not empowered with the correct tools to align operational behavior with design goals. Jeffrey wanted to re-design the personal story quests to be more forgiving, but he couldn’t do it efficiently and independently to the extent that he has to acquire this information from the players instead of just consulting some tool on his end that could provide him details on which quests contain waves of mobs of size N or greater, or which quests most people died on.

By performing Service Validation, project leaders (prior to release) and operation managers (in live environment) are able to identify gaps in the service that do not support, but rather deviate, from the design goal, and appropriately fix these. Tying into the previous example, if Service Validation was done, then people would have had foresight that there might come a time when they have to modify a set of quests in personal story lines with a specific criteria, and ensure that the appropriate tools and processes are deployed on to the people who can resolve these issues (in this case, the content design team).

Knowledge Management. This is the process of ensuring that knowledge gained within the organization is documented, stored, and used to improve the service, guide decision making and strategy planning. ANet, you already have one very fantastic resource to get inputs from – these forums. People have been providing constructive feedback and criticism, and here and there, some acknowledgement from game designers can be found. However, it seems that Knowledge Management process is not being used because this information is not being translated into results. You already know the feedback to one-time events from Halloween and from other games. You already know what happens when you hold an event in a single place. You already know (or I hope you do know) that killing Champion Karkas are boring as watching paint dry. The bigger question now is, why are these information not being translated into better decisions?

By collecting, organizing, and categorizing information from various sources available to you (forums, customer feedback, incident tickets, server load and capacity, business intelligence from your databases, etc), you should have been able to design the events better – and by better I mean in a way that is enjoyable to the majority. Sure, the new area is lovely, but at the moment, they’re just eye candy – all form, no substance.

Change Evaluation. Let me try putting this in my own words – the reality is that IT companies do not have enough budget, time or resources to implement all requested changes. This is where change evaluation comes in. Change evaluation is how you decide which changes you will prioritize based on expected risk (impact to live environment), cost, urgency, and importance. I do believe that you are performing Change Evaluation, but my personal opinion is that you’re not doing it correctly, and that you’re doing it to the detriment of your company.

Let me explain. In relation to bug fixes, I would like to call in question your team’s prioritization in development efforts. I do not pretend to know why you choose to prioritize class balance changes over bug fixes (as class balance changes are not as urgent or as critical as a majority of reported bugs requiring fixes), but let me tell you the impression that this is creating in the minds of your customers. It says to your customers, “we really don’t care if your class is broken, we care more that class X gets nerfed while our favorite class Y continues to be viable in PvE, PvP, the moon, the stars, and the universe and galaxies beyond” (apologize for the barely masked contempt but we all know X = mesmers and necros, Y = warriors).

Even the way classes are nerfed do not seem to undergo proper change evaluation in terms of strategic planning. Classes are nerfed due to PvP, but no consideration is given for the PvE aspect. Bugs that continually make certain concept builds unworkable, in the meantime, still continually collect dust in the forum bug lists.

My suggestion is this – more transparent communication. Class balance is a very very delicate issue. Players tend to identify strongly and vehemently with the classes they are playing. You should not view this as negative by any means, but rather, pat yourselves on the back for creating classes that people want to play. Leverage on this, and make it work for you, and for us.

If you are going to make changes, walk us through your thought and design process. WoW made this same mistake in their early years where it seemed like they were just swinging the nerf bat repeatedly on certain classes, but later on, their game designers switched tunes and started communicating more openly, even inviting discussion with the community on why certain abilities have been nerfed. Guess what? I, for one, learned to trust them again. This is what you have to do. You sorely need to earn the trust of your player base back. Between the Lost Shores fiasco, the mile long buglists that remain unresolved, and the heavy-handed moderation in these forums, your customers are looking for reassurance that they made the right choice – that this game is still worth their time and money.

Let me digress a bit. I am trying my best to be as objective as possible while I type this, but let me just say that if I could write a heart-rending Shakesperean impassioned plea – it would be a plea for this – transparency. Tell us what your plans us so that there will no longer be any second-guessing, no more class forums with redundant threads discussing the same things over and over and over again wondering about the if’s and why’s and when’s with no resolution in sight.

That being said, let’s move on to the next area.

4. Service Operation – Service Operation (SO) aims to provide best practice for achieving the delivery of agreed levels of services both to end-users and the customers. Service operation is the part of the lifecycle where the services and value is actually directly delivered.

List of covered processes:
a. Event management – ²
b. Incident management – ⁴
c. Request fulfillment – ⁴
d. Problem management – ¹
e. Access management – ³

My Overall Rating: ³

Comments: As is the pattern, let me focus on the positive side first. A quick check in the Account and Technical Support forums should already give you an idea of why Incident Management and Request Fullfillment got the highest marks in this area. Gaile Gray and her team handle issues and address concerns in a very professional, orderly manner. A user-friendly system exists (support.guildwars2.com) where users can raise account-related and tech-support-related incidents. I have raised some tickets to the support queue as well and I find the resolution times and handling of my concerns acceptable.

I gave a score of 3-Acceptable for Access Management. I want to give more, but Access Management deals with ensuring that only people who have the correct access and authorization can use the service, and the continuing problem with spammers (and to a lesser extent now, botters) put a dent in this. However, improvement in this area have been noticed. Please keep up the good work.

Event management deals with an organization or a system’s ability to respond to exceptional conditions. This is different from capacity management in that event management concerns itself with the process and structure of escalation/resolution in case of abnormal conditions (any type), while capacity management deals with planning for expected capacity. Let me put this in the context of the recent Lost Shores weekend.

From a capacity management point of view, the analysis should have come first and foremostly from the Marketing deparment. The Lost Shores event was a widely announced and hyped event. Given this, the capacity planner should have taken concurrent system logins + total number of GW2 accounts sold and registered, and used this to analyze how much capacity the servers can handle versus the expected number of people attending the event. If there were doubts that the system can handle it, this should have been raised to Game Design (to design the event in such a way that it will not eat up bandwidth or processing load) and/or Infrastructure Support (to make sure that all servers are ready and able to handle expected load) and/or Procurement (to purchase more servers to handled the load). In short, capacity management happens BEFORE the fact.

Event management triggers during the abnormal state. It ensures that controls are in place to escalate issues and restore normal operations. I am giving a rating of 2-Poor in this area as evidenced by the slow/non-response of ANet to the bugs in the recent lost shores event and the community dividing issues surrounding the event, and the continuing silence/non-response to customer feedback on the event.

An event management system that works is evidenced by transparent communications from leadership with regards to the current status of the system, a reassurance that normal operations will be swiftly restored, and most importantly, concrete action within an acceptable (to the customer) time period.

Finally, problem management. What could I say here that I have not already said in the Service Transition section? Oh yes. Please have a proper Problem Management process in place. I don’t know if it would be too much to ask but give us BUG ID’s we can track. This way, we know if, indeed, “hard work”, as Robert Hrouda would put it, is being done. It would give your customers concrete reassurance that these issues are acknowledged and are being addressed. That would also greatly help your forum moderators by alleviating the sheer volume of topics being opened due to game bugs.

We, as your customers, are as invested in the improvement of this game as you are. We want to see those bugs fixed definitely more than you do since we are the ones who paid for the game. I say this with no sense of entitlement, but with the common sense logic of wanting to get what I paid for.

5. Continual service improvement (CSI) – The perspective of CSI on improvement is the business perspective of service quality, even though CSI aims to improve process effectiveness, efficiency and cost effectiveness of the IT processes through the whole lifecycle.

My Overall Rating: [N/D]

Comments: This is purely internal to your organization. I’m just hoping you have this in one form or the other because this is just as important to your organization as the rest of the other parts of the Service Lifecycle.

—

A Special Note to the Forum Moderators:

Dear Moderators,

I am trying to be as civil and constructive as possible in this thread by providing feedback and suggestions based on what I am being paid in real life to do, and I am imploring, as a fellow human being, that you do not yet again mistake the content, intent, and context of this post as a “complaint” thread as have been repeatedly done in various posts throughout this weekend.

Believe it or not, the headaches you have been trying to assuage in these forums all weekend long are tied in one way or the other to one or more issues with organizational, development, and project management processes related to those detailed above. It is unfortunate that you’re getting the tail-end of the sting by being the first line of support, but let me just remind you that as forum moderators, you should be working WITH your community to bring harmony to this forums, and not against it.

My only request is, should you deem it necessary to close this thread, to ensure that you communicate what I have written here to someone within your organization capable of enacting change and making decisions instead of just relegating it to the forum trashbin as you have done countless “complaint” threads that actually contain valid feedback.

Everything I have said here are observations, feedback, and suggestions, and you would not hurt to evaluate their credibility, and assess the improvement opportunities that your organization can garner from this feedback.

This thread took 4 odd hours to write. Please do not let that effort be in vain. This should be proof, among others, that I DO care for this game, and I WANT this game to be better.

—

P.S. It is 5 in the morning in my timezone. I apologize in advance for any typos, wrong tenses, spelling mistakes, etc. English is not my first language – I’m just saying that just to clarify, not to excuse myself from poor grammar.