Guild Wars 2

Wouldn’t you be able to implement that restriction as an extra boolean flag “has_had_sell_orders” for each record on the database itself?

The in-game trading post would then filter records where
(has_had_sell_orders = true) to prevent abuse. The public API would not have the same restriction.

DarkSpirit.7046:

I assume that the discovery process is different between /items and /prices (they better be!).

I’m starting to believe that when a tradeable item is first discovered, the game should immediately make that item known to the commerce back-end in addition to the items back-end.

Id for /items has a single meaning: item_id.
Id for /prices has two meanings: listing_id and item_id.

When /prices says “invalid id”, it only refers to one meaning: invalid listing_id. Is everyone 100% clear on these concepts?

There are actually item_ids for which a listing_id is marked invalid when it shouldn’t be. Agony infusion +15 is the perfect example. It is tradeable, so it should have a listing_id. Even if that listing has no current offers, and thus a price of 0 gold.

So the commerce back-end does not have any record of items up until the first time that they are offered for sale by a player? I think that’s the real problem.

I don’t know how item “discovery” works exactly, but that same mechanism should probably create an empty price listing for the newly discovered item.

Spoiler:
The error should be 400 across all possible implementations. The reasoning is that even though the "id" value right now is implemented as a foreign key for the "item_id", it is not appropriate for the API to return 404, because that would be a design that depends on a specific implementation.

DarkSpirit.7046:

Why do you have to add a new property item_id when there is already id?

Just so there are no misunderstandings. This object…

{  
   "id":24,
   "buys":{ ... },
   "sells":{ ... }
}

 
is semantically equal to this object…

{  
   "id":24,
   "item_id":24,
   "buys":{ ... },
   "sells":{ ... }
}

The point of my post was that the primary key (“id”) is not part of the entity. It is simply an implementation detail. Implementation details can be changed without changing the design. In my example, I split the column “id” (PK, FK) into two separate columns “id” (PK) and “item_id” (FK). Just changing the implementation, not the design.

Once you understand all that, try to answer the original (strictly hypothetical) question: if the “id” column is not a foreign key for “item_id”, what should the error code be when all given price ids are non-existent?

Everything I’ve said so far is based entirely on observations. The id is NOT the same as the item id. I can explain it to you, but I can’t understand it for you.

Let’s do a thought experiment. First, explicitly add the item_id to the output.

{  
   "id":24,
   "item_id":24,
   "buys":{  
      "quantity":1501,
      "unit_price":2144
   },
   "sells":{  
      "quantity":305,
      "unit_price":4598
   }
}

 
This representation more accurately shows how the API is structured internally (primary key == foreign key pointing at item_id).

For the experiment, change the primary key scheme to something more arbitrary. I’ll pick auto-incrementing numbers that start at 1.

{  
   "id":1,
   "item_id":24,
   "buys":{  
      "quantity":1501,
      "unit_price":2144
   },
   "sells":{  
      "quantity":305,
      "unit_price":4598
   }
}

Now that the id is no longer equal to the item_id, what should the error code be when all ids are invalid?

/v2/commerce/prices?ids=24

Assume that listing_id 24 does not already exist in the output of /v2/commerce/prices.

Your argument is based on the assumption that the listing service knows (or should know) about both listings and items. It doesn’t know about items, in the exact same way that the quaggans service doesn’t know about items either.

https://api.guildwars2.com/v2/quaggans?ids=18753

At the risk of sounding like a broken record, the commerce service uses LISTING ids, not item ids. (and definitely not quaggan ids)

It doesn’t say that there is no item with id 18753. It says that there is no listing with id 18753. Just forget that they are using item ids as listing ids. They could have picked any other number.

Okay, but these are the IDs of the actual buy/sell offers, not the ID of the item being bought/sold. An item does not have to have a listing.

You seem to be under the impression that an item will not appear in /v2/commerce unless it has outstanding buy or sell offers.

valid ids: https://api.guildwars2.com/v2/commerce/prices
invalid ids: https://api.guildwars2.com/v2/items unless also on the /v2/commerce list

HTTP 404 doesn’t exclude the possibility of the item being listed at a later time. Why are you including /v2/item IDs in your request for /v2/commerce endpoints?

There are only two categories:

• valid: ids returned by the /v2/commerce/prices endpoint
• invalid: everything else

Another thought: the ids returned by /v2/commerce are not item ids, but listing ids that also happen to be item ids. A valid listing id is always a valid item id, but not the other way around.

Items that have the “AccountBound” flag can never appear on the trading post, so 404 would be less accurate than 400.

Use ids=all to get the exact same behavior as for /v1

https://api.guildwars2.com/v2/worlds?ids=all

Thanks. Take your time.

xD sorry. What would be a more democratic delay between each request?

Could we get support for the HEAD verb? Sometimes I just want to retrieve the X-Page headers for any given page_size.

https://ochronus.com/http-head-request-good-uses/

pseudo code:

headers = HTTP_HEAD("/v2/items?page_size=42");

page_count = headers  ["X-Page-Total"];

for (index = 0; index < page_count; index++) {
    page = HTTP_GET("/v2/items?page_size=42&page=" + index);
}

Hey, is anyone else running into HTTP “speed bumps”? If I send too many requests, I start seeing HTTP 502 (Gateway Timeout) and I have to wait a minute or two before I can send new requests. Just wondering if this is a technical limitation, or if we’re being throttled.

edit

Must have been a network congestion thing. It doesn’t happen now, even though I tweaked my code to send even more requests. My best record is a full download of the items database for all languages in just under 32 seconds. I’m impressed by how well the API responds to that many concurrent requests.

There’s some confusion here about what items are shown in the search results of the in-game TP and what items are not. In particular, items that don’t match your search filters are never shown. In this case, the given item fails the “Only Show Available” filter.

tl;dr
Remove the “Only Show Available” filter, which is now a default filter for every new search.

Some map types that I was able to gather (MumbleContext mapType):

2 = PvP maps
4 = Instanced map (dungeons, story, home instance, activity)
5 = PvE (all explorable zones except Dry Top)
9 = Eternal Battlegrounds
10 = Blue Borderlands
11 = Green Borderlands
12 = Red Borderlands
15 = EotM
16 = Dry Top

I wouldn’t be surprised if there are more map types, one for every number not already on my list (1, 3, 6, 7, 8 ,13, 14).

Could the first 4 bits actually be a map type instead of a continent ID? What makes EotM different from other WvW maps?

0000 → Instance
0001 → PvE
1000 → WvW
1100 → WvW (EotM)
???? → PvP

The MumbleContext struct has a mapType property that may come in use. I don’t have any data on it though.

Getting the item details db schema right is pretty difficult. But even if you nailed the design, filling your perfectly designed database with API data has its own challenges.

Some items contain references to other entities that may not already be in your database. Examples include upgradable items that reference upgrade item details, recipe sheets that reference recipe details, and dyes that reference color details.

Trying to insert an item like that results in an integrity constraint violation, because the referenced item doesn’t exist in the database. So what do you do? You insert the referenced entity first, and then insert the item that holds the reference.

The opposite can also happen. Multiple combat items can reference the same upgrade item. Trying to insert the referenced upgrade item results in a duplicate key violation, because the referenced item already exists in the database. So what do you do? Some database engines provide an option to silently ignore duplicate key errors. For the ones that don’t provide that option, change the query to insert only if the key doesn’t exist already.

Long story short… the code can (will) get pretty messy, and it’s not your fault.

That makes sense. Dungeons are instanced, so they wouldn’t put them on a megaserver shard.

I still don’t know about the other maps though.

Some shard_id values that I recorded so far (logged in from the Ring of Fire server):

10000000000000000000000000000111, Eternal Battlegrounds
11000000000000000000000000000101, Edge of the Mists
10000000000000000000000000000111, Borderlands
00010000000000000000000000000100, Dry Top
00010000000000000000000000000001, Queensdale
00010000000000000000000000000001, Gendarran Fields
00010000000000000000000000001011, Harathi Hinterlands
00010000000000000000000000000001, Lornar’s Pass
00010000000000000000000000001010, Divinity’s Reach
00010000000000000000000000001001, Lion’s Arch
00010000000000000000000000000001, The Grove
00010000000000000000000000000010, Rata Sum
00010000000000000000000000000001, Black Citadel
00010000000000000000000000000001, Hoelbrak
00010000000000000000000000000001, Southsun Cove
00010000000000000000000000011111, Sparkfly Fen
00010000000000000000000000000011, Straits of Devastation
00010000000000000000000000000001, Caledon Forest
00000000000000000000011111010101, Twilight Arbor

There is definitely a pattern here. I’m just not seeing it. Maybe if we can complete the data for all maps.

Has anybody gathered any new information over the last couple of weeks? I already know that the value of “world_id” is now set to the same value as “shardId” of struct “MumbleContext”. It appears that this value is to be used as a bit mask, not an actual ID. I can confirm that the “shardId” and “worldId” change as you move between different parts of the game, but I haven’t cracked the pattern yet.

That’s not how that works. There is no numeric data type that uses 17 bits in any programming language that I know of. The next step up from 16 bit is usually 32 bit, so you have to design for that. Furthermore, my request respects every precondition that is imposed by the API. What about my request makes it “junk”? The fact that my identifiers don’t exist? Maybe I know that, but my code doesn’t.

The maximum length of a valid query string is exactly 2207 bytes.

• 2000 bytes for the total length of all IDs combined
• 199 bytes for the delimiters
• 8 bytes for the optional “&lang=xx” parameter

Valid in the sense that it satisfies all preconditions. Not valid in the sense that any of the IDs actually exist. So the filter should be adjusted from 2048 to 2207. The power of RFC 2616 compels you!

It’s just weird having to write workarounds for limitations that exist only at the transport level. Validation should only occur at the API level. And for that purpose, it shouldn’t matter if the request is encoded as a query string, or flown into the office on a piece of paper by a carrier pidgeon.

The one thing nobody has addressed so far is the 10 digit precision problem.

I can devise a request that passes validation for every rule in place right now:

• No more than 200 identifiers
• All identifiers are numeric, 32 bit integers
• No two identifiers are the same

…yet my request will result in a crash.

http://pastebin.com/Pipvc4VS

I understand perfectly well why this is happening. You need to understand why this shouldn’t be happening. Not unless you introduce another validation rule that invalidates my request regardless of the maximum allowed query string. But what sort of rule would that be?

Keep in mind that they have to be API-level validation rules. That means forget that the query string even existed before it reached the API. So no checking for length.

Let’s break this down into manageable chunks.

Alcarin.9024:

You cannot have a json response from the API if the API cannot read your request. Your request is stopped by ISS, API never know that you made a request.

This behavior is caused by a security feature that can be disabled. I’m not trying to steal your data, so why this has to be enabled is beyond me.

Alcarin.9024:

If you get a 4xx error, you have to understand why your request is wrong, not why you dont receive an extensive response from the server that explain why your request is wrong.

The request isn’t necessarily wrong. Look at /v1/world_names.json. It returns a 4xx code even though the request is perfectly fine.

Alcarin.9024:

In this case your request have 2 errors, url is longer than maximum allowed, and you are requesting more than double (550) of max number of items requestable in a single request (200).

Really think about why we should get a proper error response for 201 IDs, but not for 550 IDs. What is the difference? I’d be glad to hear it.

Alcarin.9024:

Here the problem is in your request, not in the reply from the server, you should be sure to not do such type of request, because you know that those requests are not allowed.

You have to draw a line somewhere between what is and isn’t violating the interface. Sending more than 200 IDs is disabled for many good reasons, but it is nevertheless a syntactically and semantically well-formed request. Responding with text/html when the protocol is to respond with application/json however is a clear violation.

Guys, the problem was never the status code. The problem is the text/html content-type along with the HTML body.

if (!response.StatusCode.IsSuccessStatusCode())
{
	if (response.ContentType.StartsWith("application/json"))
	{
		// Throw an exception with a meaningful error description
	}
	else
	{
		// Crash with a meaningless error description, because the content-type is not JSON
	}
}

Having client side validation is good. Only having client side validation is bad. Our library comes with options to disable parameter validation in release mode (to save some CPU cycles), so there will still be times when a user passes some ridiculous value that doesn’t work, but doesn’t actually violate the protocol.

I can stop a user from passing more than 200 identifiers, but I can’t stop a user from passing 200 times the maximum value for a 32 bit integer (2147483647). The precision here is 10, so that alone adds up to 2000 characters. That’s still less than 2048, but then you’re forgetting about the delimiters.

I think this behavior is caused by a setting in the Request Filtering module (http://www.iis.net/configreference/system.webserver/security/requestfiltering). The article perfectly matches the behavior that I’m seeing.

When request filtering blocks an HTTP request, IIS 7 will return an HTTP 404 error to the client and log the HTTP status with a unique substatus that identifies the reason that the request was denied.

So, to disable the filter:

• Open IIS Manager and connect to the gateway server that routes the requests
• Click the root node of the server in the “Connections” panel
• Double click “Request filtering” in the “Features View” panel
• Click “Edit Feature Settings…” in the “Actions” panel
• Change the maximum URL and query string length to 4294967295 (max), or something more sensible than 2048/4096

I’m not sure if this is even worth looking at, but there’s still that bug where IIS crashes with a text/html response when the parameter list gets too big. Probably because IIS/ARR and the private API do not agree on the maximum URL length.

Example: http://pastebin.com/nXSR8kSb

If I’m getting this right, the idea is to have a program running that periodically scans the official API for previously unknown constants. So unlike the wiki, the custom API would not require manual intervention.

I can’t check on my phone, but I don’t believe that ID 1 returns an error code. Just HTTP 200 and a garbage value.

That’s just our error logic tripping over the HTML response when it was expecting JSON instead. Nothing much that I can do about besides ensuring that the content type header is application/json before attempting to parse the error content.

I uploaded version 0.9.11.0 of the library. This is a bugfix release that, among others, fixes the problem with the ‘FindAll()’ method on the prices service.

https://gw2dotnet.codeplex.com/releases/view/133644

Pat Cavit.9234:

StevenL.3761:

/v2/commerce/listings returns literal “null” strings when a bulk request specifies non-existent or account-bound item identifiers.

https://api.guildwars2.com/v2/commerce/listings?ids=-2,-3,6,24

You might have to refresh the page a couple of times to see it.

I tried two different serialization engines, and both fail to correctly recognize these values as null.

We’re taking a look at this and have a plan for how to fix. That’s a fun one, good catch!

It’s better now, but there are still some inconsistencies that make my code crash.

Right now, the listings service returns garbage values instead of an error for some identifiers:

OK: https://api.guildwars2.com/v2/commerce/listings/0
Not OK: https://api.guildwars2.com/v2/commerce/listings/1

Same story for the prices service.

OK: https://api.guildwars2.com/v2/commerce/prices/0
Not OK: https://api.guildwars2.com/v2/commerce/prices/1

• Open your solution
• Select your main project in the solution explorer
• Project | Add Existing Item… | GW2.NET.csproj
• Project | Add Existing Item… | GW2.NET.Core.csproj
• Project | Add Reference… | Solution | Projects | GW2.NET, GW2.NET.Core

edit

I see the problem now. The GW2.NET.Core.csproj configuration file wasn’t source controlled for some reason.

You only have to reference GW2.NET.dll and GW2.NET.Core.dll for that. Don’t reference the Contracts.dll libraries. You can reference other libraries, but you don’t have to.

I’m taking a look at it. Bug confirmed, but I’m not sure why this is happening. Both “Find()” and “FindAll()” use the exact same serialization engine. The only difference is that one object is part of a collection, the other isn’t.

Edit

Turns out it is a copy-paste-programming error. The ‘FindAll()’ method is talking to the /commerce/listings service instead of the /commerce/prices service. Thanks for the feedback!

I would prefer schemas over custom converters. I think that you could write a schema that only matches objects with a specific value for the “type” property. So you’d write one schema for each possible type, then use those schemas to determine which class to use for serialization.

Your documentation was very helpful though. In the end, I ended up with this class layout: https://gw2dotnet.codeplex.com/SourceControl/latest#GW2.NET/Code/GW2.NET.Core/V2/Items.Json/DetailsDataContract.cs

Can you give me the collection of identifiers that causes the problem? I’ll take a look at what’s going on.

Does the listing have buy offers but no sell offers? Discard that listing. It’s not supposed to be listed.

Hello again

You can now download version 0.9.10.0 of the library. This release adds support for the /v2/items service, and fixes some issues with the implementation for the /v1/item_details.json service.

https://gw2dotnet.codeplex.com/releases/view/133598

Enjoy!