Guild Wars 2

Mystic Forge recipes are not in the API. Only crafting.

Unfortunately many of the mystic forge recipes are still shrouded in mystery and even the current information on the wiki is not entirely correct.

I may be wrong, but I assume ArenaNet would not release those information as they prefer players to experiment on the mystic forge and share info.

Please bring mystic forge recipes to the recipes API! Please! It gets annoying :o

DarkSpirit.7046:

You would have to match the output_item_id from the recipe list.

But that would mean you have to download all recipe_details of each recipe to search for the output_item_id of the item that you need to craft. Since you can only request the recipe_details of 1 recipe at a time it is going to be a performance hit for both server and client.

It would only be a performance hit if you made an API request each time . Guild Wars 2 doesn’t release new items that, so you could easily create a cache, or alternatively create a database and sync it with theirs (using the API). This would even eliminate the need for future requests.

DarkSpirit.7046:

I suggest we either:
1. Allow output_item_id as a parameter to recipe_details.json.

Yes please.

DarkSpirit.7046:

2. Like LadyRhonwyn suggests, have a recipe_id field in the item record. That would also mean having a unique value for this field in items that are not craftable.

This could be a bit tricky, as we don’t know how the implementation details of the way items are handled, it’s very variable, but it could cause stress on the server.

DarkSpirit.7046:

3. Have a way for recipe_details.json to return all recipes in 1 call. Leaving it to the client to search for the required output_item_id.

This would create a similar problem that you’re trying to fix:

Even though the amount of data per recipe is quite minuscule (I’ll estimate it’s between 200 and 300 bytes), imagine multiplying that by 7620 (at the time this post was written), this easily blows up to 1,905,000 bytes (~1.8 Mb), which is a much much bigger load.

However, the very size is not the only problem, but the problem is that someone could misuse this feature, to make a whole bunch of requests repeatedly (which adds stress on the transfer layer and the database for GW2 servers).

Of course, you could argue that this type of request could be made once after each update and used for syncing or caching the data (like a similar observation to your first suggestion), but they can’t be certain users would use it that way, which leaves them at risk of increasing load on their servers.

Your last idea would be very good, if ArenaNet decided to implement the practice of API points (or something similar).

TL;DR: Batch detail downloads, queries for only objects that have changed.

I have a number of suggestions on the technical/feed side that would significantly improve mobile app performance and capabilities. These are largely motivated by the fact that mobile applications really to keep data from the API (items, recipes, etc.) stored locally in a database. This is for a number of reasons.

1. Mobile applications should really be usable without a network connection. That’s pretty much becoming a standard requirement for apps these days.

2. Functionality I’m implementing would prohibitively expensive in terms of network turnaround. Right now as I’m testing it takes almost 20 minutes to download full item detail for the ~30k items in the game. That’s not even including images, or the ~6k recipes to make them. So advanced search capabilities (filtering by rarity for example) would be impossible without the full item set requires all the data if there’s no search API.

3. Hitting your servers for information that rarely (if ever) changes is an unnecessary load on your infrastructure. Reducing the number of requests would simply make the service faster for everyone and save you infrastructure costs. Given the search example, I can do search much faster and efficiently with local data (and with better functionality).

4. Mobile applications should able query for changed data (item details, for example) periodically so that we don’t have to re-seed the local database and release an app update. User’s simply aren’t going to sit around for 20 minutes while the app reloads everything and sees what changed and that’s the only option right now.

So, given all that, here are my suggestions.

1. First off, for v2 of the API I would suggest significant changes to the endpoints, response formats, etc. It would make use of the API much clearer, reduce the number of endpoints, and make applications more robust. Feel free to PM me if you want to talk about that.

2. Any API endpoint that returns a large number of IDs that require another endpoint call for the actual object detail (items for example) really needs to have a variant that returns all of the detail for the items in one response. Having to do 30k item detail endpoint calls must be crushing your servers and certainly not performing well for application users considering it takes 20 minutes to download.

3. Add a changedSince query parameter that specifies returning only ids/objects that have changed since a certain time. Unfortunately there’s no standard URL-friendly format. I’d suggest YYYY-MM-DD in UTC. If your database doesn’t keep track of when an item is updated, you might want to try to convince your DBA’s to add one. It’s a pretty standard content management feature.

4. Add an endpoint that returns an array of all the IDs/signatures that are valid for the render service. Add a changedSince parameter as well. If that’s not feasible, add it to the render endpoint call and return an empty document to avoid the ~6k download and the need to compare it on the client side to the stored asset.

Sorry for the wall of text. I think that’s about it for now. These features would make mobile developers’ lives much easier and a much better experience for end-users.

Keep up the good work.

Lazarus.9716:

Right now as I’m testing it takes almost 20 minutes to download full item detail for the ~30k items in the game.

Wait WHAT? Only 20 minutes? A full update takes us about 20 hours (for all 4 languages of course at ~2-3 seconds per request, wiki check and recipe update not included)
I still think, the most important feature would be an API which provides a list of the changed items/events/recipes since the last patch, so that we can safely do incremental rather than full updates after updates where we don’t exactly know what has changed.

Yeah, the items call returns in about 2-3 seconds, and the ~30k details requests process at ~24/second. I’m only doing English right now as I just created the data model, and also not doing recipe or other ancillary requests. But it sounds like your performance is way off mine. It’s possible that they’re throttling based on requesting IP address or domain if you’ve hit their service a lot or for a while.

But, yeah, change information would be good for us and for them. The biggest thing, though, is the N+1 endpoint calls. My rough estimate is that a single response would only be about 18MB, could easily be cached on their side and would download in seconds rather than minutes or hours in your case.

The 20 minutes, btw is on ethernet on my dev machine. I would hate to even think how long it would take on LTE or even 3G mobile service. (Remember, we yanks live in the mobile stone age compared to Europe.)

Here is what I did for my trading post app, I downloaded the entire items and recipes databases into JSON files then I deserialize them async upon app startup. Chances are, the items would not change but more items could be added at a later date. If I find an item in the TP that is not in my items database, the code query the API server on the Internet. If the item is found from the API server, then my database is updated automatically.

This way, the app “learns” and updates its database through usage. And of course, I also try to release new versions of the app with updated items/recipes databases from time to time.

The obvious gap in a system like DarkSpirit’s is, of course, when an already known item/recipe changes. This happened to a lot of them on Sept. 3, when all existing equipment with Magic Find had their attributes removed, upgrade components that affected MF had their attributes/effects modified, and the recipes that crafted MF equipment were updated with a new output_item_id.

This is the gap that most of us (I assume) are covering by re-downloading the full item/recipe APIs after every release, just to be certain we catch all possible changes. Any system that could tell us which items had changed would make this step much more efficient. One option that I remember being mentioned a while back is to add an MD5 hash of the item/recipe’s JSON data to the API (maybe in a new endpoint like items_md5.json/recipes_md5.json). That way, we can easily tell which items/recipes changed by comparing their MD5 hash between releases.

That is exactly what I do. I download the json and get an md5 hash. I then compare to my database to see if the hash changed. If it has than I update.

Only one problem tho. I still need to download every item detail.

Adding an md5 hash to the items.json would be a huge help.

Dr Ishmael.9685:

The obvious gap in a system like DarkSpirit’s is, of course, when an already known item/recipe changes. This happened to a lot of them on Sept. 3, when all existing equipment with Magic Find had their attributes removed, upgrade components that affected MF had their attributes/effects modified, and the recipes that crafted MF equipment were updated with a new output_item_id.

I would call that an exception rather than the norm. It is rare for them to change so many items at one go. Having said that, it is probably wise to do a full item/recipe update after a massive item patch release like that one.

If you operate under the assumption that the item/recipe data themselves rarely change, then that should not be a big issue.

Drakma.1549:

That is exactly what I do. I download the json and get an md5 hash. I then compare to my database to see if the hash changed. If it has than I update.

Only one problem tho. I still need to download every item detail.

Adding an md5 hash to the items.json would be a huge help.

If you still need to download every item detail, then there is no advantage in calculating the md5 hash is there? Afterall, the performance bottleneck is probably the internet access to download every item.

DarkSpirit.7046:

Dr Ishmael.9685:

The obvious gap in a system like DarkSpirit’s is, of course, when an already known item/recipe changes. This happened to a lot of them on Sept. 3, when all existing equipment with Magic Find had their attributes removed, upgrade components that affected MF had their attributes/effects modified, and the recipes that crafted MF equipment were updated with a new output_item_id.

I would call that an exception rather than the norm. It is rare for them to change so many items at one go. Having said that, it is probably wise to do a full item/recipe update after a massive item patch release like that one.

If you operate under the assumption that the item/recipe data themselves rarely change, then that should not be a big issue.

You guys seem not to know about translation issues and stuff ^.^
At least the german translations and possibly also the french and spanish change with nearly every patch and so they give us a hard time in the wikis to find and move all that stuff. So once more the suggestion: An API which delivers all IDs of changed items for a given build (and therefore an API which delivers the recent build numbers – this could be done by the already existing build.json)

I am aware of the translation issues, Smiley. I just used the MF change as an example because it was recent and still on everyone’s minds.

I belive at least you are aware, Ish. My comment pointed more at this:

I would call that an exception rather than the norm. It is rare for them to change so many items at one go.

smiley.1438:

I belive at least you are aware, Ish. My comment pointed more at this:

I would call that an exception rather than the norm. It is rare for them to change so many items at one go.

I am also aware of the other languages. How often do multiple items change? I am not referring to items being added as they are discovered by players, I am referring to the existing items themselves being changed.

Other than that one-time MF patch, do the items themselves change often? If they do, then there isn’t much of a point to optimize performance by saving all items/recipes as json files in the first place.

There have been countless renamings of several items in the german translation (not only) recently. Some items have been renamed more than once. So if you’d like to maintain a database where you’re able to search items by name, a full update after patches seems to be mandatory.

If the items are renamed from the api, then I presume they were cascading down from the renaming of the items in the game code. Assuming the patch note is detail enough, then we should expect a list of items that are impacted in the German client, unfortunately they are not. I can certainly understand why people are asking for this.

Suggestion: Indicate time-gated recipes, for example the recipe to create Lump of Mithrillium.

DarkSpirit.7046:

Suggestion: Indicate time-gated recipes, for example the recipe to create Lump of Mithrillium.

+1

The “flags” Array is under-used here anyway

smiley.1438:

Lazarus.9716:

Right now as I’m testing it takes almost 20 minutes to download full item detail for the ~30k items in the game.

Wait WHAT? Only 20 minutes? A full update takes us about 20 hours (for all 4 languages of course at ~2-3 seconds per request, wiki check and recipe update not included)
I still think, the most important feature would be an API which provides a list of the changed items/events/recipes since the last patch, so that we can safely do incremental rather than full updates after updates where we don’t exactly know what has changed.

I’ve been starting a tool of my own (in .net), and right now I’m getting ~300 item_details/minute with a naive implementation, synchronous from 1 machine, using 5-6 kB/s down, 1 kB/s up. That’s just under 100 minutes for the full item set per language, or 6.5 hours for 4 languages. Unless flood control sets in at some point, getting the item_details should not be very time-consuming with let’s say 10 parallel tasks, ~40 minutes.

By the way, “1 big download with changes since the last patch” would be useful, but not to everyone. Let’s say someone misses the first patch for example, a stand-alone app that updates manually, or if 2 patches with item changes happened very close to each other (same day or so); or if you are just starting out populating your local database; or you simply want a fresh local database for whatever reason. In those cases, a big download with simply the current state of all items would be best.

I made this more clear in this post: https://forum-en.gw2archive.eu/forum/community/api/API-Suggestion-Items-Recipes-and-Crafting/2863618

tl;dr: an API which delivers the IDs of changed items starting from a given build number. build.json could be extended and used for this.

HackworthGW.8251:

smiley.1438:

Lazarus.9716:

Right now as I’m testing it takes almost 20 minutes to download full item detail for the ~30k items in the game.

Wait WHAT? Only 20 minutes? A full update takes us about 20 hours (for all 4 languages of course at ~2-3 seconds per request, wiki check and recipe update not included)
I still think, the most important feature would be an API which provides a list of the changed items/events/recipes since the last patch, so that we can safely do incremental rather than full updates after updates where we don’t exactly know what has changed.

I’ve been starting a tool of my own (in .net), and right now I’m getting ~300 item_details/minute with a naive implementation, synchronous from 1 machine, using 5-6 kB/s down, 1 kB/s up. That’s just under 100 minutes for the full item set per language, or 6.5 hours for 4 languages. Unless flood control sets in at some point, getting the item_details should not be very time-consuming with let’s say 10 parallel tasks, ~40 minutes.

By the way, “1 big download with changes since the last patch” would be useful, but not to everyone. Let’s say someone misses the first patch for example, a stand-alone app that updates manually, or if 2 patches with item changes happened very close to each other (same day or so); or if you are just starting out populating your local database; or you simply want a fresh local database for whatever reason. In those cases, a big download with simply the current state of all items would be best.

I wrote some code out in .Net to test this on my own against the recipe API. Using threading (with multiple web requests each on their own thread; concurrently), I was able to download the entire ~7.7k recipes + recipe details + output item details + ingredient details (about 30,800 requests to the API) in about 30 minutes. It was just some code I quickly hammered out, but I could probably clean it to shave it down to 20 minutes.

Of course, there’s always the bottleneck of the internet itself, and I do wonder at what point Anet’s servers will perceive a massive amount of near-instantaneous traffic as a DDOS attack. I would assume that over 1,000 API requests per minute isn’t appreciated by them anyhow, and it’s not code I’d be willing to use again (even infrequently.)

Honestly, in the current implementations, the recipe/item APIs are bad imo. There’s no real single item search outside of knowing the recipe/item ids (i.e. no request by name (like the guild API supports), by crafting profession, etc.) Inefficient and lack of serving data in a meaningful and useful manner. This is why I’ve yet to release my Recipe/Item related tools to my application’s users. I won’t force a large download on them to their computers (containing the entire recipe/item data that might need to be frequently re-downloaded/updated), and neither do I wish to host a synced database on my own servers to do what the API is supposed to be doing.

Really, it’s poor design (be it the over 7.7k recipes in the recipe list or 30k+ items in the item list) to serve just a list of ids that then necessitate making individual requests for each individual item’s/recipe’s details (that then each might also necessitate requesting multiple sub-items.)

Bandlero.6312:

There’s no real single item search outside of knowing the recipe/item ids (i.e. no request by name (like the guild API supports), by crafting profession, etc.)

Requesting by name wouldn’t make sense for items, because item names are not unique and you could get any number of items returned. Guild names must be unique, so you know requesting a guild name will return exactly 1 guild (or 0 if it doesn’t exist).

Dr Ishmael.9685:

Bandlero.6312:

There’s no real single item search outside of knowing the recipe/item ids (i.e. no request by name (like the guild API supports), by crafting profession, etc.)

Requesting by name wouldn’t make sense for items, because item names are not unique and you could get any number of items returned. Guild names must be unique, so you know requesting a guild name will return exactly 1 guild (or 0 if it doesn’t exist).

Yea, for single items and names to work each item would have to have a unique name.

What I meant by requesting by name is that you could pass a parameter (for example) like Rabid, and have the API return to you a subset of the items containing Rabid in the name. Instead of serving the entire list of items and recursively going through items.json and item_details.json just to locate items with Rabid.

We definitely need more support for additional optional parameters to pass to the API.

I assume your actual intent with that is the ability to query all items with Rabid attributes? You’re still kitten a bit in that, due to named exotics and karma items that don’t include the stat prefix in their names.

Bandlero.6312:

I wrote some code out in .Net to test this on my own against the recipe API. Using threading (with multiple web requests each on their own thread; concurrently), I was able to download the entire ~7.7k recipes + recipe details + output item details + ingredient details (about 30,800 requests to the API) in about 30 minutes. It was just some code I quickly hammered out, but I could probably clean it to shave it down to 20 minutes.

On what kind of connection?

If I use my home connection the full recipes + output items + ingredients takes around 40 minutes but on my web host it takes around 18m. Exact same code.

Looking at the data I have it should only be 15,738 requests. Where is your extra 15k coming from?

Khisanth.2948:

Bandlero.6312:

I wrote some code out in .Net to test this on my own against the recipe API. Using threading (with multiple web requests each on their own thread; concurrently), I was able to download the entire ~7.7k recipes + recipe details + output item details + ingredient details (about 30,800 requests to the API) in about 30 minutes. It was just some code I quickly hammered out, but I could probably clean it to shave it down to 20 minutes.

On what kind of connection?

If I use my home connection the full recipes + output items + ingredients takes around 40 minutes but on my web host it takes around 18m. Exact same code.

Looking at the data I have it should only be 15,738 requests. Where is your extra 15k coming from?

For my testing, I used the recipes API (grabbing that file is 1 request), then grabbed the details for each recipe (about 7.7k requests), grabbed the output item details (another ~7.7k requests), then grabbed the ingredient details for each recipe – each recipe has on average 3 ingredients for an additional ~15k to 23k requests. This test did not take into account caching already “known” data and was just a brute force test of the speed/time to download data from this API (i.e. like if this was a first time downloading from the API).

My connection is a dedicated business DSL with a 45 Mbps down / 10 Mbps up.

And to the person (sorry I forgot the name as I write this) that replied to my thought about searching items by name – yea you would still potentially receive a large amount of items to have to iterate through; however, it would still be a much smaller subset of data, more quicker, and more efficient than the current method of having to search through all 7.7k recipes (and the outputs, etc.) or 30k+ items to locate each item with “rabid.” Even 200 items with “rabid” is less than having to iterate through the entire recipes and its structure or the items.

If the APIs functioned a lot more like the Trading Post does in game, that would be the improvement. We should have URL parameters on the APIs that allow for things such as, “?item_name=rabid”, “?item_level=80”, “?item_rarity=legendary”, “?item_type=armor”, etc.; that allows us to retrieve only a subset of the data instead of having to iterate through the entire collection. Honestly, I’m assuming there’s an SQL database driving the back-end of the API, so handling these sorts of requests would minimal additional load on the servers. SELECT * FROM [Items] WHERE [Name] LIKE ‘rabid’.

I don’t know, perhaps my degree in ISE/DBA is making me over-think this, but a system like this is exactly what I’m trained/educated to analyze and fix.

I would like to suggest/request that when looking up armor pieces, the item details API shows the number of dyes that can be applied to them.

hi guys,
thanks for sharing your API knowledge.
I was wondering how the karma cost and vendor cost can be found? I manage to get all the information using the official APIs and spidy APIs, but no way I can get these remaining informations.
Any idea would be appreciated.

DasPatAttack.2516:

I was wondering how the karma cost and vendor cost can be found? I manage to get all the information using the official APIs and spidy APIs, but no way I can get these remaining informations.
Any idea would be appreciated.

There’s currently no official (or legal) way to get the price of an item. I think this has been requested more than once over here. → see first post

smiley.1438:

I made this more clear in this post: https://forum-en.gw2archive.eu/forum/community/api/API-Suggestion-Items-Recipes-and-Crafting/2863618

tl;dr: an API which delivers the IDs of changed items starting from a given build number. build.json could be extended and used for this.

Push – Great idea. We really need this. “Changelog” for Skills would be great too.

I hope I am in the correct place to add a suggestion for the skin_details.

It would be great to have an attribute skinset or something like that.

This way you could group the skins by their set (for example: all embroidered items). Currently I am missing this option in the game as well as in the API.

Until this is implemented (if it will be) there is no other way than to take all but the last word of the skin as name of the skin set.

Has anyone experienced if this approach is more or less reliable?
Has anyone another idea to get the name of the skin sets?

Leylin.3901:

I hope I am in the correct place to add a suggestion for the skin_details.

It would be great to have an attribute skinset or something like that.

This way you could group the skins by their set (for example: all embroidered items). Currently I am missing this option in the game as well as in the API.

Until this is implemented (if it will be) there is no other way than to take all but the last word of the skin as name of the skin set.

Has anyone experienced if this approach is more or less reliable?
Has anyone another idea to get the name of the skin sets?

I’ve been digging into skins for months now, and I’ve come to one conclusion regarding sets: sets are somewhat arbitrary.

1. Most skins do conform to having one obvious set name. However, most skins were also part of at least two sets – one for PvE, and one for PvP. Membership in both sets was not always guaranteed. For example, the Bandit weapons overlap with Makeshift weapons, but Makeshift Spear has no PvE counterpart and remains named Makeshift instead of Bandit.

2. Flame weapons did not include SteamFire until SteamFire was renamed a few months ago, even though they shared obvious visual similarities.

3. The Phantasmal set on the wiki only includes weapons that appear on Mesmer phantasms, exclude several that share the same obvious purple smoke characteristic, and NONE HAVE PHANTASMAL IN THEIR NAME.

There’s plenty more like this. So you can do what I do and just go with what seems obvious, because none of these sets are officially established outside of the collections and cultural merchants.

All the resent changes… I welcome them, most r very useful and if not they don’t bother me.
But with the introduction of the endless tonics also came a shortage in space. If you want to collect (all) tonics it could start costing you space. So my solution: create, like has been done with the mini’s and finishers, a separate tab were you can store all you endless tonics and when you want to use one, you take it out ( a copy) so it becomes like a collectable.

Don’t really know if this is the right place for this suggestion but….

Some further followup on skinsets: there are several examples of skins using similar or identical models to skins in existing sets that don’t match the naming conventions:

Glyphic: X6-31 Beta (Glyphic Speargun with differing partical effects and color)
Verdant: Rockweed Spire (Verdant Trident with leaves on the haft and different color)
Flame: Flamebelcher (Flame Rifle as speargun), SteamFire (Flame Staff as trident)
Aureate: The Maelstrom (Aureate Staff as trident)
Shiverpeak: OooOoo Shooter (Shiverpeak Arquebus as speargun)
Legionnaire: Gladium Trident (Legionnaire Staff as trident)
Tribal: Drakestrike (Tribal Rifle as speargun)

IME, people are a little too hardheaded to let something like being a complete and obvious copy of another set member be good enough to allow membership in a set when the names don’t match. So, while I might include the wiki skins entries in the same sets, I haven’t bothered with adjusting the membership of the actual weapons.