HEADS UP: rate limiting is coming

HEADS UP: rate limiting is coming

in API Development

Posted by: Lawton Campbell

Lawton Campbell

Web Programmer

Next

Due to increased popularity, the API needs to have rate limits added to ensure that it can continue functioning properly. This is going to be a bumpy transition, but here’s what I’m thinking so far —

  • The rate limit is per-IP,
  • Global across all endpoints (e.g., separate endpoints do not have separate rate limits),
  • Is (probably) tracked on a per-minute basis, and
  • Is continuously updated, rather than resetting at a specific interval.

The main change is that, upon reaching the rate limit, more requests just return “HTTP 429 Limit Exceeded” with the response body {"error":“rate limit exceeded”}. Rate-limited requests still count against the limit (this is an implementation detail of the existing backend service that manages limits). Additionally, responses will contain a new header:

  • X-Rate-Limit-Limit is the maximum number of requests that may be issued within a minute. This is a static value doesn’t contain count for requests you have already sent.

At some point in the future I’d like to add X-Rate-Limit-Remaining, which would contain the actual number of requests that can still be made before hitting the limit, but that needs more extensive backend changes to support.

For the actual limit, the current number I’m thinking of is 600 requests/minute. I think that’s enough to pull all of the trading post listings every 30 seconds (which is how long they’re cached for). The limit is a rolling limit, so if an application makes 600 requests, then waits for 30 seconds, it should be able to make another 300 requests without going over.

This isn’t set in stone yet, so let me know if you have feedback/concerns/questions. Rate limiting does need to be implemented before the API can be turned back on, though, otherwise the API will just continue falling over. I would like to get everything back up and running on Monday, but the schedule might slip a bit if something unexpected happens.

HEADS UP: rate limiting is coming

in API Development

Posted by: Drant.5902

Drant.5902

https://api.guildwars2.com/v2/items/12345
and
https://api.guildwars2.com/v2/items/?ids=12345,12346,12347,12348,12349 (up to 200 IDs)
Both are considered one request, right?

I guess more developers will have to learn the “bulk expansion” feature if they haven’t already.

HEADS UP: rate limiting is coming

in API Development

Posted by: Lawton Campbell

Previous

Lawton Campbell

Web Programmer

Next

Yeah, those are both single requests.

HEADS UP: rate limiting is coming

in API Development

Posted by: Tao Ythaq.6582

Tao Ythaq.6582

This is a very good idea : it will force people to have good practises.
I am the gw2tool.net developper and I hate wasting computer resources. I made a huge work to build my own caching and so on …
I was horrified when I saw a few years ago the wasting of requests everywhere in the community.

600 requests/minute is nice, but I want to know if authenticated endpoints (/v2/account* for example) count in this limit ?

I guess yes, but I need to know, because I made a public tool and if a lot of people use it, I will work on a way to spread authenticated requests to avoid peaks (I previously already added a random margin into my cache retention to spread global item requests).

HEADS UP: rate limiting is coming

in API Development

Posted by: Lawton Campbell

Previous

Lawton Campbell

Web Programmer

Next

600 requests/minute is nice, but I want to know if authenticated endpoints (/v2/account* for example) count in this limit ?

I guess yes, but I need to know, because I made a public tool and if a lot of people use it, I will work on a way to spread authenticated requests to avoid peaks (I previously already added a random margin into my cache retention to spread global item requests).

They do count towards the limit. It’s altogether possible that 600 reqs/minute is too low a number; if your application can’t fit within that, let me know what numbers would work better. The goal here is to break as few things as possible while being able to keep the APIs from crashing.

HEADS UP: rate limiting is coming

in API Development

Posted by: Drant.5902

Drant.5902

Some users have hundreds of thousands of Trading Post transactions. My client-side program has “page_size=200” in the transactions request URL, but the “X-Page-Total” header (number of requests to be made) may be in the thousands, so if a user somehow manages to download over 600 requests in under a minute, a throttling algorithm would have to kick in so the limit isn’t reached. Is there a better way to retrieve these mass transactions?

Other than that, with caching and bulk expanded IDs, requests could be kept under that limit. But we don’t have the API back up yet to see how that 600 req/min performs in practice.

HEADS UP: rate limiting is coming

in API Development

Posted by: Lawton Campbell

Previous

Lawton Campbell

Web Programmer

Next

Some users have hundreds of thousands of Trading Post transactions.

what.

Well, that’s not a use-case I considered.

600 requests/minute with 200 results/request is 120000 results/minute — depending on how the data’s being presented adding a throttle and a progress bar to the client might be okay? It’s unfortunate the data is structured the way it is — it would be a lot more efficient for everyone if there was a better way to communicate a diff of what’s been bought/sold/listed since a given point in time, but alas.

It’s definitely going to be a bit bumpy though, I’m planning on keeping a close eye on things when this gets turned on.

HEADS UP: rate limiting is coming

in API Development

Posted by: Tao Ythaq.6582

Tao Ythaq.6582

600 requests/minute is nice, but I want to know if authenticated endpoints (/v2/account* for example) count in this limit ?

I guess yes, but I need to know, because I made a public tool and if a lot of people use it, I will work on a way to spread authenticated requests to avoid peaks (I previously already added a random margin into my cache retention to spread global item requests).

They do count towards the limit. It’s altogether possible that 600 reqs/minute is too low a number; if your application can’t fit within that, let me know what numbers would work better. The goal here is to break as few things as possible while being able to keep the APIs from crashing.

At the moment there are not enough people using gw2tool.net everyday to reach the limit. But I will implement another algorthm to spread requests to avoid to reach the limit, don’t worry, I don’t need more than 600/min.

HEADS UP: rate limiting is coming

in API Development

Posted by: Nightlark.4029

Nightlark.4029

One thought is that you could perform those API requests client side (so that the origin IP is of the person using your site), then have the client send the received information to your server. That’s assuming that access to account information is limited to the owner of the API key so that it isn’t exposed to other users, and that the site has a need of storing historical info for later reference (or caching purposes for people with really high-frequency trading).

HEADS UP: rate limiting is coming

in API Development

Posted by: TPMN.1483

TPMN.1483

Let’s correct this a few users have millions of listings on the Trading Post – I know I’m one of them.

[MYTH] The Mythical Dragons -PvX http://mythdragons.enjin.com

HEADS UP: rate limiting is coming

in API Development

Posted by: Wanze.8410

Wanze.8410

Some users have hundreds of thousands of Trading Post transactions.

what.

Well, that’s not a use-case I considered.

600 requests/minute with 200 results/request is 120000 results/minute — depending on how the data’s being presented adding a throttle and a progress bar to the client might be okay? It’s unfortunate the data is structured the way it is — it would be a lot more efficient for everyone if there was a better way to communicate a diff of what’s been bought/sold/listed since a given point in time, but alas.

It’s definitely going to be a bit bumpy though, I’m planning on keeping a close eye on things when this gets turned on.

I am a high volume trader with about 1 million items listed on the TP and probably even more items bought within the last 90 days. I dont have much knowledge about programming, so I am wondering, how your proposed limit will effect my account performance. Will it only influence the utilization of my account api keys on 3rd party tools like gw2bltc, if I want to load my full listing overview or purchase history or also my trading post performance in game?

Trading Post performance has gone way down for me in recent weeks and months to a point that it sometimes takes a couple of seconds for an item info screen to load. Not sure, if this is somehow related, as the game is laggy in general.

Anyways, if you need any specific feedback from a high volume trader, once you switch the API on again, let me know and I will try to help.

And thanks for the weekend shift, the game is only half as fun for me without the API.

Tin Foil [HATS]-Hardcore BLTC-PvP Guild
Bloin – Running around, tagging Keeps, getting whack on Scoobie Snacks.

HEADS UP: rate limiting is coming

in API Development

Posted by: DarkMagister.7429

DarkMagister.7429

Well, the change is reasonable (man, we could get prices every minute and that’s not a problem? I downloaded prices every 5 minutes not to be stress, and only for those craft-used items).

I was afraid cases like re-creating locally saved items and recipes information would hit the limit, but:

At this moment I have 55k items and 12k recipes in lists. 67k / 200 = 335 requests.

So yeah, the limit looks fine. But for the above-mentioned cases with lots of pages of trading history (I don’t download all my history, I download only till the transaction that is already present in my data).

HEADS UP: rate limiting is coming

in API Development

Posted by: Zok.4956

Zok.4956

This isn’t set in stone yet, so let me know if you have feedback/concerns/questions. Rate limiting does need to be implemented before the API can be turned back on, though, otherwise the API will just continue falling over. I would like to get everything back up and running on Monday, but the schedule might slip a bit if something unexpected happens.

For gw2gh.com I implemented rate limiting. Before every new api request the API client checks if it should wait a few secs or microsecs before sending the next API request.

I think this is a fine-grade rate limit from the client side and it is better than burst-api-calls (send 300 api calls … wait 30-50 secs, send 300 api-calls, wait 30-50 secs) and it hopefully reduces the burden on the server side.

However, to check if it works as expected, the API has to be turned on again.

I understand, that at first the rate limit counting is per IP-address because it is easiest to implement on the server side.

But this puts backend-server-based-api-calls (where all API-calls for all users of the website come from i.e. PHP with the same server-ip) at a disadvantage in comparison to client-side-api-calls (where the API-calls come from i.e. JavaScript with the IP of the browser of each user ).

So a website that does do client-side/Javascript API calls requests can make parallel requests for i.e. 10.000 users at the same time (10.000 IPs) where a website that does backend-server/PHP API calls has to do the requests for the 10.000 users sequential, one after the other (only 1 IP), which will take a lot longer.

But the same use case (requesting API data for 10.000 users) should in general have the same rate-limit constrains.

Hopefully, in the future, there will be an “api-app-key” for every app/website and the rate-limit will be evenly counted per app/website and not per ip?

HEADS UP: rate limiting is coming

in API Development

Posted by: Valento.9852

Valento.9852

Eh… Lawton, was the outage caused by someone request thousands of times?

Attempts at ele specs:
Shaman
Conjurer

HEADS UP: rate limiting is coming

in API Development

Posted by: Leablo.2651

Leablo.2651

To clarify a possible point of confusion in some of the above posts, 1 item is not equal to 1 transaction. A transaction where somebody buys or sells 1 item is the same as if they bought 250 in a stack. If you’re a bulk trader, your average load (based on naive mathematics) across 1 million individual items will be about 8000 transactions.

HEADS UP: rate limiting is coming

in API Development

Posted by: DarkMagister.7429

DarkMagister.7429

To clarify a possible point of confusion in some of the above posts, 1 item is not equal to 1 transaction. A transaction where somebody buys or sells 1 item is the same as if they bought 250 in a stack. If you’re a bulk trader, your average load (based on naive mathematics) across 1 million individual items will be about 8000 transactions.

However if you are selling a stack of 250 items, and those are bought by 250 different people – you get 250 transactions in Sell History, which you get in 1,25 requests.
So if you sell/buy lots of small items, your Buy/Sell history can become rather long.
Isn’kitten

(edited by DarkMagister.7429)

HEADS UP: rate limiting is coming

in API Development

Posted by: Leablo.2651

Leablo.2651

That’s why I gave the average. The ideal would be half that.

HEADS UP: rate limiting is coming

in API Development

Posted by: Lawton Campbell

Previous

Lawton Campbell

Web Programmer

Next

Will it only influence the utilization of my account api keys on 3rd party tools like gw2bltc, if I want to load my full listing overview or purchase history or also my trading post performance in game?

This will only affect API keys on 3rd party tools. This has no effect on anything in-game.

Hopefully, in the future, there will be an “api-app-key” for every app/website and the rate-limit will be evenly counted per app/website and not per ip?

It’s kind of a hassle on my end, but that’s what we’re probably going to have to do. The main logic is that, since we’re doing IP-based rate limiting, it’s trivial to get around by just buying a $2.50 VPS or twelve. That’s silly and wasteful, so we’ll probably have a signup process for getting IPs whitelisted for a higher limit.

Whitelisting is a super annoying approach for everyone, but it lets me explicitly tag requests with the originating site and graph them and so forth. So if something goes totally haywire I can look at the chart and have a better clue what’s going wrong, which is something I can’t easily do today.

Eh… Lawton, was the outage caused by someone request thousands of times?

Tens of thousands. This is my fault for not implementing rate-limiting sooner :<

HEADS UP: rate limiting is coming

in API Development

Posted by: lynnae.4095

lynnae.4095

an unintended bonus of explicit rate limiting is that I will no longer have to guess what’s a reasonable request rate. Being the mostly nice person that I am, I try to minimize my requests, however once my new backend is up and running, I’d like to make calls more often.

A rate limit helps me plan as well

HEADS UP: rate limiting is coming

in API Development

Posted by: Illconceived Was Na.9781

Illconceived Was Na.9781

@Lawton: thanks for working on the weekend to address the issues. And thanks for thinking ahead down the road to how this affects the popular websites, high volume individuals, and the rest of us. I appreciate the detailed and frequent updates.

PS the wiki is another consumer of API data; I have no idea how that affects volume.

John Smith: “you should kill monsters, because killing monsters is awesome.”

HEADS UP: rate limiting is coming

in API Development

Posted by: Chieftain Alex.2851

Chieftain Alex.2851

The wiki doesn’t store any responses at all, and makes all API requests with widgets using the bulk expanded API queries (i.e. most pages should only make one request maximum) via the viewer’s IP.

Theoretically we could implement some client-side per session caching with sessionstorage (e.g. it could check for local information before performing another api query, and used cached results for 30 minutes).

-Guild Wars 2 Wiki.

HEADS UP: rate limiting is coming

in API Development

Posted by: Xultair.7954

Xultair.7954

Do we know if the problem is individual users or popular apps that may be poorly designed? Is there a way to require an app provide the api key/account that generated the request so the app’s volume can be split out by account to track if its individuals within an app or the app itself?

I’m just trying to think how I’d figure out where the volume was coming from if all of the requests were being logged and it should be easy enough to track the requesting IP/app but if that IP was required to supply additional information such as an api key that can be grouped by its governing account you can then see if the volume by a given app is high across all of its users or if its limited to a handful of users and then you could rate limit the users directly.

(edited by Xultair.7954)

HEADS UP: rate limiting is coming

in API Development

Posted by: Lawton Campbell

Previous

Lawton Campbell

Web Programmer

Next

Do we know if the problem is individual users or popular apps that may be poorly designed?

Not currently. Due to volume, we don’t keep detailed request logs; I know how many requests each endpoint handles, but not where the requests are coming from.

Is there a way to require an app provide the api key/account that generated the request so the app’s volume can be split out by account to track if its individuals within an app or the app itself?

Not currently; probably going to add something like that in for applications to request higher rate limits, but not require it for access. Not requiring this means there’s no barrier to entry, which is something I prefer. Requiring it for high-usage apps means I can track request volume on a per-application basis and can get a better idea of what happens when something goes wrong.

Note that since we’re going to use IP-based rate-limiting, this is mostly moot for client-side requests. If you’ve got a browser app sending more than 600 requests/minute that’s … probably not the best way of going about it. Increased rate limits only really make sense when you’re doing a bunch of stuff on a server (with a fixed IP, hopefully) like persisting historical data, tracking aggregate metrics, providing indexing, etc.

I still need to work out all the details though, so I don’t really have much on how the higher rate-limit bits will work yet.

HEADS UP: rate limiting is coming

in API Development

Posted by: Kythan Myr.4719

Kythan Myr.4719

I really hope that the wait is short. I did not realize how much I depend on the use of APIs until it was down. I understand the need to limit and have often wondered the load on the servers. Thanks for the notice, I’m just concerned the wait will turn into days, then weeks, then months and then into years with the system never really returning. I can’t count how many times these “temporary” things in the programming world never come back.

HEADS UP: rate limiting is coming

in API Development

Posted by: zeronomore.2381

zeronomore.2381

Really wonder how this will work for the item api. I cache all the items and only refresh them once a month or less. But with 55k items, it would take a rather long time to cache them. Dont think it would be great idea to let users request the data, but if i have to spend hours trying to get all data, it’s easier to let users just hit you with requests from their own ip. This would result in an insane request increase instead of decrease. Please think this trough.

Just read the whole thing, seems it would be no problem. Already using the ‘ids=’ on all requests. This has a max of 200 items if i remember correctly, so 275 requests would be all that’s needed ^^

(edited by zeronomore.2381)

HEADS UP: rate limiting is coming

in API Development

Posted by: WeAreTwO.9780

WeAreTwO.9780

Would it be legit if I use my own server for static API requests and only use the official API for account bounded requests? My Server would then have its own database of items for example and only updates this database like every 24h? This would help reducing traffic, but I’m not sure if this would count as stealing content or intellectual property.

HEADS UP: rate limiting is coming

in API Development

Posted by: Lawton Campbell

Previous

Lawton Campbell

Web Programmer

Next

Would it be legit if I use my own server for static API requests and only use the official API for account bounded requests? My Server would then have its own database of items for example and only updates this database like every 24h? This would help reducing traffic, but I’m not sure if this would count as stealing content or intellectual property.

This isn’t an official answer, but from my reading of the content terms of use that’s perfectly fine. There’s a lot of value-add that get for running your own copy of e.g. the items database — for example, searching items by name.

HEADS UP: rate limiting is coming

in API Development

Posted by: Drant.5902

Drant.5902

In order to lower my program’s requests, I’m trying to bulk expand the public (not account) guild details. According to the api-cdi
the URL below should work, but why does it not?

https://api.guildwars2.com/v2/guilds?ids=4BBB52AA-D768-4FC6-8EDE-C299F2822F0F,CB2347E3-076F-438D-9C94-E53F0967A7E9,B63949DA-C5C0-E611-80D4-E4115BE8E905,9A732AAF-5D86-4800-943D-92C81820F80A

Retrieving one guild detail at a time works:
https://api.guildwars2.com/v2/guild/4BBB52AA-D768-4FC6-8EDE-C299F2822F0F

I’ve tried both “v2/guild?ids=” and “v2/guilds?ids=”. What is the proper format for retrieving multiple guild details with a single URL request?

HEADS UP: rate limiting is coming

in API Development

Posted by: Lawton Campbell

Previous

Lawton Campbell

Web Programmer

Next

The guilds endpoint isn’t bulk-expanded. The current implementation only lets you request one guild at a time.

That documentation is a bit wrong; I’ll update it. (EDIT: blam)

(edited by Lawton Campbell.8517)

HEADS UP: rate limiting is coming

in API Development

Posted by: Drant.5902

Drant.5902

From https://api.guildwars2.com/v2

All APIs offer bulk expansion. There are several ways to do so. The simplest
is by specifying multiple ids via query parameter, as in:
/v2/colors?ids=1,2,3

Will v2/guild be bulk expandable in the future?

HEADS UP: rate limiting is coming

in API Development

Posted by: DarkMagister.7429

DarkMagister.7429

Would it be legit if I use my own server for static API requests and only use the official API for account bounded requests? My Server would then have its own database of items for example and only updates this database like every 24h?

I really think everyone should go this way.
It requires less time, more search variety, and so on and so forth.
For my software I planned to check for the change in version name, to trigger a database rebuild, but found it is too often, so just press a button after major updates.

HEADS UP: rate limiting is coming

in API Development

Posted by: ProperDave.7425

ProperDave.7425

Would it be legit if I use my own server for static API requests and only use the official API for account bounded requests? My Server would then have its own database of items for example and only updates this database like every 24h?

I really think everyone should go this way.
It requires less time, more search variety, and so on and so forth.
For my software I planned to check for the change in version name, to trigger a database rebuild, but found it is too often, so just press a button after major updates.

Caching responses from the GW2API is a good way to go – it gives your own projects more fault-tolerance for when situations like this occur.
For my part, I try and cache everything I pull from the API for at least 24 hours, and vary request intervals back to the API so that I don’t flood the servers with requests.
Who on Tyria would need to make 600 requests a minute? That’s sounds like bad programming to me. My earlier versions of Plotty were a little unforgiving on the API calls, but I’ve hopefully fixed all that in the latest builds. :o

— ProperDave of Aurora Glade [EU] | http://auroragla.de
Guild Font Pack and Emblem Generator | http://gw2.properdave.com
Plotty GW2 GPS and custom GW2 Maps | http://gw2maps.com

HEADS UP: rate limiting is coming

in API Development

Posted by: Lawton Campbell

Previous

Lawton Campbell

Web Programmer

Next

Will v2/guild be bulk expandable in the future?

Probably not. Bulk expansion only makes sense when all ids can be enumerated, and guild IDs are not enumerable for technical reasons (e.g., there’s no master list of all guild IDs that’s accessible to the API frontend).

Who on Tyria would need to make 600 requests a minute? That’s sounds like bad programming to me.

Eh, /v2/commerce/listings is updated every ~30 seconds and takes ~150 requests to pull all the data from. Sites that additionally maintain a large pool of API keys (e.g., for grabbing /v2/pvp/games) also might require higher request rates.

600 requests/minute is certainly a lot, but there are definitely cases where it’s not enough.

For static endpoints though, I highly highly recommend just pulling the data into your own database every time /v2/build updates. None of the API’s backend data is stored in a relational database so there’s a lot of things the API will never be able to provide — but that’s easily overcome by just throwing it into e.g. PostgreSQL and building your own indexes.

HEADS UP: rate limiting is coming

in API Development

Posted by: SlippyCheeze.5483

SlippyCheeze.5483

Would it be legit if I use my own server for static API requests and only use the official API for account bounded requests? My Server would then have its own database of items for example and only updates this database like every 24h?

I really think everyone should go this way.
It requires less time, more search variety, and so on and so forth.
For my software I planned to check for the change in version name, to trigger a database rebuild, but found it is too often, so just press a button after major updates.

Most of it is static enough that a background job updating the records, and simply serving “outdated” content that is almost certainly identical, while it runs … works fine.

HEADS UP: rate limiting is coming

in API Development

Posted by: Tshakaar.6429

Tshakaar.6429

Ohh, happy to see the API is back, and more, one of my suggestion starting to be implemented.
Thank you Lawton!

I always believed all endpoints was refresh on a 5minute timer. I recently learned, mainly because i wasn’t interested in TP before, /v2/commerce/listings has a 30s refresh.
Could these timers be reported in https://api.guildwars2.com/v2 details?
And maybe a timestamp in here to count from it to know the refreshes?
And to finish, le’ts be crazy, the number of requests/minute left for the actual IP?

HEADS UP: rate limiting is coming

in API Development

Posted by: ProperDave.7425

ProperDave.7425

Who on Tyria would need to make 600 requests a minute? That’s sounds like bad programming to me.

Eh, /v2/commerce/listings is updated every ~30 seconds and takes ~150 requests to pull all the data from. Sites that additionally maintain a large pool of API keys (e.g., for grabbing /v2/pvp/games) also might require higher request rates.

600 requests/minute is certainly a lot, but there are definitely cases where it’s not enough.

This inspires me to write some general purpose best-practice guides on API consumption that I’ll see if I can sneak into the references on the Wiki’s API page when I’m done.

And also; Keep up the good work Lawton! We really do appreciate it!

— ProperDave of Aurora Glade [EU] | http://auroragla.de
Guild Font Pack and Emblem Generator | http://gw2.properdave.com
Plotty GW2 GPS and custom GW2 Maps | http://gw2maps.com

HEADS UP: rate limiting is coming

in API Development

Posted by: darthmaim.6017

darthmaim.6017

I always believed all endpoints was refresh on a 5minute timer. I recently learned, mainly because i wasn’t interested in TP before, /v2/commerce/listings has a 30s refresh.
Could these timers be reported in https://api.guildwars2.com/v2 details?

Each response contains cache headers with the correct times used by the endpoint.

And to finish, le’ts be crazy, the number of requests/minute left for the actual IP?

This is currently not possible but will come one day.

HEADS UP: rate limiting is coming

in API Development

Posted by: Tshakaar.6429

Tshakaar.6429

Each response contains cache headers with the correct times used by the endpoint.

From what i see, it’s just the timestamp of when i received the json, not when it was cached on the server.

HEADS UP: rate limiting is coming

in API Development

Posted by: SlippyCheeze.5483

SlippyCheeze.5483

Each response contains cache headers with the correct times used by the endpoint.

From what i see, it’s just the timestamp of when i received the json, not when it was cached on the server.

I just did a quick check, and at least the achievement endpoint tells me:

< Expires: Fri, 24 Mar 2017 15:54:35 GMT

when requested at 15:50 GMT. so, it’s telling me I can hold it for a little under four minutes… not an amazing cache lifetime for api.guildwars2.com/v2/achievements?id=1 but not zero either.

HEADS UP: rate limiting is coming

in API Development

Posted by: Lawton Campbell

Previous

Lawton Campbell

Web Programmer

Next

I just did a quick check, and at least the achievement endpoint tells me:

< Expires: Fri, 24 Mar 2017 15:54:35 GMT

when requested at 15:50 GMT. so, it’s telling me I can hold it for a little under four minutes… not an amazing cache lifetime for api.guildwars2.com/v2/achievements?id=1 but not zero either.

I should probably increase that to 24h and flush the cache when the build updates. Hmm.

HEADS UP: rate limiting is coming

in API Development

Posted by: Tshakaar.6429

Tshakaar.6429

SlippyCheeze, teach me your sorcery. Where do you see that? I’m using network tab of dev tool from Firefox.

HEADS UP: rate limiting is coming

in API Development

Posted by: KillerRabbit.1946

KillerRabbit.1946

SlippyCheeze, teach me your sorcery. Where do you see that? I’m using network tab of dev tool from Firefox.

On the network tab you should look for the Response headers, and within that the one called ‘Expires’

HEADS UP: rate limiting is coming

in API Development

Posted by: Tshakaar.6429

Tshakaar.6429

Hmm, ok, i tried more endpoints.
I’m surprised to see this “Expires” on /v2/backstory/answers but not on /v2/account or /v2/characters/:id/core or /v2/account/wallet…
So, i can’t count on this to know how much time to keep data in cache.

HEADS UP: rate limiting is coming

in API Development

Posted by: SlippyCheeze.5483

SlippyCheeze.5483

SlippyCheeze, teach me your sorcery. Where do you see that? I’m using network tab of dev tool from Firefox.

I used `curl -V` on it, but it should be present as the Expires header in the response. (I think the format is human readable by default, btw, but if not it’ll be a unix timestamp. How spoilekittenhat I don’t even need remember that any more, huh?

HEADS UP: rate limiting is coming

in API Development

Posted by: SlippyCheeze.5483

SlippyCheeze.5483

Hmm, ok, i tried more endpoints.
I’m surprised to see this “Expires” on /v2/backstory/answers but not on /v2/account or /v2/characters/:id/core or /v2/account/wallet…
So, i can’t count on this to know how much time to keep data in cache.

You can read the official rules about HTTP cache headers for details of how this works if you want, but…

…short answer is: no Expires or ETag header => no caching. (Unless you have some out of band knowledge about it. but see my next comment to our friendly dev about that one…)

HEADS UP: rate limiting is coming

in API Development

Posted by: SlippyCheeze.5483

SlippyCheeze.5483

I just did a quick check, and at least the achievement endpoint tells me:

< Expires: Fri, 24 Mar 2017 15:54:35 GMT

when requested at 15:50 GMT. so, it’s telling me I can hold it for a little under four minutes… not an amazing cache lifetime for api.guildwars2.com/v2/achievements?id=1 but not zero either.

I should probably increase that to 24h and flush the cache when the build updates. Hmm.

Is there any chance, Lawton, that you could stick the build number in the ETag header, or something that included the build number? That is probably the most useful cache control indicator for this.

In a past life I used an etag header of “etag: ${build}:YYYY-MM-DD” to track something we didn’t want to live more than 24 hours in a cache, but might want to invalidate sooner, as a cheap way to do that without having to do anything that required deeper knowledge of the content.

(In that case, because we wanted to do cache control and rate limiting in a separate front-end service, that would spare our “real” front-end the cost of fielding lots of “etag match 304” responses, for the usual sort of horrible reasons. It had cheap access to the build number, and local access to the date, so it could tell if the etag was valid or not and respond without coordination with the things that could actually generate the answer.)

HEADS UP: rate limiting is coming

in API Development

Posted by: Lawton Campbell

Previous

Lawton Campbell

Web Programmer

Next

Is there any chance, Lawton, that you could stick the build number in the ETag header, or something that included the build number? That is probably the most useful cache control indicator for this.

In a past life I used an etag header of “etag: ${build}:YYYY-MM-DD” to track something we didn’t want to live more than 24 hours in a cache, but might want to invalidate sooner, as a cheap way to do that without having to do anything that required deeper knowledge of the content.

The API doesn’t currently support either HEAD requests or If-Modified-Since headers, so putting it in an etag feels really wonky to me. How does just having X-Build-Id sound instead?

HEADS UP: rate limiting is coming

in API Development

Posted by: SlippyCheeze.5483

SlippyCheeze.5483

Is there any chance, Lawton, that you could stick the build number in the ETag header, or something that included the build number? That is probably the most useful cache control indicator for this.

In a past life I used an etag header of “etag: ${build}:YYYY-MM-DD” to track something we didn’t want to live more than 24 hours in a cache, but might want to invalidate sooner, as a cheap way to do that without having to do anything that required deeper knowledge of the content.

The API doesn’t currently support either HEAD requests or If-Modified-Since headers, so putting it in an etag feels really wonky to me. How does just having X-Build-Id sound instead?

I’d rather just see nothing until you did add those features, if you ever do. If we need to know out of band that we should check the build ID and flush or refresh a local cache of, eg, skill info, then we can just know that. The extra header doesn’t bring that much value, I think.

HEADS UP: rate limiting is coming

in API Development

Posted by: Neural.1824

Neural.1824

Not sure if this is information that has changed, but as it is related to limiting and proper use of the API…

How often is data from the Trading Post updated such that it can be obtained via API?

Where are my gem sales? I want gem sales! Nerf EVERYTHING!

HEADS UP: rate limiting is coming

in API Development

Posted by: Lawton Campbell

Previous

Lawton Campbell

Web Programmer

How often is data from the Trading Post updated such that it can be obtained via API?

/v2/commerce/listings is every 20 minutes.
/v2/commerce/prices is every 2 minutes.
/v2/commerce/transactions is every 1 minute.

These change on occasion, refer to the Cache-Control and Expires headers.

(edited by Lawton Campbell.8517)