Showing Posts For Lawton Campbell:

Achievement points totally wrong

in API Development

Posted by: Lawton Campbell

Previous

Lawton Campbell

Web Programmer

Next

There are 574 achievement points from SAB achievements currently missing from the API. This will be fixed eventually.

I think the change to fix this is part of this issue which should go out with the next release.

API characters error 400

in API Development

Posted by: Lawton Campbell

Previous

Lawton Campbell

Web Programmer

Next

Huh, might have been related to the hotfix. Is this still not working for you? If it’s not, can you PM me your API key?

Is there a limit on number of calls?

in API Development

Posted by: Lawton Campbell

Previous

Lawton Campbell

Web Programmer

Next

Ah yeah that’s a good point, I’ll see if I can figure out how to make a sticky post.

The rate limit is currently 600 requests/minute and is exposed in the X-Rate-Limit-Limit response header. It’s per-IP address.

Suggestion: LFG API

in API Development

Posted by: Lawton Campbell

Previous

Lawton Campbell

Web Programmer

Next

LFG results wouldn’t necessarily need that long of a cache time.

The long cache times are only necessary for character/account blobs for architectural reasons.

About Apps and Legal Issues

in API Development

Posted by: Lawton Campbell

Previous

Lawton Campbell

Web Programmer

Next

Check out the Content Terms of Use. The API section answers most of your questions.

Draconis Mons map image tiles

in API Development

Posted by: Lawton Campbell

Previous

Lawton Campbell

Web Programmer

Next

I’ll regenerate the tiles today; thanks for the reminder.

Not sure … why/how those tiles got corrupted. Lots of places that might have happened; good news is that tile regeneration overwrites the old tiles, so hopefully the issue magically disappears.

EDIT: literally everything went wrong regenerating the tiles, so I’m only now starting to upload them to S3. I’ll switch the cloudfront distribution over tomorrow morning; sorry about the delay.

EDIT2: New tiles should be up. I spent a couple minutes looking for artifacts, but couldn’t find any. Lemme know if you find any corrupted tiles and I’ll try to figure out where they broke.

(edited by Lawton Campbell.8517)

Suggestion: LFG API

in API Development

Posted by: Lawton Campbell

Previous

Lawton Campbell

Web Programmer

Next

Seems reasonable. Here’s a tracking issue.

Listings question

in API Development

Posted by: Lawton Campbell

Previous

Lawton Campbell

Web Programmer

Next

Ever considered making a compressed bulk file available hourly for the most common endpoints?

Our stack makes it pretty difficult to do that; currently the APIs don’t have any sane way to trigger periodic events.

Listings question

in API Development

Posted by: Lawton Campbell

Previous

Lawton Campbell

Web Programmer

Next

You totally can make it a get request with ids separated by commas. Also the results can be paginated if you don’t want to deal with ids.

To confirm, that’s limited to 200 ids per result?

Correct.

I also read somewhere that ids=all was valid, but doesn’t appear to work for items, recipes, commerce/prices, or commerce/listings?

Is there a better way to bulk download price/listing data (ie: prepared cache) somewhere?

Typically, ids=all is only available on endpoints that have <500 resources.

IMO, the best way to fetch them all is to paginate through the listings using ?page_size=200&page=N, with N starting at 0 and ending at the value of the X-Page-Total response header. This will require 121 requests/scrape.

API for (daily) merchants?

in API Development

Posted by: Lawton Campbell

Previous

Lawton Campbell

Web Programmer

Next

Hmm, that actually might be feasible. There’s an existing issue for vendors in general, but for a handful of daily-rotating vendors (which will need to be hardcoded anyway) the whole whitelisting bits aren’t necessary. Will definitely look into it.

WvW kills supposed to be cached for 5 mins?

in API Development

Posted by: Lawton Campbell

Previous

Lawton Campbell

Web Programmer

Next

So, the kill/death counts are cached in the exact same blob as the map state — however, they’re collected using a different mechanism than the map state. The way I implemented it, there’s actually an API-specific backend server that watches logs coming off of the map instances. Every skirmish (I think? it might be on an interval), each WvW map instance flushes some transient state to logs, including kill/death counts; those logs are picked up by the API backend server and aggregated.

I have some minor changes that might be ready for the next release that watch for individual on-kill notifications (in addition to the periodic aggregated one) — that should improve the latency a bit.

tl;dr architectural, but I’m working on it.

HEADS UP: rate limiting is coming

in API Development

Posted by: Lawton Campbell

Previous

Lawton Campbell

Web Programmer

Next

How often is data from the Trading Post updated such that it can be obtained via API?

/v2/commerce/listings is every 20 minutes.
/v2/commerce/prices is every 2 minutes.
/v2/commerce/transactions is every 1 minute.

These change on occasion, refer to the Cache-Control and Expires headers.

(edited by Lawton Campbell.8517)

Character inventory API refresh interval

in API Development

Posted by: Lawton Campbell

Previous

Lawton Campbell

Web Programmer

Next

I don’t recall if it’s documented anywhere (pre-coffee reply), but the cache time is 5-10 minutes (nested caches).

WvW wrong matches and data 2017-04-14

in API Development

Posted by: Lawton Campbell

Previous

Lawton Campbell

Web Programmer

Next

Had to come into the office (can’t VPN into a BSOD’d machine, apparently) but it should be fixed now.

Endpoint for temporal festivals like SAB?

in API Development

Posted by: Lawton Campbell

Previous

Lawton Campbell

Web Programmer

Next

Seems reasonable. I’m pretty sure the previous festivals won’t have any data — prior to SAB, festivals weren’t automatic and needed a game build to turn on/off. SAB is the first one with structured data that the API can expose. Tracking issue.

/v2/wvw/matches - Kills are not updating

in API Development

Posted by: Lawton Campbell

Previous

Lawton Campbell

Web Programmer

Next

The kills are implemented via watching event logs from the map instance servers. Our event logging infrastructure had an outage earlier today (hardware failure), so the kill counts stopped being updated.

I don’t believe there’s a way to recover the lost data, unfortunately.

World Boss daily kills?

in API Development

Posted by: Lawton Campbell

Previous

Lawton Campbell

Web Programmer

Next

Created a tracking issue for this, but it’s can of worms. The cats/nodes/SAB/etc are a bit simpler since they’re totally disconnected systems, whereas a world boss endpoint would want the event data, the spawn timers, etc. Which isn’t infeasible, it’s just more work to thread everything through the needle.

Any chance of SAB character progression?

in API Development

Posted by: Lawton Campbell

Previous

Lawton Campbell

Web Programmer

Next

It’s been awhile since I did a Friday afternoon live deploy, but I just turned on /v2/characters/:id/sab which dumps the data. It requires the characters and progression permissions. I’ll add some examples to the github repo on Monday, but in the meanwhile here are the possible unlock values.

(also songs might be broken).

EDIT: deployed a fix for the songs.

(edited by Lawton Campbell.8517)

Probably Off-Topic: Login & Localization-Bug

in API Development

Posted by: Lawton Campbell

Previous

Lawton Campbell

Web Programmer

Next

While you’re at it Lawton, if you can confirm if it is or not in relation with: https://www.reddit.com/r/Guildwars2/comments/63d0nw/log_in_problems_when_using_shortcuts/

I honestly have no idea. The UI layer doesn’t have any responsibility whatsoever. I tried repro’ing it locally (assuming they’re using ‘-email xyz -password xyz -nopatchui’) with the live client but couldn’t get it to crash.

It’ll definitely crash when there’s an unfetched patch since ‘-nopatchui’ skips part of the patching process, but it’s been that way for years.

Polling times

in API Development

Posted by: Lawton Campbell

Previous

Lawton Campbell

Web Programmer

Next

The API doesn’t currently expose any online/offline status.

Adding an online count to the guild endpoint seems totally reasonable though; I went ahead and opened a tracking issue for this.

Any chance of SAB character progression?

in API Development

Posted by: Lawton Campbell

Previous

Lawton Campbell

Web Programmer

Next

There’s an open issue for this; the idea was brought up too late for me to get an implementation through the release process before SAB ends this year.

Definitely on the table though, the data seemed pretty straightforward to extract when I looked at it.

API Maintenance: back up @ 5PM PST.

in API Development

Posted by: Lawton Campbell

Previous

Lawton Campbell

Web Programmer

Next

One of the API backend servers is having a hardware issue; we’re migrating the services to a different machine. By a stroke of bad luck, this machine hosts one of the singleton services that the API needs to function (specifically: the item/recipe/etc whitelists) so everything is broken while we fix the issue.

Should be back up at or around 5PM PST.

EDIT: Nothing horrible happened and they should be back online now.

(edited by Lawton Campbell.8517)

Probably Off-Topic: Login & Localization-Bug

in API Development

Posted by: Lawton Campbell

Previous

Lawton Campbell

Web Programmer

Next

So, the launcher is implemented with web tech (it’s using Coherent UI under the covers) — but it’s loading HTML/CSS/Javascript from the .dat file rather than fetching it over the network (with the exception of the newsfeed which pull RSS down).

I’m thinking that maybe some bits on the native side might have changed in a subtle way — the language is stored in the local preferences (in gw2local.dat) which the web bits can’t read directly. Guessing that the ordering changed a bit or something else wonky happened since AFAIK we haven’t made any UI changes. I’ll ask around.

Character portrait: it's been a little while

in API Development

Posted by: Lawton Campbell

Previous

Lawton Campbell

Web Programmer

Next

I spent a couple of days working on it.

Let’s just say that I bumped my 1 month estimate to “3 months” ;__;

Any API update for WvW Skirmishing?

in API Development

Posted by: Lawton Campbell

Previous

Lawton Campbell

Web Programmer

Next

So I noticed victory points were added to wvw/matches endpoint but not wvw/matches/scores is there a reason for this? Also isn’t the warscore a pointless thing to have considering its victory points being fed into glicko2. Are we going to get victory points tied into scores within the API?

The omission from wvw/matches/scores is a bug. Tracking issue. Overall warscore might be semi-pointless, but I don’t want to break backwards compat. Once .skirmishes is available, you’ll be able to pull the current skirmish warscore.

Guild members home world

in API Development

Posted by: Lawton Campbell

Previous

Lawton Campbell

Web Programmer

Next

You’ll need an API key for each guild member to get that data from the API. I’m not sure what the game client shows.

HEADS UP: rate limiting is coming

in API Development

Posted by: Lawton Campbell

Previous

Lawton Campbell

Web Programmer

Next

Is there any chance, Lawton, that you could stick the build number in the ETag header, or something that included the build number? That is probably the most useful cache control indicator for this.

In a past life I used an etag header of “etag: ${build}:YYYY-MM-DD” to track something we didn’t want to live more than 24 hours in a cache, but might want to invalidate sooner, as a cheap way to do that without having to do anything that required deeper knowledge of the content.

The API doesn’t currently support either HEAD requests or If-Modified-Since headers, so putting it in an etag feels really wonky to me. How does just having X-Build-Id sound instead?

HEADS UP: rate limiting is coming

in API Development

Posted by: Lawton Campbell

Previous

Lawton Campbell

Web Programmer

Next

I just did a quick check, and at least the achievement endpoint tells me:

< Expires: Fri, 24 Mar 2017 15:54:35 GMT

when requested at 15:50 GMT. so, it’s telling me I can hold it for a little under four minutes… not an amazing cache lifetime for api.guildwars2.com/v2/achievements?id=1 but not zero either.

I should probably increase that to 24h and flush the cache when the build updates. Hmm.

Trading Post API Question

in API Development

Posted by: Lawton Campbell

Previous

Lawton Campbell

Web Programmer

Next

So — the long answer is that the backend component that implements the trading post’s matching engine doesn’t store enough data for you to implement this 100%. You can get most of the way there, but some stuff might slip through the cracks.

Some initial notes:

  • For every order that you create (buy or sell) a single record is made in /v2/commerce/transactions/current/{buys,sells}. It’s assigned a unique id and a creation date.
  • Every time someone else’s order matches with yours, a record is made in /v2/commerce/transactions/history/{buys,sells}. The id of the historical record does not match the id of the your matching order.
  • When someone else’s order matches yours (but does not fully consume it), your current order’s quantity is adjusted. This leaves your original order available to be matched by future orders.
  • The “created” date between current/history will match up.
  • When an order is cancelled, it’s purged from current without leaving a record in history.

An example walkthrough, assuming only two people are interacting with a trading post that starts out empty:

  • You make a buy order for 250 Mithril Ore for 5c/unit. You’ll now have a record in /v2/commerce/transactions/current/buys that looks like:

{ “id” : 1, “created” : “timestamp1”, “item_id” : 19700, “quantity” : 250, “unit_price” : 5 }

You haven’t sold anything yet, so your history will be empty.

  • If you then list a sell order for mithril at 50c/unit (because buy low sell high), your current/buys will be unchanged but you’ll have a record in current/sells:

{ “id” : 2, “created” : “timestamp2”, “item_id” : 19700, “quantity” : 99, “unit_price” : 50 }

  • Then if someone comes along and purchases 1 of your mithril ore @ 50c, the current/sells entry will be amended to reflect the updated quantity, and you’ll have a record in history/sells noting that there was a purchase:

current/sells:
{ “id” : 2, “created” : “timestamp2”, “item_id” : 19700, “quantity” : 98, “unit_price” : 50 }

history/sells:
{ “id” : 3, “created” : “timestamp2”, “purchased” : “timestamp3”, “item_id” : 19700, “quantity” : 1, “unit_price” : 50 }

It should be noted here that the “created” timestamp is the same between the current order and historical order — but not necessarily unique. It should be enough to correlate between the two using item_id/unit_price, noting that you may have multiple listings for the same item_id/unit_price (which can safely be aggregated on your end). You’re going to lose track of some paid exchange fees one way or another.

  • If someone makes another purchase of your sale listing, a new history/sell record is made:

current/sells
{ “id” : 2, “created” : “timestamp2”, “item_id” : 19700, “quantity” : 93, “unit_price” : 50 }

history/sells:
{ “id” : 3, “created” : “timestamp2”, “purchased” : “timestamp3”, “item_id” : 19700, “quantity” : 1, “unit_price” : 50 }
{ “id” : 4, “created” : “timestamp2”, “purchased” : “timestamp4”, “item_id” : 19700, “quantity” : 5, “unit_price” : 50 }

The entry in current/sells will be removed once either (A) you cancel the order, or (B) the quantity reaches zero. Determining which happened requires you to store the old version of current/sells locally, notice that the record no longer exists, then search history/sells for matching transaction that would matches it. More tersely, you basically have to reconcile the current transactions with the historical logs on your end.

Anyway, I realize this is kind of a ramble and probably doesn’t help much, but it’s a rough overview of what the API is exposing and how you can get actual data out of it. You’ll have to ask a more specific question if you want a more specific answer, I’m afraid.

Thief stolen skills aren't whitelisted

in API Development

Posted by: Lawton Campbell

Previous

Lawton Campbell

Web Programmer

Next

Tracking issue. There’s a bunch of one-offs that need to be whitelisted, will try to tackle that when I’ve got some time.

HEADS UP: rate limiting is coming

in API Development

Posted by: Lawton Campbell

Previous

Lawton Campbell

Web Programmer

Next

Will v2/guild be bulk expandable in the future?

Probably not. Bulk expansion only makes sense when all ids can be enumerated, and guild IDs are not enumerable for technical reasons (e.g., there’s no master list of all guild IDs that’s accessible to the API frontend).

Who on Tyria would need to make 600 requests a minute? That’s sounds like bad programming to me.

Eh, /v2/commerce/listings is updated every ~30 seconds and takes ~150 requests to pull all the data from. Sites that additionally maintain a large pool of API keys (e.g., for grabbing /v2/pvp/games) also might require higher request rates.

600 requests/minute is certainly a lot, but there are definitely cases where it’s not enough.

For static endpoints though, I highly highly recommend just pulling the data into your own database every time /v2/build updates. None of the API’s backend data is stored in a relational database so there’s a lot of things the API will never be able to provide — but that’s easily overcome by just throwing it into e.g. PostgreSQL and building your own indexes.

HEADS UP: rate limiting is coming

in API Development

Posted by: Lawton Campbell

Previous

Lawton Campbell

Web Programmer

Next

The guilds endpoint isn’t bulk-expanded. The current implementation only lets you request one guild at a time.

That documentation is a bit wrong; I’ll update it. (EDIT: blam)

(edited by Lawton Campbell.8517)

HEADS UP: API temporarily down

in API Development

Posted by: Lawton Campbell

Previous

Lawton Campbell

Web Programmer

Next

And the ratelimit is bumped up to 600 reqs/minute.

As far as I can see, everything is mostly humming along okay.

HEADS UP: API temporarily down

in API Development

Posted by: Lawton Campbell

Previous

Lawton Campbell

Web Programmer

Next

Okay, the API is tentatively back up with a limit of 60 reqs/minute while I double-check some things.

Will bump that up to the full 600 reqs/min in an hour or so when I’m confident that nothing has caught fire (which might happen, but doesn’t look likely at this point).

HEADS UP: rate limiting is coming

in API Development

Posted by: Lawton Campbell

Previous

Lawton Campbell

Web Programmer

Next

Would it be legit if I use my own server for static API requests and only use the official API for account bounded requests? My Server would then have its own database of items for example and only updates this database like every 24h? This would help reducing traffic, but I’m not sure if this would count as stealing content or intellectual property.

This isn’t an official answer, but from my reading of the content terms of use that’s perfectly fine. There’s a lot of value-add that get for running your own copy of e.g. the items database — for example, searching items by name.

HEADS UP: API temporarily down

in API Development

Posted by: Lawton Campbell

Previous

Lawton Campbell

Web Programmer

Next

I’m really hoping for today; pretty sure I’ve got all the kinks worked out.

HEADS UP: API temporarily down

in API Development

Posted by: Lawton Campbell

Previous

Lawton Campbell

Web Programmer

Next

Do some sort of caching, that, when API is down, can still dispense all the resources that have been frozen on take down?

Please consider this. Static endpoints such as /recipes, /recipes/search, /items, /achievements etc are all down, and they don’t have to be.

Those endpoints aren’t actually static. Greatly increased request volume to those endpoints is what caused these crashes, unfortunately.

any info when this will be up? i need my tradepost stuff ;(

I’m trying to get it back up today; I still need to get someone to review my changes to make sure I don’t cascade the failure to other systems if I turn everything back on.

Could there please have a rough Eta as to how long the API site will be down for? I understand that if ya give an ETA and things dont turn out the way ya expect it may take longer. Any info would be appreciated.

ETA was yesterday, except during testing there was a small bug in my implementation that caused it to use the wrong instance of the centralized rate-limiting service (i.e. it used the one that everything else uses instead of the API-specific one). So I punted on the deploy until we’re sure that the changes aren’t going to overwhelm anything.

HEADS UP: API temporarily down

in API Development

Posted by: Lawton Campbell

Previous

Lawton Campbell

Web Programmer

Next

Distributed systems are hard

Tomorrow might be a better day.

HEADS UP: rate limiting is coming

in API Development

Posted by: Lawton Campbell

Previous

Lawton Campbell

Web Programmer

Next

Do we know if the problem is individual users or popular apps that may be poorly designed?

Not currently. Due to volume, we don’t keep detailed request logs; I know how many requests each endpoint handles, but not where the requests are coming from.

Is there a way to require an app provide the api key/account that generated the request so the app’s volume can be split out by account to track if its individuals within an app or the app itself?

Not currently; probably going to add something like that in for applications to request higher rate limits, but not require it for access. Not requiring this means there’s no barrier to entry, which is something I prefer. Requiring it for high-usage apps means I can track request volume on a per-application basis and can get a better idea of what happens when something goes wrong.

Note that since we’re going to use IP-based rate-limiting, this is mostly moot for client-side requests. If you’ve got a browser app sending more than 600 requests/minute that’s … probably not the best way of going about it. Increased rate limits only really make sense when you’re doing a bunch of stuff on a server (with a fixed IP, hopefully) like persisting historical data, tracking aggregate metrics, providing indexing, etc.

I still need to work out all the details though, so I don’t really have much on how the higher rate-limit bits will work yet.

HEADS UP: rate limiting is coming

in API Development

Posted by: Lawton Campbell

Previous

Lawton Campbell

Web Programmer

Next

Will it only influence the utilization of my account api keys on 3rd party tools like gw2bltc, if I want to load my full listing overview or purchase history or also my trading post performance in game?

This will only affect API keys on 3rd party tools. This has no effect on anything in-game.

Hopefully, in the future, there will be an “api-app-key” for every app/website and the rate-limit will be evenly counted per app/website and not per ip?

It’s kind of a hassle on my end, but that’s what we’re probably going to have to do. The main logic is that, since we’re doing IP-based rate limiting, it’s trivial to get around by just buying a $2.50 VPS or twelve. That’s silly and wasteful, so we’ll probably have a signup process for getting IPs whitelisted for a higher limit.

Whitelisting is a super annoying approach for everyone, but it lets me explicitly tag requests with the originating site and graph them and so forth. So if something goes totally haywire I can look at the chart and have a better clue what’s going wrong, which is something I can’t easily do today.

Eh… Lawton, was the outage caused by someone request thousands of times?

Tens of thousands. This is my fault for not implementing rate-limiting sooner :<

HEADS UP: API temporarily down

in API Development

Posted by: Lawton Campbell

Previous

Lawton Campbell

Web Programmer

Next

Which timezone? I made the config change at 3/18/2017 7:06 PM PDT (UTC-7); and it normally takes a few minutes to propagate out.

HEADS UP: rate limiting is coming

in API Development

Posted by: Lawton Campbell

Previous

Lawton Campbell

Web Programmer

Next

Some users have hundreds of thousands of Trading Post transactions.

what.

Well, that’s not a use-case I considered.

600 requests/minute with 200 results/request is 120000 results/minute — depending on how the data’s being presented adding a throttle and a progress bar to the client might be okay? It’s unfortunate the data is structured the way it is — it would be a lot more efficient for everyone if there was a better way to communicate a diff of what’s been bought/sold/listed since a given point in time, but alas.

It’s definitely going to be a bit bumpy though, I’m planning on keeping a close eye on things when this gets turned on.

HEADS UP: API temporarily down

in API Development

Posted by: Lawton Campbell

Previous

Lawton Campbell

Web Programmer

Next

Regarding the sporiadic up nature, queicherius provided a graph and that correlates strongly with a minor config change I made (to turn the APIs back on for the office IP addresses). I’m not sure why/????/how there’s a correlation there, but I’ve undid it (so the behavior should go back to the solid red bar of “down”.

This week is going to be a lot of fun.

(EDIT: boo the forums strip emoji).

HEADS UP: rate limiting is coming

in API Development

Posted by: Lawton Campbell

Previous

Lawton Campbell

Web Programmer

Next

600 requests/minute is nice, but I want to know if authenticated endpoints (/v2/account* for example) count in this limit ?

I guess yes, but I need to know, because I made a public tool and if a lot of people use it, I will work on a way to spread authenticated requests to avoid peaks (I previously already added a random margin into my cache retention to spread global item requests).

They do count towards the limit. It’s altogether possible that 600 reqs/minute is too low a number; if your application can’t fit within that, let me know what numbers would work better. The goal here is to break as few things as possible while being able to keep the APIs from crashing.

HEADS UP: API temporarily down

in API Development

Posted by: Lawton Campbell

Previous

Lawton Campbell

Web Programmer

Next

looks like, at least certain, endpoints have been up and down today. How big is the fire? ie would you appreciate it if people held off work on new features today that would hit the endpoints that seem to be up (or were up last time my services ran)

They should be entirely down, except around 24h ago when I briefly turned them on to get some metrics out.

Which endpoints seem to be up? I’m terrified.

HEADS UP: API temporarily down

in API Development

Posted by: Lawton Campbell

Previous

Lawton Campbell

Web Programmer

Next

One small suggestion. If the service is off, maybe you could send a 503 header service unavailable so we can better handle this case in the future.

I should adjust that. Been turning off the APIs by disabling all endpoints via config; I should add an explicit off switch that sends the proper response.

Thanks for the heads up!
Is there any way to determine the API’s current status? Something like https://api.guildwars2.com/v2/status ? That would be really nice.

The root (https://api.guildwars2.com) kind of works like this? When the v1/v2 endpoints are disabled they’re removed from the list. But using an explicit status code to indicate “the world is on fire” is probably easier on the client side.

Does this have any relation to the recent issues with the in-game TP where requests seem to stall and have to be repeated? That has been happening for some weeks now.

AFAIK the API failure was totally self-contained and shouldn’t have affected other systems.

HEADS UP: rate limiting is coming

in API Development

Posted by: Lawton Campbell

Previous

Lawton Campbell

Web Programmer

Next

Yeah, those are both single requests.

HEADS UP: rate limiting is coming

in API Development

Posted by: Lawton Campbell

Previous

Lawton Campbell

Web Programmer

Next

Due to increased popularity, the API needs to have rate limits added to ensure that it can continue functioning properly. This is going to be a bumpy transition, but here’s what I’m thinking so far —

  • The rate limit is per-IP,
  • Global across all endpoints (e.g., separate endpoints do not have separate rate limits),
  • Is (probably) tracked on a per-minute basis, and
  • Is continuously updated, rather than resetting at a specific interval.

The main change is that, upon reaching the rate limit, more requests just return “HTTP 429 Limit Exceeded” with the response body {"error":“rate limit exceeded”}. Rate-limited requests still count against the limit (this is an implementation detail of the existing backend service that manages limits). Additionally, responses will contain a new header:

  • X-Rate-Limit-Limit is the maximum number of requests that may be issued within a minute. This is a static value doesn’t contain count for requests you have already sent.

At some point in the future I’d like to add X-Rate-Limit-Remaining, which would contain the actual number of requests that can still be made before hitting the limit, but that needs more extensive backend changes to support.

For the actual limit, the current number I’m thinking of is 600 requests/minute. I think that’s enough to pull all of the trading post listings every 30 seconds (which is how long they’re cached for). The limit is a rolling limit, so if an application makes 600 requests, then waits for 30 seconds, it should be able to make another 300 requests without going over.

This isn’t set in stone yet, so let me know if you have feedback/concerns/questions. Rate limiting does need to be implemented before the API can be turned back on, though, otherwise the API will just continue falling over. I would like to get everything back up and running on Monday, but the schedule might slip a bit if something unexpected happens.

HEADS UP: API temporarily down

in API Development

Posted by: Lawton Campbell

Previous

Lawton Campbell

Web Programmer

Next

Just disabled all of the APIs.

Not entirely sure of the reason, but the Javascript contexts for the API frontend servers are constantly crashing. Due to the holiday I won’t have the bandwidth to debug the problem until tomorrow evening, so I’ve temporarily taken the APIs down in the meanwhile. I should be able to figure out what’s going wrong tomorrow and probably get everything back up by Sunday.

I’ll post updates to this thread if something goes horribly wrong.

Sorry about the downtime

EDIT: Going to be down until Monday at least; going to add explicit rate-limiting to the API. I’ll have some more details later, still need to run some numbers.

(edited by Lawton Campbell.8517)

Some endpoints i'd like.

in API Development

Posted by: Lawton Campbell

Previous

Lawton Campbell

Web Programmer

Next

Made a pull request containing a first-pass attempt at exposing the localized story data. The personal story is a bit wonky in that it’s branching — which is kind of weird to expose due to the implementation. There’s a handful of additional endpoints that will need to be made in /v2/characters (and maybe /v2/account?) to expose which story paths have been chosen, as well as which stories/chapters have been unlocked. And which have been completed, naturally.

Comments/suggestions are always welcome; especially RE: expressing chapter branches.