Wesley Beary of Anchor speaks with host Sam Taggart about designing APIs with a particular emphasis on user experience. Wesley discusses what it means to be an “API connoisseur”— paying attention to what makes the APIs we consume enjoyable or frustrating and then taking those lessons and using them when we design our own APIs. Wesley and Sam also explore the many challenges developers face when designing APIs such as coming up with good abstractions, testing, getting user feedback, documentation, security, and versioning. They address both CLI and web APIs.
This episode is sponsored by Fly.io.
Show Notes
Related Episodes
- SE Radio 612: Eyal Solomon on API Consumption Management
- SE Radio 542: Brendan Callum on Contract-Driven APIs
- SE Radio 476: Leonid Shevtsov on Transactional Email
- SE Radio 383: Neil Madden on Securing Your API
- SE Radio 376: Justin Richer on API Security with Oauth2
Other References
Transcript
Transcript brought to you by IEEE Software magazine.
This transcript was automatically generated. To suggest improvements in the text, please contact [email protected] and include the episode number and URL.
Sam Taggart 00:00:51 This is Sam Taggart for SE Radio. I’m here today with Wesley Beary. Wesley Beary is co-founder of Anchor and has worked at a variety of tech companies, including Salesforce, Heroku, and Engine Yard as an engineer and developer advocate. Today we’re going to talk about API design. We’ve talked about various aspects of APIs in previous episodes such as 612 and 542. We’ll include links to those episodes as well as a few others in the show notes. Welcome, Wesley.
Wesley Beary 00:01:16 Thanks. Thanks for having me.
Sam Taggart 00:01:17 Let’s get started by defining what is an API.
Wesley Beary 00:01:21 I think technically Application Protocol Interface, if I remember, right? That sounds right, really it’s the interface that we expose to outside users. So most of my experience in particular is with HTP driven APIs and so usually it’s, somebody’s making a request over the web to me that they want some work done in some way, shape or form, and that’s how they tell me what and how I should do it.
Sam Taggart 00:01:45 So I think APIs actually exist at various levels because I think of an API as I’ve got a software module in the API is the things that I call to make it do things. And those command line interfaces are also a form of an API because you can script them and make them programmatic. And as you mentioned, a lot of people talk about web interfaces as APIs.
Wesley Beary 00:02:05 Sure, yep.
Sam Taggart 00:02:06 I’ve heard you in a talk before mention your own acronym for APIs called Assumptions Probably Incorrect. I found that rather funny. Can you talk about how you came up with that and what it means?
Wesley Beary 00:02:16 Sure. It kind of speaks to I suppose all those different levels of APIs that you were mentioning, whether it’s in a library, on the CLI, on the web that at least in my own experience, when I try to make a new interface for something, it’s this sort of no plan survives contact with the enemy kind of thing. I’ll have a plan, I’ll put together this interface and then I’ll try to use it and realize that I’ve forgotten six or eight things or something that there’s some edge cases that weren’t considered, that basically I’ll end up with something that the experience isn’t nearly what I hoped it would be, that it just doesn’t pan out. So it’s one of the reasons why more and more I push towards trying to do lighter weight things, I guess? Trying to basically have prototypes or other things that I can work against, don’t spend all the time to implement the interface front to back with full tests or something, only to find out that I’ve made a huge error.
Wesley Beary 00:03:02 So doing what I can to get more hands-on experience sooner, because for me at least I find it really hard to just think through all of that in the abstract. So that’s kind of what I’m talking about with assumptions probably incorrect. If I try to just do it abstractly upfront, I tend to miss things. So I feel really getting into the nitty gritty of it and trying it and using it in anger as people say, actually using it the way you think a user would use it quickly reveals the things that you didn’t think about or overlooked.
Sam Taggart 00:03:28 So if I understand correctly, you’re advocating for more of an iterative or agile style of development?
Wesley Beary 00:03:34 Yeah. I definitely think that that makes a lot of sense in this context. You can get a foundation of something together quickly, but the edges always end up for me at least being rough until I’ve had some real experience with it and can get some real feedback on what is working and what isn’t working. So more agile is the way to go.
Sam Taggart 00:03:51 Do you have any specific examples that come to mind of times when your assumptions were not quite correct?
Wesley Beary 00:03:57 I just made a mistake related to that this morning when I was working on some API design, so I guess that’s fresh where I was going in and we were working on something related to specifying domain names and what something can and can’t do basically by which domains is able to operate on. And I sort of quickly looked in the existing API and checked to see what precedents we might have. And there was an example of an array of domain strings and I was, great, I’ll just use that. But it turns out that I had had a conversation with my colleague that I’d totally forgotten about, in which he had pointed out that that’s just one of the things that we want to check against. And so I needed to actually to have an array where there was both a type and a value.
Wesley Beary 00:04:38 So one of the possibilities is A DNS type with a domain name, which is very similar to what was there already, but just putting an array of domain names would’ve been too limiting and it would’ve been very difficult to change later. So just making that small tweak to something where there was a type and a value will give us a lot more flexibility in the future. Where again, I was just sort of assuming, oh, well I’ll just follow the existing example, not thinking about the broader implications in these other use cases that I just, again, didn’t happen to be in mind at the time. It can be hard to get all the context all into your brain all at once, sometimes I think with interfaces. So we end up piecemeal recalling parts that you had previously forgotten.
Sam Taggart 00:05:11 That leads into my next question, which you kind of already answered, but the question was, I’ve heard you talk about one of the major problems with API designs that were too close to the problem. And I think that’s a good example. You’re looking at the narrow thing of, I just want to filter some domain names, but you take a step back and youíre, well, maybe people would also want to filter other things.
Wesley Beary 00:05:30 Yah and in that case, the concrete example is we aren’t going to support it away, but we expect in the future people might want to filter on IP addresses instead of domain names, right? That’s a very concrete way that that might flip. But, in terms of being too close to the problem, I think that comes up all the time. You know the domain that your interface is for probably inside and out, and oftentimes users don’t. And so even if you expose it in a way that seems straightforward to you, that’s based on the assumption that they have a bunch of knowledge that they might not have or that they have a bunch of experience that they might not have. So I think in a lot of cases you have to think about not just what technically can do all of the things, but what is going to be easy to discover or easy to understand or easy to pick up if you’ve never done this before. And sometimes maybe that means parts of the interface that are just for brand new users. That maybe you don’t expect experienced users will really use it that much at all. But it provides some extra support basically for people that don’t have the experience yet to help get them to that point.
Sam Taggart 00:06:28 Maybe like a really quick up and getting started script. What pops into my mind is wire guard. I don’t know if you’ve used it, but there’s a wire guard quick up or something that that you just run it and it just does the thing, but then more advanced users might want to specify more parameters. Does that kind of fall into your bucket?
Wesley Beary 00:06:43 Yeah, I think that’s a great example where there is an easy way, but it’s not the way for everyone necessarily. I think at least in the past, I’ve sometimes gotten caught up in this, try to make something that will be technically good enough for everyone, but then because it’s so flexible and perhaps so complicated, it ends up kind of being not that good for anyone. It’s too confusing. Yeah, I know what I want to do, but I can’t figure out how to do it with this thing because this thing is trying to let me do any possible thing and I don’t want to do any possible thing. I want to do this one narrow thing. So sometimes providing something that just does that narrow thing, if there’s enough users that need and want that can be a better experience than the something for everyone.
Sam Taggart 00:07:21 I find softwares never have a problem introducing more complexity?
Wesley Beary 00:07:24 That’s for sure.
Sam Taggart 00:07:25 We talked about agile slash iterative development. What are the best ways you’ve found to get feedback from users?
Wesley Beary 00:07:34 I think it’s a bit of a tangent I guess, but for a little while, and I hope to maybe come back to this at some point, but for a little while I was working on a board game and there’s something that I learned from that context, from the advice that I heard from other board game designers, which is one of the best things you can do is what they call a blind play test. The idea is basically you set somebody down with this board game that you’re working on and give them the rule book and say, please try to figure it out. I’m just going to watch what you’re doing and I’m not going to answer your questions because if I do, it will sort of invalidate the test. I want to just know if you’re able to figure this out. I think in a lot of cases with APIs the same kind of thing if you can manage it, this is similar to other UX testing things that people sometimes do.
Wesley Beary 00:08:12 Here is the documentation for this interface, here’s the interface. Maybe here is the task that I want you to try to accomplish. Just speak as though you’re thinking aloud and try to accomplish that. I just want to see what happens. And then you can see, oh, that’s interesting. I did this with five different people and all of them got stuck on this one thing. Probably means that there’s something that isn’t quite there. Or when they were talking about how they were thinking about it, they all use different terminology than I would use. That’s interesting. Maybe that means that the way that the documentation is written isn’t quite right because my user base and I don’t have quite the same terminology for this thing. Or maybe I should mention both terms, things like that. There’s a lot of things that become revealed that other things don’t seem to work as well for.
Wesley Beary 00:08:52 Because whether you want to or not, if you’re answering questions and really interacting with someone, you’re kind of leading the witness. So if you notice that it seems theyíre struggling a little bit, you’ll be likely to offer them help and then that’s great for in the moment. But it turns out that if everybody else is going to struggle at that point, you’re usually not going to be in the room to notice that they’re having a hard time and help them through it. So, I think it can be challenging to get in that position basically and say, I’m sorry, this is maybe not going to be very fun for you, but it’s going to be very valuable. Can you please just try to struggle through it a little bit? I’m not going to leave you struggling forever, but I do want you to try to push through at least a little.
Sam Taggart 00:09:27 I wonder if it might be worth recording those so that there’s nobody else in the room so you’re not tempted to influence the witness?
Wesley Beary 00:09:34 I definitely think so. If you have the setup for it, that’s a great approach.
Sam Taggart 00:09:37 Sometimes APIs end up just being a simple CRUD type access for database type stuff. What are the potential issues with that? And do you have any examples?
Wesley Beary 00:09:48 Sure, there’s a couple things. One relates to some of what we were already talking about. So I think one of the dangers I guess with CRUD is that you get into that trying to have something that tries to cover every single case. And so then some of the individual things can end up becoming incredibly complicated. As an example, I’ve seen and worked on APIs where update gets pretty overloaded, where what update is doing depends a lot on which particular parameters you pass in. And it’s not necessarily just that behind the scenes a database record gets changed, but it might actually kick off other follow-on operations and things that, depending on what the particular values are. That, again can get really complicated and make it very difficult for both the implementers and operators as well as the people that are writing stuff to use it to just reason about what in the world actually happens when I call update?
Wesley Beary 00:10:37 Itís the dreaded, it depends. I feel the longer you’ve been in software, the more you’ve probably heard somebody say that, but that’s sort of what happens when you call update? Well it depends. That’s the buzzword of complicated software I guess in a lot of cases. Complexity, you know? So what I have in a lot of cases is to provide CRUD because you’re going to need it, but then think about actually offering other actions or other capabilities for the resources. The other example is kind of what you’re talking about with wire guard. Where you want to set up and tear down and see which wire guard setups you have. But sometimes you want to just very simply turn it on. And not necessarily worry about all the nitty gritty of having created one and doing whatever was necessary.
Wesley Beary 00:11:17 There’s some other things that too, or another examples with servers. If you want to reset the server, how are you supposed to do that If it’s just CRUD? Do you call update and change this state to restarting or something that feels a little bit awkward versus an action that just is, please restart the server. So, I think those are some of the edges of things that are not just updating a record but actually cause something to occur. And also potentially things that are more workflow based or provide a particular use case some support.
Sam Taggart 00:11:47 Can you expand a little bit on separating actions from state and reading state versus doing things?
Wesley Beary 00:11:54 Sure. The way that I’ve tended to do it, which I feel Iíve gotten in arguments with people about a lot, so I wouldn’t say it’s that popular is in a standard sort of rest interface. A lot of times you’ll see slash resources slash an identifier and then slash maybe restart or something for the server example. And what I to do is to segregate those, like the kind of way that I to think about it in a lot of ways is it’s okay to do different things, but ideally each of the things that you’re doing you do consistently, and you make it clear where the separation is. And so what I to do for something that would be like slash servers slash ID slash actions slash restart. And so then there’s a clear sort of namespace basically underneath a particular server where all of the actions live together and it makes it easier to understand and identify at a glance then that this is something that follows the action pattern.
Wesley Beary 00:12:46 So when you’re documenting and stuff, you can basically say trout operations work this. When you call update on a server, you should expect to get back a serialization of that server. If you call read on a server, it should also be a serialization of a server, but actions on a server don’t necessarily follow those rules. It might be you just get back a reference to a job that you can check on later or something else, not necessarily a server serialization. So again, having that clear delineation that that it’s separate. The other thing that’s nice is if you have a nested resources where it might be server slash something slash volume slash something by putting all the actions together into one name space, it also helps to prevent polluting that sub name space, if that makes sense. So you don’t have to worry about accidentally creating an action that has a name that conflicts with some other thing that you want to now nest there. All the actions are together nicely of here are all the methods you can call on this thing versus here are the sub resources or whatever. If that makes sense.
Sam Taggart 00:13:40 That makes a lot of sense. Another question just popped into my mind. You were talking about URLs. How do you handle versioning? I know a lot of URLs tend to have slash V1 or slash V2. Is that a practice you recommend? Is that something you caution against?
Wesley Beary 00:13:56 Boy, how long have you got? It’s one of those kinds of problems where I think it depends a lot on what you need or want. So first off, what I have done at least sometimes in the past is push for that to be more in the headers. It feels from the semantics of HTP perspective, that feels like a header kind of information. It’s a meta information about the request you’re making, but almost everyone I’ve talked to does not like this idea. Among other things, there’s a lot of clients and stuff where headers are just a bit of a pain to use, especially if you’re needing to use them on basically every request and putting in the path just is better for usability. So more often than not, I have done it more in the path. I think the thing that gets really tricky is if I had my way, I would prefer probably to version even different endpoints with different numbers instead of having to version the entire API.
Wesley Beary 00:14:46 But that again becomes a usability nightmare basically instantly. But it’s hard because otherwise it’s this sort of thing of Iíve made a change in this one endpoint that is a breaking change. I would to roll this out but it’s not backwards compatible. Do I have to roll a whole major version of this API? And if I do roll a whole major version of this API do I need to also continue to operate the old version of the API? If so for how long? an example of that is that I’ve always found interesting, but I don’t know that I would want to do it is Stripe, at least historically. I believe their version numbers were all based on basically dates. And so when you first set up your Stripe integration, it would lock so that any API requests you make unless you change this value, get the API as it was on that day basically.
Wesley Beary 00:15:29 And so they’re not rolling out a new version every day or whatever, but it’ll be whatever the most recent version was when you start using the API is the API you log to. And then they try to basically maintain that API indefinitely. So you can at some point move up to a newer version or whatever, but the old version continues to be maintained. That’s where it gets really hairy really is once an API is out in the world that people are using, making changes to it suddenly is much, much more painful. You can’t go and change their code for them and so you don’t want to cause them pain. So, versioning is in a lot of ways to be avoided basically I think as much as possible. But obviously you can’t do that entirely. What I’ve seen in a lot of cases I think that works okay, is mostly focusing on major versions, trying to only do non braking changes, and then having a very long deprecation period if you do move to a new version and then doing probably like brownouts basically where you temporarily turn the API off.
Wesley Beary 00:16:20 So anybody that’s still using it that didn’t see the announcements, we’ll get errors and have a chance to recover and respond to it before the API is totally gone. But again, it’s onerous. So mostly I try to avoid it if I can, which you can’t really, but as much as you can, you try to avoid it.
Sam Taggart 00:16:36 It occurs to me that the biggest pain point might be changes on the backend that require you change the database. So these two things are no longer associated. Now old API endpoints are trying to make that association. It’s no longer there. Is that something?
Wesley Beary 00:16:48 That kind of stuff can be a huge pain. If you are in a situation where the backend has changed drastically, and the API just doesn’t quite match. You end up basically having to maintain all kinds of things that have just logic to convert from one version to another. So I think in some cases I’ve kind of seen itís here is the function that takes an API call from version one and translates it to version two and here’s the call that translates it from version two to version three. And you kind of end up in this thing where it just sort of takes the request and chains it up through as many layers as it needs to try to translate it into something that would be understood in the modern context. But of course maintaining the testing and infrastructure to actually operate all of those things is non-trivial.
Wesley Beary 00:17:25 It’s a real headache. You could think about it with other software that you work on or whatever. Itís what if you had to maintain not only whatever the latest version was, but like every version of that you’ve ever written. It’s a real headache. So you have to make some tradeoffs somewhere. You can’t really maintain every version. So which ones do you choose to maintain or not and how long do you continue to maintain the old ones before you just pull the plug and say we did what we could do. But if you are still not up to date. Sorry, it’s just time to move on.
Sam Taggart 00:17:53 Another question that popped into my mind as we’re talking about upgrading APIs and changing them, have you used the Strangler Fig pattern? Does that sound familiar to you?
Wesley Beary 00:18:03 I have heard of it. I don’t know if I’ve used it. I didn’t call it that at the time. I don’t think, I mean, it’s kind of the idea that you remove more and more of the thing until there’s almost none left. You kind of replace it with something else. Is that accurate or how would you describe it?
Sam Taggart 00:18:17 So I would describe it as you’ve got your old backend and you’ve got a new backend and you kind of run them in parallel and compare the results and then ship the new one and then eventually after they’re the same for a significant amount of time, then you remove the old one kind of thing.
Wesley Beary 00:18:30 Ah yeah, that makes a lot of sense. I don’t have that much experience doing that in particular, but I’ve seen that, and I would be interested in trying that approach if I was working on a problem like that. I just don’t happen to have a problem like that that I’ve worked on recently at least.
Sam Taggart 00:18:44 I’ve heard you use the term API connoisseur. What does that mean to you?
Wesley Beary 00:18:49 Sure. I think it’s kind of how sometimes I just drink a cup of coffee to drink a cup of coffee. But I’d say in general that’s an area where I feel I’m a bit of a connoisseur at least. I know some things about different regions that coffees are roasted in and some things about different ways that they might be fermented or roasted, and I have some opinions about that. So when I have my cup of coffee or whatever I’m thinking about what is it that I like or dislike about this particular brew? Would I want to do this again? Would I want to change something? And so in the same way with APIs I think, especially early on when I was working on them often would just be like, well I have been assigned the task at my work of integrating with this thing so I’m just going to do it.
Wesley Beary 00:19:28 I’m just going to figure out what needs to happen. I’m going to do it and I’m going to move on. And later I found it interesting, got to more of a point of okay yes I’m going to do that, but at the same time I can say what of this API did I really like? What was interesting, what was a pain in the butt in the end when I started doing it? What are the parts that I like and dislike about this? Why do I like and dislike those? And I think it’s very helpful then to start to develop a sort of taste for it. So it’s not just oh, well it’s an API, it is what it is, I’m going to work with it and I’m going to move on. I think especially there’s this challenge in that if you’re designing APIs you don’t get that many opportunities to actually do it basically?
Wesley Beary 00:20:08 Especially with web APIs, with other APIs you get a few more bytes at the apple so to speak. But I think most people, if you’re designing web APIs, how many web APIs are you going to make in your whole career? Maybe six or eight or something at most. Probably for a lot of people it’s more two or three. And so you’re not going to have as many chances to make the mistakes and realize that you could or should have done it differently and try again. And so rather than doing that in the costly way of just the APIs you produce, I think there’s a lot of value and trying to do what you can to be critical and learn from the APIs that are already out there to say what’s an API you really, okay, why? What are the things that you really about it? And help to get that to inform what you do in your own designs.
Sam Taggart 00:20:48 So as a connoisseur, do you have an example of a well-done API that you really like?
Wesley Beary 00:20:53 Yeah. So in terms of my more recent work, a lot of what I’ve been working on has been CLI stuff. And so an example of a CLI that I often will look to just to see how they did it and to get inspiration is the GH CLI, which is GitHub’s, CLI. And itís anything when you’re being critical of whatever, it’s like I wouldn’t do everything exactly the way that they did, but I mostly quite what they did. And so again, if Iím, I think I want to maybe take from that example and do something what they did, I’ll run through that command six or eight times and with different parameters and try to really think about what is it that I actually about this? What is the thing that I because I’m not implementing the exact same operation on my side. I’m just doing something that is maybe inspired by that. So getting a feel for it and then trying to bring that back and have something that also has some of the same, I guess feel and same taste so to speak has been productive.
Sam Taggart 00:21:45 Along those lines, is there a particular API you’ve dread using?
Wesley Beary 00:21:48 the most obvious example probably also from my recent work would be the openness SLCLI. what are the arguments? I don’t even know. I must look them up every time and then still have not the highest confidence that I’ve done the thing. They’re just very confusing, very cryptic I guess you might say.
Sam Taggart 00:22:04 The one I find most frustrating I think is Git is rough.
Wesley Beary 00:22:07 I’m not going to lie. that’s come up before when I’ve talked to people of I what Git does. I don’t necessarily the interface that it provides to me to actually do it for the core stuff it’s fine. I’ve done it so many times that it’s muscle Emory almost. But for anything beyond that, again itís back to documentation and then I’m not sure that I did it correctly. I don’t know how many rebates I’ve ended up messing up because yours and theirs don’t mean what I think that they do, and the order of things really matters and it’s not super user-friendly despite being very powerful.
Sam Taggart 00:22:36 So two things I dis about it in particular, and you mentioned one of them is the naming the R’s and there’s but also reset, revert and restore all kind of set things back to a previous state but in different ways and it’s very, it’s nice that they’re all the same. Because you remember they all set something back, but trying to remember which one sets what back is really hard and the inconsistency as well. because I keep remembering that for branches and tags you delete them but for sure remotes you remove them and that I’m always wait, which 1:00 am I doing here? So those little polish things.
Wesley Beary 00:23:09 , and it’s something that I think about a lot when I’m working on interfaces of just for me at least when I see something that where it’s in this one case it says remove and this other case it says destroy, then I’m left thinking is there a meaningful difference between these two? Is it just coincidence that they happen to have a different name or does remove really indicate something different than delete? Maybe remove means that it’s not sort of connected anymore. There’s not an association anymore but the object still exists. Or maybe it just deletes the object, and they called it two different things for no apparent reason. It makes me question how much I understand the interface in the first place. Which is not what you want your users to be doing. Do I even know what I’m doing here? I thought I did but now you’re making me question everything. That’s the worst case really.
Sam Taggart 00:23:49 I always go back to everything’s done for a reason. It doesn’t always necessarily make sense. It’s not always a great reason, but when you don’t know the reason, youíre left questioning well what was the reason?
Wesley Beary 00:23:57 And can I ever understand what that reason was? Do I need to understand what that reason was?
Sam Taggart 00:24:02 Is it important enough to waste time trying to figure it out? Exactly. Speaking of surprises, what is the principle of least surprise?
Wesley Beary 00:24:11 I think it comes from Matts the creator of Ruby. So a lot of my background is in Ruby programming and it’s something that I’ve seen talked about in the context of Matts. So basically ideally your interface is set up in such a way that if you basically learn one part of it and you move to another part, you should be able to guess basically how things work. So talking about our get example, delete versus remove, that violates I guess the principle of least surprise ideally, if you learn that getting rid of an object is done by calling delete and you move to a new object, the principle of least surprise should say basically you should be able to call delete and it should work, ? Or perhaps give you a meaningful message or instruct you as to how you should do it otherwise? you shouldn’t be surprised of, wait a second, why is this called remove? that’s just again, not a good experience for the user. So trying to just minimize that as much as possible, make as much transfer between things as you can.
Sam Taggart 00:25:03 To your comment about giving hints. I will give get credit, it’s gotten better over the years at giving hints. when you’re in the middle of AASE or something, you can do the get status and it gives you a little hints on what to do. There’s still kind of cryptic, but at least it’s something.
Wesley Beary 00:25:16 that’s been some of my worst experiences with various tools I think is, this sort of, you’ve done something wrong, please try harder, please try again. It’s, well what? What do you mean I don’t, I don’t understand what I did wrong in the first place? What am I, how am I supposed to deal with this?
Sam Taggart 00:25:28 My favorite programming language lab, you has you get file not found error, but it never tells you what file it can’t file. Which is always interesting because you’re, oh you’re looking for it, you can’t find it, why can’t you tell me so I can go find it?
Wesley Beary 00:25:41 Yep. The dreaded something went wrong, you’re, great, thanks for letting me know. I guess.
Sam Taggart 00:25:47 So talking about principle least surprise, there’s this idea of interrogation versus implicit discoverability that seems kind of related. Can you talk about that a little?
Wesley Beary 00:25:56 Sure. the idea with discoverability I think is more to that sort of I guess? I wasn’t sure what this was going to be. So I tried a thing, and it worked out versus more I called a method and it listed to me all of the options and then I chose one of those options. So it reveals to you the possibility. I don’t know. this is one of the areas where I guess in some ways, I would rather not choose probably it seems a kind of improv theater both and situation. I think there are times where it’s very helpful to be able to say, can you just tell me which operations are available here? But ideally you could also just guess and be fine, different users resonate more with one or the other. Some users maybe would never feel guessing, maybe that’s just not their thing. And sometimes you would never guess the thing because it’s an operation that only exists on that one object so you never would’ve seen it before. So, I think there’s a use case for both.
Sam Taggart 00:26:46 Do you have any specific tips for documenting or describing APIs? And as I’m thinking about this, is it better to have separate documents for that or would you rather interrogate the API to figure out what’s there or do we need both?
Wesley Beary 00:26:59 I’ve done different things over time. These days I lean towards using open API to write a spec basically that describes what the API should be doing. And then from there, thereís a lot of great tooling that will generate nice HTML documentation from that that you can give out to users. And then we even use a lot of tooling to take that same spec to be able to run our tests against it. So we have in testing basically if you make an API call that does not somehow match with the spec, that will just raise an error automatically so that we can make sure that basically what we’re documenting lines up with what we’re testing, which lines up with what we’re actually presenting. And that has been really helpful to have that be something that’s basically machine readable because then you can do interesting things with it as opposed to literally just a marked-on file somewhere that says what it does. Which is more prone to drifting away from what is in actuality happening.
Sam Taggart 00:27:48 Stale docs definitely cause a lot of problems.
Wesley Beary 00:27:51 Yes, for sure.
Sam Taggart 00:27:52 Going back to our principle least surprise a little bit, how do you balance being new and innovative and doing things in a different way and sticking to what people are used to and what they expect? You have an example of that?
Wesley Beary 00:28:04 Sure. I wouldn’t say that I’ve always gotten that balance correct. I think some of it goes to what the needs of user, I think doing anything innovative, is highly unly to be something that will fit into the principle of Lisa prize. if they’ve never seen it before, they’re not going to guess it. That’s just not how that works. So there’s a couple parts to it. One is obviously if you are going to do something new, at least do it consistently. Don’t just do not something new in one place and do something else that is new to do the same kind of thing somewhere else. try to have a pattern basically. So that’s one of the things that I guess I lean towards is try to have a new pattern, not a new one-off.
Wesley Beary 00:28:44 So what we were talking about with having the actions be nested in grouped together. That’s a pattern that you can repeat? So you could have, anytime there are actions anywhere in the API, they appear this. And here’s the set of rules that apply to actions so you can understand what to expect there. But, I think some of it just comes down to how much pain is this causing or not? How much confusion is it causing or not? Is it worth trying to do something a little bit different to ease that? And there’s a lot of cases where more and more I lean towards doing the again, both and thing. Of maybe I can provide the old way and the new way and so then I can make an argument for why the new way is better. And if people are interested and buy that argument and have those needs, they’ll probably use the new way. And if they donít, they can still use the old way. it’s not all or nothing. you have to get ramped up to my weird thing that might or might not be good and let’s in order to even use my API.
Sam Taggart 00:29:33 Would an example of that maybe be running a GraphQL server and a regular CRUD server side by side or something that.
Wesley Beary 00:29:39 Quite possibly. And sometimes that’s just not reasonable or realistic. Because the operational cost of running more than one thing is high. It’s some of the assumptions that crud versus GraphQL make may be incompatible in terms of how you’re even storing the data. things that. But, I think that that could be a reasonable example. You
Sam Taggart 00:29:57 Mentioned being a Ruby developer, you have a ruby gem called fog. Do you want to talk about that a little bit and what it does?
Wesley Beary 00:30:04 Sure. This is kind of the, from the early days of cloud, I was a Ruby developer, I was working at cloud infrastructure startups, and I was finding that in when AWS even was quite new and there were some other competitors just coming online that the clients available to me to interact with, those were left a lot to be desired. I guess they either were non-existent or just not very good yet. And so I initially set out just to add clients for a couple of the new AWS services that I didn’t see any clients for yet. I think I want to say maybe simple DB or something was the first one that I did, which I’ve not even heard anybody talk about in a really long time, I don’t think. But at the time I was just, this is interesting. And so, I kind of dived into writing these clients just because cloud seemed exciting and I wanted to learn more and I wanted to be able to use it.
Wesley Beary 00:30:55 And again, I just, the tools weren’t there. So at first it was just me dabbling with that and then over time it grew to be a lot of clients for a lot of different APIs as my interest continued to grow or people asked me if I could add support for something else and then eventually I worked to start building abstractions on top of that so that you could have some degree of interoperability, sort of the adapter pattern basically of saying I would a server. Okay, well which cloud provider would you a server on? And have it be able to do at least many of the things to make it interoperable between those providers. So it would understand that when you say you want a server on Rackspace, it’s this API call that you need to make, but if you want to server on AWS, it’s this other different API call that you might need to make.
Sam Taggart 00:31:36 That is definitely something that I’ve struggled with a lot is having things that objects that do similar things but have different APIs and trying to come up with good abstractions around that. What lessons have you learned from fog that might help other people with that?
Wesley Beary 00:31:49 Sure. It depends a lot on the nature of the objects I guess. And I think this kind of relates maybe also to the action pattern I was talking about actually with just something I kind of stumbled upon much later. But the closer it is to basically just crud, the easier it is going to be to have something that is more interoperable, a repeatable pattern. So a good example of that would be Cloud storage. Whether it’s S3 or the many other S3 clones and competitors that now exist. the interface to most of them is pretty comparable. You create folders to contain files, you create files within those folders, you fetch them, you update them that’s kind of it. So creating a more uniform interface to all of those is at least relatively straightforward.
Wesley Beary 00:32:34 The thing where it got much trickier is the example I gave of servers, when people say server, they don’t always, always even seem to mean the same thing. there’s a lot of different things that could mean in terms of is it bare metal, is it virtualized, is it somewhere in between? Does it have volumes that are attached that you need to do something with? Do you put an image onto it? Do you put a docker thing onto it, there’s a lot of different things that could mean. So the challenge there definitely be became, it felt the interface that resulted was sort of this lowest common denominator thing where technically a can Buddha server, but is it actually a meaningful useful server if you use the generic interface? Or do you have to get into all these particulars for that particular service in order to actually get it to do what you want? At which point how much value is this really adding? So I don’t know, it was an interesting thing to explore and there are parts of it that were great, but there’s other parts that, just because the services are specialized and they’re trying to differentiate from each other or whatever, they keep pulling away from you all the time. So it’s hard to just keep them close enough to continue to be meaningful and interesting. I’ve
Sam Taggart 00:33:31 Run into that a lot in my work because I have a variety of instruments and say a digital multimeter. And people choose specific multimeters because they want this one specific piece of functionality. And then if you put that in the generic interface, then you run into problems and you’re breaking all the solid rules or whatever and things that.
Wesley Beary 00:33:49 It’s, that’s where it gets really tricky is, when there’s differentiation, how do you deal with that? And of course these are competing companies so they’re trying hard to differentiate. So it’s this never-ending battle basically to try to figure out some way to make all of those pieces fit together as they continue to try to pull apart.
Sam Taggart 00:34:05 Let’s change topics a little bit and talk about anchor and local host.
Wesley Beary 00:34:11 Sure.
Sam Taggart 00:34:11 Can you talk a little bit about what Anchor is and what it does and what need you’re trying to fill?
Wesley Beary 00:34:16 Relates to our discussion earlier of how open SSL is kind of a beast to work with. Increasingly people see the need and value to do encryption and to have certificates, but they’re still kind of a pain to work on and work with. And so the goal with Anchor is really to make them more accessible and easier to use so that everybody can encrypt whenever and wherever they need to. And so we’re slowly but surely trying to chip away at that problem and provide more capabilities to do that more readily.
Sam Taggart 00:34:47 I think ease of use is really important when it comes to security stuff this. I was listening to a; somebody talk about security the other day and she mentioned that a simple thing you can do is when you Google how to do something to just add a how to do it securely onto the end. Because everybody always puts the easy answer, oh I can’t access this file, what should I do? Oh, change the permissions and make them wide open. Or my SSL certificate’s not working. Oh well here’s how you disable the SSL certificate.
Wesley Beary 00:35:12 Yes. The number of times I’ve seen the response to security questions be, here’s how you turn off HTPS, I can’t even begin to count. And it just kind of hurts me, makes me die a little bit more inside every time I see that because it’s not a good sustainable, reasonable approach but it gets the problem solved quickly so it’s tempting.
Sam Taggart 00:35:31 So turn off the firewall or something similar.
Wesley Beary 00:35:34 Yep, for sure.
Sam Taggart 00:35:35 Those are often, good as a first step just to see, okay, that’s what’s causing my problem. But if you stop there and say, oh, I solved my problem, you walk away, well you’ve just created more problems.
Wesley Beary 00:35:45 For sure.
Sam Taggart 00:35:47 So how does your, you have some command line tools and stuff to make things easier for developers. How does it help solve that problem of, oh, I could just do this. Is there an equally easy solution with your command line tools to solve some of those problems?
Wesley Beary 00:36:01 In particular with the anchor CLI, we’ve developed what we call LCL host. And so the idea with that is to try as much as possible to make it really easy for you to get encryption working on your local machine. So you’re running your app server in whichever language and you want it to serve secure content to your browser on your machine. Generally this is, challenging because a lot of the ways that you would get certificates normally let’s encrypt or things like that, you need to do DNS management and other things and, most people probably don’t necessarily have DNS pointing to their local machine in a way that they can easily do that. And so historically you would basically have to provision the dreaded self-signed certs which again, you have to use cryptic commands to actually generate them, but then can also be challenging because now you’ve solved that problem for you.
Wesley Beary 00:36:53 But if you’re on a team of developers, which most people probably are, this is still a problem for everyone else on the team. And there’s a lot of examples for instance, if you’re trying to use Stripe or Apple Pay or something and you’re trying to develop on that, you can’t use those interfaces in your browser if it’s not a secure context. So just doesn’t work. So the idea with a lot of the LCL host stuff is to make it as easy as possible to do that. So we have a single command basically that you can run that just goes through the steps that are necessary to provision a certificate for your machine to install it where it needs to be in your trust stores so that your browser will recognize and use it and other tools will recognize and use it. And then within just a few minutes you’re up and running and you can see again whatever your web app is delivered through a secure context so that you can get back to doing whatever you need to do.
Sam Taggart 00:37:38 So previously we talked about web APIs a lot. Do you see a major difference between web APIs and CLI APIs?
Wesley Beary 00:37:47 Sometimes, it depends a lot on just what the use cases are. So there are parts of our API that we’ve developed basically because the LCL command that we were just talking about is trying to accomplish certain things. I’m not sure if end users normally would be trying to do those things or to be doing them in quite the same way. So I think it might make sense to for instance, provide the normal CRUD operations across your different resources because eventually everybody’s going to need them or whatever. But maybe to have particular actions or additional endpoints that are specifically for the CLI, because again, it just needs to do things sometimes that other people don’t necessarily need to do or care to do. And so, I don’t know, it’s just a mix, it’s just whatever is needed for one or the other.
Wesley Beary 00:38:30 But I think the other thing is we have for better or worse designed certain CLI actions that actually require multiple API calls beneath behind the scenes. And that can get hairy because now it’s not impossible, but it’s very difficult to have transactional boundaries around all those actions. So it’s very easy then to end up in situations where things end up in weird partial states. Because two of the changes succeeded and then the third one failed. And so now do you roll that all back or something. Whereas if you just created a single API endpoint that was basically directly tied to that particular CLI action, it would be much easier to wrap all of the things in transaction boundary. Does that make sense?
Sam Taggart 00:39:09 If I’m understanding correctly, in this particular case, your CLI is actually calling an API behind the scenes a web API. Okay.
Wesley Beary 00:39:16 So we definitely have cases where the CLI itself is calling a web API and doing multiple different things and usually from the way it’s set up that’s just fine. But, there are some edge cases where we’ve ended up in outliers where, there’s stuff that’s in a weird state because only some of the commands succeeded because we don’t have a specific API on the website that is wrapping all of that in a transaction.
Sam Taggart 00:39:39 So they don’t match one-to-one necessarily,
Wesley Beary 00:39:41 There’s a lot of cases where the CLI is actually doing a few things for you instead of just one big thing.
Sam Taggart 00:39:47 Is that the case of then giving them the easy solution that is run this one CLI command versus making several different API calls for the more advanced user?
Wesley Beary 00:39:56 That is the case. And weíve already seen that in some cases too, where it’s sort of we are doing a somewhat more complicated thing in the CLI to make it easier for the users, but now we’re starting to identify that, oh actually this is just something that users are going to want to do more generally. And so now we’re moving towards developing some new API endpoints that will do the more complicated thing that the CLI used to do and then move the CLI to use that endpoint instead of, again, having it do four operations and take all the return values from all of that and knit them together in some particular way and then give you the output from that. Itís maybe that should just be one API call that does all of that. So, we continue to learn, we continue to make mistakes, we continue to tweak and iterate again, trying to be Agile, we talked about.
Sam Taggart 00:40:37 That kind of gels with this idea that I’ve heard a lot lately of pushing the complexity down and making things easier in the sense that the typical use case is to do these three things all in order and itís the 90% use case make a method or make an action that does just that.
Wesley Beary 00:40:53 And I don’t know that I have gotten all the way into doing that in the web API as much as I might, but we’ve definitely done that a lot in the CLII would say where, for instance, the LCL command that we mentioned, behind the scenes it actually runs, I think it’s either four or five CLI commands and you can technically run each of those in order yourself, but in many cases, why would you? let the tool do this for me. And, just generally I think, I think more and more about this notion of things like crud, things actions, but also having other action, other possibilities that are more a workflow almost, so that’s the sort of notion of there’s actually these three actions that people very commonly want to do together. So maybe we should have a sort of workflow that encompasses those set of things and just can run end to end and maybe it includes some hints or extra information that you wouldn’t get if you ran them individually because this is a router thing or maybe it doesn’t, I don’t know.
Wesley Beary 00:41:43 But it gives you some context to provide something specifically to suit that use case in a way that is more tailored, that provides a nicer experience that gives the person just what they need. Especially again when it’s a super common use case that happens day in, day out.
Sam Taggart 00:41:57 So maybe they do step one and then it prompts, do you want to continue to step two or you can just cancel out or something that? So wrapping things up, talk that I heard you give recently, you mentioned an HCDP API design guide. Where can people get that and what is it and what does it cover?
Wesley Beary 00:42:15 Sure. It’s on the inter agent I-N-T-E-R, agent GitHub. And it was from when I worked at Roku. So at Heroku I was the lead designer for the version three API. And so coming out of that, we tried to basically kind of create a style guide. I donít know if you’re familiar with the notion of style guides from say writing or whatever. There’s the, is it the Chicago style guide and thereís the AP style guide and things like that that kind of say, here’s when you should or shouldn’t use commas or apostrophes or dashes. So the idea was kind of trying to copy from something that and at least in part it was geared towards us doing that internally, I’ve designed this API, I now want to try to explain to you how and why I’ve done certain things and how you should do them so that it’ll be easier for other people to contribute and participate and for us to continue to have something that’s consistent, to have less surprises, better principle of least surprise, things like that.
Wesley Beary 00:43:13 And having done that, we decided that a lot of the advice was generically useful and interesting and so it would be good to share that more broadly. And so we opened it up and at least to the public and, it tries to cover just a lot of things talks about the actions pattern that I mentioned. It talks about versioning and ideas and recommendations for that and just kind of tries to cover a lot of different things of how to try to again have an API that’s consistent that people can kind of understand and navigate that it doesn’t have to much surprise. Ideally. Just first principles for a lot of that stuff to hopefully help make it easier for people to learn from my taste. It’s a cookbook almost or something of I had developed this taste for APIs or whatever and I’m trying to explain to someone how and why you might want to follow along with those examples.
Sam Taggart 00:43:57 Do you have any idea of the adoption of this guide? Do you know anyone who’s been using it?
Wesley Beary 00:44:03 I got a bunch of stars on GitHub. That’s a good sign. Probably. I’ve been told by people that they found it very interesting and influential at the time because not a lot of other people had tried to talk openly about that kind of stuff. you could go and see what somebody’s API was, but you didn’t really see an explanation of how or why and so I gather it was influential. But, just through anecdotes really and I guess through the star counts. So I hope it helps people. That’s why we do stuff like that.
Sam Taggart 00:44:32 Great. Is there anything else about APIs that we haven’t talked about that you would like to mention?
Wesley Beary 00:44:37 another relating to open API and things like that too. The other part of that that I found to be quite useful that we didn’t talk about is mocking, which is if you have an open API spec, there are a lot of nice tools actually that will basically start a server that will provide all of the API endpoints that you have to find that you can actually start to write a client against and get back. Not real but realistic responses. I found that to be super helpful. Again, going to the notion of trying to make the mistakes in the cheapest way possible. So a lot of my flow now is sort of like write a spec that is what I think I need, then start to write a client against it and that usually helps me to reveal quite quickly what I forgot, what I overlooked, and then adjusting the spec and then actually go into the real implementation of it once that spec is settled down. It isn’t a way that I developed historically and having stumbled upon that, I found it useful. So I think that’s a good one to keep in mind for people.
Sam Taggart 00:45:32 So in that case then you write the client or a sample client before you actually write the server? Is that Yes? Okay?
Wesley Beary 00:45:39 For sure because again, I don’t want to end up in the situation where I do all of the work of writing the server code and then realized that I had fundamental misunderstandings about what I was trying to do. And so doing the client will often help me to get there faster. So whether that’s doing some, in my case, again, I was working on CLI stuff a lot. I would start to write out the CLI operation and make it work at least in the shape of things working before I’d ever implemented the server. And that would quickly reveal to me, oh, I left out two or three fields that really ought to be here, or this doesn’t get me this piece of information that I needed. So do I add this to this API endpoint?
Wesley Beary 00:46:15 Do I make another call to another API endpoint? Do I make a new API endpoint that includes both of them? What do I need to do to get to where I need to be? Definitely has helped me to again, be more agile basically, because I can iterate before doing the server implementation as well. Because at the end of the day, you’re going to have to implement the server and the client and probably the spec and so, that just feels the faster way. Usually the client is simpler in my experience than the server is. So work on the spec first, which is it’s the simplest and cheapest, then the client second because it’s the next simplest and cheapest, I guess. And then defer the server till the last because you have the most information then. So you have least likely to make mistakes more to be able to just push through and get it done.
Sam Taggart 00:46:57 That sounds great advice. Thank you.
Wesley Beary 00:47:00 Thank you.
Sam Taggart 00:47:01 For SE Radio, this is Sam Taggart. Thanks for joining us. [End of Audio]
