This is the first instalment of a series of articles providing you with technical guidance on API technology – or “application programming interface”.
This content will be published according to two formulas:
- Here, as episodic content, one new instalment every about two weeks, and
- After the series’ completion, in early January, as a single document that you can download and read offline.
This unabridged version will also be available in French and German.
Each instalment will also be followed by e-learning modules on the same topics available here, to make them come alive and more memorable.
The themes we are going to touch extend across the full spectrum of what is useful to know, from the basics to the more advanced topics. We will start with a general introduction, providing general context, objectives and plans for further development. This will be followed by a general overview of API technologies – introducing the basics about API construction. We are going then to go deeper in detail on a few mainstreams ways of “doing API’s”: RESTful interfaces first, and the more modern GraphQL later. Finally – we are going to cover API security design, and how to document API’s, describing all requirements for a comprehensive documentation of an API and its behaviour.
So, stay tuned and don’t forget to discuss the materials by commenting on each instalment, or with your peers in our forum.
1. Overview of API technologies – Part 1 of 2
As software are complex systems it is important to structure software into manageable pieces. The basic units are modules and libraries in programming. Later larger, loosely coupled components have been implemented as distributed systems. The communication between the distributed components was still the abstraction of calling procedures of a foreign module using “remote procedure calls” (RPC) over the network. These RPCs have been mostly restricted to a homogeneous programming environment and specific framework, e.g. “Java RMI”. As the software systems became bigger and bigger the distributed components evolved into independent services and the “Service-oriented Architecture” (SOA) paradigm became popular. Each service has a clear functional focus and responsibility. This aspect became more sharpened with the appearance of the second generation of SOA pushed by the “Domain Driven Design”1 (DDD) and “microservices”2, where the border of a service is defined primarily by its core domain, its well-defined interfaces (API) and the organizational responsibility through a fully accountable team for a service. The evolution from simply distributed components to full-blown services strengthen the significance of the connecting APIs and has shown that different kinds of APIs are needed from a technical point of view. To explain this, see figure 1, showing the paradigm of “Self-Contained Systems” (SCS).
Figure 1: Self-Contained Systems
Based on the results of the “Domain Driven Design” one or more micro services implement a domain as a “service” with clear bounds. The service can be bundled with the data it manages. A service provides one or more APIs to other services. Especially, a service can be bundled with a micro frontend to provide a GUI to the users. A bundle of logic, data and frontend is called “Self-contained System”3 (SCS). For a domain-based service or SCS one team should be completely responsible. The figure points out that there are different ways services and SCSs can communicate to each other. A weblink in the GUI can be used to jump a new business transaction in the user interface, e.g. showing data, creating a new item and much more. The classical approach is that services communicate using a REST interface. But there can also signal events in an asynchronous mode to each other. In addition to a fixed function call between services a GraphQL interface can be provided to a frontend or other services to read and update data in a flexible manner. All these interface types are explained in detail in the later sections.
To make sure that such a complex system will be maintainable and the services reusable in the long term all these interfaces must be well-defined. Well-defined means that an API must
have a comprehensive formal specification as well as a comprehensive documentation, which were complete, understandable, consistent and correct
have a comprehensive formal specification as well as a comprehensive documentation, where both must be compliant with the given architectural policies and be usable by third parties hassle-free and
have a complete set of written binding requirements, which don’t leave spaces for interpretation
Before, there exist only a few programming languages and runtime environments. But nowadays there is a zoo of programming languages and runtime ecosystems. The viewpoint is to use the best tool to solve a problem. So, the programming world became polyglot. This evolution demands programming language independent, interoperable, stable and widely adaptable realization of APIs and its clients.
How should you proceed to define and implement an API? It is worth remembering that an API is effectively an interface designed to support the transfer of data between multiple systems or services, irrespective of the programming languages that these systems use. Therefore, interoperability, i.e. the ability to work together with a variety of systems and programming languages, should always rank highly. In practice, this means that well-designed APIs should ensure that systems involved in the transfer of data can use different programming languages. You can best achieve this by programming APIs with a neutral schema definition language, e.g. the “Web Service Description Language”4 (WSDL) for WS-SOAP-based APIs. But while this approach is undoubtedly preferable and dogmatically superior, it is also fraud with practical difficulties. In reality, most APIs are not defined by following an "API first" approach 5 that designs APIs in agnostic terms. Instead, specifications and documentations are frequently derived from the source code of the framework used by the service or system to which the API is attached; e.g. a Java class definition. At first, this is a quicker approach to defining suitable APIs for the originating system, but it also means that these APIs are programming language and framework specific - and often poorly documented. As a result of this unfortunate status quo, there is a strong demand to revert to the "API first" approach - assuming that this will help developers to use APIs more easily and, thus, successfully. Following the "API first" approach means that the API should be created and documented first, using a rich schema definition language to generate the required code (in the preferred language) and to write comprehensive documentation. APIs that follow this principle will eventually be more interoperable - and, thus, will simply be better APIs.
In the following paragraphs, we explain and show in detail, how to achieve these goals.
1.1 Basic concepts
To understand modern API technologies, we explain in this section some fundamental basic concepts used by the APIs. In the beginning of distributed systems, a programmer has to understand the basic idea of the well-known principle of procedure calls and the idea of server and client to implement a “Remote Procedure Call” over a network.
With the idea of the “Web” the “Hypertext Transfer Protocol” (HTTP) and the web links as “Uniform Resource Locator”6 (URL) rise up. The HTTP protocol defines the basic transport layer for modern APIs. It is a text-based protocol, which could be easily read and understood by humans. Besides the basic control of the communication session, it allows client and server to exchange meta-information in the header and the payload in the body within a request / response cycle. The HTTP communication can be secured by encapsulating a communication session with a “Transport Layer Security” (TLS) mechanism. Nowadays HTTPS is a must to ensure that external and even internal users can’t intercept your communication.
In general, an URL is simply a universal form of an address on the Internet. In the context of modern APIs, a URL could address entry points and functions of a server or address a (multimedia) resource. An URL could also contain basic parameters used by functions/methods defined in the API. WS-SOAP, Event-Bus and GraphQL are examples for using the URL only for the address of the entry point. WS-REST and Weblinks use the full power of URLs and the HTTP(S) protocol to define the API functionality.
1.1.3 Data representation format
Besides the communication protocol and addressing the data representation format of the payload in the body is a core aspect of an API. Basically, you can use HTML forms for structured data, plain text or (Base64) encoded data (Binary Large Object, Blob). In practice there are several data representation formats in use for deeply structured data:
XML (Extensible Markup Language7), a strongly typed language to represent structured data using tags and attributes, e.g. <message lang=”de”>Hello</message>.
YAML (YAML Ain’t Markup Language9, a lightweight object notation using indented lines and properties, e.g.
While XML is principally used by the WS-SOAP approach, uses WS-REST and GraphQL primarily the more lightweight approaches JSON and YAML.
1.1.4 Predefined methods
The old-fashioned RPC approach provides procedures and functions in a very generic way, e.g. a function just adding to numbers. In contrast, business services handle more complex business objects. In contrast, the origin of the “Web” handles resources (data items, multimedia objects, business objects, …) in a uniform way. Therefore, the HTTP protocol predefines methods on these resources, which should be primarily used to define an API:
PUT, PATCH (update)
With these concepts in mind let us explore the different types of APIs in the next section.
1.2 API types
As shown in figure 1 Microservices and Self-contained Systems are communicating over various kinds of APIs. In the following subsections, we give an overview of the most important API types.
The “Simple Object Access Protocol” (SOAP or WS-SOAP) is a standardized protocol10 for remote procedure calls (RPC). RPCs trigger an operation between two systems using the HTTP(S) protocol and the XML data representation. HTTP(S) serves as the underlying transport protocol to access the entry point of a SOAP-service. The message which actually implements the RPC is contained in standardized XML data.
SOAP is still widely used, mainly because it serves the framework-specific interoperability requirements of existing applications that follow the "Service-oriented Architecture" (SOA) approach. It is also perceived as beneficial thanks to the huge SOAP-ecosystem of technical components and implementation guidelines. These include, for example, security services and mandatory technology policies that projects should consider.
A major challenge of SOAP is that APIs should use an agnostic schema definition language. This is the best starting point to create truly interoperable interface code 11 independent from the framework requirements of the system to which the API is attached. As mentioned earlier, the reality is, however, almost the exact opposite. API schemas will often be generated from the implementation code of a system, e.g. a Java class definition in an implementation (or framework) specific way.
XML and SOAP are still very present today, implemented in various existing or legacy interfaces and implementations. But they are also heavyweight technologies that are increasingly replaced by applications employing a RESTful or micro service approach. This API guide focusses on more modern approaches and, therefore, does not explore SOAP in more detail.
1.3 Coming next
In the next instalment of this series, we are completing this overview of API technologies before starting going deeper into the detail. We will introduce you to a few mainstreams ways of “doing API’s”: RESTful interfaces first, and the more modern GraphQL later. The overview completes by covering the basic principles of API security design, and how to document them and their behaviour.
- 1. A. Avram, F. Marinescu: Domain Driven Design Quickly, see https://www.infoq.com/minibooks/domain-driven-design-quickly/
- 2. M. Fowler: Microservices - a definition of this new architectural term https://www.martinfowler.com/articles/microservices.html
- 3. Self-Contained Systems - Assembling software from independent systems, see https://scs-architecture.org/index.html
- 4. Web Services Description Language (WSDL) Version 2.0 Part 1: Core Language, see https://www.w3.org/TR/wsdl20/
- 5. API University: Understanding API First Design, see https://www.programmableweb.com/api-university/understanding-api-first-design
- 6. Berners-Lee, Tim (21 March 1994). "Uniform Resource Locators (URL): A syntax for the Expression of Access Information on Objects on the Network", see http://www.w3.org/Addressing/URL/url-spec.txt
- 7. Extensible Markup Language (XML) 1.0, see https://www.w3.org/TR/REC-xml/
- 8. Introducing JSON, see https://www.json.org/json-en.html
- 9. YAML Ain't Markup Language YAML™) Version 1.2, see https://yaml.org/
- 10. SOAP Version 1.2, see https://www.w3.org/TR/soap/
- 11. The mapping of a SOAP specification to a concrete language and framework is not in all cases bijective and simple.