Archived [2020-03-31] - Directive on Management of Information Technology

Information technology (IT) enables the federal government to effect operations and service transformation. IT matters strategically for increasing government productivity and enhancing government services to the public for the benefit of citizens, businesses, taxpayers and employees.

Date modified: 2019-08-02

Supporting tools

Guidelines:

Standard:

Mandatory procedures:

Hierarchy

Directive on Management of Information Technology

View complete hierarchy

Appendix D - Mandatory Procedures for Application Programming Interfaces

D.1 Effective date
- D.1.1These procedures take effect on December 1, 2018.
D.2 Procedures
- D.2.1This procedures provide details on the requirements set out in section 6 of the Directive on Management of Information Technology.
- D.2.2Mandatory procedures are as follows:
  - D.2.2.1Follow the Government of Canada Digital Standards – APIs must be developed following the Government of Canada's Digital Standards, specifically :
    - D.2.2.1.1Design with users by:
      - D.2.2.1.1.1Working with the developers who are expected to consume your API to ensure the interface specification meets their needs and addresses any constraints or limitations they may have.
      - D.2.2.1.1.2Building APIs against the business requirements consuming systems are intended to support and not the backend data structures they're accessing.
    - D.2.2.1.2Empower staff to deliver better services by:
      - D.2.2.1.2.1Ensuring the necessary tooling, training, and processes are in place to support a robust and agile API development and lifecycle management process.
      - D.2.2.1.2.2Adopt Continuous Integration and Delivery (CI/CD) and Test Driven Development (TDD) practices supported by automation tools and integrated security testing. This provides the basis for DevOps adoption as maturity improves.
    - D.2.2.1.3Work in the open by default by:
      - D.2.2.1.3.1Building APIs which can be publically consumed and allow reuse when working with non-sensitive data.
    - D.2.2.1.4Use open standards and solutions by:
      - D.2.2.1.4.1Exposing APIs using industry accepted open standards, while vendor proprietary protocols and data schemas must be avoided.
      - D.2.2.1.4.2Leveraging open source tools and frameworks to implement the API where possible.
    - D.2.2.1.5Iterate and improve frequently by:
      - D.2.2.1.5.1Design APIs with reuse in mind, but don't anticipate or guess at future requirements.
      - D.2.2.1.5.2Ensure APIs are designed in such a way as to enable iterations as new requirements and use cases emerge, while providing a reasonable level of backwards compatibility.
  - D.2.2.2Build APIs following the RESTful model by default. Representational State Transfer (REST) is effectively the standard for integration with cloud services and is also the standard set by the majority of other Governments with mature API programs. Follow industry best practices when designing and developing your RESTful API by:
    - D.2.2.2.1Represent resources as Uniform Resource Locators (URLs) by ensuring URLs represent entities and business objects, not operations on those entities and objects (i.e., avoid verbs inside the URL string).
    - D.2.2.2.2Use JavaScript Object Notation (JSON) and other JSON-based representations (e.g., JSON-LD) as the message structure wherever possible and apply the following practices:
      - D.2.2.2.2.1Form responses as a JSON object and not an array. Arrays can limit the ability to include metadata about results and limit the APIs ability to add additional top-level keys in the future.
      - D.2.2.2.2.1Avoid unpredictable (i.e., dynamic) object keys such as those derived from data as this adds friction for clients.
      - D.2.2.2.2.3Select a single grammar case for object keys, such as "underscore" or "CamelCase" and use it consistently.
    - D.2.2.2.3Ensure each verb represents a single operation on a given resource and avoid using request parameters to pass additional operations. However, the following are appropriate uses of HyperText Transfer Protocol (HTTP) verbs in the context of a RESTful API:
      GET (retrieve or query a resource),
      
      POST (create a new resource or initiate an action),
      
      PUT (update or replaced an existing resource), and
      
      DELETE (remove a resource)
    - D.2.2.2.4Use Uniform Resource Identifiers (URIs) to uniquely identify data that is returned as a part of a response so future operations can be performed on it, and iterations can leverage existing resources with minimal rework.
    - D.2.2.2.5Content negotiation must be done using the agent-driven approach via HTTP headers, including:
      - D.2.2.2.5.1ACCEPT and CONTENT-TYPE request headers are mandatory
      - D.2.2.2.5.2AUTHORIZATION header is mandatory for secured APIs
      - D.2.2.2.5.3The API Key must be passed in the header rather than the URI
      - D.2.2.2.5.4API keys/tokens must be securely setup and used appropriately
      - D.2.2.2.5.5The response must contain the CONTENT-TYPE header.
    - D.2.2.2.5.6Publish using Simple Object Access Protocol (SOAP)/Extensible Markup Language (XML) APIs only if there are technical constraints on either the provider or consumer sides. All SOAP endpoints should follow SOAP 1.2 specifications and be WS-I Basic Profile 2.0 compliant.
  - D.2.2.3Ensure APIs respond with message schemas that are well-defined, easy to understand, and consume by applying the following practices:
    - D.2.2.3.1Leveraging industry recognized common information models (e.g., NIEM, HR-JSON, HL7) where possible. If you have to define your own information model, create a model which is technology and platform agnostic rather than simply reusing a vendor proprietary schema. The appropriate use of common information models must follow Government of Canada Data Standards.
    - D.2.2.3.2Exposing raw data structures (e.g., rowsets, table arrays, LDAP DN) from backend systems must be limited to open data, reporting, and statistical APIs only, and strictly prohibited on master data, transactional, or business APIs. APIs should abstract backend physical data representation from the consumer.
    - D.2.2.3.3Avoiding relational data structures such as JSON and XML as they are hierarchical data structures and are not well suited to representing relational data. Relational data schemas should be flattened based on the perspective of the API consumer.
    - D.2.2.3.4Ensuring the constraints of a data schema are apparent when reading the schema definition. Generic data structures such as key-value pairs and generic fields are prohibited due to the inability to test API compatibility at the contract level.
    - D.2.2.3.5Avoiding building custom error codes and schemes which require heavy parsing by the consuming system. Conform to HTTP status codes when building RESTful APIs and conform to SOAP 1.2 faults when building SOAP APIs.
    - D.2.2.3.6Abstracting internal technical details whereby responses, including error messages, should abstract technical details which the API consumer has no visibility into. Internal technical errors, thread dumps, and process identifiers etc. must all be kept out of response data.
    - D.2.2.3.7Ensuring the interaction between the API consumer and provider is stateless. APIs must not expect any concept of session or management of state on the part of the consumer (e.g., passing session IDs). Any interactions where multiple APIs are called in a repeatable sequence to create a singular business interaction should be implemented as a composite API to avoid the burden of consumer-side orchestration.
  - D.2.2.4Validate your API design by consuming it with a production application within your organization.
    - D.2.2.4.1Build once for multiple channels by designing APIs so they can be consumed by internal Government of Canada systems, trusted partners, and external parties (i.e., public). Design should allow for different data access profiles to be applied, either to the API or at a proxy layer, without the need to build additional APIs.
    - D.2.2.4.2Pilot internally first by building APIs in parallel with an internal use case which would integrate with the API and use this internal pilot to validate the API implementation before publishing it for external use.
  - D.2.2.5Ensuring the security of the API when designing and implementing any API which provides access to protected or privileged data. As a baseline minimum, the following security control practices must be followed for any API other than those exposing public data (e.g., open data):
    - D.2.2.5.1Enforce secure communications by ensuring that sensitive data is never sent over an insecure or unencrypted connection, and where possible non-sensitive data should also be sent over a secure connection. Enable TLS 1.2 or subsequent versions, in accordance with CSE guidance.
    - D.2.2.5.2Design APIs to be resistant to attacks by ensuring that all APIs are designed and implemented to be resistant to common API attacks such as buffer overflows and SQL injection. Treat all submitted data as untrusted and validate before processing. Leverage schema and data models for ensuring correct data validation.
    - D.2.2.5.3Do not include sensitive data in request URLs as request URL strings can be tracked and compromised even with transport encryption. If a query involves sensitive data elements (e.g., SIN), pass the query parameters as a JSON message payload rather than in the URL request string.
    - D.2.2.5.4Protect access to APIs by implementing an access control scheme that protects APIs from being improperly invoked, including unauthorized function and data references. Always authenticate and authorize before any operation to ensure access to APIs are restricted to permitted individuals and/or systems. Use open standards such as OpenID Connect and Open Authorization 2.0 (OAuth 2.0) for RESTful APIs, and Security Assertion Markup Language 2.0 (SAML 2.0) for SOAP APIs. Ensure that the API key/secret is adequately protected. Open data APIs must be secured with an API key to allow for usage tracking and provide the ability to identify and prevent potential malicious use. Open data APIs must be secured with an API key to allow for usage tracking and provide the ability to identify and prevent potential malicious use.
    - D.2.2.5.5Apply secure token management practices whereby token-based authentication is strongly recommended and is mandatory for any APIs published to be consumed across the Government of Canada and/or externally. Use industry standard tokens; do not create custom tokens; and avoid using vendor proprietary token schemes. JSON Web Token (JWT) is required for RESTful API interactions. WS-Security SAML Token Profile is required for SOAP APIs. All access tokens must expire within a reasonable amount of time (i.e., less than 24 hours). In the case of SAML, the assertion expiry must be set to control the validity period of the entire authentication and authorization session.
    - D.2.2.5.6Use gateways and proxies instead of whitelists when exposing APIs to the internet. Use a secure gateway layer to provide a security control point instead of simply whitelisting inbound Internet Protocol addresses (IPs). The API Store's gateway functionality may be used. When consuming external APIs, route flows through a forward (egress) proxy instead of using IP address whitelisting on the outbound firewall.
    - D.2.2.5.7Integrate and automate security testing to validate any new changes to API source code and to ensure robustness of requested changes. Assess the change impact and conduct testing accordingly. Periodic audits of API access may be required depending on the nature of the data and its usage.
    - D.2.2.5.8Audit access to sensitive data whereby access to APIs dealing with sensitive and/or personal data must be logged for future audit and reviewed on a regular basis. Access logs must include as a minimum both the system and individual identifiers attempting to access the API along with the timestamp.
    - D.2.2.5.9Log and monitor for performance and activity by tracking usage and monitoring for suspicious activity including abnormal access patterns such as after-hours requests, large data requests, etc. Use logging standards (e.g. common event format) and integrate logs centrally. Identify dependencies and monitor for vulnerabilities, especially those for uploaded run-times that work as part of the API. Suspicious events must be sent to the appropriate security operations capability or authority in compliance with Government of Canada Cyber Security policies and Government of Canada Cyber Security Event Management Plan.
  - D.2.2.6Use consistent encoding and meta-data to ensure APIs are interoperable across organizations and maintain data consistency by implementing the following practices when defining the API:
    - D.2.2.6.1Use Unicode Transformation Format-8 (UTF-8) as the standard encoding type for all text and textual representations of data through APIs. It must be adhered to for all APIs published across the GC and externally. Other encodings may be used for single purpose and/or intra-organizational APIs if and only if there are technical limitations to using UTF-8.
    - D.2.2.6.2Use consistent date-time format by International Organization for Standardization 8601 (ISO 8601), in Coordinated Universal Time (UTC), as the standard date-time format for data and timestamp fields in APIs published across the GC and externally. The date format is <yyyy-mm-dd> while timestamp format is <yyyy-mm-dd>T<hh:mm:ss>Z. Any other representation of time in the source system must be converted to this format by the API.
    - D.2.2.6.3Support official languages whereby all English or French content returned as data are to be nested with BCP-47 language codes used as keys, specifically "en" and "fr". External facing APIs must reply with content in the requested language if the backend data support it. APIs must interpret the ACCEPT-LANGUAGE HTTP header and return the appropriate content. If the header is not set, then content in both languages should be returned. In the case of unilingual data and systems, every effort should be made to ensure that the language is appropriately indicated in the response message.
  - D.2.2.7Evolve and support the API throughout its lifecycle by practicing the following:
    - D.2.2.7.1Build iteratively by releasing new versions of the API as requirements change and/or new requirements are introduced. Actively solicit feedback from the API consumers to understand whether the API is providing appropriate value and make adjustments in future iterations.
      - D.2.2.7.1.1Version each and every change to the API no matter how small, following the v<Major>.<Minor>.<Patch> versioning structure whereby:
        
        Major = Significant release which is likely to break backwards compatibility
        
        Minor = Addition of optional attributes or new functionality that is backwards compatible, but should be tested
        
        Patch = Internal fix which should not impact the schema and/or contract of the API
        
        For example, going from v1.1.0 to v1.1.1 would allow a simple deploy-in-place upgrade as it is a patch, while going from v1.1.0 to v2.0.0 would be a major release and would require the legacy version to be kept while consumers test and migrate to the new version.
      - D.2.2.7.1.2The URL must reflect only the major version (e.g., v3). Versions must not be passed as a parameter or in the request header to force the API consumer to explicitly identify the version and to avoid defaulting to an incompatible version. Minor and patch versions do not need to be in the URL as they should not break backwards compatibility, but they should be clearly identified in the contract, interface documentation, and response message.
    - D.2.2.7.2Respect existing consumer dependencies by supporting at least one previous major version (i.e., N-1) to ensure consuming systems have time to migrate to the latest version of the API. Communicate your development roadmap with consuming teams and work with them to understand the impact of any major changes. Set clear deprecation policies and timelines up front so consumers understand how long they have to migrate to each new release before the legacy one is placed offline. Coordinate any necessary testing on all minor and major releases.
    - D.2.2.7.3Publish a designated point of contact to any teams consuming your API to the API Store. If the API is published for GC-wide or external use, publish a support email account. A phone number should also be provided for high-criticality APIs.
    - D.2.2.7.4Each API should be accompanied with a clearly defined Service Level Agreement (SLA). At a minimum, the SLA should define the following:
      - Support hours (e.g., 24/7, 9/5, coast to coast business hours)
      - Service availability (e.g., 99%)
      - Support response time (e.g., within the hour, 24 hours, best effort)
      - Scheduled outages (e.g., nightly, weekly, every 2nd Sunday evening)
      - Throughput limit (e.g., 100 requests per second per consumer)
      - Message size limit (e.g., <1Mb per request)
  - D.2.2.8Measure and publish API benchmarks whereby API performance should be benchmarked periodically to ensure the performance and capacity continually meets existing and projected business needs. The following actions should be taken:
    - D.2.2.8.1Run performance and load tests against the API to determine the response time and throughput during reasonable load as well as identify performance thresholds beyond which the API becomes unstable. Performance tests should be integrated into the development cycle, preferably through an automated continuous integration and continuous deployment (CI/CD) pipeline to ensure they're done on each new release.
    - D.2.2.8.2Performance summaries (e.g., average response time and associated throughput, max stable throughput) should be published alongside the API contract and SLA. This should be updated on every release.
    - D.2.2.8.3Runtime performance should be monitored and reported on to identify trends and to ensure the API has appropriate amount of capacity to meet usage demand.
    - D.2.2.8.4Throttling mechanisms should be implemented to control throughput against the stated SLA to guard against unexpected spikes in activity. It is better to reject requests that exceed the pre-defined throughput limits than to let the API crash.
  - D.2.2.9Use and design APIs sensibly by recognizing that APIs are not the solution for all integration scenarios. The following considerations should be made when deciding to implement an API as well as designing what types of queries are allowed to ensure the integration architecture is appropriate and sustainable:
    - D.2.2.9.1Query-based APIs (pull pattern) are preferred over data sink APIs (push pattern). Having consuming systems query APIs based on specific parameters ensures that only data that's required in the context of a business process or transaction is being passed. This approach also ensures that data access can be more finely controlled from a security perspective. Data sink APIs promote mass data synchronization and proliferation, which is antithesis to what an API-centric architecture is intended for. Bulk data integration techniques and tools should be used for data synchronization patterns and only when absolutely necessary.
    - D.2.2.9.2Restrict the use of wildcard queries in APIs as they can be dangerous from a data performance perspective. If wildcard characters are allowed, ensure there are restrictions on which and how many parameters can have wildcard input to prevent large data query sizes. It is much safer to reject a query with too many wildcards than to timeout on a database query on the backend.
    - D.2.2.9.3APIs exposing large datasets must support some form of data segmentation. The following are some common patterns for pagination along with appropriate use cases:
      - page and per_page – Best used to navigate large static datasets (e.g., reference data) where the same set of data is likely to be returned given the same page reference over time.
      - offset and limit – Best used for APIs fronting Structured Query Language (SQL)-based backends where the offset represents the data cursor on a given indexed column.
      - since and limit – Best used for queries where the consumer is interested in the delta since the last query and the backend data structure is indexed based on time.
    - D.2.2.9.4The ability to inject consumer defined query strings or objects into an API must be limited to open data, reporting, and statistical APIs only, and strictly prohibited on master data, transactional or business APIs. Dynamic and open queries create dangerous attack surfaces for APIs. It's better to invest more effort in identifying all the valid query use cases and design the API to specifically meet them. GraphQL can be used for statistical, analytics, and reporting purposes, but should not be used to support business transactions. OData should only be used if there are technical limitations by the backend system and only for an organization's internal APIs.
    - D.2.2.9.5Apply special considerations for bulk datasets as there will be scenarios where APIs will need to be involved in making bulk datasets available between systems or to the public. In those scenarios, apply the following considerations:
      - D.2.2.9.5.1Smaller datasets should be returned in low overhead formats (e.g., Comma-separated values (CSV) or JSON) rather than XML. The use of compressed file attachments should be avoided, especially when consuming external APIs as they may bypass malicious content scanning mechanisms.
      - D.2.2.9.5.2An API may be implemented as a trigger to initiate an out-of-band interface (e.g., managed file transfer) more appropriate for moving large data volumes.
      - D.2.2.9.5.3If the dataset is published on file servers already available to the consumer, an API could be implemented to return a link to a specific file based on specific request parameters.
  - D.2.2.10APIs must be published to be discoverable. How each API is to be consumed must be clearly documented. The documentation must be concise and up to date. The following practices must be adhered to in order to ensure APIs are documented appropriately without incurring the excess burden of managing companion documentation:
    - D.2.2.10.1All APIs should be published to the API Store for the purposes of discovery and lifecycle management. APIs must be appropriately tagged to indicate whether they are for intra-departmental, Government of Canada internal, or public consumption.
    - D.2.2.10.2Use OpenAPI, a machine-readable interface specification for RESTful APIs, to document RESTful API contracts. There are open source tools (e.g., Swagger) which can then generate human-readable documentation from this specification which avoids the need to create and maintain separate documentation.
    - D.2.2.10.3Each SOAP API must be accompanied with a Web Services Description Language (WSDL) contract. The WSDL is a machine-readable specification which allows the API consumer developer to generate the consumer code.
    - D.2.2.10.4Publish unit tests and test data as the most effective way to document what an API is supposed to do alongside the API contract. This becomes easier if Test-Driven-Development (TDD) methodology was followed in the development of the API.
    - D.2.2.10.5Avoid heavy companion documents. The need for a large separate document explaining each method and attribute is usually an indication that the API is too large, generic, or complex. Consider breaking the API down into smaller components and making the message structure more constrained.

Report a problem or mistake on this page

Date modified:: 2017-08-24

Search and menus

Search

Archived [2020-03-31] - Directive on Management of Information Technology

Supporting tools

Hierarchy

Archives

Appendix D - Mandatory Procedures for Application Programming Interfaces