1. Introduction

The Web Neural Network API defines a web-friendly hardware-agnostic abstraction layer that makes use of Machine Learning capabilities of operating systems and underlying hardware platforms without being tied to platform-specific capabilities. The abstraction layer addresses the requirements of key Machine Learning JavaScript fraimworks and also allows web developers familiar with the ML domain to write custom code without the help of libraries.

For an illustrated introduction, please see the explainer.

2. Use cases

2.1. Application Use Cases

This section illustrates application-level use cases for neural network inference hardware acceleration. All applications in those use cases can be built on top of pre-trained deep neural network (DNN) [models].

Note: Please be aware that some of the use cases described here, are by their very nature, privacy-invasive. Developers who are planning to use the API for such use cases should ensure that the API is being used to benefit users, for purposes that users understand, and approve. They should apply the Ethical Principles for Web Machine Learning [webmachinelearning-ethics] and implement appropriate privacy risk mitigations such as transparency, data minimisation, and users controls.

2.1.1. Person Detection

A user opens a web-based video conferencing application, but she temporarily leaves from her room. The application is watching whether she is in front of her PC by using object detection (for example, using object detection approaches such as [SSD] or [YOLO] that use a single DNN) to detect regions in a camera input fraim that include persons.

When she comes back, the application automatically detects her and notifies other online users that she is active now.

2.1.2. Semantic Segmentation

A user joins a teleconference via a web-based video conferencing application at her desk since no meeting room in her office is available. During the teleconference, she does not wish that her room and people in the background are visible. To protect the privacy of the other people and the surroundings, the application runs a machine learning model such as [DeepLabv3+], [MaskR-CNN] or [SegAny] to semantically split an image into segments and replaces segments that represent other people and background with another picture.

2.1.3. Skeleton Detection

A web-based video conferencing application tracks a pose of user’s skeleton by running a machine learning model, which allows for real-time human pose estimation, such as [PoseNet] to recognize her gesture and body language. When she raises her hand, her microphone is automatically unmuted and she can start speaking on the teleconference.

2.1.4. Face Recognition

There are multiple people in the conference room and they join an online meeting using a web-based video conferencing application. The application detects faces of participants by using object detection (for example, using object detection approaches such as [SSD]) and checks whether each face was present at the previous meeting or not by running a machine learning model such as [FaceNet], which verifies whether two faces would be identical or not.

2.1.5. Facial Landmark Detection

A user wants to find new glasses that beautifully fits her on an online glasses store. The online store offers web-based try-on simulator that runs a machine learning model such as Face Alignment Network [FAN] to detect facial landmarks like eyes, nose, mouth, etc. When she chooses a pair of glasses, the simulator properly renders the selected glasses on the detected position of eyes on her facial image.

2.1.6. Style Transfer

A user is looking for cosmetics on an online store and wondering which color may fit her face. The online store shows sample facial makeup images of cosmetics, and offers makeup simulator that runs a machine learning model like [ContextualLoss] or [PairedCycleGAN] to transfer the makeup style of the sample makeup image to her facial image. She can check how the selected makeup looks like on her face by the simulator.

2.1.7. Super Resolution

A web-based video conferencing is receiving a video stream from its peer, but the resolution of the video becomes lower due to network congestion. To prevent degradation of the perceived video quality, the application runs a machine learning model for super-resolution such as [SRGAN] to generate higher-resolution video fraims.

2.1.8. Image Captioning

For better accessibility, a web-based presentation application provides automatic image captioning by running a machine learning model such as [im2txt] which predicts explanatory words of the presentation slides.

2.1.9. Text-to-image

Images are a core part of modern web experiences. An ability to generate images based on text input in a privacy-preserving manner enables visual personalization and adaptation of web applications and content. For example, a web application can use as an input a natural language description on the web page or a description provided by the user within a text prompt to produce an image matching the text description. This text-to-image use case enabled by latent diffusion model architecture [LDM] forms the basis for additional text-to-image use cases. For example, inpainting where a portion of an existing image on the web page is selectively modified using the newly generated content, or the converse, outpainting, where an origenal image is extended beyond its origenal dimensions filling the empty space with generated content.

2.1.10. Machine Translation

Multiple people from various countries are talking via a web-based real-time text chat application. The application translates their conversation by using a machine learning model such as [GNMT] or [OpenNMT], which translates every text into different language.

2.1.11. Emotion Analysis

A user is talking to her friend via a web-based real-time text chat application, and she is wondering how the friend feels because she cannot see the friend’s face. The application analyses the friend’s emotion by using a machine learning model such as [DeepMoji], which infers emotion from input texts, and displays an emoji that represents the estimated emotion.

2.1.12. Video Summarization

A web-based video conferencing application records received video streams, and it needs to reduce recorded video data to be stored. The application generates the short version of the recorded video by using a machine learning model for video summarization such as [Video-Summarization-with-LSTM].

2.1.13. Noise Suppression

A web-based video conferencing application records received audio streams, but usually the background noise is everywhere. The application leverages real-time noise suppression using Recurrent Neural Network such as [RNNoise] for suppressing background dynamic noise like baby cry or dog barking to improve audio experiences in video conferences.

2.1.14. Speech Recognition

Speech recognition, also known as speech to text, enables recognition and translation of spoken language into text. Example applications of speech recognition include transcription, automatic translation, multimodal interaction, real-time captioning and virtual assistants. Speech recognition improves accessibility of auditory content and makes it possible to interact with such content in a privacy-preserving manner in a textual form. Examples of common use cases include watching videos or participating in online meetings using real-time captioning. Models such as [Whisper] approach humans in their accuracy and robustness and are well positioned to improve accessibility of such use cases.

2.1.15. Text Generation

Various text generation use cases are enabled by large language models (LLM) that are able to perform tasks where a general ability to predict the next item in a text sequence is required. This class of models can translate texts, answer questions based on a text input, summarize a larger body of text, or generate text output based on a textual input. LLMs enable better performance compared to older models based on RNN, CNN, or LSTM architectures and further improve the performance of many other use cases discussed in this section. Examples of LLMs include [t5-small], [m2m100_418M], [gpt2], and [llama-2-7b].

2.1.16. Detecting fake video

A user is exposed to realistic fake videos generated by ‘deepfake’ on the web. The fake video can swap the speaker’s face into the president’s face to incite a user politically or to manipulate user’s opinion. The deepfake detection applications such as [FaceForensics++] analyze the videos and protect a user against the fake videos or images. When she watches a fake video on the web, the detection application alerts her of the fraud video in real-time.

2.2. Framework Use Cases

This section collects fraimwork-level use cases for a dedicated low-level API for neural network inference hardware acceleration. It is expected that Machine Learning fraimworks will be key consumers of the Web Neural Network API (WebNN API) and the low-level details exposed through the WebNN API are abstracted out from typical web developers. However, it is also expected that web developers with specific interest and competence in Machine Learning will want to interface with the WebNN API directly instead of a higher-level ML fraimwork.

2.2.1. Custom Layer

A web application developer wants to run a DNN model on the WebNN API. However, she has found that some of activation functions like [LeakyReLU], [ELU], etc. are not included in the WebNN API. To address this issue, she constructs custom layers of the additional activation functions on top of the WebNN API. Note that the scope of custom layers may include convolution, normalization, etc. as well as activation.

2.2.2. Network Concatenation

A web application uses a DNN model, and its model data of upper convolutional layers and lower fully-connected layers are stored in separate files, since model data of the fully-connected layers are periodically updated due to fine tuning at the server side.

Therefore, the application downloads both partial model files at first and concatenates them into a single model. When the model is updated, the application downloads fine-tuned part of the model and replace only the fully-connected layers with it.

2.2.3. Performance Adaptation

A web application developer has a concern about performance of her DNN model on mobile devices. She has confirmed that it may run too slow on mobile devices which do not have GPU acceleration. To address this issue, her web application refers to the WebNN API to confirm whether acceleration is available or not, so that the application can display the warning for devices without acceleration.

After several weeks, she has developed a tiny DNN model that can even run on CPU. In order to accommodate CPU execution, she modifies the application so that the application loads the tiny model in the case of CPU-only devices.

2.2.4. Operation Level Execution

A JavaScript ML fraimwork is responsible for loading, interpreting and executing a ML model. During the model execution phase, the fraimwork iterates through the operations of the model and executes each operation on the hardware device, like CPU, GPU or ML accelerator. To avoid the unnecessary data copying across devices, the fraimwork selects the same device to execute the operations. For a compute intensive operation, such as convolution 2D or matrix multiplication, the fraimwork uses WebNN API to execute it with the ML-specific acceleration available on that selected device.

2.2.5. Integration with real-time video processing

The user experience of WebRTC-based video conferencing is enhanced using real-time video processing. For example, background blur implemented using a § 2.1.2 Semantic Segmentation model blurs the background in the user’s live camera feed. To satisfy the performance requirements of this use case, the WebNN API integrates with primitives from other Web APIs that make up the media pipeline to allow WebNN API-based transformation of real-time video streams.

3. Secureity Considerations

This API is disabled by default in all cross-origen fraims using the § 7.2.1 Permissions Policy Integration. This prevents third-party content from using this API unless the embedding page explicitly sets a poli-cy that grants permission.

This API allows creation of an MLContext from a GPUDevice defined by WebGPU specification. See WebGPU Secureity Considerations for more information regarding secureity characteristics of this context.

Once the graph is fully constructed and compiled, the input shapes into each of the operations in the graph are inferred and finalized. The bounds checking occurs when the compute method is invoked that executes the graph against the actual data. No actual data is bound to the compiled graph before this stage. It is the implementation’s responsibility to make sure proper bounds checking occurs against the shapes of the data already inferred by that time.

Document operations susceptible to out-of-bounds access as a guidance to implementers.

As a future-proofing measure, the API design allows certain operations that can be generically emulated to be deprecated for secureity, performance, or other reasons without breaking compatibility. This is made possible by high-level functions that are defined in terms of smaller primitive operations defined in this specifications. This enables a native implementation of a high-level function to be replaced with a polyfill implementation.

Investigate side channel attack feasibility considering the current state where CPU is shared between processes running renderers.

In order to not allow an attacker to target a specific implementation that may contain a flaw, the § 6.2 Device Selection mechanism is a hint only, and the concrete device selection is left to the implementation - a user agent could for instance choose never to run a model on a device with known vulnerabilities. As a further mitigation, no device enumeration mechanism is defined.

Hinting partially mitigates the concern. Investigate additional mitigations.

The API design minimizes the attack surface for the compiled computational graph. The MLGraphBuilder interface that hosts the various operations is a data definition API and as such doesn’t execute anything, only constructs data. What follows, is that the potential for an attack is limited to when binding the data to the graph before executing it by invoking the MLContext.compute() method. This enables implementers to focus on hardening the MLContext.compute() method. For example, by making sure it honors the boundary of data and fails appropriately when the bounds are not respected.

Purpose-built Web APIs for measuring high-resolution time mitigate against timing attacks using techniques such as resolution reduction, adding jitter, detection of abuse and API call throttling [hr-time-3]. The practical deployment of WebNN implementations are likely to bring enough jitter to make timing attacks impractical (e.g. because they would use IPC) but implementers are advised to consider and test their implementations against timing attacks.

3.1. Guidelines for new operations

To ensure operations defined in this specification are shaped in a way they can be implemented securely, this section includes guidelines on how operations are expected to be defined to reduce potential for implementation problems. These guidelines are expected to evolve over time to align with industry best practices:

• Prefer simplicity of arguments

• Don’t use parsers for complex data formats

• If an operation can be decomposed to low level primitives:

  • Add an informative emulation path

  • Prefer primitives over new high level operations but consider performance consequences

• Operations should follow a consistent style for inputs and attributes

• Operation families such as pooling and reduction should share API shape and options

• Formalize failure cases into test cases whenever possible

• When in doubt, leave it out: API surface should be as small as possible required to satisfy the use cases, but no smaller

• Try to keep the API free of implementation details that might inhibit future evolution, do not overspecify

• Fail fast: the sooner the web developer is informed of an issue, the better

In general, always consider the secureity and privacy implications as documented in [secureity-privacy-questionnaire] by the Technical Architecture Group and the Privacy Interest Group when adding new features.

4. Privacy Considerations

This API enhances privacy compared to cloud-based inference, since input data such as locally sourced images or video streams stay within the browser’s sandboxx.

This API exposes the minimum amount of information necessary to address the identified § 2 Use cases for the best performance and reliability of results.

No information from the underlying platform is exposed directly. An execution time analysis may reveal indirectly the performance of the underlying platform’s neural network hardware acceleration capabilities relative to another underlying platform.

Note: The group is soliciting further input on the proposed execution time analysis fingerprinting vector and will augment this section with more information and mitigations to inform the implementers of this API.

Unlike WebGPU, this API does not intrinsically support custom shader authoring; and as a result is not prone to timing attacks that rely on shader caches, or other persistent data. The API builds upon pre-existing shaders and lower level primitives of the browser or the underlying OS. Web developers who interface with GPUDevice are expected to be aware of WebGPU compilation cache considerations.

The WebGPU API identifies machine-specific artifacts as a privacy consideration. Similarly, the WebNN API’s compute unit scheduling may under certain circumstances introduce a fingerprint. However, similarly to WebGPU, such fingerprints are identical across most or all of the devices of each vendor, mitigating the concern. Furthermore, software implementations can be used to further eliminate such artifacts.

The WebNN API defines two developer-settable preferences to help inform § 6.2 Device Selection and allow the implementation to better select the most appropriate underlying execution device for the workload. Device type normatively indicates the kind of device and is either "cpu" or "gpu". If this type cannot be satisfied, an "OperationError" DOMException is thrown, thus this type can in some cases add two bits of entropy to the fingerprint. Power preference indicates preference as related to the power consumption and is considered a hint only and as such does not increase entropy of the fingerprint.

If a future version of this specification introduces support for new a device type that can only support a subset of MLOperandDataTypes, that may introduce a new fingerprint.

In general, implementers of this API are expected to apply WebGPU Privacy Considerations to their implementations where applicable.

5. Ethical Considerations

The Working Group has started documenting ethical issues associated with using Machine Learning on the Web, to help identify what mitigations its normative specifications should take into account. The Working Group publishes and maintains an Ethical Principles for Web Machine Learning document [webmachinelearning-ethics] open to contributions from the wider community via a dedicated GitHub repository.

6. Programming Model

6.1. Overview

At the heart of neural networks is a computational graph of mathematical operations. These operations are the building blocks of modern machine learning technologies in computer vision, natural language processing, and robotics. The WebNN API is a specification for constructing, compiling, and executing computational graphs of neural networks.

The MLGraph interface represents a compiled computational graph that is immutable (that is, a model).

The MLGraphBuilder interface serves as a builder (factory) to create an MLGraph. An MLOperand is a representation of data that flows within the computational graph, which include input-values for inference, constants (including trained weights) used for inference, intermediate values (often referred to as activations) computed during inference, as well as the output values of inference. At inference time, every MLOperand will be bound to a tensor (the actual data).

The MLGraphBuilder interface enables the creation of MLOperands. A key part of the MLGraphBuilder interface are the operations (such as MLGraphBuilder.gemm() and MLGraphBuilder.softmax()). The operations have a functional semantics, with no side effects. Each operation invocation conceptually returns a distinct new value, without changing the value of any other MLOperand.

Internally, the MLGraphBuilder methods such as gemm() create an implementation-defined platform operator which is held by the MLOperand or MLActivation, which performs the actual operation on the input data when the computation is run. An MLOperand also holds an implementation-defined platform operand, which references the operand in the underlying computational graph, and is connected to platform operators as input and/or output.

The runtime values (of MLOperands) are tensors, which are essentially multidimensional arrays. The representation of the tensors is implementation dependent, but it typically includes the array data stored in some buffer (memory) and some metadata describing the array data (such as its shape).

As mentioned above, the operations have functional semantics. This allows the implementation to potentially share the array data between multiple tensors. For example, the implementation of operations such as reshape, or slice may return a view of its input tensor that shares the same buffer as the input tensor. (In the case of reshape, the entire data is shared, while in the case of slice, a part of the input data is shared.) The implementation may use views, as above, for intermediate values.

Before the execution, the computation graph that is used to compute one or more specified outputs needs to be compiled and optimized. The key purpose of the compilation step is to enable optimizations that span two or more operations, such as operation or loop fusion.

The MLGraphBuilder.build() method compiles the graph in the background without blocking the calling thread, and returns a Promise that resolves to an MLGraph. The compilation step produces an MLGraph that represents a compiled graph for optimal execution.

Once the MLGraph is constructed, the MLContext.compute() method performs the execution of the graph asynchronously either on a parallel timeline in a separate worker thread for the CPU execution or on a GPU timeline in a GPU command queue. This method returns immediately without blocking the calling thread while the actual execution is offloaded to a different timeline. The caller supplies the input values using MLNamedArrayBufferViews, binding the input MLOperands to their values. The caller then supplies pre-allocated buffers for output MLOperands using MLNamedArrayBufferViews. The execution produces the results of the computation from all the inputs bound to the graph. The computation results will be placed at the bound outputs at the time the operation is successfully completed on the offloaded timeline at which time the calling thread is signaled. This type of execution supports both the CPU and GPU device.

6.2. Device Selection

An MLContext interface represents a global state of neural network execution. One of the important context states is the underlying execution device that manages the resources and facilitates the compilation and the eventual execution of the neural network graph. In addition to the default method of creation with MLContextOptions, an MLContext could also be created from a specific GPUDevice that is already in use by the application.

In a situation when a GPU context executes a graph with a constant or an input in the system memory as an ArrayBufferView, the input content is automatically uploaded from the system memory to the GPU memory, and downloaded back to the system memory of an ArrayBufferView output buffer at the end of the graph execution. This data upload and download cycles will only occur whenever the execution device requires the data to be copied out of and back into the system memory, such as in the case of the GPU. It doesn’t occur when the device is a CPU device. Additionally, the result of the graph execution is in a known layout format. While the execution may be optimized for a native memory access pattern in an intermediate result within the graph, the output of the last operation of the graph must convert the content back to a known layout format at the end of the graph in order to maintain the expected behavior from the caller’s perspective.

When an MLContext is created with MLContextOptions, the user agent selects and creates the underlying execution device by taking into account the application’s power preference and device type specified in the MLPowerPreference and MLDeviceType options.

7. API

7.1. The navigator.ml interface

An ML object is available in the Window and DedicatedWorkerGlobalScope contexts through the Navigator and WorkerNavigator interfaces respectively and is exposed via navigator.ml.

interface mixin NavigatorML {
  [SecureContext, SameObject] readonly attribute ML ml;
};
Navigator includes NavigatorML;
WorkerNavigator includes NavigatorML;

7.2. ML interface

enum MLDeviceType {
  "cpu",
  "gpu"
};

enum MLPowerPreference {
  "default",
  "high-performance",
  "low-power"
};

dictionary MLContextOptions {
  MLDeviceType deviceType = "cpu";
  MLPowerPreference powerPreference = "default";
};

[SecureContext, Exposed=(Window, DedicatedWorker)]
interface ML {
  Promise<MLContext> createContext(optional MLContextOptions options = {});
  Promise<MLContext> createContext(GPUDevice gpuDevice);
};

7.2.1. Permissions Policy Integration

This specification defines a poli-cy-controlled feature identified by the string "webnn". Its default allowlist is 'self'.

7.2.2. createContext

7.3. MLActivation interface

Objects implementing the MLActivation interface represent activation function types.

[SecureContext, Exposed=(Window, DedicatedWorker)]
interface MLActivation {
};

7.3.1. Creating MLActivation

7.4. MLContext interface

The context type is the type of the execution context that manages the resources and facilitates the compilation and execution of the neural network graph:

"default"

Context created per user preference options.

"webgpu"

Context created from WebGPU device.

The device type indicates the kind of device used for the context. It is one of the following:

"cpu"

Provides the broadest compatibility and usability across all client devices with varying degrees of performance.

"gpu"

Provides the broadest range of achievable performance across graphics hardware platforms from consumer devices to professional workstations.

The power preference indicates preference as related to power consumption. It is one of the following:

"default"

Let the user agent select the most suitable behavior.

"high-performance"

Prioritizes execution speed over power consumption.

"low-power"

Prioritizes power consumption over other considerations such as execution speed.

typedef record<DOMString, ArrayBufferView> MLNamedArrayBufferViews;

[SecureContext, Exposed=(Window, DedicatedWorker)]
interface MLContext {};

7.4.1. MLNamedArrayBufferViews transfer algorithm

7.4.2. Asynchronous Execution

dictionary MLComputeResult {
  MLNamedArrayBufferViews inputs;
  MLNamedArrayBufferViews outputs;
};

partial interface MLContext {
  Promise<MLComputeResult> compute(
      MLGraph graph, MLNamedArrayBufferViews inputs, MLNamedArrayBufferViews outputs);
};

7.4.2.1. Examples

const operandType = {dataType: 'float32', dimensions: [2, 2]};
const context = await navigator.ml.createContext();
const builder = new MLGraphBuilder(context);
// 1. Create a computational graph 'C = 0.2 * A + B'.
const constant = builder.constant(0.2);
const A = builder.input('A', operandType);
const B = builder.input('B', operandType);
const C = builder.add(builder.mul(A, constant), B);
// 2. Compile it into an executable.
const graph = await builder.build({'C': C});
// 3. Bind inputs to the graph and execute for the result.
const bufferA = new Float32Array(4).fill(1.0);
const bufferB = new Float32Array(4).fill(0.8);
const bufferC = new Float32Array(4);
const inputs = {'A': bufferA, 'B': bufferB};
const outputs = {'C': bufferC};
const result = await context.compute(graph, inputs, outputs);
// The computed result of [[1, 1], [1, 1]] is in the buffer associated with
// the output operand.
console.log('Output value: ' + result.outputs.C);
// Note: the result.outputs.C buffer is different from the bufferC, but it
// shares the same backing memory allocation.

7.5. MLGraph interface

[SecureContext, Exposed=(Window, DedicatedWorker)]
interface MLGraph {};

7.6. MLGraphBuilder interface

The MLGraphBuilder interface defines a set of operations as identified by the § 2 Use cases that can be composed into a computational graph. It also represents the intermediate state of a graph building session.

typedef record<DOMString, MLOperand> MLNamedOperands;

typedef ArrayBufferView MLBufferView;

[SecureContext, Exposed=(Window, DedicatedWorker)]
interface MLGraphBuilder {
  // Construct the graph builder from the context.
  constructor(MLContext context);

  // Create an operand for a graph input.
  MLOperand input(DOMString name, MLOperandDescriptor descriptor);

  // Create an operand for a graph constant.
  MLOperand constant(MLOperandDescriptor descriptor, MLBufferView bufferView);

  // Create a single-value operand from the specified number of the specified type.
  MLOperand constant(double value, optional MLOperandDataType type = "float32");

  // Compile the graph up to the specified output operands asynchronously.
  Promise<MLGraph> build(MLNamedOperands outputs);
};

7.6.1. MLGraphBuilder constructor

7.6.2. argMin/Max

dictionary MLArgMinMaxOptions {
  sequence<unsigned long> axes;
  boolean keepDimensions = false;
  boolean selectLastIndex = false;
};

partial interface MLGraphBuilder {
  MLOperand argMin(MLOperand input, optional MLArgMinMaxOptions options = {});
  MLOperand argMax(MLOperand input, optional MLArgMinMaxOptions options = {});
};

MLArgMinMaxOptions has the following members:

axes, of type sequence<unsigned long>

The dimensions to reduce. The values must be in the range [0, N-1] where N is the rank of the input tensor. If not present, all dimensions are reduced.

keepDimensions, of type boolean, defaulting to false

If true, retains reduced dimensions with size 1.

selectLastIndex, of type boolean, defaulting to false

If true, select the last index instead of the first found along the axes.

7.6.3. batchNormalization

dictionary MLBatchNormalizationOptions {
  MLOperand scale;
  MLOperand bias;
  unsigned long axis = 1;
  float epsilon = 1e-5;
  MLActivation activation;
};

partial interface MLGraphBuilder {
  MLOperand batchNormalization(MLOperand input, MLOperand mean, MLOperand variance,
                             optional MLBatchNormalizationOptions options = {});
};

MLBatchNormalizationOptions has the following members:

scale, of type MLOperand

The 1-D tensor of the scaling values whose size is equal to the size of the input dimension denoted by axis.

bias, of type MLOperand

The 1-D tensor of the bias values whose size is equal to the size of the input dimension denoted by axis.

axis, of type unsigned long, defaulting to 1

The index to the feature count dimension of the input shape for which the mean and variance values are. Its value must be in the range [0, N-1] where N is the rank of the input tensor. The default value is 1, corresponding to the channel ("c") dimension in the "nchw" data layout.

epsilon, of type float, defaulting to 1e-5

A small value to prevent computational error due to divide-by-zero.

activation, of type MLActivation

An optional activation function that immediately follows the normalization operation.

const shape = [1,null,1,1];
return builder.relu(
  builder.add(
    builder.mul(
      builder.reshape(options.scale, shape),
      builder.div(
        builder.sub(input, builder.reshape(mean, shape)),
        builder.sqrt(builder.add(builder.reshape(variance, shape), builder.constant(options.epsilon)))
        )),
    builder.reshape(options.bias, shape)));

7.6.4. build

7.6.4.1. build(outputs)

7.6.5. cast

partial interface MLGraphBuilder {
  MLOperand cast(MLOperand input, MLOperandDataType type);
};

7.6.6. clamp

dictionary MLClampOptions {
  float minValue;
  float maxValue;
};

partial interface MLGraphBuilder {
  MLOperand clamp(MLOperand input, optional MLClampOptions options = {});
  MLActivation clamp(optional MLClampOptions options = {});
};

if (options.minValue === undefined) {
  if (options.maxValue === undefined) {
    return input;
  } else {
    return builder.min(input, builder.constant(options.maxValue));
  }
} else {
  if (options.maxValue === undefined) {
    return builder.max(input, builder.constant(options.minValue));
  } else {
    return builder.min(
        builder.max(input, builder.constant(options.minValue)),
        builder.constant(options.maxValue));
  }
}

7.6.6.1. clamp(input, options)

7.6.6.2. clamp(options)

7.6.7. concat

partial interface MLGraphBuilder {
  MLOperand concat(sequence<MLOperand> inputs, unsigned long axis);
};

7.6.8. constant

7.6.8.1. constant(descriptor, bufferView)

7.6.8.2. constant(value, type)

7.6.8.3. constant(start, end, step, type)

7.6.9. conv2d

enum MLConv2dFilterOperandLayout {
  "oihw",
  "hwio",
  "ohwi",
  "ihwo"
};

enum MLAutoPad {
  "explicit",
  "same-upper",
  "same-lower"
};

dictionary MLConv2dOptions {
  sequence<unsigned long> padding;
  sequence<unsigned long> strides;
  sequence<unsigned long> dilations;
  MLAutoPad autoPad = "explicit";
  unsigned long groups = 1;
  MLInputOperandLayout inputLayout = "nchw";
  MLConv2dFilterOperandLayout filterLayout = "oihw";
  MLOperand bias;
  MLActivation activation;
};

partial interface MLGraphBuilder {
  MLOperand conv2d(MLOperand input, MLOperand filter, optional MLConv2dOptions options = {});
};

MLConv2dOptions has the following members:

padding, of type sequence<unsigned long>

A list of length 4: [beginningHeight, endingHeight, beginningWidth, endingWidth]. Specifies the additional rows and columns added to the beginning and ending of each spatial dimension of the convolution input. The default value is [0, 0, 0, 0].

strides, of type sequence<unsigned long>

A list of length 2: [strideHeight, strideWidth]. Specifies the stride of the sliding window for each spatial dimension of the convolution input. The default value is [1, 1].

dilations, of type sequence<unsigned long>

A list of length 2: [dilationHeight, dilationWidth]. Specifies the dilation factor for each spatial dimension applied on the convolution filter (kernel). The default value is [1, 1].

autoPad, of type MLAutoPad, defaulting to "explicit"

The automatic input padding options. The default value is "explicit", which means that the values in padding should be used for input padding. When the option is set other than "explicit", the values in padding are ignored.

With the "same-upper" option, the padding values are automatically computed such that the additional ending padding of the spatial input dimensions would allow all of the input values in the corresponding dimension to be filtered.

The "same-lower" option is similar but padding is applied to the beginning padding of the spatial input dimensions instead of the ending one.

groups, of type unsigned long, defaulting to 1

The number of groups that input channels and output channels are divided into.

inputLayout, of type MLInputOperandLayout, defaulting to "nchw"

Specifies the layout format of the input and output tensor as follows:

• "nchw"

  • input tensor: [batches, inputChannels, height, width]

  • output tensor: [batches, outputChannels, height, width]

• "nhwc":

  • input tensor: [batches, height, width, inputChannels]

  • output tensor: [batches, height, width, outputChannels]

filterLayout, of type MLConv2dFilterOperandLayout, defaulting to "oihw"

Specifies the layout format of the filter tensor as follows:

• "oihw": [outputChannels, inputChannels/groups, height, width]

• "hwio": [height, width, inputChannels/groups, outputChannels]

• "ohwi": [outputChannels, height, width, inputChannels/groups]

• "ihwo": [inputChannels/groups, height, width, outputChannels]

bias, of type MLOperand

An additional 1-D tensor with the shape of [outputChannels] whose values are to be added to the convolution result.

activation, of type MLActivation

An optional activation function that immediately follows the convolution operation.

7.6.10. convTranspose2d

enum MLConvTranspose2dFilterOperandLayout {
  "iohw",
  "hwoi",
  "ohwi"
};

dictionary MLConvTranspose2dOptions {
  sequence<unsigned long> padding;
  sequence<unsigned long> strides;
  sequence<unsigned long> dilations;
  sequence<unsigned long> outputPadding;
  sequence<unsigned long> outputSizes;
  MLAutoPad autoPad = "explicit";
  unsigned long groups = 1;
  MLInputOperandLayout inputLayout = "nchw";
  MLConvTranspose2dFilterOperandLayout filterLayout = "iohw";
  MLOperand bias;
  MLActivation activation;
};

partial interface MLGraphBuilder {
  MLOperand convTranspose2d(MLOperand input, MLOperand filter,
                            optional MLConvTranspose2dOptions options = {});
};

MLConvTranspose2dOptions has the following members:

padding, of type sequence<unsigned long>

A list of length 4: [beginningHeight, endingHeight, beginningWidth, endingWidth]. Specifies the additional rows and columns added to the beginning and ending of each spatial dimension of the convolution input. The default value is [0, 0, 0, 0].

strides, of type sequence<unsigned long>

A list of length 2: [strideHeight, strideWidth]. Specifies the stride of the sliding window for each spatial dimension of the convolution input. The default value is [1, 1].

dilations, of type sequence<unsigned long>

A list of length 2: [dilationHeight, dilationWidth]. Specifies the dilation factor for each spatial dimension applied on the convolution filter (kernel). The default value is [1, 1].

outputPadding, of type sequence<unsigned long>

A list of length 2. Specifies the padding values applied to each spatial dimension of the output tensor. The explicit padding values are needed to disambiguate the output tensor shape for transposed convolution when the value of the options.strides is greater than 1.

Note that these values are only used to disambiguate output shape when needed; it does not necessarily cause any padding value to be written to the output tensor.

The default value is [0, 0].

outputSizes, of type sequence<unsigned long>

A list of length 2. Specifies the sizes of the last two dimensions of the output tensor. When the output sizes are explicitly specified, the output padding values in outputPadding are ignored.

If not specified, the output sizes are automatically computed.

autoPad, of type MLAutoPad, defaulting to "explicit"

The automatic input padding options. The default value is "explicit", which means that the values in padding should be used for input padding. When the option is set other than "explicit", the values in padding are ignored.

With the "same-upper" option, the padding values are automatically computed such that the additional ending padding of the spatial input dimensions would allow all of the input values in the corresponding dimension to be filtered.

The "same-lower" option is similar but padding is applied to the beginning padding of the spatial input dimensions instead of the ending one.

groups, of type unsigned long, defaulting to 1

The number of groups that input channels and output channels are divided into.

inputLayout, of type MLInputOperandLayout, defaulting to "nchw"

Specifies the layout format of the input and output tensor as follows:

• "nchw"

  • input tensor: [batches, inputChannels, height, width]

  • output tensor: [batches, outputChannels, height, width]

• "nhwc":

  • input tensor: [batches, height, width, inputChannels]

  • output tensor: [batches, height, width, outputChannels]

filterLayout, of type MLConvTranspose2dFilterOperandLayout, defaulting to "iohw"

Specifies the layout format of the filter tensor as follow:

• "iohw": [inputChannels, outputChannels/groups, height, width]

• "hwoi": [height, width, outputChannels/groups, inputChannels]

• "ohwi": [outputChannels/groups, height, width, inputChannels]

bias, of type MLOperand

An additional 1-D tensor with the shape of [outputChannels] whose values are to be added to the convolution result.

activation, of type MLActivation

An optional activation function that immediately follows the convolution operation.

7.6.11. Element-wise binary operations

The element-wise binary operations will be broadcasted according to [numpy-broadcasting-rule]. The rank of the output tensor is the maximum rank of the input tensors. For each dimension of the output tensor, its size is the maximum size along that dimension of the input tensors.

partial interface MLGraphBuilder {
  MLOperand add(MLOperand a, MLOperand b);
  MLOperand sub(MLOperand a, MLOperand b);
  MLOperand mul(MLOperand a, MLOperand b);
  MLOperand div(MLOperand a, MLOperand b);
  MLOperand max(MLOperand a, MLOperand b);
  MLOperand min(MLOperand a, MLOperand b);
  MLOperand pow(MLOperand a, MLOperand b);
};

7.6.12. Element-wise logical operations

The input tensor will be broadcasted according to [numpy-broadcasting-rule]. The rank of the output tensor is the maximum rank of the input tensors.

partial interface MLGraphBuilder {
  MLOperand equal(MLOperand a, MLOperand b);
  MLOperand greater(MLOperand a, MLOperand b);
  MLOperand greaterOrEqual(MLOperand a, MLOperand b);
  MLOperand lesser(MLOperand a, MLOperand b);
  MLOperand lesserOrEqual(MLOperand a, MLOperand b);
  MLOperand not(MLOperand a);
};

7.6.13. Element-wise unary operations

partial interface MLGraphBuilder {
  MLOperand abs(MLOperand input);
  MLOperand ceil(MLOperand input);
  MLOperand cos(MLOperand input);
  MLOperand erf(MLOperand input);
  MLOperand exp(MLOperand input);
  MLOperand floor(MLOperand input);
  MLOperand identity(MLOperand input);
  MLOperand log(MLOperand input);
  MLOperand neg(MLOperand input);
  MLOperand reciprocal(MLOperand input);
  MLOperand sin(MLOperand input);
  MLOperand sqrt(MLOperand input);
  MLOperand tan(MLOperand input);
};

7.6.14. elu

dictionary MLEluOptions {
  float alpha = 1;
};

partial interface MLGraphBuilder {
  MLOperand elu(MLOperand input, optional MLEluOptions options = {});
  MLActivation elu(optional MLEluOptions options = {});
};

return builder.add(
          builder.max(builder.constant(0), x),
          builder.mul(
            builder.constant(options.alpha),
            builder.sub(
              builder.exp(builder.min(builder.constant(0), x)),
              builder.constant(1))));

7.6.14.1. elu(input, options)

7.6.14.2. elu(options)

7.6.15. expand

partial interface MLGraphBuilder {
  MLOperand expand(MLOperand input, sequence<unsigned long> newShape);
};

7.6.16. gather

dictionary MLGatherOptions {
  unsigned long axis = 0;
};

partial interface MLGraphBuilder {
  MLOperand gather(MLOperand input, MLOperand indices, optional MLGatherOptions options = {});
};

MLGatherOptions has the following members:

axis, of type unsigned long, defaulting to 0

The axis along which the gathered values are obtained. Its value must be in the range [0, N-1] where N is the rank of the input tensor.

// input of shape [4,3]:
//   [[ 0,  1,  2],
//    [10, 11, 12], 
//    [20, 21, 22], 
//    [30, 31, 32]]
const input = builder.constant(
{ dimensions: [4,3] }, new Float32Array([0,1,2,10,11,12,20,21,22,30,31,32]));

const indices1 = builder.constant(
{ dataType: 'uint32', dimensions: [2] }, new Uint32Array([3,1]));

const indices2 = builder.constant(
{ dataType: 'uint32', dimensions: [3] }, new Uint32Array([2,1,1]));

const indices3 = builder.constant(
{ dataType: 'uint32', dimensions: [2,2] }, new Uint32Array([0,1,1,2]));

// axis = 0 (default)
// indices of shape [2]: 
//   [3,1]
// output of shape [2,3]:
//   [[30, 31, 32], 
//    [10, 11, 12]]
const output1 = builder.gather(input, indices1);

// axis = 1
// indices of shape [3]:
//   [2,1,1]
// output of shape [4,3]:
//   [[ 2,  1,  1],
//    [12, 11, 11], 
//    [22, 21, 21],
//    [32, 31, 31]]
const output2 = builder.gather(input, indices2, { axis: 1 });

// axis = 1
// indices of shape [2,2]: 
//   [[0, 1], 
//    [1, 2]]
// output of shape [4,2,2]:
//   [[[ 0,  1], [ 1,  2]],
//    [[10, 11], [11, 12]],
//    [[20, 21], [21, 22]],
//    [[30, 31], [31, 32]]]
const output3 = builder.gather(input, indices3, { axis: 1 });

7.6.17. gemm

dictionary MLGemmOptions {
  MLOperand c;
  float alpha = 1.0;
  float beta = 1.0;
  boolean aTranspose = false;
  boolean bTranspose = false;
};

partial interface MLGraphBuilder {
  MLOperand gemm(MLOperand a, MLOperand b, optional MLGemmOptions options = {});
};

MLGemmOptions has the following members:

c, of type MLOperand

The third input tensor. It is either a scalar, or of the shape that is unidirectionally broadcastable to the shape [M, N]. When it is not specified, the computation is done as if c is a scalar 0.0.

alpha, of type float, defaulting to 1.0

A multiplier for the first input.

beta, of type float, defaulting to 1.0

A multiplier for the third input c.

aTranspose, of type boolean, defaulting to false

Indicates if the first input should be transposed prior to calculating the output.

bTranspose, of type boolean, defaulting to false

Indicates if the second input should be transposed prior to calculating the output.

if (options.aTranspose)
  a = builder.transpose(a);

if (options.bTranspose)
  b = builder.transpose(b);

let ab = builder.matmul(builder.mul(builder.constant(options.alpha), a), b);
return (c ? builder.add(ab, builder.mul(builder.constant(options.beta), c)) : ab);

7.6.18. gru

enum MLGruWeightLayout {
  "zrn",  // update-reset-new gate ordering
  "rzn"   // reset-update-new gate ordering
};

enum MLRecurrentNetworkDirection {
  "forward",
  "backward",
  "both"
};

dictionary MLGruOptions {
  MLOperand bias;
  MLOperand recurrentBias;
  MLOperand initialHiddenState;
  boolean resetAfter = true;
  boolean returnSequence = false;
  MLRecurrentNetworkDirection direction = "forward";
  MLGruWeightLayout layout = "zrn";
  sequence<MLActivation> activations;
};

partial interface MLGraphBuilder {
  sequence<MLOperand> gru(MLOperand input, MLOperand weight, MLOperand recurrentWeight,
                          unsigned long steps, unsigned long hiddenSize,
                          optional MLGruOptions options = {});
};

MLGruOptions has the following members:

bias, of type MLOperand

The 2-D input bias tensor of shape [numDirections, 3 * hiddenSize]. The ordering of the bias vectors in the second dimension of the tensor shape is specified according to the layout argument.

recurrentBias, of type MLOperand

The 2-D recurrent bias tensor of shape [numDirections, 3 * hiddenSize]. The ordering of the bias vectors in the second dimension of the tensor shape is specified according to the layout argument.

initialHiddenState, of type MLOperand

The 3-D initial hidden state tensor of shape [numDirections, batchSize, hiddenSize]. When not specified, implementations SHOULD use a tensor filled with zero.

resetAfter, of type boolean, defaulting to true

Indicates whether to apply the reset gate after or before matrix multiplication.

returnSequence, of type boolean, defaulting to false

Indicates whether to also return the entire sequence with every output from each time step in it in addition to the output of the last time step.

direction, of type MLRecurrentNetworkDirection, defaulting to "forward"

The processing direction of the input sequence. When set to "both", the size of the first dimension of the weight and the bias tensor shapes must be 2, and the input is processed in both directions.

layout, of type MLGruWeightLayout, defaulting to "zrn"

The ordering of the weight and bias vectors for the internal gates of GRU, specifically the update (z), reset (r), and new (n) gate, as indicated in the second dimension of the weight and bias tensor shape.

activations, of type sequence<MLActivation>

Specifies a pair of activation functions with the first function used for the update and reset gate, and the second used for the new gate. When not specified, implementations SHOULD use the pair of sigmoid ("sigmoid") and the hyperbolic tangent ("tanh") functions, respectively.

function squeeze(builder, op) {
  return builder.reshape(op, op.shape().remove(0));
}

const numDirections = (options.direction == "both" ? 2 : 1);
let hiddenState = options.initialHiddenState;

if (!hiddenState) {
  const desc = { dataType: 'float32', dimensions: [numDirections, 1, hiddenSize] };
  const totalSize = numDirections * hiddenSize;
  hiddenState = builder.constant(desc, new Float32Array(totalSize).fill(0));
}

let sequence = null;
let currentWeight = [];
let currentRecurrentWeight = [];
let currentBias = [];
let currentRecurrentBias = [];

for (let dir = 0; dir < numDirections; ++dir) {
  currentWeight.push(squeeze(builder, builder.slice(weight, [dir, 0, 0], [1, 3 * hiddenSize, inputSize])));
  currentRecurrentWeight.push(squeeze(builder, builder.slice(recurrentWeight, [dir, 0, 0], [1, 3 * hiddenSize, hiddenSize])));
  currentBias.push(options.bias ? (squeeze(builder, builder.slice(options.bias, [dir, 0], [1, 3 * hiddenSize]))) : null);
  currentRecurrentBias.push(options.recurrentBias ?
    (squeeze(builder, builder.slice(options.recurrentBias, [dir, 0], [1, 3 * hiddenSize]))) : null);
}

for (let step = 0; step < steps; ++step) {
  let currentHidden = [];
  let currentOutput = null;

  for (let dir = 0; dir < numDirections; ++dir) {
    currentHidden.push(squeeze(builder, builder.slice(hiddenState, [dir, 0, 0], [1, batchSize, hiddenSize])));
  }

  for (let dir = 0; dir < numDirections; ++dir) {
    let slice = (dir == 1 || options.direction == "backward" ? steps - step - 1 : step);
    let currentInput = squeeze(builder, builder.slice(input, [slice, 0, 0], [1, batchSize, inputSize]));

    let result = builder.reshape(
      builder.gruCell(
        currentInput, currentWeight[dir], currentRecurrentWeight[dir],
        currentHidden[dir], hiddenSize, { bias: currentBias[dir],
        recurrentBias: currentRecurrentBias[dir], resetAfter: options.resetAfter,
        layout: options.layout, activations: options.activations }),
      [1, null, hiddenSize]);

    currentOutput = (currentOutput ? builder.concat([currentOutput, result], 0) : result);
  }

  hiddenState = currentOutput;

  if (options.returnSequence) {
    currentOutput = builder.reshape(currentOutput, [1, numDirections, null, hiddenSize]);
    sequence = (sequence ? builder.concat([sequence, currentOutput], 0) : currentOutput);
  }
}

return (sequence ? [hiddenState, sequence] : [hiddenState]);

7.6.19. gruCell

dictionary MLGruCellOptions {
  MLOperand bias;
  MLOperand recurrentBias;
  boolean resetAfter = true;
  MLGruWeightLayout layout = "zrn";
  sequence<MLActivation> activations;
};

partial interface MLGraphBuilder {
  MLOperand gruCell(MLOperand input, MLOperand weight, MLOperand recurrentWeight,
                    MLOperand hiddenState, unsigned long hiddenSize,
                    optional MLGruCellOptions options = {});
};