xviz/docs/protocol-formats/binary-protocol.md

XVIZ Binary Container Protocol Format

XVIZ comes with encoding and parsing support for a binary container format based on the GLB container for glTF. In this format, each message is packaged in a binary container that contains at least two "chunks" as defined by the GLB specification.

• a JSON chunk containing the JSON encoding of semantic parts of the data
• a BIN chunk containing compact, back-to-back binary representations of large numeric arrays, images etc.

An intended benefit of the binary format is that large segments of data can be sent and processed natively. Encoding XVIZ inside a glTF GLB format allows XVIZ to leverage the features available for assets.

Current Status

The current usage in the XVIZ context is to store point clouds and images in the BIN chunk of the GLB. The intention is to expand this to large vertex arrays and streams of data that can benefit from binary storage.

XVIZ currently has limited support for glTF. We are using the @loaders.gl/gltf node module so our support will be tied to this library.

XVIZ encoding in GLB

XVIZ in GLB is implemented with 2 changes over a standard GLB.

The xviz Property in the JSON chunk

The standard XVIZ JSON object is represented as a top-level property in the JSON chunk.

Use of JSON Pointers to reference GLB assets

Inside the xviz property any entry that is stored in the accompaning GLB BINARY structures are represented by a JSON Pointer, specifically URI Fragment format for JSON pointers. This format has a '#' prefix, followed by a JSON Pointer path where the root is the JSON CHUNK of the GLB container.

For example, the JSON Pointer #/accessors/0 would point to the first entry of the accessors property on the root JSON object in the JSON Chunk.

Using JSON Pointers allows the XVIZ data to reference the container and reuse the glTF specification for assets.

[... header]
{
  "accessors": [
    {
      "bufferView": 0,
      "type": "VEC4",
      "componentType": 5121,
      "count": 123397
    },
    {
      "bufferView": 1,
      "type": "VEC3",
      "componentType": 5126,
      "count": 123397
    }
  ],
  "bufferViews": [
    {
      "buffer": 0,
      "byteOffset": 0,
      "byteLength": 493588
    },
    {
      "buffer": 0,
      "byteOffset": 493588,
      "byteLength": 1480764
    }
    {
      "buffer": 0,
      "byteOffset": 493588,
      "byteLength": 1480764
    },
    {
      "buffer": 0,
      "byteOffset": 2164912
      "byteLength": 190560
    }
  ],
  "images": [
    {
      "bufferView":2,
      "mimeType":"image/png",l
      "width":397,
      "height":120
    }
  ],
  "xviz": {
    "type": "xviz/status_update",
    "data": {
      "update_type": "snapshot",
      "updates": [
        {
          "primitives": {
            "/lidar": {
              "points": [
                {
                  "colors": "#/accessors/0",
                  "points": "#/accessors/1"
                }
              ]
            },
            "/camera": {
              "images": [
                {
                  "data": "#/images/0",
                  "width_px": 397,
                  "height": 120,
                  "position": [0, 0, 0]
                }
              ]
            }
          }
        }
      ]
    }
  }
}
[... binary chunk start
 // first 493588 bytes are pointed at by 'bufferViews[0]'
 // next run is pointed at by 'bufferViews[1]'
 ... binary chunk end]

XVIZ API References

Encoding Support

XVIZWriter by default will output the GLB binary encoding of the XVIZ data. This will be output for both Metadata and Frames, but not the FrameIndex, which is always output as JSON.

Parsing Support

parseStreamMessage will parse the data and handle GLB encoded XVIZ as well as other encodings of the data.

XVIZ parsing functions will decode the binary container, parse the JSON and resolve binary references. The application will get a "patched" JSON structure, with the difference from the basic JSON protocol format being that certain arrays will be compact typed arrays instead of classic JavaScript arrays.

If an attribute has been hydrated from binary then it will be transformed into the corresponding TypeArray. Typed arrays do not support nesting so all numbers will be laid out flat and the application needs to know how many values represent one element, for instance 3 values represent the x, y, z coordinates of a point.

References

• glTF 2 Poster
• glTF 2 Spec

Remarks

Note on endianness:

• glTF is little endian: GLB header is little endian. glTF Buffer contents are specified to be little endian.
• Essentially all of the current web is little endian, so potential big-endian issues are igored for now.