Schema Building Guide

The output schema you define on a respective pipeline allows bem to effectively take a myriad of different inputs to normalize into your desired output structure. General best practices and conventions for defining JSON Schema go a long way here, but below are recommendations from the bem team to improve accuracy and reliability of your outputs so you can utilize the best of what bem has to offer.

Provide descriptions for every field

This is the most powerful change you can make to affect the output and accuracy of your model. bem won't know what you don't tell it (before you provide feedback), so natural language descriptions indicating the purpose of the field, how it could be potentially presented in your input data, and example values go a long way to improve how bem semantically analyzes your input.

Ensure you have types set on every field

Typing of fields helps bem better analyze your input against how you intend to utilize your output data. In addition to helping provide you which reliably structured output you can depend on in a variety of contexts, typing can also help surface bad input or invalid data that may be coming through your pipeline which get highlighted as “invalid properties” that you can build your own error handling and fault tolerance around.

Specify which fields you consider are required

Similar to ensuring typing, setting required fields will help bem know to focus on identifying context around those fields during the transformation process. Just like with typing on fields, fields that are required but can't be filled in from the input data will get highlighted back to you as “invalid properties” that you can build your own error handling and fault tolerance around; we can help you find those needles in the haystack as you sort through a deluge of inputs.

Provide formatting hints for fields if necessary

If a field you want to populate has a fixed format, you can either specify the format as a regular expression in the conventional JSON Schema pattern field. We've also seen great results from specifying a pattern in natural language in description fields, but if you have more stringent formatting expectations we'd recommend setting a regex pattern.

Set enums for fields you know have a certain set of possible values

Enum values help constrain the set of valid values that bem transforms into a given property. At the moment, this isn't a “strict” constraint but generally helps bem understand the intent behind the desired transformation. Think of it as a stronger way to indicate desired output than providing example values in a description.

Example Schema

Below is an example schema showcasing the above best practices that can be used to normalize inputs from a variety of commercial vehicle electronic logging device (ELD) providers.

{
  "type": "object",
  "title": "Fleet Trip Summary",
  "$schema": "http://json-schema.org/draft-07/schema#",
  "required": [
    "fleetId",
    "tripSummary",
    "compliance",
    "operationalEfficiency"
  ],
  "properties": {
    "fleetId": {
      "type": "string",
      "description": "Unique identifier for Upa; fleet."
    },
    "compliance": {
      "type": "object",
      "required": [
        "hoursOfServiceCompliance",
        "notes"
      ],
      "properties": {
        "notes": {
          "type": "string",
          "description": "Additional notes on compliance."
        },
        "hoursOfServiceCompliance": {
          "type": "string",
          "description": "Compliance status with hours of service."
        }
      }
    },
    "tripSummary": {
      "type": "object",
      "required": [
        "tripId",
        "vehicle",
        "driver",
        "start",
        "end",
        "distanceCovered",
        "fuelUsage",
        "incidents"
      ],
      "properties": {
        "end": {
          "type": "object",
          "required": [
            "time",
            "location",
            "odometerEnd"
          ],
          "properties": {
            "time": {
              "type": "string",
              "format": "date-time",
              "description": "End time of the trip."
            },
            "location": {
              "type": "string",
              "description": "End location of the trip."
            },
            "odometerEnd": {
              "type": "integer",
              "description": "Odometer reading at the end of the trip."
            }
          }
        },
        "start": {
          "type": "object",
          "required": [
            "time",
            "location",
            "odometerStart"
          ],
          "properties": {
            "time": {
              "type": "string",
              "format": "date-time",
              "description": "Start time of the trip."
            },
            "location": {
              "type": "string",
              "description": "Start location of the trip."
            },
            "odometerStart": {
              "type": "integer",
              "description": "Odometer reading at the start of the trip."
            }
          }
        },
        "driver": {
          "type": "object",
          "required": [
            "id",
            "name"
          ],
          "properties": {
            "id": {
              "type": "string",
              "description": "Unique identifier for the driver."
            },
            "name": {
              "type": "string",
              "description": "Name of the driver."
            }
          }
        },
        "tripId": {
          "type": "string",
          "description": "Unique identifier for the trip."
        },
        "vehicle": {
          "type": "object",
          "required": [
            "id",
            "details"
          ],
          "properties": {
            "id": {
              "type": "string",
              "description": "Unique identifier for the vehicle."
            },
            "details": {
              "type": "string",
              "description": "Description of the vehicle including make, model, and year."
            }
          }
        },
        "fuelUsage": {
          "type": "object",
          "required": [
            "totalGallons",
            "averagePricePerGallon",
            "totalCost"
          ],
          "properties": {
            "totalCost": {
              "type": "string",
              "description": "Total cost of fuel."
            },
            "totalGallons": {
              "type": "number",
              "description": "Total gallons of fuel used."
            },
            "averagePricePerGallon": {
              "type": "string",
              "description": "Average price per gallon of fuel."
            }
          }
        },
        "incidents": {
          "type": "array",
          "items": {
            "type": "object",
            "required": [
              "type",
              "time",
              "location",
              "details"
            ],
            "properties": {
              "time": {
                "type": "string",
                "format": "date-time",
                "description": "Time of the incident."
              },
              "type": {
                "type": "string",
                "description": "Type of incident."
              },
              "details": {
                "type": "string",
                "description": "Detailed description of the incident."
              },
              "location": {
                "type": "string",
                "description": "Location of the incident."
              }
            }
          }
        },
        "distanceCovered": {
          "type": "string",
          "description": "Total distance covered during the trip."
        }
      }
    },
    "operationalEfficiency": {
      "type": "object",
      "required": [
        "totalEngineHours",
        "idleTime",
        "efficiencyRating"
      ],
      "properties": {
        "idleTime": {
          "type": "string",
          "description": "Total idle time during the trip."
        },
        "efficiencyRating": {
          "type": "string",
          "description": "Efficiency rating of the trip."
        },
        "totalEngineHours": {
          "type": "string",
          "description": "Total engine hours for the trip."
        }
      }
    }
  }
}