PySpark

Translates JSON Schema to PySpark StructType definitions (.py).

Example

Input (JSON Schema):

{
  "type": "object",
  "properties": {
    "name": { "type": "string" },
    "age": { "type": "integer" }
  }
}

Output (PySpark):

import pyspark.sql.types as T

users_schema = T.StructType([
    T.StructField("name", T.StringType(), nullable=True),
    T.StructField("age", T.LongType(), nullable=True),
])

Supported JSON Schema Features

Type Keywords

• type
• enum
• const

Type Values

• string
• integer
• number
• boolean
• array
• object
• null

Schema Composition

• allOf
• anyOf
• oneOf
• not

Object Keywords

• properties
• required
• additionalProperties
• patternProperties
• propertyNames
• minProperties / maxProperties
• unevaluatedProperties
• dependentRequired

Array Keywords

• items
• prefixItems
• contains
• minItems / maxItems
• uniqueItems
• unevaluatedItems
• maxContains / minContains

Numeric Validation

• minimum / maximum
• exclusiveMinimum / exclusiveMaximum
• multipleOf

String Validation

• minLength / maxLength
• pattern

References & Definitions

• $ref
• $defs
• $id
• $anchor
• $dynamicRef / $dynamicAnchor

String Formats

• date
• date-time
• time
• duration
• uuid
• uri / uri-reference / uri-template
• iri / iri-reference
• email / idn-email
• hostname / idn-hostname
• ipv4 / ipv6
• json-pointer / relative-json-pointer
• regex

Annotations

• description
• title
• default
• deprecated
• readOnly / writeOnly
• examples

Conditional

• if / then / else
• dependentSchemas

Content

• contentEncoding
• contentMediaType
• contentSchema