Sunday, August 18, 2013

Quest for the best scene format

A graphics engine needs to know how to compose a scene from a given set of resources, such as: meshes, skeletons, and textures. This is what scene file is for - it's a document that describes relationship between basic resources, joining them into a system that can be effectively processed by the code. This file is composed either by hand (if the scene is small), or by the exporter script from a 3D modelling program. During the evolution of KRI engine the scene format changed several times. I'll try to review the development history and analyze various formats I used, based on the personal experience.

0. Composed in code: kri-1, kri-2

This is where we all start: just slapping entities on the screen directly.
Lang:    C++
    -no export stage
    -no need to validate
    -no parsing
    -non extensible

1. Custom binary: kri-3

All scene data and resources were stored in a single binary file of a custom format.
Lang:    Boo
    -no external libs
    -fast parsing
    -non human-readable -> difficult to debug
    -difficult to validate
    -resources are not separate

2. XML: kri-web

XML is a well-known document format, it has the greatest language/tool support. Besides, that's what we used at my former employer company.
Lang:    Dart, XML
    -built-in validation with Schema
    -support for default values
    -need to keep Schema synchronized with exporter/loader
    -too verbose
    -bloated loading code (no 1:1 data representation)
    -not clear what to put into attributes -> design ambiguity

3. JSON: claymore-engine

This is where I discovered JSON, and it immediately appealed to me because of the simple syntax and its 1:1 reflection with the data. Fortunately, this is the only format Rust had a built-in support for. However, it turned out to be a poor choice for the scene description due to the lack of heterogeneous structures.
Lang:    Rust, JSON
    -no heterogeneous structures -> difficult to read

From there I started looking for something like JSON but to describe the document instead of the data. I looked into YAML, which seemed nice, a bit more complex, and not supported by Rust. Then I found Candle Object Notation, which seemed like a non-ambiguous and compact version of XML, plus the 1:1 mapping to data. However, the format is not that well documented and supported... "It would be nice to have the same object initialization syntax as Rust" - I thought when this idea hit me like a train: why not use Rust then?

4. Rust: k-engine

Let's just export a rust source file, which will be compiled with the application.
Lang:    Rust
    -free validation (your code is the spec)
    -no run-time parsing -> instant loading, no run-time errors
    -no need to learn new syntax
    -compact (no external file needed to run)
    -need to compile the scene
    -bound to the language

This approach seems to be the perfect solution for my scene format. The only thing that worries me is that it depends on Rust compile times. Though, we can still parse Rust code at run time, if we want, while still verifying it at compile time. You can see an actual export result here. It is compact, easy to read, and elegant.