Reduce redundancy in generated schema

[marcprux]

Some structures in the generated vega-lite-schema.json are repeated multiple times. For example, the timeUnit enumeration is repeated verbatim 13 times throughout the schema. Detecting this redundancy and emitting repeated structures to a shared defs section of the schema would make it much more compact.

TODO:

• add enum support to typescript-json-schema
• change build process to generate schema using new typescript-json-schema
• remove old schema generation process

[marcprux]

I would be interested in working on this, but I was first wondering if it should be part of #927.

[kanitw]

Yes, this should be a part of #927. In addition, our vl.schema.instantiate method, which instantiate default values for the spec, also uses default property in the schema. Current vl.schema.instantiate does not support traversing defs section of the schema. So that's another part of work to be done for this to work.

[domoritz]

@mprudhom It would be awesome if you could look into this! In addition to using defs, we think that the json schema should be generated from typescript rather than manually by hand. In fact, I worked a bit on https://github.com/YousefED/typescript-json-schema to add support for union types and it's almost ready to be used for our purposes. What's missing right now is support for enums.

I suggest that we first look into the ts -> json schema generation (which can use defs) and then modify the instantiate method to use references.

If you want to talk about this in more detail, we could discuss on skype.

[marcprux]

@domoritz Are you sure that typescript-json-schema supports storing shared schemas in a separate definitions section of the schema? Glancing over the code I don't see anywhere that will generate any $ref schema attributes.

[domoritz]

Sorry, I didn't realize this hasn't been merged yet. There is a branch referenced from YousefED/typescript-json-schema#3. It only needs a few more changes to be ready.

[domoritz]

Relevant #927

[domoritz]

@mprudhom would you like to work on this? It would be incredibly helpful for us!

[marcprux]

@domoritz I'll take a stab at it. Will it be OK for vega-lite to rely on my own fork of typescript-json-schema? As you mention, it will at least need support for enums.

[domoritz]

Sure! Although @ YousefED has been extremely responsive so I think we will be able to merge quickly.

Since we are refactoring some things around the schema in VL right now, the first step would be to finish ref support (YousefED/typescript-json-schema#3) and then make sure that enums work.

Also note that TS 1.8 added support for string literals and we might use them some time in the future. I don't think they are treated as enums internally and may produce horrible code with a lot of oneOfs.

[marcprux]

I implemented support for enums in typescript-json-schema. You can take a look at the generated schema by installing my branch:

npm install 'glimpseio/typescript-json-schema#%24ref-support'

Then generating the new schema:

node_modules/typescript-json-schema/bin/typescript-json-schema src/schema/schema.ts Spec > vega-lite-schema2.json

Most of the sample specs pass validation:

node_modules/z-schema/bin/z-schema vega-lite-schema2.json examples/specs/*.json

@domoritz Before I proceed, can you take a look at the TODO list I put in this issue's description and let me know if it sounds good to you?

[domoritz]

Fantastic! The validation errors seem to be mostly because the ts schema is not complete. We use any in a couple of places and the compiler assumes that objects are expected.

migrate "const TIMEUNITS" to "enum TimeUnit" in src/timeunit.ts

Looks like the only difficulty here is that vl shorthand uses the actual array. But to be honest, I think we can just change that code and always output the complete time unit. In fact, I think its necessary now that we have yearmonth. cc @kanitw

We are planning to extract the shorthand into its own repo anyway. I'll see whether we can speed that up.

update string literal references to use new enum types across vega-lite

What does this mean? We don't use string literals in vl yet but once TS supports microsoft/TypeScript#6554, we might. Check out https://github.com/Microsoft/TypeScript/wiki/What's-new-in-TypeScript#string-literal-types.

Before we change too much in vl, let's try to get YousefED/typescript-json-schema#3 and your changes into the main repo. This means

• Add command line options to use refs
• Add command line option to not create a ref for the root. This should be the default.
• Maybe even fix the logic so that two entities with the same name but in different namespaces don't create the same ref entity.
• Send a PR

Should we move these before the changes in vl?

Also, just as a side note, we are changing the way default values are used so the instantiate methods or the schema in vega lite are probably not necessary any more. This simplifies our life and makes your contributions to schema generation from interfaces even more important. See #1119

[marcprux]

OK, I've submitted a PR with CLI support for using refs & root refs at YousefED/typescript-json-schema#15. You can see it in action by grabbing my branch and running:

./bin/typescript-json-schema -useRef -noRootRef ../vega-lite-schema/src/schema/schema.ts Spec 

Is the name-conflict issue an existent problem in the vega-lite schema? I don't notice any cases where it overlapping schema references were being created.

[domoritz]

Is the name-conflict issue an existent problem in the vega-lite schema?

Not that I know of.

[marcprux]

In that case, I'll defer implementing it in order to keep the PR as simple as possible.

Should I submit a vega-lite PR now to use the new schema generation against my branch of typescript-json-schema, or should we wait to see if #1119 gets merged soon?