lint: `default` and `$ref` causing early process exit

[jeremyfiel]

running jsonschema lint specs/common/ on a folder full of schemas... I received this error but no reference to the file that has the invalid reference

error: Could not resolve the reference to an external schema
file:///C:/Users/fieljere/repos/api-specs/specs/common/indicatorType_v01.json

This is likely because you forgot to import such schema using --resolve/-r

the schema referenced is very simple and there is no reference inside of it. So the error comes from another schema trying to reference this schema.

{
  "$schema": "http://json-schema.org/draft-04/schema#",
  "title": "IndicatorType",
  "description": "Boolean value, typically used to represent an indicator style attribute",
  "type": "boolean"
}

[jeremyfiel]

i tried adding verbose mode and this is the output

Linting: C:\Users\fieljere\repos\api-specs\specs\common\codeListType_v01.json
C:\Users\fieljere\repos\api-specs\specs\common\codeListType_v01.json:
In Draft 7 and older dialects, keywords sibling to $ref are never evaluated (draft_ref_siblings)
at schema location "/properties/exclusiveIndicator"
error: Could not resolve the reference to an external schema
file:///C:/Users/fieljere/repos/api-specs/specs/common/indicatorType_v01.json

This is likely because you forgot to import such schema using --resolve/-r

{
	"$schema": "http://json-schema.org/draft-04/schema#",
	"title": "CodeListType",
	"description": "Code list structure to represent standard (company level) as well as service provider specific codes restricting the values for certain API fields and components.  The API component must be of either 'code' or 'simpleCode' type to be able to leverage the code lists",
	"type": "object",
	"properties": {
		"codeListTitle": {
			"description": "Code list title",
			"type": "string"
		},
		"exclusiveIndicator": {
			"description": "False indicates that the user may enter their own value for the code.  That is, the code list is supported and extensions are also supported.  True means that only the codeList values are supported. If value is false then the UI will allow a dialog box for the shortName.  The system will generate a key (codeValue) for the user entered value.  The system would need to manage these user defined values.",
			"$ref": "./indicatorType_v01.json"
		},
		"multiSelectionIndicator": {
			"description": "True indicate user can select multiple items from list items and submits as array of codes. Default behaviour allows to select single item",
			"$ref": "./indicatorType_v01.json",
			"default": false
		},
		"listItems": {
			"type": "array",
			"items": {
				"$ref": "./codeListItemType_v01.json"
			}
		},
		"links": {
			"type": "array",
			"items": {
				"$ref": "./linkType_v01.json"
			}
		}
	},
	"additionalProperties": false
}

apart from the obvious ref siblings of description and default, these relative references are valid and the indicatorType_v01.json is in the same folder as this schema, so it should resolve.

[jeremyfiel]

further testing appears to reveal the default definition is causing the process to exit. Believe that should only be an error output and the process should continue linting the remaining schemas.

---

EDIT:
interestingly, it's failing to resolve the external schema in adjacent subschemas, it's not throwing an error for the schema with the $ref and default definition.

take this example:

{
	"$schema": "http://json-schema.org/draft-04/schema#",
	"title": "routingPolicy",
	"description": "Represents the routing policy for a service registration instance. The policy can only contain ONE of the following: userProfileIndicator set to true, or an attributeRoutingPolicy definition, or a podRoutingPolicy definition.",
	"type": "object",
	"oneOf": [
		{
			"type": "object",
			"properties": {
				"userProfileIndicator": {
					"description": "If set to true the requests for the assoicated service will be routed based on the information returned from the User Profile Infrastructure",
					"type": "boolean",
					"default": false
				}
			},
			"additionalProperties": false
		},
		{
			"type": "object",
			"properties": {
				"attributeRoutingPolicy": {
					"$ref": "./attributeRoutingPolicyType_v01.json"
				}
			},
			"additionalProperties": false
		},
		{
			"type": "object",
			"properties": {
				"podRoutingPolicy": {
					"$ref": "./podRoutingPolicyType_v01.json"
				}
			},
			"additionalProperties": false
		}
	]
}

jsonschema lint -v specs/common/infrastructure//api-gateway/routingPolicyType_v01.json
Linting: C:\Users\fieljere\repos\api-specs\specs\common\infrastructure\api-gateway\routingPolicyType_v01.json
error: Could not resolve the reference to an external schema
file:///C:/Users/fieljere/repos/api-specs/specs/common/infrastructure/api-gateway/attributeRoutingPolicyType_v01.json

This is likely because you forgot to import such schema using --resolve/-r

[jviotti]

Hey @jeremyfiel , thanks for raising this one up too. So the thing is that:

• If default or examples is encountered, we will try to validate the values against the current subschema, and thus attempt full JSON Schema evaluation
• But JSON Schema evaluation requires resolving references, so that's a case where you might get a --resolve issue even though it was fine in other cases
• Finally, the CLI never attempts to read file-based references by default, instead forcing the user to manually import any schema into the resolution context using --resolve

But maybe the 3rd point is a bit too restrictive? If the reference is a file reference and not just a URI, then maybe it is OK to attempt to just load it? @jeremyfiel What do you think? Your opinion as a user would be great to make sure the tool makes sense.

[jviotti]

Believe that should only be an error output and the process should continue linting the remaining schemas.

This is fair too. So you say that if lint is choking on a schema because of a missing reference, it should report it, but also continue with other schemas?

[jeremyfiel]

• If default or examples is encountered, we will try to validate the values against the current subschema, and thus attempt full JSON Schema evaluation
• But JSON Schema evaluation requires resolving references, so that's a case where you might get a --resolve issue even though it was fine in other cases

If that's the case, the first example I showed with $ref and sibling default should resolve ok and default: false should validate successfully Although, because it's draft-04 and no sibling refs, maybe it's not working as expected because the current subschema is a different context than the $ref.

Actually, for draft-04 schemas, if you encounter $ref, all siblings should be ignored, including default, so you shouldn't attempt to validate the default keyword because it's not apart of the $ref.

• Finally, the CLI never attempts to read file-based references by default, instead forcing the user to manually import any schema into the resolution context using --resolve

That's concerning for us, because we use 100% file refs, no id/$id referencing. I'm not sure that's the behavior though because it appears many of our $refs are being resolved and linted. maybe i'm wrong.

But maybe the 3rd point is a bit too restrictive? If the reference is a file reference and not just a URI, then maybe it is OK to attempt to just load it? @jeremyfiel What do you think? Your opinion as a user would be great to make sure the tool makes sense.

I would expect ref resolution for local or id/$id. http resolution maybe as an option, i know it's not part of the spec.

[jeremyfiel]

Believe that should only be an error output and the process should continue linting the remaining schemas.

This is fair too. So you say that if lint is choking on a schema because of a missing reference, it should report it, but also continue with other schemas?

yes, i would expect to be able to have error output for the entire command. I'm willing to bet a lot of users will try to lint a lot of schemas at once. i.e. in one or more directories

we have a component library with more than 2k schemas and many many more not part of the component library but as private types for specific apis.

consider a file structure like this where we have a common folder for the component library and any api can reference those for reuse. then we have individual api folders where they have their own schemas such as: response, request and each api folder can define private type schemas for their own use, but not for reuse by other apis.

specs

common
indicatorType_v01.json
core
> my_favorite_api
> v1
> schemas
response-schema.json
> types
.json

we use bash expansion a lot so something like jsonschema lint specs/{core,hr,hcm}/**/**/schemas* would be a command we may use to lint all the schemas folders in those parent folders.

[jviotti]

If that's the case, the first example I showed with $ref and sibling default should resolve ok and default: false should validate successfully Although, because it's draft-04 and no sibling refs, maybe it's not working as expected because the current subschema is a different context than the $ref.

I'm not sure I follow this part. Can you elaborate?

Actually, for draft-04 schemas, if you encounter $ref, all siblings should be ignored, including default, so you shouldn't attempt to validate the default keyword because it's not apart of the $ref.

That is correct and a good point. I think the default / examples validation linting rules don't take this case into account. Good catch. I'll fix it.

That's concerning for us, because we use 100% file refs, no id/$id referencing. I'm not sure that's the behavior though because it appears many of our $refs are being resolved and linted. maybe i'm wrong.

So the way it works at the moment for your case is:

• If a schema has no $id/id, the tool derives an implicit file:// URI for it. So for example, it might choose file:///C:/Users/user/schema.json
• If such schema has a relative reference ./foo.json, the resulting URI will be file:///C:/Users/user/foo.json
• But if you want file:///C:/Users/user/foo.json to be considered by the schema resolution logic, you have to manually --resolve the file that results in file:///C:/Users/user/foo.json

But of course on that third part, we know that is a file path and there is no reason why we couldn't at least try to read it. Of course the same logic does not make sense for non file:// URIs, but in that case, it might be OK.

I would expect ref resolution for local or id/$id. http resolution maybe as an option, i know it's not part of the spec.

So in the tool $ref works for URI based references and local file references, though the latter are just interpreted as URIs without any extra meaning (i.e. you still need to --resolve them). But I think we can handle that a bit different for file-based refs indeed, to make it easier for users.

We also have HTTP resolution, but only if you explicitly pass --http, for security reasons, so that the CLI doesn't fire a lot of HTTP requests unless you actually want that.

yes, i would expect to be able to have error output for the entire command. I'm willing to bet a lot of users will try to lint a lot of schemas at once. i.e. in one or more directories

Fair enough! Should be easy to fix.

[jeremyfiel]

If that's the case, the first example I showed with $ref and sibling default should resolve ok and default: false should validate successfully Although, because it's draft-04 and no sibling refs, maybe it's not working as expected because the current subschema is a different context than the $ref.

I'm not sure I follow this part. Can you elaborate?

So I think your fix to evaluate the draft used will fix it.

But to elaborate, the resolution of $ref takes precedence over the description and default keywords. we use them because they are annotations and don't affect the validation output. We know they are not per the spec.
So my point was to say, the linter should only consider the valid keywords per the draft spec, in this case, ignore description and default and thus ignore the default keyword validation logic you have.
The first point is the linter was failing on the ref resolution of the indicatorType_v01json and it appears to be because of the default: false definition was trying to evaluate the schema.
The second point was to say, if you are evaluating default: false for this schema, because there are no assertions, default: false should throw an error for invalid default value, because an empty schema would be equivalent to default: true

{
	"$schema": "http://json-schema.org/draft-04/schema#",
	"properties": {
		"multiSelectionIndicator": {
			"description": "True indicate user can select multiple items from list items and submits as array of codes. Default behaviour allows to select single item",
			"$ref": "./indicatorType_v01.json",
			"default": false
		}
	}
}

hope that makes sense.

[jviotti]

Your comment makes sense!

because an empty schema would be equivalent to default: true

I don't think that's true. If a schema has no constraints, in theory any JSON value could be a valid default. It doesn't need to be a boolean or the true value. false would be a good fit as well. Any JSON value would do.

[jviotti]

Here is an initial PR fixing the linting sibling to $ref in Blaze: sourcemeta/blaze#494. I'll try to do the rest tomorrow

[jeremyfiel]

because an empty schema would be equivalent to default: true

I don't think that's true. If a schema has no constraints, in theory any JSON value could be a valid default. It doesn't need to be a boolean or the true value. false would be a good fit as well. Any JSON value would do.

i suppose that's correct too!