Edawson/scdl schema

[edawson]

Description

This MR implements a strict schema-defined header for SCDL archives. This header stores metadata about the archive and its composite arrays, including a version, the array lengths and data types, and information about the RowFeatureIndexes. This adds the features necessary to fix #999 as well as implement simple bit-packing of the rowptr, colptr, and data arrays. It also should make SCDL more secure, enable strict compatibility checking, and open the door to more performance improvements.

Note: I am still wiring up the header to the archive. I will make a note here when the MR is ready.

Type of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Refactor
• Documentation update
• Other (please describe):

CI Pipeline Configuration

Configure CI behavior by applying the relevant labels:

• SKIP_CI - Skip all continuous integration tests
• INCLUDE_NOTEBOOKS_TESTS - Execute notebook validation tests in pytest
• INCLUDE_SLOW_TESTS - Execute tests labelled as slow in pytest for extensive testing

Note

By default, the notebooks validation tests are skipped unless explicitly enabled.

Authorizing CI Runs

We use copy-pr-bot to manage authorization of CI
runs on NVIDIA's compute resources.

• If a pull request is opened by a trusted user and contains only trusted changes, the pull request's code will
  automatically be copied to a pull-request/ prefixed branch in the source repository (e.g. pull-request/123)
• If a pull request is opened by an untrusted user or contains untrusted changes, an NVIDIA org member must leave an
  /ok to test comment on the pull request to trigger CI. This will need to be done for each new commit.

Usage

This change is opaque to the user - the headers are not human-readable on disk. For a full description of the format and how to interact with it, see the schema directory in SCDL's source directory.

Pre-submit Checklist

• I have tested these changes locally
• I have updated the documentation accordingly
• I have added/updated tests as needed
• All existing tests pass successfully

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[yzhang123]

/ok to test 93b78e3

[copy-pr-bot]

/ok to test 93b78e3

@yzhang123, there was an error processing your request: E2

See the following link for more information: https://docs.gha-runners.nvidia.com/cpr/e/2/

[yzhang123]

\ok to test 93b78e3

[yzhang123]

/ok to test 93b78e3

[copy-pr-bot]

/ok to test 93b78e3

@yzhang123, there was an error processing your request: E2

See the following link for more information: https://docs.gha-runners.nvidia.com/cpr/e/2/

[yzhang123]

/ok to test e768f60

[yzhang123]

/ok to test cb53ac7

[codecov-commenter]

❌ 45 Tests Failed:

Tests completed	Failed	Passed	Skipped
1180	45	1135	52

View the top 3 failed test(s) by shortest run time

sub-packages/bionemo-scdl/tests/bionemo/scdl/schema/test_header.py::TestSchemaCompliance::test_core_header_layout

Stack Traces | 0.001s run time

self = <schema.test_header.TestSchemaCompliance object at 0x7ed9243d5700>

    def test_core_header_layout(self):
        """Test core header layout matches schema specification."""
        header = SCDLHeader()
        serialized = header.serialize()
    
        # Schema specifies 16-byte core header
        assert len(serialized) >= 16
    
        # Magic number at offset 0x00 (4 bytes)
        assert serialized[0:4] == SCDL_MAGIC_NUMBER
    
        # Version at offsets 0x04, 0x05, 0x06 (3 bytes)
        assert serialized[4] == 0  # major
        assert serialized[5] == 0  # minor
>       assert serialized[6] == 2  # point
E       assert 9 == 2

.../scdl/schema/test_header.py:681: AssertionError

sub-packages/bionemo-scdl/tests/bionemo/scdl/schema/test_header.py::TestSchemaCompliance::test_current_version_matches_schema

Stack Traces | 0.001s run time

self = <schema.test_header.TestSchemaCompliance object at 0x7ed9243d5250>

    def test_current_version_matches_schema(self):
        """Test current version matches schema documentation."""
        # Schema documents version 0.0.2
        current = CurrentSCDLVersion()
        assert current.major == 0
        assert current.minor == 0
>       assert current.point == 2
E       assert 9 == 2
E        +  where 9 = SCDLVersion(major=0, minor=0, point=9).point

.../scdl/schema/test_header.py:654: AssertionError

sub-packages/bionemo-scdl/tests/bionemo/scdl/schema/test_header.py::TestSCDLHeader::test_json_output

Stack Traces | 0.002s run time

self = <schema.test_header.TestSCDLHeader object at 0x7ed9243d45f0>

    def test_json_output(self):
        """Test JSON representation."""
        header = SCDLHeader()
        array = ArrayInfo("test.dat", 1000, ArrayDType.FLOAT32_ARRAY, (100, 10))
        header.add_array(array)
    
        json_str = header.to_json()
        json_data = json.loads(json_str)
    
        assert json_data["version"]["major"] == 0
        assert json_data["version"]["minor"] == 0
>       assert json_data["version"]["point"] == 2
E       assert 9 == 2

.../scdl/schema/test_header.py:632: AssertionError

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}

[edawson]

/ok to test: 9ecc929

[edawson]

/ok to test 73982f1