Grading Platform

Production platform for automated assessment of students’ handwritten graphs, extending the research in:

Automated Grading of Students’ Handwritten Graphs: A Comparison of Meta-Learning and Vision-Large Language Models
Parsaeifard et al., arXiv:2507.03056

This repository implements a deployable stack around the methods from the paper and the reference meta-learning code in handwritten-graph-grading: N-way K-shot prototypical / matching / relation / MAML models, plus an OpenAI multimodal LLM baseline that grades from graph images and rubric criteria via in-context learning.

Screenshots

Annotate — toggle rubric criteria (binary grade), optional model assist, navigate submissions.

Inference — few-shot prediction with support-set status, model reload, and training job history.

Architecture

                    ┌─────────────┐
                    │  frontend   │ :3080
                    └──────┬──────┘
                           │
                    ┌──────▼──────┐     ┌──────────┐
                    │  platform   │────▶│ PostgreSQL│
                    │   :8800     │     └──────────┘
                    └──┬────┬─────┘
           ┌───────────┘    └───────────┐
           ▼                            ▼
    ┌─────────────┐              ┌───────────┐
    │  inference  │◀─────────────│  Airflow  │ :8080
    │   :9001     │              └─────┬─────┘
    └──────┬──────┘                    │
           │                     ┌─────▼─────┐
    ┌──────┴──────┐              │ml-training│
    │ ml-core     │              └─────┬─────┘
    │ llm (VLLM)  │                    │
    └─────────────┘              ┌─────▼─────┐
           ▲                     │  MLflow   │ :5001
           └───── model reload ──┴───────────┘

flowchart TB
    FE["frontend :3080"]
    API["platform :8800"]
    DB[("PostgreSQL")]
    INF["inference :9001"]
    AF["airflow :8080"]
    TRN["ml-training"]
    MF["mlflow :5001"]
    CORE["ml-core"]
    VLLM["llm"]

    FE --> API
    API --> DB
    API --> INF
    API --> AF
    AF --> TRN
    TRN --> API
    TRN --> MF
    TRN --> INF
    INF --> CORE
    INF --> VLLM
    MF -.->|artifact reload| INF

Loading

Layer	Components	Role
UI	frontend/	Annotate rubrics, upload graphs, queue retraining, run inference
Platform	platform/	Courses, assignments, submissions, annotations, job queue
ML	ml-core/, llm/, inference/, ml-training/	Few-shot models, LLM adapter, serving, batch training
Ops	airflow/, mlflow/, db/	Retrain orchestration, experiment tracking, persistence

Data flow

1. Seed — redacted sample bundles in data/sample/ (training + annotation assignments).
2. Annotate — human labels set fulfilled criteria; grade = Σ 2^i over rubric bits.
3. Train — train.py CLI (export JSON → MLflow artifact). In Docker, pipeline.py / Airflow fetch export, run CLI, register model, reload inference.
4. Infer — platform builds K-shot support set from annotated course data → inference adapter predicts class / criteria.

Training (CLI-first)

ml-training/train.py is standalone — no platform or inference HTTP inside the trainer:

# 1. Freeze training data (optional offline step)
python ml-training/export.py --course-id <UUID> --output export.json

# 2. Train + log to MLflow
python ml-training/train.py \
  --export export.json \
  --artifact-dir ./run-001 \
  --mlflow-tracking-uri http://localhost:5001

In the stack, orchestration is thin:

UI / bootstrap / Airflow
        → ml-training POST /run
        → pipeline.py: fetch export → subprocess train.py → register-model → inference/reload

Quick start

Requires Docker with Compose v2 (docker compose).

cp .env.example .env   # optional OPENAI_API_KEY for openai_multimodal
docker compose up --build

URL	Service
http://localhost:3080	UI
http://localhost:8800/docs	Platform API
http://localhost:5001	MLflow
http://localhost:8080	Airflow (admin / admin)

Reset database: docker compose down -v && docker compose up --build

GPU (optional)

Meta-learning training and inference use PyTorch via ml-core. Default is CPU (ML_DEVICE=cpu).

To use a GPU:

1. Install NVIDIA Container Toolkit on the host.

2. In .env:

   ML_DEVICE=cuda

   (INFERENCE_DEVICE is still supported as a fallback alias.)

3. Pass the GPU into the ML containers. Easiest: create docker-compose.override.yml (gitignored locally):

   services:
     inference:
       gpus: all
     ml-training:
       gpus: all

4. Rebuild and start: docker compose up --build

train.py picks up the device from the environment automatically — no code changes needed. LLM (openai_multimodal) does not use local GPU.

Offline CLI training on a GPU machine:

export ML_DEVICE=cuda
python ml-training/train.py --export export.json --artifact-dir ./run-001

Model adapters

ID	Training	Family	Notes
few_shot_prototypical	MLflow	Meta-learning	Default
few_shot_matching	MLflow	Meta-learning	Matching network
few_shot_relation	MLflow	Meta-learning	Relation network
few_shot_maml	MLflow	Meta-learning	MAML
openai_multimodal	—	VLLM	In-context GPT-4o vision; predicts fulfilled criteria from image + rubric

Testing & CI

CI

Suite	Command	What it covers
Unit (Python)	pytest tests/unit tests/inference	Rubric encoding, adapter catalog, LLM prompts, inference store
Integration	pytest tests/integration -m integration	Platform API + PostgreSQL, sample bundle import
Unit (frontend)	cd frontend && npm test	Binary grade helper, API error parsing
Build	GitHub Actions	Docker images for platform & frontend

Local integration tests need PostgreSQL:

pip install -r tests/requirements.txt
export DATABASE_URL=postgresql://grading:grading@localhost:5433/grading_test
pytest tests/integration -m integration

Repository layout

platform/      FastAPI API, models, seed/import
frontend/      React UI
ml-core/       Meta-learning adapters (paper / HGG models)
llm/           OpenAI multimodal VLLM adapter
inference/     Model store, /predict, /reload
ml-training/   train.py (CLI), export.py, pipeline.py (orchestration), serve.py (HTTP)
airflow/       grading_train_reload DAG
data/sample/   Redacted sample bundles
docs/images/   README screenshots
tests/         Unit, integration, and inference tests
db/            PostgreSQL init

See data/README.md for bundle format.

License

This project is licensed under the MIT License.

Citation

@article{parsaeifard2025automated,
  title={Automated Grading of Students' Handwritten Graphs: A Comparison of Meta-Learning and Vision-Large Language Models},
  author={Parsaeifard, Behnam and others},
  journal={arXiv preprint arXiv:2507.03056},
  year={2025}
}