Open Data Certificate - Node.js Implementation

A modern, streamlined Node.js implementation of the Open Data Certificate questionnaire system, migrated from the original Ruby application.

🚀 Features

• Simplified Architecture: MongoDB-based data model with clear separation of concerns
• Numeric Achievement Levels: Streamlined 0-4 level system (none, basic, pilot, standard, exemplar)
• RESTful API: Clean, modern API design
• Real-time Level Calculation: Achievement levels calculated on-demand
• Migration Support: Complete data migration from Ruby PostgreSQL to MongoDB
• JSON-based Survey Definition: Flexible questionnaire structure using JSON files

📊 Architecture Overview

Data Model

Survey (Questionnaire Definition)
├── Sections (Legal, Technical, etc.)
│   └── Elements (Questions & Logic)

Dataset (Data Being Certified)
├── Metadata (title, URL, etc.)
└── Owner Information

Certificate (Questionnaire Attempt + Result)
├── References: surveyId, datasetId, userId
├── Responses (Map of question → answer)
├── Achievement Level (0–4)
└── Publication State (draft, published, archived, superseded)

Achievement Levels

Levels are stored per-survey in Survey.levels as a map keyed by level index (0–4):

levels: {
  "0": { title, description, icon },
  "1": { title, description, icon },
  ...
}

During migration, defaults are set:

• 0: “No level” — No level has yet been achieved (icon: images/badges/no_level_badge.png)
• 1: “Bronze” — A fantastic start… (icon: images/badges/raw_level_badge.png)
• 2: “Silver” — Extra effort went in… (icon: images/badges/pilot_level_badge.png)
• 3: “Gold” — Regularly published open data… (icon: images/badges/standard_level_badge.png)
• 4: “Platinum” — Above and beyond… (icon: images/badges/exemplar_level_badge.png)

The certificate view reads the survey’s levels to display the badge and description. The textual name is also mapped to a friendly string at render time.

🛠️ Setup Instructions

Prerequisites

• Node.js 16+
• MongoDB 4.4+
• MySQL (for migration from Ruby app)

Installation

1. Clone the repository

   git clone <repository-url>
   cd open-data-certificate-node

2. Install dependencies

   npm install

3. Environment Configuration Create a .env file (see config.env.example):

   # MongoDB
   MONGO_URL=mongodb://localhost:27017/open_data_certificate
   
   # MySQL (for migration)
   MYSQL_HOST=127.0.0.1
   MYSQL_PORT=3306
   MYSQL_DATABASE=certificates
   MYSQL_USER=root
   MYSQL_PASSWORD=password
   
   # Server
   PORT=3000
   NODE_ENV=development
   
   # Session Secret
   SESSION_SECRET=your_session_secret
   
   # ODI OAuth Credentials
   DJANGO_CLIENT_ID=your_django_client_id
   DJANGO_CLIENT_SECRET=your_django_client_secret
   DJANGO_CALLBACK_URL=/auth/django/callback
   
   # Google OAuth Credentials
   GOOGLE_CLIENT_ID=your_google_client_id
   GOOGLE_CLIENT_SECRET=your_google_client_secret
   GOOGLE_CALLBACK_URL=/auth/google/callback

4. Database Setup

   # Start MongoDB
   mongod
   
   # Start MySQL (if migrating from Ruby app)
   # Ensure your Ruby app database is accessible

5. Run Migration (Optional) If migrating from the Ruby application:

   Full Migration (recommended):

   npm run migrate:surveys:full
   # or clear existing surveys first
   npm run migrate:surveys:full:clear

   Single Survey Migration (for focused testing):

   # Default single import (falls back to ID 3252)
   npm run migrate:surveys:single
   
   # Specify a survey ID via CLI arg
   npm run migrate:surveys:single -- --id=4000
   # or
   npm run migrate:surveys:single -- --survey-id=4000
   
   # Specify via env
   SINGLE_MODE=true SINGLE_SURVEY_ID=4000 node scripts/migrateSurveys.js
   
   # Clear existing surveys first
   npm run migrate:surveys:single:clear -- --id=4000

6. Start the Application

   # Development
   npm run dev
   
   # Production
   npm start

📁 Project Structure

├── models/                 # MongoDB schemas
│   ├── Survey.js
│   ├── User.js
│   ├── Dataset.js
│   └── Certificate.js
├── controllers/            # Web controllers
│   ├── certificates.js
│   └── surveys.js
├── services/               # Business logic
│   └── levelCalculationService.js
├── routes/                 # Routes
│   ├── datasets.js         # Public + My datasets + certificates (HTML/JSON via content negotiation)
│   ├── redirects.js        # Legacy redirects (locale-prefixed)
│   └── surveys.js          # Surveys index and criteria (HTML) + survey JSON (content negotiation)
├── views/                  # EJS templates
│   ├── pages/
│   │   ├── datasets/
│   │   │   ├── index.ejs
│   │   │   └── dataset.ejs
│   │   └── certificates/
│   │       └── show.ejs
│   └── partials/
│       └── header.ejs
├── public/                 # Static assets (css, images, lib)
├── scripts/                # Migration scripts
│   ├── migrateSurveys.js
│   └── migrateCertificates.js
└── .gitignore

🔄 Migration from Ruby

The migration script extracts data from the original Ruby MySQL database and transforms it for the new MongoDB schema:

Migration Process

1. Surveys: Converts survey definitions to MongoDB format
2. Users: Migrates user accounts and preferences
3. Datasets: Transfers dataset metadata
4. Certificates: Converts questionnaire responses and achievement records

Test Migration Features

The migration script includes test mode options to validate the migration process:

• Limited Records: Test with a small subset of data
• Mixed States: Ensures variety in response set states (draft, published, archived)
• Non-Blank Data: Filters out empty records to ensure meaningful test data
• Balanced Sampling: Gets mix of admin/regular users, active/removed datasets, etc.

Running Migration

See scripts and npm commands:

Surveys:

npm run migrate:surveys:single         # imports one (default 3252) or pass --id
npm run migrate:surveys:single:clear   # clear then import one
npm run migrate:surveys:full           # imports all per SQL criteria
npm run migrate:surveys:full:clear     # clear then full import

Certificates/Datasets:

npm run migrate:certificates:single        # migrate dataset 220763 by default (or --dataset-id)
npm run migrate:certificates:full          # migrate all datasets with published certs

🎯 Key Improvements

Simplified Data Model

• Unified Response Storage: All responses stored as key-value pairs
• No Complex Joins: MongoDB document structure eliminates complex queries
• Real-time Calculations: Achievement levels calculated on-demand

Modern Architecture

• Routes: Consolidated /datasets router and legacy redirects router
• MVC-ish: Controllers for pages and drill-down tables
• Survey-driven Certificates: Certificate titles, per-question statement text, and display controls come from the survey
• Numeric Levels: Simplified 0–4 achievement system, driven by Survey.levels

Performance Benefits

• Better Scalability: MongoDB handles large datasets efficiently
• Faster Queries: Document-based queries are more efficient
• Reduced Complexity: Simplified data relationships

🌐 Route Definitions

Public web routes

• GET / → Home
• GET /about → About page
• GET /datasets → Browse published datasets (HTML) or JSON list via content negotiation
• GET /datasets/:id → Dataset drill-down (HTML) or JSON via content negotiation
• GET /datasets/:datasetId/certificates → List or redirect to a certificate for dataset
• GET /datasets/:datasetId/certificates/:certificateId → Render a certificate

Authenticated web routes

• GET /datasets/my → “My Datasets” (HTML) or JSON via content negotiation (owner; all for admin)
• GET /auth/* → Auth routes (login, profile, logout)

Legacy route redirects

• /:locale/datasets/:datasetId/certificates → 301 → /datasets/:datasetId/certificates
• /:locale/datasets/:datasetId/certificates/:certificateId → 301 → /datasets/:datasetId/certificates/:certificateId
• /:locale/datasets/:datasetId/certificate → 301 → /datasets/:datasetId/certificate
• /:locale/datasets/:datasetId/certificate/embed → 301 → /datasets/:datasetId/certificate/embed
• /:locale/datasets/:datasetId/certificate/badge.png → 301 → /datasets/:datasetId/certificate/badge.png
• /:locale/datasets/:datasetId/certificate/badge.js → 301 → /datasets/:datasetId/certificate/badge.js
• /:locale/datasets/ → 301 → /datasets/

Survey routes (HTML + JSON via content negotiation)

• GET /surveys → Surveys index page (HTML) or grouped latest-per-locale list (JSON)
• GET /surveys/:surveyId/criteria → Survey criteria page (HTML)
• GET /surveys/:surveyId → Survey JSON schema (JSON) or criteria page (HTML)

Note on content negotiation: send Accept: application/json to receive JSON; otherwise HTML is rendered.

🧪 Testing

# Run tests
npm test

# Run tests with coverage
npm run test:coverage

# Run specific test file
npm test -- --testPathPattern=levelCalculationService

🚀 Deployment

Docker

# Build image
docker build -t open-data-certificate .

# Run container
docker run -p 3000:3000 open-data-certificate

Environment Variables (production example)

NODE_ENV=production
MONGO_URL=mongodb://your-mongo-host:27017/open_data_certificate
PORT=3000
SESSION_SECRET=your_production_session_secret

# Optional: OAuth in production
DJANGO_CLIENT_ID=...
DJANGO_CLIENT_SECRET=...
DJANGO_CALLBACK_URL=/auth/django/callback
GOOGLE_CLIENT_ID=...
GOOGLE_CLIENT_SECRET=...
GOOGLE_CALLBACK_URL=/auth/google/callback

🤝 Contributing

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Add tests for new functionality
5. Submit a pull request

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

• Original Ruby implementation by the Open Data Institute
• Survey structure based on the Open Data Certificate questionnaire
• Migration patterns inspired by modern data migration practices