Chapter 6: Data Platform — GW & DW Monitoring Design Guide

6.1 Platform Architecture

The monitoring platform is the central intelligence of the system, responsible for receiving, validating, storing, analyzing, and reporting all monitoring data. A well-designed platform transforms raw sensor readings into actionable intelligence — validated trends, timely alarms, and audit-ready compliance reports. A poorly designed platform, conversely, can undermine the value of even the best field instrumentation by introducing data quality issues, alarm fatigue, or compliance gaps.

The platform architecture follows a microservices pattern, with each functional component independently scalable and maintainable. The core services include: Ingestion Service (receives and parses telemetry from all field devices); QA/QC Engine (validates data quality and applies flags); Time-Series Database (stores all validated data with efficient time-range queries); Alarm Engine (evaluates rules and dispatches notifications); Reporting Service (generates scheduled and on-demand reports); and API Gateway (provides secure access for external integrations and the web/mobile frontend).

Platform Service	Primary Function	Key Requirements	Performance Target
Ingestion Service	Receive, parse, and route telemetry from all field devices	MQTT, HTTPS, Modbus TCP, OPC UA support; backlog handling	Latency <5 s from receipt to storage
QA/QC Engine	Validate data quality, apply flags, detect anomalies	Configurable rules, false positive rate <1%, audit trail	Process 10,000 data points/second
Time-Series Database	Store and query all validated monitoring data	5-year retention, sub-second query for 1-year range	Query response <2 s for 1-year range
Alarm Engine	Evaluate alarm rules, dispatch notifications	Multi-level rules, escalation paths, suppression logic	Alarm dispatch <30 s from trigger
Reporting Service	Generate scheduled and on-demand compliance reports	Regulatory format templates, PDF/Excel export, audit trail	Report generation <60 s for 1-year period
API Gateway	Secure access for external integrations and frontend	REST API, OAuth2, rate limiting, API versioning	99.9% uptime, <200 ms response

6.2 QA/QC Data Pipeline

The QA/QC pipeline is the most critical component for data quality assurance. It operates as a multi-stage filter that transforms raw sensor readings into validated, flagged data suitable for compliance reporting and alarm evaluation. Each stage applies a specific set of checks, and data that fails a check is flagged rather than deleted — preserving the original record while clearly marking its quality status for downstream users.

The pipeline operates in four stages. Stage 1 (Range Check) verifies that each reading falls within the physically possible range for the parameter and sensor type — values outside this range are flagged as "invalid." Stage 2 (Rate-of-Change Check) detects step changes that exceed the maximum physically possible rate of change, flagging them as "spike" or "step change." Stage 3 (Persistence Check) detects stuck sensors by identifying readings that remain unchanged for longer than the expected variability period. Stage 4 (Cross-Validation) compares online sensor readings against laboratory results and redundant sensors, flagging divergences that exceed calibration tolerance.

QA/QC Stage	Check Type	Trigger Condition	Flag Applied	Action
Stage 1	Range Check	Value outside [min, max] for parameter	INVALID	Exclude from compliance reports; raise maintenance alarm
Stage 2	Rate-of-Change	Δvalue/Δtime exceeds physical maximum	SPIKE	Flag data; notify operator for investigation
Stage 3	Persistence Check	Value unchanged for >N consecutive readings	STUCK	Create calibration/maintenance work ticket
Stage 4	Cross-Validation	Divergence from lab result or redundant sensor >tolerance	SUSPECT	Hold for operator review; schedule recalibration
Stage 5	Completeness Check	Missing data beyond expected gap threshold	MISSING	Gap report; comms alarm if threshold exceeded

6.3 Alarm Engine Design

Alarm engine design is one of the most consequential aspects of monitoring system implementation. A poorly configured alarm engine — with too many alarms, poorly set thresholds, or no escalation logic — leads to alarm fatigue, where operators begin ignoring alarms because they are too frequent or too often false. Alarm fatigue is a well-documented safety risk in industrial monitoring systems and has been implicated in numerous environmental incidents where real events were missed because operators had learned to ignore the alarm system.

The recommended alarm design follows a three-level hierarchy: Level 1 (Warning) — parameter approaching threshold, no immediate action required, logged for trend analysis; Level 2 (Alarm) — parameter has exceeded threshold, operator notification required within 15 minutes, work ticket created; Level 3 (Critical) — parameter has exceeded critical threshold or multiple parameters are simultaneously alarming, immediate escalation to senior operator and management, automatic emergency response actions triggered.

Alarm Level	Trigger	Notification	Response Time	Escalation
Level 1 — Warning	80% of alarm threshold exceeded	Dashboard indicator, daily digest email	Next business day	None (logged only)
Level 2 — Alarm	Alarm threshold exceeded, QA/QC validated	SMS + email to on-call operator	≤15 min acknowledgment	To supervisor if unacknowledged after 30 min
Level 3 — Critical	Critical threshold exceeded or multi-parameter event	Phone call + SMS + email to all contacts	Immediate (<5 min)	To management and regulatory authority if required

6.4 Reporting & Compliance

The reporting module must generate reports that satisfy regulatory requirements without requiring manual data manipulation. Regulatory authorities increasingly require that compliance reports be generated directly from the monitoring system's validated data, with a clear audit trail showing that the data has not been altered after collection. This requirement drives several design decisions: immutable data storage, role-based access control for data modification, and complete audit logging of all data access and changes.

Standard report types include daily operational summaries, monthly compliance reports, annual trend analysis reports, and event-specific incident reports. All reports must include the data source, QA/QC status of each data point, calibration status at the time of measurement, and any data gaps or flags. Reports must be exportable in both PDF (for regulatory submission) and Excel (for further analysis) formats.

Compliance Reporting Checklist

All reported data points have QA/QC status flags visible
Calibration certificate references are included for each sensor
Data gaps are explicitly reported with cause and duration
Report generation timestamp and operator identity are recorded
Reports are stored with immutable checksums for audit verification
Regulatory submission status is tracked and confirmed