Building Enterprise IoT Platforms on AWS: Lessons from Connecting 10,000+ Devices

The Enterprise IoT Challenge

Enterprise IoT is fundamentally different from consumer IoT. When you're connecting thousands of commercial devices — vending machines, industrial sensors, medical equipment, fleet vehicles — the requirements are dramatically more demanding. Devices must operate reliably in hostile network environments, data must be processed in real-time for operational decisions, and the platform must scale from hundreds to tens of thousands of devices without architectural changes.

Our experience building the Costa Express IoT platform — connecting over 1,000 smart vending machines across the UK — taught us that the technology challenges are only half the problem. The other half is operational: how do you deploy firmware updates to thousands of devices without bricking them? How do you diagnose connectivity issues in a machine installed in a hospital basement with intermittent WiFi? How do you ensure data integrity when devices go offline for hours or days?

Reference Architecture: Enterprise IoT on AWS

After multiple IoT deployments, we've converged on a reference architecture that handles the common challenges while remaining flexible enough to accommodate domain-specific requirements.

Device Layer

The device layer handles local data collection, edge processing, and communication with the cloud platform. Key design decisions:

• Edge computing: Critical decisions (safety shutoffs, basic anomaly detection) happen on-device without cloud dependency. AWS IoT Greengrass enables running Lambda functions locally on devices with periodic cloud sync.
• Offline resilience: Devices buffer data locally when connectivity is lost and sync automatically when connection is restored. Message ordering and deduplication are handled at the platform level.
• Secure boot and attestation: Every device verifies its firmware integrity on startup and attests its identity to the cloud platform using X.509 certificates stored in hardware security modules.
• OTA updates: Firmware updates deployed in staged rollouts — 1% of devices first, then 10%, then 50%, then 100% — with automatic rollback if error rates exceed thresholds.

Connectivity Layer

Reliable connectivity is the hardest problem in enterprise IoT. Devices operate in environments with poor cellular coverage, congested WiFi, and intermittent power. Our connectivity layer handles:

• Protocol flexibility: MQTT for real-time telemetry (low overhead, bidirectional), HTTPS for bulk data uploads and firmware downloads, and LoRaWAN for low-power sensors in remote locations.
• Connection management: AWS IoT Core handles millions of concurrent MQTT connections with automatic scaling. Custom connection managers handle reconnection logic, backoff strategies, and connection pooling.
• Message routing: IoT Core Rules Engine routes messages to appropriate processing pipelines based on topic, content, and device metadata. Critical alerts go to real-time processing; routine telemetry goes to batch pipelines.

Data Processing Layer

IoT data comes in high volume and requires multiple processing patterns:

• Real-time stream processing: Amazon Kinesis Data Streams for sub-second processing of critical events (alerts, anomalies, threshold breaches). Kinesis Data Analytics for real-time aggregations and pattern detection.
• Batch processing: S3 + Glue for historical analysis, trend detection, and reporting. Data lands in S3 via Kinesis Firehose, is catalogued by Glue Crawlers, and queried via Athena.
• Time-series storage: Amazon Timestream for operational dashboards and short-term queries. S3 + Parquet for long-term storage and cost-effective historical analysis.

Application Layer

The application layer serves operators, maintenance teams, and business stakeholders:

• Operational dashboards: Real-time device status, health metrics, and alert management. Built on React with WebSocket connections for live updates.
• Predictive maintenance: ML models (SageMaker) trained on historical failure data predict maintenance needs 2-4 weeks before failure. Maintenance teams receive prioritised work orders based on predicted failure probability and business impact.
• Business intelligence: Aggregated data feeds into business dashboards showing utilisation, revenue per device, and operational efficiency metrics.

Lessons Learned: What We'd Do Differently

1. Invest in Device Simulation Early

You can't test an IoT platform with 10 devices and expect it to work with 10,000. We build device simulators that generate realistic traffic patterns, failure modes, and edge cases. This enables load testing, chaos engineering, and development without physical devices.

2. Design for Partial Failure

In any fleet of thousands of devices, some percentage will always be in a degraded state. The platform must handle partial data, delayed messages, and conflicting state without corrupting the overall system view. Event sourcing and CQRS patterns handle this elegantly.

3. Observability is Non-Negotiable

When a device in a remote location stops reporting, you need to diagnose the issue remotely. Is it a network problem? A firmware crash? A power failure? A hardware fault? Comprehensive device-side logging, heartbeat monitoring, and remote diagnostics capabilities are essential from day one.

4. Plan for Device Lifecycle

Devices have a lifecycle: provisioning, activation, operation, maintenance, decommissioning. Each stage has different security, data, and operational requirements. Design the platform to handle the full lifecycle, not just steady-state operation.

The Costa Express platform now processes over 2 million telemetry events daily from 1,000+ machines, with 99.97% data delivery reliability and sub-5-second alert latency. The architecture has scaled 3x since initial deployment without any structural changes.

When to Build vs. Buy

AWS IoT services provide excellent building blocks, but they're not a complete platform. You'll always need custom logic for device management workflows, business-specific analytics, and integration with existing enterprise systems. Our recommendation: use AWS IoT Core for connectivity and basic device management, but build custom application logic for everything above the connectivity layer.