AI Doesn't Always Run in the Cloud — Why 'Edge Computing' Is Creating a Whole New Engineering Career

Your car’s collision avoidance system can’t wait 200 milliseconds for a round-trip to a data center. It needs to make a decision in under 10. That’s why the AI running in your car, your smart speaker, your hospital’s imaging machine, and increasingly your factory floor doesn’t run “in the cloud” — it runs locally, on specialized hardware, on the “edge” of the network. Edge computing is one of the fastest-growing subfields in engineering, and it barely existed as a career path 10 years ago. The engineers who build edge AI systems are writing the rules of a field that’s being invented in real time.

Cloud vs. Edge: Why the Distinction Matters

To understand why edge computing exists, you need to understand what “the cloud” actually is: a collection of massive data centers, typically owned by Amazon (AWS), Microsoft (Azure), or Google (Google Cloud), that provide computing resources over the internet. When you ask ChatGPT a question, your query leaves your device, travels to a data center, gets processed by servers there, and the response travels back to you. This round-trip takes anywhere from 50 to 500 milliseconds depending on network conditions.

For most applications, this latency is fine. Asking a chatbot a question can tolerate a half-second delay. Loading a webpage can tolerate even more. But consider:

A Tesla autopilot system detecting a pedestrian stepping into the road needs to initiate braking within 10–20 milliseconds. An industrial robot arm detecting a worker’s hand in its path needs to stop within 5 milliseconds. A hospital CT scanner running AI-powered anomaly detection needs to process each image frame as it’s acquired, not after a network round-trip. A smart speaker detecting its wake word needs to respond immediately, without sending audio of every conversation to a cloud server.

For all of these applications, the cloud model doesn’t work — either because latency requirements make it physically impossible, or because privacy requirements make it legally and ethically problematic, or because network connectivity can’t be guaranteed (a factory floor, a rural agricultural sensor, a ship in the middle of the ocean).

Edge computing is the solution: running the AI computation locally, on hardware embedded in the device or deployed nearby, with enough processing power to handle the workload within the required constraints.

The Engineering Challenges of Edge AI

Building AI systems that run at the edge is fundamentally different from building AI that runs on unrestricted cloud hardware. The constraints are severe:

Power constraints. A datacenter GPU can consume 700 watts. An edge AI chip in a camera or wearable device might have a power budget of 1–5 watts. Running a useful AI model within that constraint requires extreme optimization — selecting or designing models that achieve the required capability at tiny computational cost.

Memory constraints. Datacenter servers have terabytes of RAM and fast interconnects between chips. An edge device might have 4–16 GB of memory. Fitting a model that would normally require 80 GB of GPU memory into 4 GB requires architectural choices, quantization, and model compression techniques that are active research areas.

Thermal constraints. Edge devices often can’t use active cooling. A camera mounted outdoors or a medical device can’t have fans; passive cooling through the device case is the only option. This means the hardware operating temperature is constrained by ambient conditions, which limits how hard the chip can run.

Real-time requirements. Edge AI often means hard real-time deadlines — responses that are correct but arrive 1 millisecond too late are failures. Software systems with real-time guarantees use different programming paradigms, operating systems, and validation methods than conventional software.

Connectivity constraints. Edge devices may have intermittent or no internet access. They need to operate fully autonomously, update their models periodically, and sync data when connectivity is available — a very different operational model than always-connected cloud systems.

The Hardware of Edge AI

NVIDIA’s Jetson platform — a family of small, low-power AI computing modules — is the dominant edge AI hardware platform for high-capability applications. The Jetson Orin, released in 2022, delivers 275 TOPS (trillion operations per second) in a module the size of a credit card, consuming approximately 60 watts. This is enough to run sophisticated computer vision and natural language models in real time. It’s used in robotics, autonomous vehicles, medical devices, and industrial automation.

Google’s Edge TPU — a custom chip designed for running TensorFlow Lite models — is optimized for even lower power: it can execute inference at 2 TOPS while consuming approximately 2 watts. This makes it suitable for battery-powered devices and embedded systems where the Jetson’s power budget is too high.

Qualcomm’s Snapdragon platforms — which power most Android smartphones — increasingly include dedicated AI accelerators (Neural Processing Units, or NPUs) that run on-device AI models. When Siri processes your speech on an iPhone or Google Assistant activates on a Pixel, the inference is running on an NPU in the phone, not in a data center.

Apple’s Neural Engine, integrated into every A-series and M-series chip since the A11 in 2017, is specifically designed for efficient on-device AI inference. Face ID, photo processing, speech recognition, and predictive text all run on this hardware without sending data to Apple’s servers.

The semiconductor engineering required to design these chips — which need to achieve AI computation with radically different power and area constraints than datacenter processors — is a highly specialized and extremely well-compensated discipline. We’ve covered the broader semiconductor career landscape in our piece on GPU and semiconductor careers in the AI industry.

The Research Picture

Gartner’s Hype Cycle consistently identifies edge AI as past the peak of inflated expectations and moving toward the productive trough — which means actual deployment at scale rather than speculative hype. Their 2023 and 2024 analyses found that edge AI deployments were growing 30–40% annually across manufacturing, healthcare, retail, and transportation.

IDC’s Worldwide Edge Computing Spending Guide projects that edge computing spending will grow from approximately $232 billion in 2024 to $350 billion by 2027. Importantly, AI at the edge is cited as the primary driver of this growth — not generic edge computing, but specifically AI inference workloads that must execute locally.

Academic research at MIT, Stanford, and CMU has focused on the model efficiency techniques that make edge AI viable. MIT’s TinyML initiative, which focuses on running machine learning models on microcontrollers with kilobytes of memory, has demonstrated remarkable capability from extraordinarily constrained hardware. Stanford’s HAI (Human-Centered AI) initiative has funded significant research on edge AI for healthcare applications, specifically because healthcare data privacy requirements make cloud processing problematic for many use cases.

Application	Latency Requirement	Power Constraint	Privacy Driver	Edge vs. Cloud
Automotive ADAS	<20ms	Limited (vehicle power)	No	Edge (safety-critical latency)
Medical imaging AI	<100ms	Low (portable device)	Yes (HIPAA)	Edge (privacy + latency)
Industrial robotics	<5ms	Moderate	Sometimes	Edge (hard real-time)
Smart speakers (wake word)	<50ms	Very low (battery)	Yes	Edge (always-on, privacy)
Retail analytics	<1s	Moderate	Yes (GDPR)	Edge or hybrid
Agricultural sensors	Variable	Very low (solar/battery)	No	Edge (no connectivity)

Sources: Gartner, IDC, MIT TinyML project

Compensation for edge AI engineers reflects both the scarcity of the skill set and the criticality of the applications. Mid-career embedded AI engineers — also titled Edge ML Engineer, Embedded Systems ML Engineer, or Real-Time AI Engineer — earn $145,000–$195,000 at companies like NVIDIA, Qualcomm, Apple, Tesla, or robotics startups. Senior engineers with expertise in both model optimization and embedded systems earn $195,000–$250,000. At medical device companies (where FDA regulatory expertise adds another premium), compensation is comparable.

What This Means for Your Kid

Edge computing sits at the intersection of three disciplines that rarely overlap in a single engineer: embedded systems hardware, machine learning software, and real-time systems engineering. That combination is genuinely rare, and its rarity is the career opportunity.

The kids most likely to find this field rewarding are those who are interested in physical devices — who want to know not just that AI works, but how the AI runs on the chip in their hand, how the chip manages its power budget, how the software communicates with the hardware in real time. This is engineering that you can see and touch, unlike cloud software that lives invisibly in data centers.

One of the most accessible entry points is the Raspberry Pi and Arduino ecosystem, combined with machine learning frameworks designed for resource-constrained devices. TensorFlow Lite and PyTorch Mobile allow students to run trained models on a Raspberry Pi or even on Arduino-class hardware. The experience of optimizing a computer vision model to run at 30 frames per second on a $35 computer is directly applicable to professional edge AI engineering.

This connects naturally to our broader piece on embedded systems engineering — edge computing is, at its core, advanced embedded systems engineering with AI workloads. Kids who build the foundation in embedded systems have a clear path to edge AI specialization.

Practical preparation:

• Ages 10–13: Raspberry Pi projects that connect sensors to software. The experience of a program running on a physical device — and being constrained by what that device can do — is foundational
• Ages 14–17: TensorFlow Lite or PyTorch Mobile tutorials. Run an image classifier on a Raspberry Pi. Measure how long it takes. Try to make it faster. This is the core edge AI engineering loop
• Ages 17+: Computer engineering or electrical engineering with embedded systems and machine learning coursework. Programs that offer real-time systems or embedded AI courses are increasingly available at schools like Georgia Tech, Carnegie Mellon, and University of Michigan

What to Watch Over the Next 3 Months

• Apple’s M-series and A-series chip announcements. Apple’s Neural Engine generations are benchmarks for on-device AI capability. Each new generation demonstrates the expanding frontier of what’s possible at the edge
• NVIDIA Jetson Orin NX and upcoming platform updates. NVIDIA’s edge platform roadmap shows where professional edge AI hardware is heading
• TensorFlow Lite and PyTorch Mobile framework updates. Model optimization framework improvements make it easier to deploy capable AI at the edge — their release notes are a window into what’s becoming possible
• Healthcare and industrial automation news. These two sectors are the largest deployers of edge AI, and their product announcements often describe the technical requirements that edge AI engineers are building to

The career path is emerging fast enough that a student who starts building edge AI projects at 15 will have a portfolio that few undergraduates in their cohort possess. That advantage compounds through internship applications, research opportunities, and early career positioning.

FAQ

Is edge computing the same as fog computing? Edge computing and fog computing are related but distinct. Edge computing places computation at or near the device generating data. Fog computing is an intermediate layer between the edge and the cloud — small local servers that aggregate data from many edge devices and perform more complex processing than edge devices can handle, while remaining closer than the cloud. In practice, the terms are often used interchangeably.

Do edge AI engineers need to understand hardware design? It depends on the role. Software-side edge AI engineers (model optimization, inference framework development) need to understand hardware constraints and characteristics but don’t necessarily design chips. Hardware-side roles (chip design, hardware-software co-design) require deeper hardware expertise. The most valuable engineers have enough understanding of both sides to make informed decisions.

How does edge AI relate to privacy? Significantly. AI that runs locally on a device doesn’t need to send data to a server for processing. This is the primary reason that face recognition on your iPhone and voice recognition on smart speakers process locally — it’s both a privacy feature and a latency feature. Privacy regulations like GDPR and HIPAA are driving edge AI adoption in sectors where data cannot leave the device.

Is Python used in edge AI? Python is used for model training and development, but deployed edge AI typically uses C, C++, or specialized frameworks. TensorFlow Lite and PyTorch Mobile use C++ runtimes. Some edge platforms support Python for development but optimize to C++ for production. Kids who learn Python can start in edge AI development; production work requires lower-level languages.

What companies hire edge AI engineers? Apple, NVIDIA, Qualcomm, Google, Amazon (Alexa), Tesla, Waymo, medical device companies (Medtronic, GE Healthcare, Siemens Healthineers), industrial automation companies (Fanuc, Rockwell Automation, Siemens), and a large ecosystem of robotics and IoT startups.

---

About the author Ricky Flores is the founder of HiWave Makers and an electrical engineer with 15+ years of experience building consumer technology at Apple, Samsung, and Texas Instruments. He writes about how kids learn to build, think, and create in a tech-saturated world. Read more at hiwavemakers.com.

---

Sources

1. Warden, P., Situnayake, D. TinyML: Machine Learning with TensorFlow Lite. O’Reilly Media, 2019. https://www.oreilly.com/library/view/tinyml/9781492052036/
2. Gartner. Hype Cycle for Emerging Technologies 2023. https://www.gartner.com/en/articles/what-s-new-in-the-2023-gartner-hype-cycle-for-emerging-technologies
3. IDC. Worldwide Edge Computing Spending Guide. https://www.idc.com/getdoc.jsp?containerId=IDC_P33170
4. MIT TinyML Initiative. https://tinyml.mit.edu
5. NVIDIA. Jetson Platform Overview. https://www.nvidia.com/en-us/autonomous-machines/embedded-systems/
6. Qualcomm Technologies. AI for Edge Devices. https://www.qualcomm.com/research/artificial-intelligence
7. Ray, P.P. “A survey on Internet of Things architectures.” Journal of King Saud University – Computer and Information Sciences, 2018. https://doi.org/10.1016/j.jksuci.2016.10.003