Uma arquitetura de rede neural com offloading para dispositivos com memória e processamento limitados

Abstract

Data inferece by deep neural networks (DNNs) on low cost devices, such as embedded systems and microcontrollers, may be impracticable. Given the need of storing many layers on memory, plus the use of complex arithmetic operations, many of these DNNs resorts to high-performance and dedicated CPUs and GPUs. An alternative to use these DNNs on low cost devices is to use offloading. In other words, assign the complex processing to a high-performance device, such as a virtual machine hosted on a cloud service. However, assigning the majority of the processing to a third-party device, the system becomes dependable of the cloud’s availability, exposing it to the risks of network unstability and data transmission delays. This research proposes the HOENN model (Hybrid Offloading Embedded Neural Network), a hybrid model composed by a memory-arithmetical optimized neural network for low cost devices and a DNN hosted on a virtual machine on a cloud service. In addition, the model implements a confidence metric, and a threshold value, to decide whether the device’s inference is finished on the device or on the cloud. This way, the HOENN model receives input data and infers using the optimized neural network. For each inference, a confidence value is assigned. If this value surpasses the threshold, the inference is kept and sent to the end-user. If not, this inference is discarded and the input data is sent to the cloud to a second inference by the DNN. Experiments on a low cost device – ESP32S – connected to the cloud – AWS t2.medium –, using a dataset of handwritten numeric digits, show that it is possible to reduce the amount of data sent to the cloud by 75%, with a reduction of 50% on average inference time, when compared to inferring only on the cloud, under a penalty of only 10% on system’s accuracy.