E-mail Alert
RSS
| Citation: |
Zhao BH, Cheng JW, Wu B, Gao DS, Zhou HL et al. Integrated photonic convolution acceleration core for wearable devices. Opto-Electron Sci 2, 230017 (2023). doi: 10.29026/oes.2023.230017
|
Integrated photonic convolution acceleration core for wearable devices
-
Abstract
With the advancement of deep learning and neural networks, the computational demands for applications in wearable devices have grown exponentially. However, wearable devices also have strict requirements for long battery life, low power consumption, and compact size. In this work, we propose a scalable optoelectronic computing system based on an integrated optical convolution acceleration core. This system enables high-precision computation at the speed of light, achieving 7-bit accuracy while maintaining extremely low power consumption. It also demonstrates peak throughput of 3.2 TOPS (tera operations per second) in parallel processing. We have successfully demonstrated image convolution and the typical application of an interactive first-person perspective gesture recognition application based on depth information. The system achieves a comparable recognition accuracy to traditional electronic computation in all blind tests. -
-
References
[1] Zhang SB, Li YX, Zhang S, Shahabi F, Xia S et al. Deep learning in human activity recognition with wearable sensors: a review on advances. Sensors 22, 1476 (2022). doi: 10.3390/s22041476 [2] Chang WJ, Chen LB, Chiou YZ. Design and implementation of a drowsiness-fatigue-detection system based on wearable smart glasses to increase road safety. IEEE Trans Consum Electron 64, 461–469 (2018). doi: 10.1109/TCE.2018.2872162 [3] Ramanujam E, Perumal T, Padmavathi S. Human activity recognition with smartphone and wearable sensors using deep learning techniques: a review. IEEE Sensors J 21, 13029–13040 (2021). doi: 10.1109/JSEN.2021.3069927 [4] Chen KX, Zhang DL, Yao LN, Guo B, Yu ZW et al. Deep learning for sensor-based human activity recognition: overview, challenges, and opportunities. ACM Comput Surv 54, 77 (2022). doi: 10.1145/3447744 [5] Wang JD, Chen YQ, Hao SJ, Peng XH, Hu LS. Deep learning for sensor-based activity recognition: a survey. Pattern Recognit Lett 119, 3–11 (2019). doi: 10.1016/j.patrec.2018.02.010 [6] Nweke HF, Teh YW, Al-garadi MA, Alo UR. Deep learning algorithms for human activity recognition using mobile and wearable sensor networks: state of the art and research challenges. Expert Syst Appl 105, 233–261 (2018). doi: 10.1016/j.eswa.2018.03.056 [7] Terven JR, Raducanu B, Meza-de-Luna ME, Salas J. Head-gestures mirroring detection in dyadic social interactions with computer vision-based wearable devices. Neurocomputing 175, 866–876 (2016). doi: 10.1016/j.neucom.2015.05.131 [8] Perusquía-Hernández M, Hirokawa M, Suzuki K. A wearable device for fast and subtle spontaneous smile recognition. IEEE Trans Affective Comput 8, 522–533 (2017). doi: 10.1109/TAFFC.2017.2755040 [9] Gruebler A, Suzuki K. Design of a wearable device for reading positive expressions from facial EMG signals. IEEE Trans Affective Comput 5, 227–237 (2014). doi: 10.1109/TAFFC.2014.2313557 [10] Brown TB, Mann B, Ryder N, Subbiah M, Kaplan J et al. Language models are few-shot learners. In Proceedings of the 34th International Conference on Neural Information Processing Systems 159 (ACM, 2020); http://doi.org/10.5555/3495724.3495883. [11] Hussein AI. Wearable computing: challenges of implementation and its future. In 2015 12th Learning and Technology Conference 14–19 (IEEE, 2015);http://doi.org/10.1109/LT.2015.7587224. [12] Zhou HL, Dong JJ, Cheng JW, Dong WC, Huang CR et al. Photonic matrix multiplication lights up photonic accelerator and beyond. Light Sci Appl 11, 30 (2022). doi: 10.1038/s41377-022-00717-8 [13] Shastri BJ, Tait AN, Ferreira De Lima T, Pernice WHP, Bhaskaran H et al. Photonics for artificial intelligence and neuromorphic computing. Nat Photonics 15, 102–114 (2021). doi: 10.1038/s41566-020-00754-y [14] Nahmias MA, De Lima TF, Tait AN, Peng HT, Shastri BJ et al. Photonic multiply-accumulate operations for neural networks. IEEE J Sel Top Quantum Electron 26, 1–18 (2020). doi: 10.1109/jstqe.2019.2941485 [15] Lin X, Rivenson Y, Yardimci NT, Veli M, Luo Y et al. All-optical machine learning using diffractive deep neural networks. Science 361, 1004–1008 (2018). doi: 10.1126/science.aat8084 [16] Panuski CL, Christen I, Minkov M, Brabec CJ, Trajtenberg-Mills S et al. A full degree-of-freedom spatiotemporal light modulator. Nat Photonics 16, 834–842 (2022). doi: 10.1038/s41566-022-01086-9 [17] Li JX, Hung YC, Kulce O, Mengu D, Ozcan A. Polarization multiplexed diffractive computing: all-optical implementation of a group of linear transformations through a polarization-encoded diffractive network. Light Sci Appl 11, 153 (2022). doi: 10.1038/s41377-022-00849-x [18] Xu XY, Tan MX, Corcoran B, Wu JY, Boes A et al. 11 TOPS photonic convolutional accelerator for optical neural networks. Nature 589, 44–51 (2021). doi: 10.1038/s41586-020-03063-0 [19] Bai BW, Yang QP, Shu HW, Chang L, Yang FH et al. Microcomb-based integrated photonic processing unit. Nat Commun 14, 66 (2023). doi: 10.1038/s41467-022-35506-9 [20] Shen YC, Harris NC, Skirlo S, Prabhu M, Baehr-Jones T et al. Deep learning with coherent nanophotonic circuits. Nat Photonics 11, 441–446 (2017). doi: 10.1038/nphoton.2017.93 [21] Hughes TW, Minkov M, Shi Y, Fan SH. Training of photonic neural networks through in situ backpropagation and gradient measurement. Optica 5, 864–871 (2018). doi: 10.1364/OPTICA.5.000864 [22] Zhou HL, Zhao YH, Wang X, Gao DS, Dong JJ et al. Self-configuring and reconfigurable silicon photonic signal processor. ACS Photonics 7, 792–799 (2020). doi: 10.1021/acsphotonics.9b01673 [23] Zhang H, Gu M, Jiang XD, Thompson J, Cai H et al. An optical neural chip for implementing complex-valued neural network. Nat Commun 12, 457 (2021). doi: 10.1038/s41467-020-20719-7 [24] Tait AN, De Lima TF, Zhou E, Wu AX, Nahmias MA et al. Neuromorphic photonic networks using silicon photonic weight banks. Sci Rep 7, 7430 (2017). doi: 10.1038/s41598-017-07754-z [25] Xu SF, Wang J, Yi SC, Zou WW. High-order tensor flow processing using integrated photonic circuits. Nat Commun 13, 7970 (2022). doi: 10.1038/s41467-022-35723-2 [26] Huang CR, Bilodeau S, Ferreira De Lima T, Tait AN, Ma PY et al. Demonstration of scalable microring weight bank control for large-scale photonic integrated circuits. APL Photonics 5, 040803 (2020). doi: 10.1063/1.5144121 [27] Cheng JW, He ZM, Guo YH, Wu B, Zhou HL et al. Self-calibrating microring synapse with dual-wavelength synchronization. Photonics Res 11, 347 (2023). doi: 10.1364/PRJ.478370 [28] Zhang WP, Huang CR, Peng HT, Bilodeau S, Jha A et al. Silicon microring synapses enable photonic deep learning beyond 9-bit precision. Optica 9, 579–584 (2022). doi: 10.1364/OPTICA.446100 [29] Bandini A, Zariffa J. Analysis of the hands in egocentric vision: a survey. IEEE Trans Pattern Anal Mach Intell 45, 6846–6866 (2023). doi: 10.1109/TPAMI.2020.2986648 [30] Zhang YF, Cao CQ, Cheng J, Lu HQ. EgoGesture: a new dataset and benchmark for egocentric hand gesture recognition. IEEE Trans Multimedia 20, 1038–1050 (2018). doi: 10.1109/TMM.2018.2808769 [31] Razzari L, Duchesne D, Ferrera M, Morandotti R, Chu S et al. CMOS-compatible integrated optical hyper-parametric oscillator. Nat Photonics 4, 41–45 (2010). doi: 10.1038/nphoton.2009.236 [32] Moss DJ, Morandotti R, Gaeta AL, Lipson M. New CMOS-compatible platforms based on silicon nitride and Hydex for nonlinear optics. Nat Photonics 7, 597–607 (2013). doi: 10.1038/nphoton.2013.183 [33] Kippenberg TJ, Gaeta AL, Lipson M, Gorodetsky ML. Dissipative kerr solitons in optical microresonators. Science 361, eaan8083 (2018). [34] Graves A, Wayne G, Reynolds M, Harley T, Danihelka I et al. Hybrid computing using a neural network with dynamic external memory. Nature 538, 471–476 (2016). doi: 10.1038/nature20101 -
Supplementary Information
Movie 
-
Access History
Figures(5)
Tables(1)
Article Metrics
Export File
Citation
Zhao BH, Cheng JW, Wu B, Gao DS, Zhou HL et al. Integrated photonic convolution acceleration core for wearable devices. Opto-Electron Sci 2, 230017 (2023). doi: 10.29026/oes.2023.230017
Format
Content
DownLoad:
-
Figure 1.
Schematic of a computing system based on the integrated convolution acceleration core (PCAC) chip.
-
Figure 2.
(a) Detailed photos of the packaged layout chip show the MRR array in the center, with the photonics chip on the right combined with the leads of the customized printed circuit board (PCB) for computation and control. On the left, there is an optical input/output port using a fiber V-groove, and the entire assembly is mounted on a TEC for heat dissipation. (b) The micrograph of the MRR array and detailed photo of a single MRR. (c) The transmission spectra of the MRR array. Different voltages (800–1800 mV, 100 mV/step) are applied to the third MRR. Similar results can be obtained when the voltage is applied to other MRRs. (d) The transmission rate of a single IM on the chip under voltage tuning. These curves represent the normalized W-V mapping. (e) The transmission rate of a single MRR on the chip under voltage tuning. These curves represent the normalized W-V mapping.
-
Figure 3.
(a) Experimental setup of the PCAC chip for performing convolutional operations. (b) Original image used for demonstrating the convolution effect. (c) Convolution kernels used: Bottom sobel, Top sobel, Left sobel. (d) Corresponding convolution image results.
-
Figure 4.
(a) Schematic diagram of the convolutional neural network (CNN) architecture suitable for first-person digit gesture recognition with depth information. (b) Probability of recognition for the 10 gestures after performing the convolutional layer computation using the PCAC chip as a replacement for the computer.
-
Figure 5.
(a) Scatter plot comparing measured results with calculated results for Gesture 2. (b) Probability distribution of the error offset in the experimental results, resembling a Gaussian curve. (c) Offset of each point during the computation process. (d) Results of the first layer convolution computation obtained through electronic computation. (e) Results of the first layer convolution computation obtained through optical-electronic computation using the PCAC chip.
