Block-Wise Encryption for Reliable Vision Transformer models
Article Sidebar
Main Article Content
Abstract
This article presents block-wise image encryption for the vision transformer and its applications. Perceptual image encryption for deep learning enables us not only to protect the visual information of plain images but to also embed unique features controlled with a key into images and models. However, when using conventional perceptual encryption methods, the performance of models is degraded due to the inuence of encryption. In this paper, we focus on block-wise encryption for the vision transformer, and we introduce three applications: privacy-preserving image classification, access control, and the combined use of federated learning and encrypted images. Our scheme can have the same performance as models without any encryption, and it does not require any network modification. It also allows us to easily update the secret key. In experiments, the effectiveness of the scheme is demonstrated in terms of performance degradation and access control on the CIFAR-10 and CIFAR-100 datasets.
Article Details
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
References
Y. LeCun, Y. Bengio and G. Hinton, “Deep learning,” nature, vol. 521, no. 7553, p. 436, 2015.
X. Liu, Z. Deng and Y. Yang, “Recent progress in semantic image segmentation,” Artif. Intell. Rev., vol. 52, no. 2, pp. 1089–1106, 2019.
C.-T. Huang, L. Huang, Z. Qin, H. Yuan, L. Zhou, V. Varadharajan and C.-C. J. Kuo, “Survey on securing data storage in the cloud,” APSIPA Transactions on Signal and Information Processing, vol. 3, p. e7, 2014.
M.-R. Ra, R. Govindan and A. Ortega, “P3: Toward Privacy-Preserving photo sharing,” in 10th USENIX Symposium on Networked Systems Design and Implementation (NSDI 13). Lombard, IL: USENIX Association, Apr. 2013, pp. 515–528.
R. Lagendijk, Z. Erkin and M. Barni, “Encrypted signal processing for privacy protection: Conveying the utility of homomorphic encryption and multiparty computation,” IEEE Signal Processing Magazine, vol. 30, no. 1, pp. 82–105, 2013.
M. Fredrikson, S. Jha and T. Ristenpart, “Model inversion attacks that exploit confidence information and basic countermeasures,” in Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security, ser. CCS ’15. New York, NY, USA: Association for Computing Machinery, pp. 1322–1333, 2015.
R. Shokri, M. Stronati, C. Song and V. Shmatikov, “Membership inference attacks against machine learning models,” in 2017 IEEE symposium on security and privacy (SP). IEEE, pp. 3–18, 2017.
C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. J. Goodfellow, and R. Fergus, “Intriguing properties of neural networks,” in 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, Y. Bengio and Y. LeCun, Eds., 2014.
R. S. Siva Kumar, M. Nystr ̈om, J. Lambert, A. Marshall, M. Goertzel, A. Comissoneru, M. Swann, and S. Xia, “Adversarial machine learning-industry perspectives,” in 2020 IEEE Security and Privacy Workshops (SPW), pp. 69–75, 2020.
H. Kiya, A. P. M. Maung, Y. Kinoshita, S. Imaizumi, and S. Shiota, “An overview of compressible and learnable image transformation with secret key and its applications,” APSIPA Transactions on Signal and Information Processing, vol. 11, no. 1, e11, 2022. [Online]. Available: http://dx.doi.org/10.1561/116.00000048
A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An image is worth 16×16 words: Transformers for image recognition at scale,” in International Conference on Learning Representations, 2021.
T. Chuman, W. Sirichotedumrong, and H. Kiya, “Encryption-then-compression systems using grayscale-based image encryption for jpeg images,” IEEE Transactions on Information Forensics and Security, vol. 14, no. 6, pp. 1515–1525, 2019.
T. Chuman, K. Kurihara, and H. Kiya, “On the security of block scrambling-based etc systems against jigsaw puzzle solver attacks,” in 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2157–2161, 2017.
J.Zhou,X.Liu,O.C.Au,andY.Y. Tang, “Designing an efficient image encryptionthencompression system via prediction error clustering and random permutation,” IEEE Transactions on Information Forensics and Security, vol. 9, no. 1, pp. 39–50, 2014.
M. Ghonge and K. Nimbokar, “A survey based on designing an efficient image encryptionthencompression system,” International Journal of Computer Applications, p. 8887, 2014.
T.Y.Liu,K.J.Lin,andH.C.Wu,“Ecg data encryption then compression using singular value decomposition,” IEEE Journal of Biomedical and Health Informatics, vol. 22, no. 3, pp. 707–713, 2018.
W. Liu, W. Zeng, L. Dong, and Q. Yao, “Efficient compression of encrypted grayscale images,” IEEE Transactions on Image Processing, vol. 19, no. 4, pp. 1097–1102, 2010.
R. Hu, X. Li, and B. Yang, “A new lossy compression scheme for encrypted gray-scale images,” in 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7387–7390, 2014.
M. Johnson, P. Ishwar, V. Prabhakaran, D. Schonberg, and K. Ramchandran, “On compressing encrypted data,” IEEE Transactions on Signal Processing, vol. 52, no. 10, pp. 2992–3006, 2004.
M. Gaata and F. F. Hantoosh, “An efficient image encryption technique using chaotic logistic map and rc4 stream cipher,” International Journal of Modern Trends in Engineering and Research, vol. 3, pp. 213–218, 2016.
W. Sirichotedumrong and H. Kiya, “Grayscalebased block scrambling image encryption using YCbCr color space for encryptionthencompression systems,” APSIPA Transactions on Signal and Information Processing, vol. 8, p. e7, 2019.
S. Imaizumi, Y. Izawa, R. Hirasawa, and H. Kiya, “A reversible data hiding method in compressible encrypted images,” IEICE Transactions on Fundamentals of Electronics, Communications and Computer Sciences, vol. E103.A, no. 12, pp. 1579–1588, 2020.
K. Iida and H. Kiya, “Privacy-preserving content-based image retrieval using compressible encrypted images,” IEEE Access, vol. 8, pp. 200 038–200 050, 2020.
K. Iida and H. Kiya, “An image identification scheme of encrypted jpeg images for privacypreserving photo sharing services,” 2019 IEEE International Conference on Image Processing (ICIP), pp. 4564–4568, 2019.
A. Kawamura, Y. Kinoshita, T. Nakachi, S. Shiota, and H. Kiya, “A privacy-preserving machine learning scheme using etc images,” IEICE Transactions on Fundamentals of Electronics, Communications and Computer Sciences, vol. E103.A, no. 12, pp. 1571–1578, 2020.
Y. Bandoh, T. Nakachi, and H. Kiya, “Distributed secure sparse modeling based on random unitary transform,” IEEE Access, vol. 8, pp. 211 762–211 772, 2020.
T. Nakachi, Y. Bandoh, and H. Kiya, “Secure overcomplete dictionary learning for sparse representation,” IEICE Transactions on Information and Systems, vol. E103.D, no. 1, pp. 50–58, 2020.
T. Nakachi, Y. Wang, and H. Kiya, “Privacypreserving pattern recognition using encrypted sparse representations in l0 norm minimization,” in ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2697–2701, 2020.
I. Nakamura, Y. Tonomura, and H. Kiya, “Unitary transform-based template protection and its application to l 2 -norm minimization problems,” IEICE Transactions on Information and Systems, vol. E99.D, no. 1, pp. 60–68, 2016.
T. Maekawa, A. Kwamura, T. Nakachi, and H. Kiya, “Privacy-preserving support vector machine computing using random unitary transformation,” IEICE Transactions on Fundamentals of Electronics, Communications and Computer Sciences, vol. E102.A, no. 12, pp. 1849–1855, 2019.
M. Aprilpyone and H. Kiya, “Block-wise image transformation with secret key for adversarially robust defense,” IEEE Transactions on Information Forensics and Security, vol. 16, pp. 2709–2723, 2021.
A. MaungMaung and H. Kiya, “Encryption inspired adversarial defense for visual classification,” in 2020 IEEE International Conference on Image Processing (ICIP), pp. 1681–1685, 2020.
A. MaungMaung and H. Kiya, “Ensemble of keybased models: Defense against black-box adversarial attacks,” in 2021 IEEE 10th Global Conference on Consumer Electronics (GCCE), pp. 95–98, 2021.
M. Chen and M. Wu, “Protect your deep neural networks from piracy,” in 2018 IEEE International Workshop on Information Forensics and Security (WIFS), pp. 1–7, 2018.
H. Chen, C. Fu, B. D. Rouhani, J. Zhao, and F. Koushanfar, “Deepattest: An end-toend attestation framework for deep neural networks,” 2019 ACM/IEEE 46th Annual International Symposium on Computer Architecture (ISCA), pp. 487–498, 2019.
M. Aprilpyone and H. Kiya, “A protection method of trained cnn model with a secret key from unauthorized access,” APSIPA Transactions on Signal and Information Processing, vol. 10, p. e10, 2021.
Y. Uchida, Y. Nagai, S. Sakazawa, and S. Satoh, “Embedding watermarks into deep neural networks,” in Proceedings of the 2017 ACM on International Conference on Multimedia Retrieval, ser. ICMR ’17. Association for Computing Machinery, 2017, pp. 269–277.
H. Chen, B. D. Rouhani, C. Fu, J. Zhao, and F. Koushanfar, “Deepmarks: A secure fingerprinting framework for digital rights management of deep learning models,” in Proceedings of the 2019 on International Conference on Multimedia Retrieval, ser. ICMR ’19. Association for Computing Machinery, 2019, p. 105–113.
B. D. Rouhani, H. Chen, and F. Koushanfar, “Deepsigns: A generic watermarking framework for ip protection of deep learning models,” arXiv preprint arXiv:1804.00750, 2018. [Online]. Available: https://arxiv.org/abs/1804.00750
L. Fan, K. W. Ng, C. S. Chan, and Q. Yang, “Deepip: Deep neural network intellectual property protection with passports,” in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 10, pp. 6122-6139, 2021.
Y. Adi, C. Baum, M. Cisse, B. Pinkas, and J. Keshet, “Turning your weakness into a strength: Watermarking deep neural networks by backdooring,” in Proceedings of the 27th USENIX Conference on Security Symposium, ser. SEC’18. USENIX Association, 2018, pp. 1615–1631.
J. Zhang, Z. Gu, J. Jang, H. Wu, M. P. Stoecklin, H. Huang, and I. Molloy, “Protecting intellectual property of deep neural networks with watermarking,” in Proceedings of the 2018 on Asia Conference on Computer and Communications Security, ser. ASIACCS’18. Association for Computing Machinery, 2018, pp.159–172.
S. Sakazawa, E. Myodo, K. Tasaka, and H. Yanagihara, “Visual decoding of hidden watermark in trained deep neural network,” in 2019 IEEE Conference on Multimedia Information Processing and Retrieval (MIPR), 2019, pp. 371–374.
E. Le Merrer, P. P ́erez, and G. Tr ́edan, “Adversarial frontier stitching for remote neural network watermarking,” Neural Computing and Applications, vol. 32, no. 13, pp. 9233–9244, 2019.
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Association for Computational Linguistics, Jun. 2019, pp. 4171–4186.
I. Tolstikhin, N. Houlsby, A. Kolesnikov, L. Beyer, X. Zhai, T. Unterthiner, J. Yung, A. P. Steiner, D. Keysers, J. Uszkoreit, M. Lucic, and A. Dosovitskiy, “MLP-mixer: An all-MLP architecture for vision,” in Advances in Neural Information Processing Systems, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, Eds., 2021.
H. Touvron, P. B. andMathilde Caron, M. Cord, A. El-Nouby, E. Grave, A. Joulin, G. Synnaeve, J. Verbeek, and H. J ́egou, “Resmlp: Feedforward networks for image classification with dataefficient training,” CoRR, vol. abs/2105.03404, 2021.
S. Chen, E. Xie, C. GE, R. Chen, D. Liang, and P. Luo, “CycleMLP: A MLP-like architecture for dense prediction,” in International Conference on Learning Representations, 2022.
H. Liu, Z. Dai, D. So, and Q. V. Le, “Pay attention to MLPs,” in Advances in Neural Information Processing Systems, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, Eds., 2021.
Q. Hou, Z. Jiang, L. Yuan, M.-M. Cheng, S. Yan, and J. Feng, “Vision permutator: A permutable mlp-like architecture for visual recognition,” IEEE transactions on pattern analysis and machine intelligence, vol. 45, no. 1, pp. 1328-1334, 2023.
A. Trockman and J. Z. Kolter, “Patches are all you need?,” arXiv preprint arXiv:2201.09792, 2022. [Online]. Available: https://arxiv.org/abs/2201.09792
Y. Wang and K. Plataniotis, “Face based biometric authentication with changeable and privacy preservable templates,” in 2007 Biometrics Symposium, pp. 1–6, 2007.
H. Kiya, R. Iijima, A. MaungMaung, and Y. Kinoshita, “Image and model transformation with secret key for vision transformer,” IEICE Transactions on Information and Systems, vol. E106.D, no. 1, pp. 2–11, 2023.
H. Kiya, T. Nagamori, S. Imaizumi, and S. Shiota, “Privacy-preserving semantic segmentation using vision transformer,” Journal of Imaging, vol. 8, no. 9, 2022. [Online]. Available: https://www.mdpi.com/2313-433X/8/9/233
J. Koneˇcny ́, H. B. McMahan, F. X. Yu, P. Richtarik, A. T. Suresh, and D. Bacon, “Federated learning: Strategies for improving communication efficiency,” in NIPS Workshop on Private Multi-Party Machine Learning, 2016.[Online]. Available: https://arxiv.org/abs/1610.05492
B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y. Arcas, “CommunicationEfficient Learning of Deep Networks from Decentralized Data,” in Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, ser. Proceedings of Machine Learning Research, A. Singh and J. Zhu, Eds., vol. 54. PMLR, 20–22 Apr 2017, pp. 1273–1282.
A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala, “Pytorch: An imperative style, highperformance deep learning library,” in Advances in Neural Information Processing Systems 32, H. Wallach, H. Larochelle, A. Beygelzimer, F. d’Alch ́e-Buc, E. Fox, and R. Garnett, Eds. Curran Associates, Inc., 2019, pp. 8024–8035.
M. Tanaka, “Learnable image encryption,” in 2018 IEEE International Conference on Consumer Electronics-Taiwan (ICCE-TW), pp. 1–2, 2018.
W. Sirichotedumrong, T. Maekawa, Y. Kinoshita, and H. Kiya, “Privacy-preserving deep neural networks with pixel-based image encryption considering data augmentation in the encrypted domain,” in 2019 IEEE International Conference on Image Processing (ICIP), pp. 674–678, 2019.
W. Sirichotedumrong and H. Kiya, “A ganbased image transformation scheme for privacypreserving deep neural networks,” in 2020 28th European Signal Processing Conference (EUSIPCO), pp. 745–749, 2021.
H. Ito, Y. Kinoshita, M. Aprilpyone, and H. Kiya, “Image to perturbation: An image transformation network for generating visually protected images for privacy-preserving deep neural networks,” IEEE Access, vol. 9, pp. 64 629–64 638, 2021.
T. Chuman and H. Kiya, “A jigsaw puzzle solverbased attack on image encryption using vision transformer for privacy-preserving dnns,” Information, vol. 14, no. 6, 2023. [Online]. Available: https://www.mdpi.com/2078-2489/14/6/311
T. Nagamori and H. Kiya, “ombined use of federated learning and image encryption for privacypreserving image classification with vision transformer,” arXiv preprint arXiv:2301.09255, 2023. [Online]. Available: https://arxiv.org/abs/2301.09255