Match!

Optimizing Kernel Machines Using Deep Learning

Published on Nov 1, 2018in IEEE Transactions on Neural Networks11.68
· DOI :10.1109/TNNLS.2018.2804895
Huan Song4
Estimated H-index: 4
(ASU: Arizona State University),
Jayaraman J. Thiagarajan14
Estimated H-index: 14
(LLNL: Lawrence Livermore National Laboratory)
+ 1 AuthorsAndreas Spanias28
Estimated H-index: 28
(ASU: Arizona State University)
Abstract
Building highly nonlinear and nonparametric models is central to several state-of-the-art machine learning systems. Kernel methods form an important class of techniques that induce a reproducing kernel Hilbert space (RKHS) for inferring non-linear models through the construction of similarity functions from data. These methods are particularly preferred in cases where the training data sizes are limited and when prior knowledge of the data similarities is available. Despite their usefulness, they are limited by the computational complexity and their inability to support end-to-end learning with a task-specific objective. On the other hand, deep neural networks have become the de facto solution for end-to-end inference in several learning paradigms. In this paper, we explore the idea of using deep architectures to perform kernel machine optimization, for both computational efficiency and end-to-end inferencing. To this end, we develop the deep kernel machine optimization framework, that creates an ensemble of dense embeddings using Nystrom kernel approximations and utilizes deep learning to generate task-specific representations through the fusion of the embeddings. Intuitively, the filters of the network are trained to fuse information from an ensemble of linear subspaces in the RKHS. Furthermore, we introduce the kernel dropout regularization to enable improved training convergence. Finally, we extend this framework to the multiple kernel case, by coupling a global fusion layer with pretrained deep kernel machines for each of the constituent kernels. Using case studies with limited training data, and lack of explicit feature sources, we demonstrate the effectiveness of our framework over conventional model inferencing techniques.
  • References (64)
  • Citations (10)
References64
Newest
Mar 1, 2017 in ICASSP (International Conference on Acoustics, Speech, and Signal Processing)
#1Huan Song (ASU: Arizona State University)H-Index: 4
#2Jayaraman J. Thiagarajan (LLNL: Lawrence Livermore National Laboratory)H-Index: 14
Last.Andreas Spanias (ASU: Arizona State University)H-Index: 28
view all 5 authors...
8 CitationsSource
#1Sai Zhang (ASU: Arizona State University)H-Index: 5
#2Cihan Tepedelenlioglu (ASU: Arizona State University)H-Index: 26
Last.Andreas Spanias (ASU: Arizona State University)H-Index: 28
view all 4 authors...
19 CitationsSource
Jun 30, 2016 in SDM (SIAM International Conference on Data Mining)
#1John Moeller (UofU: University of Utah)H-Index: 11
#2Sarathkrishna Swaminathan (UofU: University of Utah)H-Index: 2
Last.Suresh Venkatasubramanian (UofU: University of Utah)H-Index: 35
view all 3 authors...
2 CitationsSource
Jun 1, 2016 in CVPR (Computer Vision and Pattern Recognition)
#1Kaiming He (Microsoft)H-Index: 57
#2Xiangyu Zhang (Xi'an Jiaotong University)H-Index: 25
Last.Jian Sun (Microsoft)H-Index: 88
view all 4 authors...
14k CitationsSource
Mar 1, 2016 in ICASSP (International Conference on Acoustics, Speech, and Signal Processing)
#1Huan Songg (ASU: Arizona State University)H-Index: 1
#2Jayaraman J. Thiagarajan (LLNL: Lawrence Livermore National Laboratory)H-Index: 14
Last.Pavan Turaga (ASU: Arizona State University)H-Index: 24
view all 5 authors...
5 CitationsSource
Jan 4, 2016 in MMM (Conference on Multimedia Modeling)
#1Raghvendra Kannao (IITG: Indian Institute of Technology Guwahati)H-Index: 5
#2Prithwijit Guha (IITG: Indian Institute of Technology Guwahati)H-Index: 8
8 CitationsSource
Jan 1, 2016 in AISTATS (International Conference on Artificial Intelligence and Statistics)
#1Andrew Gordon Wilson (University Hospitals Coventry and Warwickshire NHS Trust)H-Index: 23
#2Zhiting Hu (CMU: Carnegie Mellon University)H-Index: 24
Last.Eric P. Xing (CMU: Carnegie Mellon University)H-Index: 64
view all 4 authors...
75 Citations
#1Mu Li (CMU: Carnegie Mellon University)H-Index: 16
#2Wei Bi (HKUST: Hong Kong University of Science and Technology)H-Index: 9
Last.Bao-Liang Lu (SJTU: Shanghai Jiao Tong University)H-Index: 36
view all 4 authors...
37 CitationsSource
#1Xianbiao Qi (Beijing University of Posts and Telecommunications)H-Index: 9
#2Rong Xiao (Microsoft)H-Index: 15
Last.Xiaoou Tang (CAS: Chinese Academy of Sciences)H-Index: 99
view all 6 authors...
183 CitationsSource
Sep 6, 2014 in ECCV (European Conference on Computer Vision)
#1Mehrtash Harandi (NICTA)H-Index: 27
#2Mathieu Salzmann (NICTA)H-Index: 35
Last.Hongdong Li (NICTA)H-Index: 33
view all 5 authors...
44 CitationsSource
Cited By10
Newest
#1Euijoon Ahn (USYD: University of Sydney)H-Index: 8
#2Ashnil Kumar (USYD: University of Sydney)H-Index: 13
Last.Jinman Kim (USYD: University of Sydney)H-Index: 19
view all 5 authors...
1 CitationsSource
Jul 1, 2019 in IJCNN (International Joint Conference on Neural Network)
#1Luc Giffon (AMU: Aix-Marseille University)
#2Stéphane Ayache (AMU: Aix-Marseille University)H-Index: 13
Last.Hachem Kadri (AMU: Aix-Marseille University)H-Index: 9
view all 4 authors...
Source
#1Sunil Rao (ASU: Arizona State University)H-Index: 4
#2Andreas Spanias (ASU: Arizona State University)H-Index: 28
Last.Cihan Tepedelenlioglu (ASU: Arizona State University)H-Index: 26
view all 3 authors...
1 CitationsSource
May 1, 2019 in ICASSP (International Conference on Acoustics, Speech, and Signal Processing)
#1Jie Fan (ASU: Arizona State University)H-Index: 1
#2Cihan Tepedelenlioglu (ASU: Arizona State University)H-Index: 26
Last.Andreas Spanias (ASU: Arizona State University)H-Index: 28
view all 3 authors...
Source
#1Jiewan Zheng (Beihang University)H-Index: 1
#2Xianbin Cao (Beihang University)H-Index: 13
Last.Xiangbo Su (Beihang University)H-Index: 2
view all 5 authors...
3 CitationsSource
#1Euijoon AhnH-Index: 8
#2Ashnil KumarH-Index: 13
Last.Jinman Kim (USYD: University of Sydney)H-Index: 19
view all 5 authors...
Oct 1, 2018 in ICIP (International Conference on Image Processing)
#1Sameeksha Katoch (ASU: Arizona State University)H-Index: 3
#2Pavan Turaga (ASU: Arizona State University)H-Index: 24
Last.Cihan Tepedelenlioglu (ASU: Arizona State University)H-Index: 26
view all 4 authors...
2 CitationsSource
#1Gang Zhang (GDUT: Guangdong University of Technology)H-Index: 1
#2Ling Zhong (GDUT: Guangdong University of Technology)H-Index: 1
Last.Yonghui Huang (GDUT: Guangdong University of Technology)H-Index: 1
view all 4 authors...
Source
#1Farib Khondoker (ASU: Arizona State University)H-Index: 1
#2Trevor Thornton (ASU: Arizona State University)H-Index: 23
Last.Uday Shankar Shanthamallu (ASU: Arizona State University)H-Index: 2
view all 4 authors...
Source