Download or read book Efficient Deep Neural Networks Architectures for Video Analytics Systems written by Zeinab Hakimi and published by . This book was released on 2023 with total page 0 pages. Available in PDF, EPUB and Kindle. Book excerpt: In recent years, there has been a remarkable surge in the volume of digital data across various formats and domains. For instance, modern camera systems leverage new technologies and the fusion of information from multiple views to capture high-quality images. As a result of this data explosion, there is a growing interest and demand for analyzing information using data-intensive machine learning algorithms, particularly deep neural networks (DNNs). However, despite the success of deep learning approaches in various domains, their performance on small edge devices with constrained computing power and memory are limited. The primary objective of this thesis is to design efficient intelligent vision systems that effectively overcome the limitations of deep neural networks (DNNs) when deployed on edge devices with limited resources. This work explores a variety of methods aimed at optimizing the utilization of information and context in the design of DNN architectures. By leveraging these techniques, the proposed systems aim to enhance the performance and efficiency of DNNs in resource-constrained environments. Specifically, the thesis proposes context-aware methods to differentiate between low and high quality sensors representations by incorporating the context into the CNN models and reduce the computation and communication costs of edge devices in a distributed camera system. The primary objective is to minimize the computation and communication costs associated with edge devices in a distributed camera system. In addition, the thesis proposes a fault-tolerant mechanism to address the challenges posed by abnormal and noisy data in the system, particularly due to unknown conditions. This mechanism serves as a solution to mitigate the adverse effects of such data, ensuring the reliability and robustness of the proposed system. Furthermore, a resolution-aware multi-view design is outlined to address data transmission and power challenges in embedded devices. Moreover, the thesis introduces a patch-based attention-likelihood technique, designed to enhance the recognition performance of small objects within high-resolution images. This technique effectively reduces the computational burden of handling high-resolution images on edge devices by processing sub-samples of the input patches. By selectively attending to relevant patches, the proposed approach significantly improves the overall efficiency of object recognition while maintaining a high level of accuracy. Finally, the thesis introduces an efficient task-adaptive visual transformer model specifically designed for fine-grained classification tasks on IoT devices. By optimizing the system's performance for IoT devices, it enables efficient and reliable fine-grained classification without compromising computational resources or compromising the accuracy of results. Overall, this thesis offers a comprehensive approach to overcoming the limitations associated with deploying deep neural networks (DNNs) on edge devices within visual intelligent systems.

Download or read book Efficient Processing of Deep Neural Networks written by Vivienne Sze and published by Springer Nature. This book was released on 2022-05-31 with total page 254 pages. Available in PDF, EPUB and Kindle. Book excerpt: This book provides a structured treatment of the key principles and techniques for enabling efficient processing of deep neural networks (DNNs). DNNs are currently widely used for many artificial intelligence (AI) applications, including computer vision, speech recognition, and robotics. While DNNs deliver state-of-the-art accuracy on many AI tasks, it comes at the cost of high computational complexity. Therefore, techniques that enable efficient processing of deep neural networks to improve key metrics—such as energy-efficiency, throughput, and latency—without sacrificing accuracy or increasing hardware costs are critical to enabling the wide deployment of DNNs in AI systems. The book includes background on DNN processing; a description and taxonomy of hardware architectural approaches for designing DNN accelerators; key metrics for evaluating and comparing different designs; features of DNN processing that are amenable to hardware/algorithm co-design to improve energy efficiency and throughput; and opportunities for applying new technologies. Readers will find a structured introduction to the field as well as formalization and organization of key concepts from contemporary work that provide insights that may spark new ideas.

Download or read book Efficient Processing of Deep Neural Networks written by Vivienne Sze and published by . This book was released on 2020-06-24 with total page 342 pages. Available in PDF, EPUB and Kindle. Book excerpt: This book provides a structured treatment of the key principles and techniques for enabling efficient processing of deep neural networks (DNNs). DNNs are currently widely used for many artificial intelligence (AI) applications, including computer vision, speech recognition, and robotics. While DNNs deliver state-of-the-art accuracy on many AI tasks, it comes at the cost of high computational complexity. Therefore, techniques that enable efficient processing of deep neural networks to improve key metrics-such as energy-efficiency, throughput, and latency-without sacrificing accuracy or increasing hardware costs are critical to enabling the wide deployment of DNNs in AI systems. The book includes background on DNN processing; a description and taxonomy of hardware architectural approaches for designing DNN accelerators; key metrics for evaluating and comparing different designs; features of DNN processing that are amenable to hardware/algorithm co-design to improve energy efficiency and throughput; and opportunities for applying new technologies. Readers will find a structured introduction to the field as well as formalization and organization of key concepts from contemporary work that provide insights that may spark new ideas.

Download or read book Explainable Machine Learning Models and Architectures written by Suman Lata Tripathi and published by John Wiley & Sons. This book was released on 2023-10-03 with total page 277 pages. Available in PDF, EPUB and Kindle. Book excerpt: EXPLAINABLE MACHINE LEARNING MODELS AND ARCHITECTURES This cutting-edge new volume covers the hardware architecture implementation, the software implementation approach, and the efficient hardware of machine learning applications. Machine learning and deep learning modules are now an integral part of many smart and automated systems where signal processing is performed at different levels. Signal processing in the form of text, images, or video needs large data computational operations at the desired data rate and accuracy. Large data requires more use of integrated circuit (IC) area with embedded bulk memories that further lead to more IC area. Trade-offs between power consumption, delay and IC area are always a concern of designers and researchers. New hardware architectures and accelerators are needed to explore and experiment with efficient machine-learning models. Many real-time applications like the processing of biomedical data in healthcare, smart transportation, satellite image analysis, and IoT-enabled systems have a lot of scope for improvements in terms of accuracy, speed, computational powers, and overall power consumption. This book deals with the efficient machine and deep learning models that support high-speed processors with reconfigurable architectures like graphic processing units (GPUs) and field programmable gate arrays (FPGAs), or any hybrid system. Whether for the veteran engineer or scientist working in the field or laboratory, or the student or academic, this is a must-have for any library.

Download or read book Deep Learning and Parallel Computing Environment for Bioengineering Systems written by Arun Kumar Sangaiah and published by Academic Press. This book was released on 2019-07-26 with total page 280 pages. Available in PDF, EPUB and Kindle. Book excerpt: Deep Learning and Parallel Computing Environment for Bioengineering Systems delivers a significant forum for the technical advancement of deep learning in parallel computing environment across bio-engineering diversified domains and its applications. Pursuing an interdisciplinary approach, it focuses on methods used to identify and acquire valid, potentially useful knowledge sources. Managing the gathered knowledge and applying it to multiple domains including health care, social networks, mining, recommendation systems, image processing, pattern recognition and predictions using deep learning paradigms is the major strength of this book. This book integrates the core ideas of deep learning and its applications in bio engineering application domains, to be accessible to all scholars and academicians. The proposed techniques and concepts in this book can be extended in future to accommodate changing business organizations’ needs as well as practitioners’ innovative ideas. Presents novel, in-depth research contributions from a methodological/application perspective in understanding the fusion of deep machine learning paradigms and their capabilities in solving a diverse range of problems Illustrates the state-of-the-art and recent developments in the new theories and applications of deep learning approaches applied to parallel computing environment in bioengineering systems Provides concepts and technologies that are successfully used in the implementation of today's intelligent data-centric critical systems and multi-media Cloud-Big data

Download or read book Practical Convolutional Neural Networks written by Mohit Sewak and published by Packt Publishing Ltd. This book was released on 2018-02-27 with total page 211 pages. Available in PDF, EPUB and Kindle. Book excerpt: One stop guide to implementing award-winning, and cutting-edge CNN architectures Key Features Fast-paced guide with use cases and real-world examples to get well versed with CNN techniques Implement CNN models on image classification, transfer learning, Object Detection, Instance Segmentation, GANs and more Implement powerful use-cases like image captioning, reinforcement learning for hard attention, and recurrent attention models Book Description Convolutional Neural Network (CNN) is revolutionizing several application domains such as visual recognition systems, self-driving cars, medical discoveries, innovative eCommerce and more.You will learn to create innovative solutions around image and video analytics to solve complex machine learning and computer vision related problems and implement real-life CNN models. This book starts with an overview of deep neural networkswith the example of image classification and walks you through building your first CNN for human face detector. We will learn to use concepts like transfer learning with CNN, and Auto-Encoders to build very powerful models, even when not much of supervised training data of labeled images is available. Later we build upon the learning achieved to build advanced vision related algorithms for object detection, instance segmentation, generative adversarial networks, image captioning, attention mechanisms for vision, and recurrent models for vision. By the end of this book, you should be ready to implement advanced, effective and efficient CNN models at your professional project or personal initiatives by working on complex image and video datasets. What you will learn From CNN basic building blocks to advanced concepts understand practical areas they can be applied to Build an image classifier CNN model to understand how different components interact with each other, and then learn how to optimize it Learn different algorithms that can be applied to Object Detection, and Instance Segmentation Learn advanced concepts like attention mechanisms for CNN to improve prediction accuracy Understand transfer learning and implement award-winning CNN architectures like AlexNet, VGG, GoogLeNet, ResNet and more Understand the working of generative adversarial networks and how it can create new, unseen images Who this book is for This book is for data scientists, machine learning and deep learning practitioners, Cognitive and Artificial Intelligence enthusiasts who want to move one step further in building Convolutional Neural Networks. Get hands-on experience with extreme datasets and different CNN architectures to build efficient and smart ConvNet models. Basic knowledge of deep learning concepts and Python programming language is expected.

Download or read book NVIDIA TAO Toolkit and Deep Stream SDK A Developer s Guide written by Anand Vemula and published by Anand Vemula. This book was released on with total page 36 pages. Available in PDF, EPUB and Kindle. Book excerpt: This book equips you with the skills to build and deploy custom vision AI applications for real-time video analysis. Whether you're a developer, researcher, or enthusiast, you'll gain a comprehensive understanding of NVIDIA's powerful toolkit, from training models to real-world deployment. Part 1: Introduction to Vision AI and Deep Learning Lays the groundwork for computer vision and deep learning concepts. Explains how these technologies are used in real-world applications. Introduces NVIDIA TAO and DeepStream, your one-stop shop for vision AI development. Part 2: NVIDIA TAO Toolkit - Your Vision AI Training Companion Guides you through setting up and navigating the user-friendly TAO interface. Explains how to prepare your data for efficient model training. Covers techniques for leveraging pre-trained models and adding new classes. Dives into model training optimization and explores methods for reducing model size for deployment. Teaches you how to export your trained models for seamless integration with DeepStream. Part 3: NVIDIA DeepStream SDK - Unleashing Your Vision AI in Real-Time Unveils the core functionalities and architecture of DeepStream for real-time video analytics. Explains how DeepStream leverages GStreamer, a powerful framework, for efficient data processing. Provides step-by-step guidance on building real-time video analytics pipelines using DeepStream. Explores various DeepStream plugins for common tasks like decoding, inference, and displaying results. Demonstrates how to integrate your TAO models into DeepStream pipelines for real-world applications. Part 4: Deployment and Optimization - Taking Your DeepStream Applications to the Real World Explores different deployment options for your DeepStream applications, from edge devices to cloud servers. Provides optimization techniques to ensure your applications run smoothly and efficiently. Covers methods for improving inference speed and resource utilization. Explains how to profile and debug your DeepStream pipelines for optimal performance. By combining the power of TAO for model training with DeepStream for real-time deployment, you'll be equipped to build cutting-edge vision AI applications that analyze and understand the visual world around you. Get started today and unlock the potential of real-time video analytics!

Download or read book Deep Learning for Multimedia Processing Applications written by Uzair Aslam Bhatti and published by CRC Press. This book was released on 2024-02-21 with total page 481 pages. Available in PDF, EPUB and Kindle. Book excerpt: Deep Learning for Multimedia Processing Applications is a comprehensive guide that explores the revolutionary impact of deep learning techniques in the field of multimedia processing. Written for a wide range of readers, from students to professionals, this book offers a concise and accessible overview of the application of deep learning in various multimedia domains, including image processing, video analysis, audio recognition, and natural language processing. Divided into two volumes, Volume Two delves into advanced topics such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), and generative adversarial networks (GANs), explaining their unique capabilities in multimedia tasks. Readers will discover how deep learning techniques enable accurate and efficient image recognition, object detection, semantic segmentation, and image synthesis. The book also covers video analysis techniques, including action recognition, video captioning, and video generation, highlighting the role of deep learning in extracting meaningful information from videos. Furthermore, the book explores audio processing tasks such as speech recognition, music classification, and sound event detection using deep learning models. It demonstrates how deep learning algorithms can effectively process audio data, opening up new possibilities in multimedia applications. Lastly, the book explores the integration of deep learning with natural language processing techniques, enabling systems to understand, generate, and interpret textual information in multimedia contexts. Throughout the book, practical examples, code snippets, and real-world case studies are provided to help readers gain hands-on experience in implementing deep learning solutions for multimedia processing. Deep Learning for Multimedia Processing Applications is an essential resource for anyone interested in harnessing the power of deep learning to unlock the vast potential of multimedia data.

Download or read book Deep Learning in Computer Vision written by Mahmoud Hassaballah and published by CRC Press. This book was released on 2020-03-23 with total page 261 pages. Available in PDF, EPUB and Kindle. Book excerpt: Deep learning algorithms have brought a revolution to the computer vision community by introducing non-traditional and efficient solutions to several image-related problems that had long remained unsolved or partially addressed. This book presents a collection of eleven chapters where each individual chapter explains the deep learning principles of a specific topic, introduces reviews of up-to-date techniques, and presents research findings to the computer vision community. The book covers a broad scope of topics in deep learning concepts and applications such as accelerating the convolutional neural network inference on field-programmable gate arrays, fire detection in surveillance applications, face recognition, action and activity recognition, semantic segmentation for autonomous driving, aerial imagery registration, robot vision, tumor detection, and skin lesion segmentation as well as skin melanoma classification. The content of this book has been organized such that each chapter can be read independently from the others. The book is a valuable companion for researchers, for postgraduate and possibly senior undergraduate students who are taking an advanced course in related topics, and for those who are interested in deep learning with applications in computer vision, image processing, and pattern recognition.

Download or read book Architecture Design for Highly Flexible and Energy efficient Deep Neural Network Accelerators written by Yu-Hsin Chen (Ph. D.) and published by . This book was released on 2018 with total page 147 pages. Available in PDF, EPUB and Kindle. Book excerpt: Deep neural networks (DNNs) are the backbone of modern artificial intelligence (AI). However, due to their high computational complexity and diverse shapes and sizes, dedicated accelerators that can achieve high performance and energy efficiency across a wide range of DNNs are critical for enabling AI in real-world applications. To address this, we present Eyeriss, a co-design of software and hardware architecture for DNN processing that is optimized for performance, energy efficiency and flexibility. Eyeriss features a novel Row-Stationary (RS) dataflow to minimize data movement when processing a DNN, which is the bottleneck of both performance and energy efficiency. The RS dataflow supports highly-parallel processing while fully exploiting data reuse in a multi-level memory hierarchy to optimize for the overall system energy efficiency given any DNN shape and size. It achieves 1.4x to 2.5x higher energy efficiency than other existing dataflows. To support the RS dataflow, we present two versions of the Eyeriss architecture. Eyeriss v1 targets large DNNs that have plenty of data reuse. It features a flexible mapping strategy for high performance and a multicast on-chip network (NoC) for high data reuse, and further exploits data sparsity to reduce processing element (PE) power by 45% and off-chip bandwidth by up to 1.9x. Fabricated in a 65nm CMOS, Eyeriss v1 consumes 278 mW at 34.7 fps for the CONV layers of AlexNet, which is 10× more efficient than a mobile GPU. Eyeriss v2 addresses support for the emerging compact DNNs that introduce higher variation in data reuse. It features a RS+ dataflow that improves PE utilization, and a flexible and scalable NoC that adapts to the bandwidth requirement while also exploiting available data reuse. Together, they provide over 10× higher throughput than Eyeriss v1 at 256 PEs. Eyeriss v2 also exploits sparsity and SIMD for an additional 6× increase in throughput.

Download or read book Video Data Analytics for Smart City Applications Methods and Trends written by Abhishek Singh Rathore and published by Bentham Science Publishers. This book was released on 2023-04-20 with total page 150 pages. Available in PDF, EPUB and Kindle. Book excerpt: Video data analytics is rapidly evolving and transforming the way we live in urban environments. Video Data Analytics for Smart City Applications: Methods and Trends, data science experts present a comprehensive review of the latest advances and trends in video analytics technologies and their extensive applications in smart city planning and engineering. The book covers a wide range of topics including object recognition, action recognition, violence detection, and tracking, exploring deep learning approaches and other techniques for video data analytics. It also discusses the key enabling technologies for smart cities and homes and the scope and application of smart agriculture in smart cities. Moreover, the book addresses the challenges and security issues in terahertz band for wireless communication and the empirical impact of AI and IoT on performance management. One contribution also provides a review of the progress in achieving the Jal Jeevan Mission Goals for institutional capacity building in the Indian State of Chhattisgarh. For researchers, computer scientists, data analytics professionals, smart city planners and engineers, this book provides detailed references for further reading and demonstrates how technologies are serving their use-cases in the smart city. The book highlights the advances and trends in video analytics technologies and extensively addresses key themes, making it an essential resource for anyone looking to gain a comprehensive understanding of video data analytics for smart city applications.

Download or read book Deep Learning for Robot Perception and Cognition written by Alexandros Iosifidis and published by Academic Press. This book was released on 2022-02-04 with total page 638 pages. Available in PDF, EPUB and Kindle. Book excerpt: Deep Learning for Robot Perception and Cognition introduces a broad range of topics and methods in deep learning for robot perception and cognition together with end-to-end methodologies. The book provides the conceptual and mathematical background needed for approaching a large number of robot perception and cognition tasks from an end-to-end learning point-of-view. The book is suitable for students, university and industry researchers and practitioners in Robotic Vision, Intelligent Control, Mechatronics, Deep Learning, Robotic Perception and Cognition tasks. Presents deep learning principles and methodologies Explains the principles of applying end-to-end learning in robotics applications Presents how to design and train deep learning models Shows how to apply deep learning in robot vision tasks such as object recognition, image classification, video analysis, and more Uses robotic simulation environments for training deep learning models Applies deep learning methods for different tasks ranging from planning and navigation to biosignal analysis

Download or read book Video Analytics Face and Facial Expression Recognition written by Xiang Bai and published by Springer. This book was released on 2019-01-18 with total page 87 pages. Available in PDF, EPUB and Kindle. Book excerpt: This book constitutes the proceedings of the Third Workshop on Face and Facial Expression Recognition from Real World Videos, FFER 2018, and the Second International Workshop on Deep Learning for Pattern Recognition, DLPR 2018, held at the 24th International Conference on Pattern Recognition, ICPR 2018, in Beijing, China, in August 2018. The 7 papers presented in this volume were carefully reviewed and selected from 9 submissions. They deal with topics such as histopathological images, action recognition, scene text detection, speech recognition, object classification, presentation attack detection, and driver drowsiness detection.

Download or read book Fundamentals of Deep Learning and Computer Vision written by Singh Nikhil and published by BPB Publications. This book was released on 2020-02-24 with total page 227 pages. Available in PDF, EPUB and Kindle. Book excerpt: Master Computer Vision concepts using Deep Learning with easy-to-follow steps Key Featuresa- Setting up the Python and TensorFlow environmenta- Learn core Tensorflow concepts with the latest TF version 2.0a- Learn Deep Learning for computer vision applications a- Understand different computer vision concepts and use-casesa- Understand different state-of-the-art CNN architectures a- Build deep neural networks with transfer Learning using features from pre-trained CNN modelsa- Apply computer vision concepts with easy-to-follow code in Jupyter NotebookDescriptionThis book starts with setting up a Python virtual environment with the deep learning framework TensorFlow and then introduces the fundamental concepts of TensorFlow. Before moving on to Computer Vision, you will learn about neural networks and related aspects such as loss functions, gradient descent optimization, activation functions and how backpropagation works for training multi-layer perceptrons.To understand how the Convolutional Neural Network (CNN) is used for computer vision problems, you need to learn about the basic convolution operation. You will learn how CNN is different from a multi-layer perceptron along with a thorough discussion on the different building blocks of the CNN architecture such as kernel size, stride, padding, and pooling and finally learn how to build a small CNN model. Next, you will learn about different popular CNN architectures such as AlexNet, VGGNet, Inception, and ResNets along with different object detection algorithms such as RCNN, SSD, and YOLO. The book concludes with a chapter on sequential models where you will learn about RNN, GRU, and LSTMs and their architectures and understand their applications in machine translation, image/video captioning and video classification.What will you learnThis book will help the readers to understand and apply the latest Deep Learning technologies to different interesting computer vision applications without any prior domain knowledge of image processing. Thus, helping the users to acquire new skills specific to Computer Vision and Deep Learning and build solutions to real-life problems such as Image Classification and Object Detection. Who this book is forThis book is for all the Data Science enthusiasts and practitioners who intend to learn and master Computer Vision concepts and their applications using Deep Learning. This book assumes a basic Python understanding with hands-on experience. A basic senior secondary level understanding of Mathematics will help the reader to make the best out of this book. Table of Contents1. Introduction to TensorFlow2. Introduction to Neural Networks 3. Convolutional Neural Network 4. CNN Architectures5. Sequential ModelsAbout the AuthorNikhil Singh is an accomplished data scientist and currently working as the Lead Data Scientist at Proarch IT Solutions Pvt. Ltd in London. He has experience in designing and delivering complex and innovative computer vision and NLP centred solutions for a large number of global companies. He has been an AI consultant to a few companies and mentored many apprentice Data Scientists. His LinkedIn Profile: https://www.linkedin.com/in/nikhil-singh-b953ba122/Paras Ahuja is a seasoned data science practitioner and currently working as the Lead Data Scientist at Reliance Jio in Hyderabad. He has good experience in designing and deploying deep learning-based Computer Vision and NLP-based solutions. He has experience in developing and implementing state-of-the-art automatic speech recognition systems.His LinkedIn Profile: https://www.linkedin.com/in/parasahuja

Download or read book Intelligent Image and Video Analytics written by El-Sayed M. El-Alfy and published by CRC Press. This book was released on 2023-04-12 with total page 361 pages. Available in PDF, EPUB and Kindle. Book excerpt: Provides up-to-date coverage of the state-of-the-art techniques in intelligent video analytics Explores important applications that require techniques from both artificial intelligence and computer vision Describes multimodality video analytics for different applications Examines issues related to multimodality data fusion and highlights research challenges Integrates various techniques from video processing, data mining and machine learning which has many emerging indoor and outdoor applications of smart cameras in smart environments, smart homes, and smart cities

Download or read book Deep Learning Convergence to Big Data Analytics written by Murad Khan and published by Springer. This book was released on 2018-12-30 with total page 79 pages. Available in PDF, EPUB and Kindle. Book excerpt: This book presents deep learning techniques, concepts, and algorithms to classify and analyze big data. Further, it offers an introductory level understanding of the new programming languages and tools used to analyze big data in real-time, such as Hadoop, SPARK, and GRAPHX. Big data analytics using traditional techniques face various challenges, such as fast, accurate and efficient processing of big data in real-time. In addition, the Internet of Things is progressively increasing in various fields, like smart cities, smart homes, and e-health. As the enormous number of connected devices generate huge amounts of data every day, we need sophisticated algorithms to deal, organize, and classify this data in less processing time and space. Similarly, existing techniques and algorithms for deep learning in big data field have several advantages thanks to the two main branches of the deep learning, i.e. convolution and deep belief networks. This book offers insights into these techniques and applications based on these two types of deep learning. Further, it helps students, researchers, and newcomers understand big data analytics based on deep learning approaches. It also discusses various machine learning techniques in concatenation with the deep learning paradigm to support high-end data processing, data classifications, and real-time data processing issues. The classification and presentation are kept quite simple to help the readers and students grasp the basics concepts of various deep learning paradigms and frameworks. It mainly focuses on theory rather than the mathematical background of the deep learning concepts. The book consists of 5 chapters, beginning with an introductory explanation of big data and deep learning techniques, followed by integration of big data and deep learning techniques and lastly the future directions.

Download or read book From Line Drawings to Human Actions written by Fang Wang and published by . This book was released on 2017 with total page 0 pages. Available in PDF, EPUB and Kindle. Book excerpt: In recent years, deep neural networks have been very successful in computer vision, speech recognition, and artificial intelligent systems. The rapid growth of data and fast increasing computational tools provide solid foundations for the applications which rely on the learning of large scale deep neural networks with millions of parameters. The deep learning approaches have been proved to be able to learn powerful representations of the inputs in various tasks, such as image classification, object recognition, and scene understanding. This thesis demonstrates the generality and capacity of deep learning approaches through a series of case studies including image matching and human activity understanding. In these studies, I explore the combinations of the neural network models with existing machine learning techniques and extend the deep learning approach for each task. Four related tasks are investigated: 1) image matching through similarity learning; 2) human action prediction; 3) finger force estimation in manipulation actions; and 4) bimodal learning for human action understanding. Deep neural networks have been shown to be very efficient in supervised learning. Further, in some tasks, one would like to group the features of the samples in the same category close to each other, in additional to the discriminative representation. Such kind of properties is desired in a number of applications, such as semantic retrieval, image quality measurement, and social network analysis, etc. My first study is to develop a similarity learning method based on deep neural networks for image matching between sketch images and 3D models. In this task, I propose to use Siamese network to learn similarities of sketches and develop a novel method for sketch based 3D shape retrieval. The proposed method can successfully learn the representations of sketch images as well as the similarities, then the 3D shape retrieval problem can be solved with off-the-shelf nearest neighbor methods. After studying the representation learning methods for static inputs, my focus turns to learning the representations of sequential data. To be specific, I focus on manipulation actions, because they are widely used in the daily life and play important parts in the human-robot collaboration system. Deep neural networks have been shown to be powerful to represent short video clips [Donahue et al., 2015]. However, most existing methods consider the action recognition problem as a classification task. These methods assume the inputs are pre-segmented videos and the outputs are category labels. In the scenarios such as the human-robot collaboration system, the ability to predict the ongoing human actions at an early stage is highly important. I first attempt to address this issue with a fast manipulation action prediction method. Then I build the action prediction model based on Long Short-Term Memory (LSTM) architecture. The proposed approach processes the sequential inputs as continuous signals and keeps updating the prediction of the intended action based on the learned action representations. Further, I study the relationships between visual inputs and the physical information, such as finger forces, that involved in the manipulation actions. This is motivated by recent studies in cognitive science which show that the subject's intention is strongly related to the hand movements during an action execution. Human observers can interpret other's actions in terms of movements and forces, which can be used to repeat the observed actions. If a robot system has the ability to estimate the force feedbacks, it can learn how to manipulate an object by watching human demonstrations. In this work, the finger forces are estimated by only watching the movement of hands. A modified LSTM model is used to regress the finger forces from video frames. To facilitate this study, a specially designed sensor glove has been used to collect data of finger forces, and a new dataset has been collected to provide synchronized streams of videos and finger forces. Last, I investigate the usefulness of physical information in human action recognition, which is an application of bimodal learning, where both the vision inputs and the additional information are used to learn the action representation. My study demonstrates that, by combining additional information with the vision inputs, the accuracy of human action recognition can be improved steadily. I extend the LSTM architecture to accept both video frames and sensor data as bimodal inputs to predict the action. A hallucination network is jointly trained to approximate the representations of the additional inputs. During the testing stage, the hallucination network generates approximated representations that used for classification. In this way, the proposed method does not rely on the additional inputs for testing.