Download or read book Spatio temporal Reasoning for Semantic Scene Understanding and Its Application in Recognition and Prediction of Manipulation Actions in Image Sequences written by Fatemeh Ziaeetabar and published by . This book was released on 2020 with total page 0 pages. Available in PDF, EPUB and Kindle. Book excerpt: Human activity understanding has attracted much attention in recent years, due to a key role in a wide range of applications and devices, such as human- computer interfaces, visual surveillance, video indexing, intelligent humanoid robots, ambient intelligence and more. Of particular relevance, performing manipulation actions has a significant importance due to its enormous use, especially for service, as well as industrial robots. These robots strongly benefit from a fast and predictive recognition of manipulation actions. Although, for us as humans performing these actions is a quite triv...

Download or read book Spatio Temporal Stream Reasoning with Adaptive State Stream Generation written by Daniel de Leng and published by Linköping University Electronic Press. This book was released on 2017-09-08 with total page 153 pages. Available in PDF, EPUB and Kindle. Book excerpt: A lot of today's data is generated incrementally over time by a large variety of producers. This data ranges from quantitative sensor observations produced by robot systems to complex unstructured human-generated texts on social media. With data being so abundant, making sense of these streams of data through reasoning is challenging. Reasoning over streams is particularly relevant for autonomous robotic systems that operate in a physical environment. They commonly observe this environment through incremental observations, gradually refining information about their surroundings. This makes robust management of streaming data and its refinement an important problem. Many contemporary approaches to stream reasoning focus on the issue of querying data streams in order to generate higher-level information by relying on well-known database approaches. Other approaches apply logic-based reasoning techniques, which rarely consider the provenance of their symbolic interpretations. In this thesis, we integrate techniques for logic-based spatio-temporal stream reasoning with the adaptive generation of the state streams needed to do the reasoning over. This combination deals with both the challenge of reasoning over streaming data and the problem of robustly managing streaming data and its refinement. The main contributions of this thesis are (1) a logic-based spatio-temporal reasoning technique that combines temporal reasoning with qualitative spatial reasoning; (2) an adaptive reconfiguration procedure for generating and maintaining a data stream required to perform spatio-temporal stream reasoning over; and (3) integration of these two techniques into a stream reasoning framework. The proposed spatio-temporal stream reasoning technique is able to reason with intertemporal spatial relations by leveraging landmarks. Adaptive state stream generation allows the framework to adapt in situations in which the set of available streaming resources changes. Management of streaming resources is formalised in the DyKnow model, which introduces a configuration life-cycle to adaptively generate state streams. The DyKnow-ROS stream reasoning framework is a concrete realisation of this model that extends the Robot Operating System (ROS). DyKnow-ROS has been deployed on the SoftBank Robotics NAO platform to demonstrate the system's capabilities in the context of a case study on run-time adaptive reconfiguration. The results show that the proposed system – by combining reasoning over and reasoning about streams – can robustly perform spatio-temporal stream reasoning, even when the availability of streaming resources changes.

Download or read book Spatio temporal Representation for Reasoning with Action Genome written by Kesar Murthy and published by . This book was released on 2021 with total page 38 pages. Available in PDF, EPUB and Kindle. Book excerpt: Representing Spatio-temporal information in videos has proven to be a difficult task compared to action recognition in videos involving multiple actions. A single activity consists many smaller actions that can provide better understanding of the activity. This paper tries to represent the varying information in a scene-graph format in order to answer temporal questions to obtain improved insights for the video, resulting in a directed temporal information graph. This project will use the Action Genome dataset, which is a variation of the charades dataset, to capture pairwise relationships in a graph. The model performs significantly better than the benchmark results of the dataset providing state-of-the-art results in predicate classification. The paper presents a novel Spatio-temporal scene graph for videos, represented as a directed acyclic graph that maximises the information in the scene. The results obtained in the counting task suggest some interesting finds that are described in the paper. The graph can be used for reasoning with a much lower computational requirement explored in this work among other downstream tasks such as video captioning, action recognition and more, trying to bridge the gap between videos and textual analysis.

Download or read book Scene Vision written by Kestutis Kveraga and published by MIT Press. This book was released on 2014-10-31 with total page 339 pages. Available in PDF, EPUB and Kindle. Book excerpt: Cutting-edge research on the visual cognition of scenes, covering issues that include spatial vision, context, emotion, attention, memory, and neural mechanisms underlying scene representation. For many years, researchers have studied visual recognition with objects—single, clean, clear, and isolated objects, presented to subjects at the center of the screen. In our real environment, however, objects do not appear so neatly. Our visual world is a stimulating scenery mess; fragments, colors, occlusions, motions, eye movements, context, and distraction all affect perception. In this volume, pioneering researchers address the visual cognition of scenes from neuroimaging, psychology, modeling, electrophysiology, and computer vision perspectives. Building on past research—and accepting the challenge of applying what we have learned from the study of object recognition to the visual cognition of scenes—these leading scholars consider issues of spatial vision, context, rapid perception, emotion, attention, memory, and the neural mechanisms underlying scene representation. Taken together, their contributions offer a snapshot of our current knowledge of how we understand scenes and the visual world around us. Contributors Elissa M. Aminoff, Moshe Bar, Margaret Bradley, Daniel I. Brooks, Marvin M. Chun, Ritendra Datta, Russell A. Epstein, Michèle Fabre-Thorpe, Elena Fedorovskaya, Jack L. Gallant, Helene Intraub, Dhiraj Joshi, Kestutis Kveraga, Peter J. Lang, Jia Li Xin Lu, Jiebo Luo, Quang-Tuan Luong, George L. Malcolm, Shahin Nasr, Soojin Park, Mary C. Potter, Reza Rajimehr, Dean Sabatinelli, Philippe G. Schyns, David L. Sheinberg, Heida Maria Sigurdardottir, Dustin Stansbury, Simon Thorpe, Roger Tootell, James Z. Wang

Download or read book Analysis of Human centric Activities in Video Via Qualitative Spatio temporal Reasoning written by Hajar Sadeghi Sokeh and published by . This book was released on 2015 with total page 0 pages. Available in PDF, EPUB and Kindle. Book excerpt: Applying qualitative spatio-temporal reasoning in video analysis is now a very active research topic in computer vision and artificial intelligence. Among all video analysis applications, monitoring and understanding human activities is of great interest. Many human activities can be understood by analysing the interaction between objects in space and time. Qualitative spatio-temporal reasoning encapsulates information that is useful for analysing huma-centric videos. This information can be represented in a very compact form involving interactions between objects of interest in the form of qualitative spatio-temporal relationships. This thesis focuses on three different aspects of interpreting human-centric videos; first introducing a representation of interactions between objects of interest, second determining which objects in the scene are relevant to the activity, and third recognising of human actions by applying the proposed representation model between human body joints and body parts. As a first contribution, we present an accurate and comprehensive model for representing several aspects of space over time from videos called "AngledCORE-9", a modified version of CORE-9 (proposed by Cohn et al. [2012]). This model is as efficient as CORE-9 and allows us to extract spatial information with much higher accuracy than previously possible. We evaluate our new knowledge representation method on a real video dataset to perform action clustering. Our next contribution is proposing a model for differentiating relevant from irrelevant objects to the human actions in the videos. The chief issue of recognising different human actions in videos using spatio-temporal features is that there are usually many moving objects in the scene. No existing method can successfully find the involved objects in the activity. The output of our system is a list of tracks for all possible objects in the video with their probabilities for being involved in the activity. The track with the highest probability is most likely to be the object with which the person is interacting. Knowing the involved object(s) in the activities is very advantageous. Since it can be used to improve the human action recognition rate. Finally, instead of looking at human-object interactions, we consider skeleton joints as the points of interest. Working on joints provides more information about how a person is moving to perform the activity. In this part of the thesis, we use videos with human skeletons in 3D captured by Kinect, MSR3D-action dataset. We use our proposed model "AngledCORE-9" to extract features and describe the temporal variation of these features frame by frame. We compare our results against some of the recent works on the same dataset.

Download or read book Deep Learning for Robot Perception and Cognition written by Alexandros Iosifidis and published by Academic Press. This book was released on 2022-02-04 with total page 638 pages. Available in PDF, EPUB and Kindle. Book excerpt: Deep Learning for Robot Perception and Cognition introduces a broad range of topics and methods in deep learning for robot perception and cognition together with end-to-end methodologies. The book provides the conceptual and mathematical background needed for approaching a large number of robot perception and cognition tasks from an end-to-end learning point-of-view. The book is suitable for students, university and industry researchers and practitioners in Robotic Vision, Intelligent Control, Mechatronics, Deep Learning, Robotic Perception and Cognition tasks. Presents deep learning principles and methodologies Explains the principles of applying end-to-end learning in robotics applications Presents how to design and train deep learning models Shows how to apply deep learning in robot vision tasks such as object recognition, image classification, video analysis, and more Uses robotic simulation environments for training deep learning models Applies deep learning methods for different tasks ranging from planning and navigation to biosignal analysis

Download or read book Elements of Scene Perception written by Monica S. Castelhano and published by Cambridge University Press. This book was released on 2021-11-11 with total page 156 pages. Available in PDF, EPUB and Kindle. Book excerpt: Visual cognitive processes have traditionally been examined with simplified stimuli, but generalization of these processes to the real-world is not always straightforward. Using images, computer-generated images, and virtual environments, researchers have examined processing of visual information in the real-world. Although referred to as scene perception, this research field encompasses many aspects of scene processing. Beyond the perception of visual features, scene processing is fundamentally influenced and constrained by semantic information as well as spatial layout and spatial associations with objects. In this review, we will present recent advances in how scene processing occurs within a few seconds of exposure, how scene information is retained in the long-term, and how different tasks affect attention in scene processing. By considering the characteristics of real-world scenes, as well as different time windows of processing, we can develop a fuller appreciation for the research that falls under the wider umbrella of scene processing.

Download or read book Statistical Semantic Analysis of Spatio temporal Image Sequences written by Ying Luo and published by . This book was released on 2004 with total page 108 pages. Available in PDF, EPUB and Kindle. Book excerpt:

Download or read book Topic Modeling for Discovering Spatio temporal Relationships in Motion Patterns written by Dalwinderjeet Kaur Kular and published by . This book was released on 2015 with total page 242 pages. Available in PDF, EPUB and Kindle. Book excerpt: Understanding spatio-ternporal relationships among motion patterns in a video is a key problem in computer vision with applications such as scene understanding and analysis, human-action classification, and facial expression recognition. The problem is challenging because of the noisy nature of low-level motion features and the complexities of collective dynamics of multiple activities. To perform higher-level reasoning concerning activities in a video, algorithms need to identify both spatial and temporal factors. In this work we propose to identify spatial and temporal relationships in motion patterns by probabilistic topic modeling and Granger causality. Two main approaches are presented. First, we combine probabilistic topic modeling with Granger causality. Second, we focus on inter-relationship among spatially co-occurring motion patterns using the relational topic model approach. Our experiments demonstrate that our methods discover relevant motion patterns by learning spatial patterns and their temporal relationships.

Download or read book Probabilistic Inductive Logic Programming written by Luc De Raedt and published by Springer. This book was released on 2008-02-26 with total page 348 pages. Available in PDF, EPUB and Kindle. Book excerpt: This book provides an introduction to probabilistic inductive logic programming. It places emphasis on the methods based on logic programming principles and covers formalisms and systems, implementations and applications, as well as theory.

Download or read book The Influence of Sequential Predictions on Scene Gist Recognition written by Maverick E. Smith and published by . This book was released on 2019 with total page pages. Available in PDF, EPUB and Kindle. Book excerpt: Past research has argued that scene gist, a holistic semantic representation of a scene acquired within a single fixation, is extracted using purely feed-forward mechanisms. Many scene gist recognition studies have presented scenes from multiple categories in randomized sequences. We tested whether rapid scene categorization could be facilitated by priming from sequential expectations. We created more ecologically valid, first-person viewpoint, image sequences, along spatiotemporally connected routes (e.g., an office to a parking lot). Participants identified target scenes at the end of rapid serial visual presentations. Critically, we manipulated whether targets were in coherent or randomized sequences. Target categorization was more accurate in coherent sequences than in randomized sequences. Furthermore, categorization was more accurate for a target following one or more images within the same category than following a switch between categories. Likewise, accuracy was higher for targets more visually similar to their immediately preceding primes. This suggested that prime-to-target visual similarity may explain the coherent sequence advantage. We tested this hypothesis in Experiment 2, which was identical except that target images were removed from the sequences, and participants were asked to predict the scene category of the missing target. Missing images in coherent sequences were more accurately predicted than missing images in randomized sequences, and more predictable images were identified more accurately in Experiment 1. Importantly, partial correlations revealed that image predictability and prime-to-target visual similarity independently contributed to rapid scene gist categorization accuracy suggesting sequential expectations prime and thus facilitate scene recognition processes.

Download or read book Semantic guided Visual Analysis and Synthesis with Spatio temporal Models written by and published by . This book was released on 2017 with total page 240 pages. Available in PDF, EPUB and Kindle. Book excerpt: Visual analysis is concerned with problems to identify object status or scene layout in images or videos. There are numerous concepts that are of great interest for visual analysis and understanding in the computer vision and machine learning communities. For instance, researchers have been working on developing algorithms to recognize, detect and segment objects/scenes in images. To understand such content, numerous challenges make these problems significantly challenging in the real world scenario, since objects or scenes usually appear in different conditions such as viewpoints, scales, and background noise, and even may deform with different shapes, parts or poses. In addition to images, video understanding has drawn much attention in various research areas due to the ease of obtaining video data and the importance of video applications, such as virtual reality, autonomous driving and video surveillance. Different from images, videos contain richer information in the temporal domain, thereby it also produces difficulties and requires larger computational powers to fully exploit video content. In this thesis, we propose to construct optimization frameworks for video object tracking and segmentation tasks. First, we utilize a spatial-temporal model to jointly optimize video object segmentation and optical flow estimation, and show that both results can be improved in the proposed framework. Second, we introduce a co-segmentation algorithm to further understand object semantics by considering relations between objects among a collection of videos. As a result, our proposed algorithms achieve state-of-the-art performance in video object segmentation. With such visual understanding in images and videos, the following question would be how to use them in real world applications. In this thesis, we focus on the visual synthesis problem, where it is a task for people to create or edit contents in the original data. For instance, numerous image editing problems have been studied widely, such as inpainting, harmonization and colorization. For these tasks, as the human can easily discover unrealistic artifacts after the original data is edited, one important challenge is to create realistic contents. To tackle this challenge, we propose to extract semantics by utilizing visual analysis as the guidance to improve the realism of synthesized outputs. With such guidance, we show that our visual synthesis systems produce visually pleasing and realistic results on sky replacement and object/scene composition tasks.

Download or read book Task oriented Visual Understanding for Scenes and Events written by Siyuan Qi and published by . This book was released on 2019 with total page 157 pages. Available in PDF, EPUB and Kindle. Book excerpt: Scene understanding and event understanding of humans correspond to the spatial and temporal aspects of computer vision. Such abilities serve as a foundation for humans to learn and perform tasks in the world we live in, thus motivating a task-oriented representation for machines to interpret observations of this world. Toward the goal of task-oriented scene understanding, I begin this thesis by presenting a human-centric scene synthesis algorithm. Realistic synthesis of indoor scenes is more complicated than neatly aligning objects; the scene needs to be functionally plausible, which requires the machine to understand the tasks that could be performed in the scene. Instead of directly modeling the object-object relationships, the algorithm learns the human-object relations and generate scene configurations by imagining the hidden human factors in the scene. I analyze the realisticity of the synthesized scenes, as well as its usefulness for various computer vision tasks. This framework is useful for backward inference of 3D scenes structures from images in an analysis-by-synthesis fashion; it is also useful for generating data to train various algorithms. Moving forward, I introduce a task-oriented event understanding framework for event parsing, event prediction, and task planning. In the computer vision literature, event understanding usually refers to action recognition from videos, i.e., "what is the action of the person". Task-oriented event understanding goes beyond this definition to find out the underlying driving forces of other agents. It answers questions such as intention recognition ("what is the person trying to achieve"), and intention prediction ("how the person is going to achieve the goal"), from a planning perspective. The core of this framework lies in the temporal representation for tasks that is appropriate for humans, robots, and the transfer between these two. In particular, inspired by natural language modeling, I represent the tasks by stochastic context-free grammars, which are natural choices to capture the semantics of tasks, but traditional grammar parsers (e.g., Earley parser) only take symbolic sentences as inputs. To overcome this drawback, I generalize the Earley parser to parse sequence data which is neither segmented nor labeled. This generalized Earley parser integrates a grammar parser with a classifier to find the optimal segmentation and labels. It can be used for event parsing, future predictions, as well as incorporating top-down task planning with bottom-up sensor inputs.

Download or read book Spatio temporal Modeling for Action Recognition in Videos written by Guoxi Huang and published by . This book was released on 2022 with total page 0 pages. Available in PDF, EPUB and Kindle. Book excerpt:

Download or read book Multimodal Scene Understanding written by Michael Yang and published by Academic Press. This book was released on 2019-07-17 with total page 0 pages. Available in PDF, EPUB and Kindle. Book excerpt: Multimodal Scene Understanding: Algorithms, Applications and Deep Learning presents recent advances in multi-modal computing, with a focus on computer vision and photogrammetry. It provides the latest algorithms and applications that involve combining multiple sources of information and describes the role and approaches of multi-sensory data and multi-modal deep learning. The book is ideal for researchers from the fields of computer vision, remote sensing, robotics, and photogrammetry, thus helping foster interdisciplinary interaction and collaboration between these realms. Researchers collecting and analyzing multi-sensory data collections - for example, KITTI benchmark (stereo+laser) - from different platforms, such as autonomous vehicles, surveillance cameras, UAVs, planes and satellites will find this book to be very useful.

Download or read book Exploring Temporal Information for Improved Video Understanding written by Yi Zhu and published by . This book was released on 2019 with total page 328 pages. Available in PDF, EPUB and Kindle. Book excerpt: In this dissertation, I present my work towards exploring temporal information for better video understanding. Specifically, I have worked on two problems: action recognition and semantic segmentation. For action recognition, I have proposed a framework, termed hidden two-stream networks, to learn an optimal motion representation that does not require the computation of optical flow. My framework alleviates several challenges faced in video classification, such as learning motion representations, real-time inference, multi-framerate handling, generalizability to unseen actions, etc. For semantic segmentation, I have introduced a general framework that uses video prediction models to synthesize new training samples. By scaling up the training dataset, my trained models are more accurate and robust than previous models even without modifications to the network architectures or objective functions. Along these lines of research, I have worked on several related problems. I performed the first investigation into depth for large-scale video action recognition where the depth cues are estimated from the videos themselves. I further improved my hidden two-stream networks for action recognition through several strategies, including a novel random temporal skipping data sampling method, an occlusion-aware motion estimation network and a global segment framework. For zero-shot action recognition, I proposed a pipeline using a large-scale training source to achieve a universal representation that can generalize to more realistic cross-dataset unseen action recognition scenarios. To learn better motion information in a video, I introduced several techniques to improve optical flow estimation, including guided learning, DenseNet upsampling and occlusion-aware estimation. I believe videos have much more potential to be mined, and temporal information is one of the most important cues for machines to perceive the visual world better.

Download or read book Spatio Temporal Image Analysis for Longitudinal and Time Series Image Data written by Stanley Durrleman and published by . This book was released on 2015-01-31 with total page 100 pages. Available in PDF, EPUB and Kindle. Book excerpt: