Michael D. Naish—Publications

A Centralized Omnidirectional Multi-Camera System with Peripherally-Guided Active Vision and Depth Perception
N. D. Jankovic and M. D. Naish, accepted for publication in the Proceedings of the 2007 IEEE International Conference on Networking, Sensing and Control, London, UK, April 15–17, 2007 Abstract

The growing popularity of omnidirectional vision technology has spawned numerous multi-camera designs that integrate various different camera types. This paper presents an omnidirectional vision system that combines a catadioptric camera, a fisheye camera and an active perspective camera. Aligning these cameras vertically provides a number of beneficial features, such as allowing simple peripherally-guided active vision, depth perception and a near spherical composite omnidirectional field of view. By having the active camera rotate around the outer perimeter, it can attain complete spherical access to the environment. The triangulation performance is evaluated experimentally using a target fixed to a long translation stage. Static positions of the target are estimated using a stereo pair that consists of one active perspective camera and one omnidirectional camera. Overall, the system provides sufficient accuracy to facilitate further surveillance research.

close
Active-Vision Based Multi-Sensor Surveillance—An Implementation
A. Bakhtari, M. D. Naish, M. Eskandari, E. A. Croft and B. Benhabib, IEEE Transactions on Systems, Man, and Cybernetics—Part C: Applications and Reviews, vol. 36, no. 5, pp. 668–680, September 2006. Abstract | IEEE Xplore | PDF 1.0 MB | PostScript 5.5 MB

In this paper, a novel reconfigurable surveillance system that incorporates multiple active vision sensors is presented. The proposed system has been developed for visual-servoing and other similar applications, such as tracking and state estimation, which require accurate and reliable target surveillance data. In the specific implementation case discussed herein, the position and orientation of a single target are surveyed at predetermined time instants along its unknown trajectory. The principles of dispatching, typically used in the operation of service vehicles (e.g., taxicabs and ambulances), form the basis of an effective approach to real-time sensing-system reconfiguration. Dispatching is used to select an optimal subset of dynamic sensors, to be used in a data-fusion process, and manoeuvre them in response to the motion of the object. The goal is to provide information of increased quality for the task at hand, while ensuring adequate response to future object manoeuvres.

Our experimental system is composed of a static overhead camera to predict the object’s gross motion and four mobile cameras to provide surveillance of a feature on the object (i.e., target). Object motion was simulated by placing it on an x-y table and pre-programming a path that is unknown to the surveillance system. The (future) pose predictions of the object are used by the dispatching algorithm to determine the optimal positions and bearings of the four mobile cameras. The selected cameras are independently positioned to estimate the target’s pose (a circular marker in our case) at the desired time instant. The target data obtained from the cameras, together with their own position and bearing, are fed to a fusion algorithm, where the final assessment of the target’s pose is determined. Experiments have shown that the use of dynamic sensors, together with a dispatching algorithm, tangibly improves the performance of a surveillance system.

close
Calibrating an Active Omnidirectional Vision System
N. D. Jankovic and M. D. Naish, Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, Edmonton, Alberta, pp. 3093–3098, August 2–6, 2005. Abstract | IEEE Xplore | PDF 770 kB | PostScript 4.4 MB

This paper describes a straightforward process for calibrating an active vision system containing both pinhole perspective and omnidirectional cameras. The perspective cameras can be easily calibrated using standard methods. Unfortunately, these methods are not suitable for omnidirectional cameras. Methods that rely on iterative least squares optimization, using a set of known image-world correspondences, are adopted for omnidirectional cameras. To ensure unbiased estimation of camera parameters, an omnidirectional calibration rig is employed so that nearly the entire field of view contains known calibration points. Measurement uncertainties collected from each stage of calibration are then combined to estimate the overall system uncertainty. This calibration process is evaluated experimentally by estimating the location of known points using triangulation, where the results achieved are comparable with the estimated system uncertainties.

close
Developing a Modular Active Spherical Vision System
N. D. Jankovic and M. D. Naish, Proceedings of the 2005 IEEE International Conference on Robotics and Automation, Barcelona, Spain, pp. 1234–1239, April 18–22, 2005. Abstract | IEEE Xplore | PDF 1.7 MB | PostScript 2.6 MB

This paper introduces a modular, real-time, omnidirectional, active vision system, as well as a constructed prototype. By combining omnidirectional and active pan-tilt cameras, a robust vision system is created that builds on the strengths of each camera type. The system can be easily configured to provide nearly an entire spherical field of view and independently track several targets of interest within the environment. The novel design allows the camera modules to be stacked, creating a vertical sensor structure. This vertical arrangement also provides a simple solution to the epipolar geometry and triangulation for target localization. Applications for this modular system can range from simple mobile robot navigation to complex multi-target tracking and surveillance.

close
Active Vision for the Autonomous Surveillance of Dynamic, Multi-Object Environments
A. Bakhtari, M. D. Naish and B. Benhabib, Proceedings of the 2004 ASME International Mechanical Engineering Congress and Exposition, Anaheim, California, November 11–15, 2004. Abstract | PDF 328 kB | PostScript 689 kB

This paper presents a novel method for the coordinated selection and positioning of groups of active-vision cameras for the autonomous surveillance of an object-of-interest as it travels through a multi-object workspace with an a priori unknown trajectory. Different approaches have been previously proposed to address the problem of sensor selection and control. However, these have primarily relied on off-line planning methods and only infrequently utilized on-line planning to compensate for unexpected variations in a target’s trajectory. The method proposed in this paper, on the other hand, uses a real-time dispatching algorithm, which eliminates the need for any a priori knowledge of the target’s trajectory, and, thus, is robust to unexpected variations in the environment. Experiments have shown that the use of dynamic sensors along with a dispatching algorithm can tangibly improve the performance of an active-surveillance system.

close
Sensing-System Planning for the Surveillance of Moving Objects
M. D. Naish, Ph.D. Thesis, Department of Mechanical and Industrial Engineering, University of Toronto, Toronto, Ontario, 2004. Abstract | PDF 1.9 MB | PostScript 29.3 MB

The surveillance of a manoeuvring target with multiple, mobile sensors in a coordinated manner requires a method for selecting and positioning groups of sensors in real time. In this context, this thesis outlines a comprehensive approach to planning sensor systems. The problem is addressed in two parts. The first part aims to optimally configure the sensing system for a particular task. This is accomplished through a simulation-based optimization procedure conducted off-line. Given a priori information about the expected object trajectories, the sensor set and initial sensor poses are determined such that the effectiveness of the sensing system is maximized. Using the optimally configured sensing system as a starting point, the second part updates the sensor poses and coordination strategies in real-time, reacting to the object motion. The principles of dispatching, as used for the effective operation of service vehicles, are adopted for this purpose. The predicted object trajectory is first discretized into a number of demand instants (data acquisition times), to which groups of sensors are assigned, respectively. Dispatching uses two complementary strategies. The coordination strategy determines which sensors will be assigned to a demand, while the positioning strategy specifies the pose of all sensors. The proposed approach aims to improve the quality of surveillance data in three ways: (1) The assigned sensors are manoeuvred into "optimal" sensing positions, (2) the uncertainty of the measured data is mitigated through sensor fusion, and (3) the poses of the unassigned sensors are adjusted to ensure that the surveillance system can react to future object manoeuvres. Two different sensor dispatching methods are developed. One uses heuristics, while the other learns appropriate strategies using reinforcement learning techniques. In addition, a method by which the two approaches may be used synergistically is introduced. Comprehensive simulations and experiments demonstrate the advantages of dispatching dynamic sensors over similar static-sensor surveillance systems.

close
Sensing-System Reconfiguration: A Comparison of On-line Methods
M. D. Naish, E. A. Croft and B. Benhabib, Proceedings of the 14th International Conference on Flexible Automation and Intelligent Manufacturing, vol. 1, Toronto, Ontario, pp. 368–375, July 12–14, 2004. Abstract | PDF 329 kB | PostScript 715 kB

This paper investigates the performance of two dispatching approaches applied to the real-time coordination of multiple, mobile sensors. The sensing system is targeted towards the surveillance of objects in the context of autonomous manufacturing systems. Sensors are assigned and manoeuvred to collect data at specific points on the object trajectory. A technique based on reinforcement learning (RL) is compared to a heuristic dispatching method and a system that does not use dispatching at all. Through a number of simulation examples, it is shown that, on average, the RL-based dispatcher achieves very similar, if not slightly better, performance than the heuristic dispatcher. Both approaches appear to provide a benefit over non-dispatching systems, thereby validating the efficacy of the dispatching approach, despite very different underlying implementations.

close
Object Surveillance Using Reinforcement Learning Based Sensor Dispatching
M. D. Naish, E. A. Croft and B. Benhabib, Proceedings of the 2004 IEEE International Conference on Robotics and Automation, vol. 1, New Orleans, Louisiana, pp. 71–76, April 26–May 1, 2004. Abstract | IEEE Xplore | PDF 249 kB | PostScript 919 kB

This paper outlines an approach to the coordination of multiple mobile sensors for the surveillance of a single moving target. A real-time dispatching algorithm is used to select and position groups of sensors in response to the observed object motion. The aim is to provide robust, high-quality data while ensuring that the system can react to unexpected object manoeuvres. Sensors are assigned to collect data at specific points on the object trajectory. A dispatching strategy learned via reinforcement learning is used to control the sensor poses with respect to these points. In using the learned strategy, each sensor adopts an egocentric view of the system state to determine the most appropriate action. Simulations demonstrate the performance of the RL-based dispatcher, in comparison to similar static-sensor systems.

close
A Multi-Sensor Surveillance System for Active-Vision Based Object Localization
A. Bakhtari, M. Eskandari, M. D. Naish and B. Benhabib, Proceedings of the 2003 IEEE International Conference on Systems, Man and Cybernetics, vol. 1, Washington, D.C., pp. 1013–1018, October 5–8, 2003. Abstract | IEEE Xplore | PDF 743 kB | PostScript 4.1 MB

In this paper, the implementation of a novel surveillance system that incorporates multiple active vision sensors controlled by a real-time dispatching algorithm is presented. The proposed system improves reliability and accuracy of target surveillance – tracking systems used for visual-servoing and other similar applications.

Experiments using a dispatched system have shown that the use of dynamic sensors can improve the performance of a surveillance system, primarily, due to the following factors: (i) decrease in the uncertainty associated with the object’s estimated pose, (ii) increase in robustness of the system due to its ability to cope with a wider range of a priori unknown object trajectories, and (iii) increase in reliability through sensory fault tolerance.

close
Coordinated Dispatching of Proximity Sensors for the Surveillance of Manoeuvring Targets
M. D. Naish, E. A. Croft and B. Benhabib, Journal of Robotics and Computer Integrated Manufacturing, vol. 19, no. 3, pp. 283–299, June 2003. Abstract | scholarsportal | PDF 855 kB | PostScript 1.8 MB

The surveillance of a manoeuvring target with multiple sensors in a coordinated manner requires a method for selecting and positioning groups of sensors in real time. Herein, the principles of dispatching, as used for the effective operation of service vehicles, are considered. The object trajectory is first discretized into a number of demand instants (data acquisition times), to which groups of sensors are assigned, respectively. Heuristic rules are used to determine the composition of each sensor group by evaluating the potential contribution of each sensor. In the case of dynamic sensors, the position of each sensor with respect to the target is also speciﬁed. Our proposed approach aims to improve the quality of the surveillance data in three ways: (1) The assigned sensors are manoeuvred into ‘‘optimal’’ sensing positions, (2) the uncertainty of the measured data is mitigated through sensor fusion, and (3) the poses of the unassigned sensors are adjusted to ensure that the surveillance system can react to future object manoeuvres. If a priori target trajectory information is available, the system performance maybe further improved by optimizing the initial pose of each sensor off-line. The advantages of dispatching dynamic sensors over similar static-sensor systems are demonstrated through comprehensive simulations.

close
Dispatching of Coordinated Proximity Sensors for Object Surveillance
M. D. Naish, E. A. Croft and B. Benhabib, Proceedings of the 2001 ASME International Mechanical Engineering Congress and Exposition, New York, New York, DSC-01-01, November 11–16, 2001. Abstract | PDF 309 kB | PostScript 1.9 MB

This paper presents a method of selecting and positioning groups of sensors in a coordinated manner for the surveillance of a maneuvering object. The object trajectory is discretized into a number of demand instants (data acquisition times) to which groups of sensors are assigned, respectively. Heuristic rules are used to evaluate the suitability of each sensor for servicing (observing) a demand instant, determine the composition of the sensor group, and, in the case of dynamic sensors, specify the position of each sensor with respect to the object. This approach aims to improve the quality of the surveillance data in three ways: (1) the assigned sensors are maneuvered into “optimal” sensing positions, (2) the uncertainty of the measured data is mitigated through sensor fusion, and (3) the poses of the unassigned sensors are adjusted to ensure that sensing-system can react to object maneuvers. Simulations with proximity sensors demonstrate the advantages of dispatching dynamic sensors over similar static-sensor systems.

close
Simulation-Based Sensing-System Configuration for Dynamic Dispatching
M. D. Naish, E. A. Croft and B. Benhabib, Proceedings of the 2001 IEEE International Conference on Systems, Man and Cybernetics, vol. 5, Tucson, Arizona, pp. 2964–2969, October 7–10, 2001. Abstract | IEEE Xplore | PDF 894 kB | PostScript 1.5 MB

This paper presents a methodology for determining the initial configuration of a set of sensors for a surveillance task. It serves to complement a dynamic dispatching methodology, which selects and maneuvers subsets of sensors to achieve optimal data acquisition in real-time. Specifically, given a priori information about the expected object trajectory, the initial sensor poses are determined such that the sensing-system effectiveness is maximized. This is achieved using the a constrained, non-linear, direct search method in combination with simulations of the sensing-system performance. (i.e., dynamic dispatching to adjust the sensor poses in response to the object motion.)

close
Dynamic Dispatching of Coordinated Sensors
M. D. Naish, E. A. Croft and B. Benhabib, Proceedings of the 2000 IEEE International Conference on Systems, Man and Cybernetics, vol. 5, Nashville, Tennessee, pp. 3318–3323, October 8–11, 2000. Abstract | IEEE Xplore | PDF 168 kB | PostScript 233 kB

Sensory data must be collected in-real time for the majority of autonomous decision making tasks, such as target tracking, surveillance and navigation. The use of multiple sensors may significantly improve the quality and robustness of the data. Given an environment containing a set of mobile sensors, capable of altering their position and orientation, this work addresses the problem of selecting and maneuvering subsets of these sensors for optimal data acquisition in real-time. A heuristic approach to the dispatching problem suitable for on-line implementation is illustrated by a computer-simulated example.

close
ELSA: A Multisensor Integration Architecture for Industrial Grading Tasks
M. D. Naish and E. A. Croft, MECHATRONICS special issue on "Developments in intelligent mechatronic systems", vol. 10, iss. 1–2, pp. 19–51, February–March 2000. Abstract | scholarsportal | PDF 315 kB | PostScript 1.5 MB

This paper presents the topology of the Extended Logical Sensor Architecture (ELSA) for multisensor integration and the methodology for constructing industrial sensor integration systems based on this architecture. ELSA has been developed for industrial applications, particularly, the on-line grading and classification of non-uniform food products. It addresses a number of issues specific to industrial inspection. The system must be modular and scalable to accommodate new processes and changing customer demands. It must be easy to understand so that non-expert users can construct, modify, and maintain the system. Furthermore, a data representation scheme which allows for the quantification of product deviations from an ideal model is required.

To address these needs, the sensors are encapsulated by a logical sensor model, providing robustness and flexibility. The construction methodology is based upon the object model which represents object classifications through combinations of primary features weighted by fuzzy membership functions. The features guide the selection of sensors and processing routines; the classifications determine the rulebase used by the inference engine for process decisions. Although inspection is the focus of this work, it is intended to be applicable to a variety of automation tasks which may benefit from a multiple sensor perception system.

close
Multisensor Industrial Inspection and Grading Using ELSA
M. D. Naish and E. A. Croft, Proceedings of the 1999 IEEE/ASME International Conference on Advanced Intelligent Mechatronics (AIM’99), Atlanta, GA, pp. 938–943, September 19–23, 1999. Abstract | IEEE Xplore | PDF 203 kB | PostScript 971 kB

The Extended Logical Sensor Architecture (ELSA) has been developed for industrial applications, particularly, the on-line grading and classification of non-uniform food products. This architecture addresses a number of issues specific to industrial inspection including modularity, scalability, and design by non-expert users. To address these needs, the sensors are encapsulated by a logical sensor model, providing robustness and flexibility. The construction methodology is based upon the object model which represents object classifications through combinations of primary features weighted by fuzzy membership functions. The features guide the selection of sensors and processing routines; the classifications determine the rulebase used by the inference engine for process decisions.

close
ELSA: An Intelligent Multisensor Integration Architecture for Industrial Grading Tasks
M. D. Naish, M.A.Sc. Thesis Department of Mechanical Engineering, University of British Columbia, Vancouver, British Columbia, 1999. Abstract | PDF 1.2 MB | PostScript 9.2 MB

The Extended Logical Sensor Architecture (ELSA) for multisensor integration has been developed for industrial applications, particularly, the on-line grading and classification of non-uniform food products. It addresses a number of issues specific to industrial inspection. The system must be modular and scalable to accommodate new processes and changing customer demands. It must be easy to understand so that non-expert users can construct, modify, and maintain the system.

The object model used by ELSA is particularly suited to the representation of non-uniform products, or any object for which classification is desired. Objects are represented by a connected graph structure; object nodes represent salient features of the object. Object classifications are defined by linking to primary features, which may be composed of a number of lower-level subfeatures.

Sensors and processing algorithms are encapsulated by an logical sensor model, providing robustness and flexibility. This is achieved by separating sensors from their functional use within a system. The hierarchical structure of the architecture allows modification with minimal disturbance to other components.

The construction methodology enables domain experts, who lack signal processing knowledge, to design and understand a sensor system for their particular application. This is achieved through a formal design process that addresses functional requirements in a systematic way. Each stage involves the extraction and utilization of the user's expert knowledge about the process and desired outcomes. Specification of the requirements leads to the identification of primary features and object classifications. Primary features are expanded into subfeatures. Logical sensors are then chosen to provide each of the features defined by the object model; this in turn determines what physical sensors are required by the system. The object classifications determine the rulebase used by the inference engine to infer process decisions.

close
An Open Architecture for Intelligent Multisensor Integration in Industrial Applications
M. D. Naish and E. A. Croft, Proceedings of the SPIE International Conference on Architectures, Networks, and Intelligent Systems for Manufacturing Integration, vol. 3203, Pittsburgh, Pennsylvania, pp. 33–44, October 13–17, 1997. Abstract | SPIE digital library | PDF 116 kB | PostScript 248 kB

An open architecture framework for intelligent multisensor integration in an industrial environment is being developed. This framework allows for the computational evaluation and understanding of sensor uncertainty and data validity through the comparison of sensor data in a common format.

A logical sensor model is used to represent both real and abstract sensors within the architecture. This allows for the unobtrusive addition or replacement of sensors. All logical sensor outputs are accompanied by a corresponding confidence level. These confidences are used to dynamically allocate valid sensor readings for use by higher-level sensors.

Sensory information is passed to an inference engine which uses user-selectable and adjustable fuzzy logic and/or neural network modules to provide the required decision making intelligence. This architecture may be applied to a broad range of industrial applications, especially those involving non-uniform product grading.

close
Data Representation and Organization for an Industrial Multisensor Integration Architecture
M. D. Naish and E. A. Croft, Proceedings of the 1997 IEEE International Conference on Systems, Man and Cybernetics, Orlando, Florida, pp. 821–826, October 12–15, 1997. Abstract | IEEE Xplore | PDF 166 kB | PostScript 1.1 MB

An open architecture for intelligent multisensor integration in an industrial environment is being developed. A logical sensor model is used to represent both real and abstract sensors within the architecture, allowing for the ready addition or replacement of sensors. Processing algorithms are also encapsulated by logical sensors. Objects are modeled using a connected graph structure wherein each node represents a salient feature of the object. Interactive training is used to determine the logical sensors required to extract desired features from objects. Extracted features are identified by the user and become part of the model. Once trained, the system can use object models for identification and classification purposes.

close