This thesis presents a visual tracking framework, Component-based Tracking. This framework combines low-level visual clues and constraints in a systematic way. It is based on a probabilistic graphical model to infer the marginal probability density function of the state of a tracked target. In this framework, all potential functions are approximated as weighted sums of multivariate Gaussians so probabilistic inference by message passing is equivalent to hypotheses combination and evaluation. The possibility of low level detectors not providing useful information is taken into account by the introduction of a null hypothesis. Simulation is conducted to verify the framework. In addition, a new kind of image-level region tracking method is studied. The estimation error of transformation parameters from using kernels is analyzed, and algorithms are proposed to select and construct optimal kernels to minimize the estimation errors of various types of geometric transformations. The algorithms are verified and compared against the method of sum-of-square differences (SSD) by both simulation and using real images. This optimal kernel tracker can be either used by itself or as a part of the component-based tracking framework. Finally, an experimental application on face tracking by the component-based tracking framework with optimal kernels is developed. Its performance is analyzed and compared against that of particle filters with various numbers of particles.