Techniques and systems are provided for processing video data. For example, techniques and systems are provided for determining costs for blob trackers and blobs. A blob can be detected in a video frame. The blob includes pixels of at least a portion of a foreground object. A physical distance between a blob tracker and the blob can be determined. A size ratio between the blob tracker and the blob can also be determined. A cost between the blob tracker and the blob can then be determined using the physical distance and the size ratio. In some cases, a spatial relationship between the blob tracker and the blob is determined, in which case the physical distance can be determined based on the spatial relationship. Blob trackers can be associated with blobs based on the determined costs between the blob trackers and the blobs.