Pedestrian tracking studies have been facilitated by a large amount of surveillance apparatus in the city while also raising public privacy concerns. In this paper, we propose X-Tracking, a privacy-aware pedestrian tracking paradigm designed for vision systems in Smart City. It allows low-cost compatibility with existing surveillance architecture. To protect entities' privacy, X-Tracking uses video pre-processing with desensitization so that identity information is unexposed to the tracking algorithm. We implement system-level privacy protection by redesigning the tracking framework that decouples all services based on a single responsibility principle. Then, we elaborate on the roles, behaviors, and protocols used in the new system and illustrate how the paradigm strikes a favorable balance between privacy protection and convenience services. Furthermore, we propose a new tracking task that aims to track humans in masking surveillance video. It is comparable to previous tracking tasks but considering the target with a distorted appearance poses new challenges for visual tracking. Finally, we evaluate the baseline algorithm on the task with a demo dataset.