User association is one of the key techniques of the resource management in the heterogeneous cellular networks (HetNets). Interference management plays an important role on the system performance improvement in interference limited cellular networks. In this paper, we address the user association problem from the view of interference management of HetNets. Especially, we consider the co-channel interference from both intra-tier and inter-tier, and propose an interference matrix based user association scheme. We use interference matrix to obtain the leakage plus noise ratio (SLNR) of each users in the HetNets. In the proposed scheme, the optimization problem of user association is modeled jointly taking the SLNR of users and traffic load of base stations into consideration, which gives a tradeoff between interference mitigation and load balancing. A simple but useful heuristic algorithm is given to obtain a suboptimal solution of the user association problem. We give the simulation results of the proposed scheme on the system throughput and user fairness, which can effectively achieve the tradeoff between interference mitigation and load balancing.