Joint Power and Modulation Optimization in Two-User Non-Orthogonal Multiple Access Channels: A Minimum Error Probability Approach

In this paper, we consider the joint power and modulation optimization in two-user non-orthogonal multiple access (NOMA) channels based on error-rate performance criterion and finite-alphabet inputs. Specifically, with joint maximum-likelihood detection being employed at each user's receiver, the users' transmission power levels are jointly determined with the 4-QAM constellation of one of the users being rotated such that the probability of dominant pairwise error event in the NOMA channel is minimized. Toward solving the joint power and modulation problem, our results are twofold. First, with $M$ antennas at the receiver, closed-form expressions for the optimal angles as a function of $M$, the channel coefficients, and the users' power levels are derived. Second, closed-form expressions are derived for the joint power allocation with optimal rotation by casting the joint optimization as a piecewise convex optimization problem over its sub-domains. Using the proposed joint optimization, the error-rate performance of NOMA improves significantly and it outperforms traditional NOMA power allocation methods for the selected transmission rates. The proposed approach thus represents a rather distinct approach in NOMA optimization, and it is applicable in the uplink NOMA where users share the same time/frequency channel or in the downlink NOMA where users are multiplexed in the power domain.