Quantile Regression Gets a Vector Boost

Author: Denis Avetisyan

A new duality theory extends quantile regression to handle vector-valued responses, offering a powerful approach to modeling complex data.

This work establishes strong duality and provides a closed-form solution for entropic Vector Quantile Regression under Gaussian marginals, leveraging optimal transport and Wasserstein distance.

While extending quantile regression to vector-valued responses presents computational challenges, this paper, ‘Entropic vector quantile regression: Duality and Gaussian case’, develops a rigorous duality theory for entropic Vector Quantile Regression (VQR). Establishing strong duality and dual attainment for possibly unbounded marginals, we demonstrate a closed-form solution when marginals are Gaussian, recovering a Gaussian optimal coupling and quantifying the approximation rate to unregularized VQR. These results not only deepen our understanding of entropic regularization within optimal transport but also raise the question of whether similar closed-form solutions can be found for other marginal distributions in VQR.

Beyond Scalar Quantiles: Embracing Vector-Valued Uncertainty

Traditional quantile regression, a powerful tool for understanding the distribution of predicted values, encounters significant challenges when applied to vector-valued responses – situations where predictions aren’t single numbers, but sets of interconnected values. The core issue lies in the absence of a natural ordering for these multi-dimensional outputs; unlike a single variable where values can be simply ranked, comparing entire vectors requires defining a meaningful distance or preference. This limitation hinders the ability to accurately assess risk and make predictions in fields such as financial modeling, spatial statistics, and image analysis, where outputs are inherently multi-dimensional. For example, predicting both the mean and the variance of a financial asset requires considering both values simultaneously, something standard quantile regression isn’t equipped to handle effectively without imposing arbitrary ordering schemes.

The inability of standard quantile regression to effectively handle multi-dimensional responses presents significant challenges for accurate predictive modeling and risk analysis. Traditional methods rely on ordering outcomes to define quantiles – a straightforward process for single variables. However, when dealing with vector-valued data – such as predicting multiple correlated financial assets or forecasting several environmental factors simultaneously – establishing a natural ordering becomes problematic. This deficiency limits the precision of risk assessments, particularly in fields like finance and econometrics where understanding the distribution of potential losses across multiple dimensions is crucial. Consequently, predictions generated using standard quantile regression may fail to capture the full spectrum of possible outcomes, potentially leading to underestimation of risk and suboptimal decision-making in complex, multi-output scenarios.

Vector Quantile Regression addresses the limitations of traditional quantile regression when dealing with multi-dimensional response variables. Unlike scalar quantiles which rely on a simple ordering, VQR employs the mathematical framework of optimal transport to establish a meaningful notion of quantiles for vectors. This innovative approach determines the ‘quantile region’ – the set of vectors associated with a particular quantile level – by minimizing the ‘transport cost’ required to move probability mass from the predicted distribution to the observed data. Effectively, VQR doesn’t just predict a single vector value for a given quantile, but rather a region encompassing plausible outcomes, allowing for a more nuanced and robust assessment of risk and uncertainty in scenarios where the output isn’t a single number, but a complex vector of characteristics. This is particularly useful in fields like finance, environmental modeling, and medical diagnostics where understanding the full distribution of potential outcomes is crucial.

Optimal Transport: A Mathematically Rigorous Measure of Dissimilarity

Optimal Transport (OT) offers a mathematically grounded approach to quantifying the distance between probability distributions, differing from methods like Kullback-Leibler divergence which may be undefined or infinite when distributions have non-overlapping support. The core concept involves finding the minimum “cost” to transform one distribution into another, where cost is determined by a ground metric – typically $L^p$ distance – applied to individual data points. This results in the Wasserstein distance – also known as the Earth Mover’s Distance – which represents the minimum amount of “work” required to morph one distribution into the other. Crucially, the Wasserstein distance is a true metric, satisfying properties like non-negativity, symmetry, and the triangle inequality, and provides a geometrically interpretable measure of dissimilarity even when distributions are disjoint or have differing supports.

Vector Quantized-Representation (VQR) employs Optimal Transport (OT) to establish a cost function quantifying the dissimilarity between predicted and observed vector responses. This cost function, based on the Wasserstein distance $W(P,Q)$ , assesses the minimum cost of transforming the probability distribution of predicted vectors $P$ into the distribution of observed vectors $Q$ . Specifically, the cost is calculated by finding the optimal transport plan – a mapping between predicted and observed vectors – that minimizes the cumulative cost, typically using a ground distance like the squared Euclidean distance. This OT-based cost provides a more robust and meaningful measure of dissimilarity than traditional metrics like mean squared error, particularly when dealing with multi-modal distributions or distributions with differing supports.

The Brenier map provides a computationally efficient solution for determining the optimal transport plan between two probability distributions. Traditional optimal transport calculations require solving a linear program, which scales poorly with dimensionality. The Brenier map, specifically leveraging the $c(x) = |x|^2$ cost function, guarantees the existence of a gradient field that defines a transport map Γ from one distribution to the other. This map effectively specifies how mass is moved to minimize the total transport cost. By solving a Monge-Ampère equation to find Γ, the Wasserstein distance can be computed without explicitly iterating over all possible transport plans. This reduction in computational complexity is critical for the scalability of Vector Quantized-Variational Autoencoder (VQR) implementations, enabling the efficient calculation of dissimilarity metrics between high-dimensional vector responses.

Stabilizing the Solution: Entropic Regularization and Duality

Entropic regularization stabilizes the Vector Quantized Regression (VQR) problem by incorporating a penalty term based on the Kullback-Leibler (KL) divergence $D_{KL}(p||q)$ . This penalty, applied to the transport plan, discourages solutions that assign probability mass to distant quantiles, thereby mitigating instability during optimization. Specifically, the addition of this term ensures the existence of a unique optimal solution, addressing the non-uniqueness that can occur in standard VQR formulations. The strength of the regularization is controlled by a temperature parameter; lower temperatures increase the penalty and promote smoother, more stable transport plans, while higher temperatures reduce the penalty and allow for more flexible solutions.

Entropic regularization promotes smoother transport plans by discouraging highly concentrated probability mass assignments during the optimal transport calculation. This is achieved by minimizing the Kullback-Leibler (KL) divergence between the estimated transport plan and a uniform distribution, effectively penalizing solutions with sharp discontinuities. Consequently, the estimated quantiles become less sensitive to minor perturbations in the input data, enhancing the robustness of the quantile regression estimates and providing more stable and reliable results, particularly in scenarios with noisy or incomplete data.

Reformulating the Vector Quantized Regression (VQR) problem as a dual program facilitates analytical tractability and provides a foundation for establishing optimality conditions. This dual formulation, grounded in established convex optimization theory, allows for the derivation of strong duality results – meaning the primal and dual objective values are equal at optimal solutions – and conditions guaranteeing dual attainment, where a primal optimal solution can be recovered from a dual optimal solution. Specifically, strong duality holds under certain constraints on the problem data, and dual attainment is ensured by satisfying appropriate constraint qualifications, enabling efficient computation and rigorous performance analysis of the VQR estimator.

Theoretical Foundations: Rigor and Convergence

Coercivity of the dual problem, rigorously demonstrated through analytical methods, ensures the existence of a solution that minimizes the functional and maintains stability. Specifically, coercivity establishes a lower bound on the functional’s values, preventing them from decreasing indefinitely and thus guaranteeing a minimum is attainable. This property is crucial for proving convergence of optimization algorithms used to solve the problem, as it confirms that iterative procedures will not diverge and will ultimately converge to a stable, minimizing solution. The established coercivity provides a foundational mathematical guarantee for the well-posedness and solvability of the dual formulation.

The Beta-Sub-Weibull distribution facilitates the proof of crucial properties regarding the Kullback-Leibler (KL) divergence by providing a flexible framework for modeling probability distributions with varying degrees of concentration. Specifically, this distribution allows for a controlled analysis of the KL divergence’s behavior under perturbation, enabling the derivation of bounds on its sensitivity and rate of change. Its parameters allow for the explicit characterization of distributional tails and shapes, which are critical in establishing convergence rates and stability conditions related to optimization problems involving the KL divergence. This approach contrasts with using generic distributions, offering increased precision in bounding the error terms and guaranteeing the existence of solutions in related optimization contexts.

Theoretical analysis establishes the convergence rate between the entropic regularization solution and the unregularized optimal transport solution. Specifically, the 2-Wasserstein distance, a metric quantifying the difference between probability distributions, converges at a rate of ε, where ε represents the regularization parameter. Furthermore, the error introduced by the regularization process itself is demonstrated to be of order $O(ε²)$ . This indicates that the error scales quadratically with the regularization parameter, implying a diminishing effect on the overall solution accuracy as ε approaches zero.

Expanding the Scope: Applications and Future Directions

The Variational Quantum Refinement (VQR) framework extends beyond theoretical exploration, offering a surprisingly versatile toolkit for tackling challenges in fields as disparate as financial risk management and the pursuit of robust machine learning algorithms. Its adaptability stems from the ability to efficiently approximate solutions to complex optimization problems, crucial for tasks like portfolio optimization where minimizing risk while maximizing returns requires navigating high-dimensional spaces. Similarly, in machine learning, VQR’s capacity to handle noisy or incomplete data allows for the development of algorithms less susceptible to adversarial attacks or real-world imperfections. This broad applicability positions VQR not merely as a quantum algorithm, but as a foundational method with potential for integration into a variety of computational workflows, promising enhanced performance and resilience across numerous domains.

The selection of the Gaussian distribution as a foundational test case for Variational Quantum Refinement (VQR) isn’t merely academic; it underscores the method’s practical potential. Given the prevalence of Gaussian, or normally distributed, data across numerous scientific and engineering disciplines – from signal processing and statistical physics to financial modeling and machine learning – VQR’s demonstrated efficacy with this distribution suggests broad applicability. Successfully refining solutions within a Gaussian framework establishes a strong baseline for tackling more complex, real-world datasets that often contain, or are built upon, Gaussian components. This initial validation signifies that VQR isn’t limited to contrived scenarios but possesses the flexibility to address challenges arising in domains where Gaussian statistics are fundamental, paving the way for robust performance across diverse applications.

A key strength of the proposed Variational Quantum Refinement (VQR) method lies in its provable accuracy; the 2-Wasserstein distance – a metric for comparing probability distributions – between the approximate solution generated by VQR and the true, optimal solution is rigorously bounded. Specifically, this distance is shown to be less than or equal to $εtr(L⁻¹Λ₀) + O(ε²)$ , where ε represents the refinement parameter controlling the trade-off between computational cost and accuracy. This bound demonstrates that the error introduced by the approximation scales linearly with $εtr(L⁻¹Λ₀)$ and exhibits quadratic convergence as ε approaches zero, guaranteeing a controllable and increasingly precise solution. The result establishes a theoretical foundation for VQR’s effectiveness, assuring users of its reliability even when dealing with complex optimization problems where finding the absolute optimum is computationally intractable.

The pursuit of optimal transport, as demonstrated within this work on entropic vector quantile regression, echoes a fundamental principle: identifying invariants as complexity increases. Let N approach infinity – what remains invariant? The paper rigorously establishes strong duality, a form of invariance, within the quantile regression framework. This duality guarantees a predictable relationship between primal and dual solutions, regardless of the dimensionality of the vector-valued responses. As Max Planck observed, “A new scientific truth does not triumph by convincing its opponents but by the opponents dying out.” Similarly, the elegance of this duality theory lies not merely in its functional correctness-providing a closed-form solution under Gaussian marginals-but in its provable invariance, a mathematical truth that transcends specific datasets or algorithmic implementations. The focus on Wasserstein distance and entropic regularization further reinforces this pursuit of invariant properties within a complex statistical landscape.

Further Horizons

The established duality, while elegant, merely shifts the computational burden. Closed-form solutions, even under Gaussian assumptions, are transient victories. The true challenge lies not in finding a coupling, but in demonstrating its robustness to the inevitable imperfections of real-world data. One suspects the Gaussian case, so conveniently tractable, masks a far more complex landscape of non-Gaussian marginals – a landscape where duality may falter, and approximation becomes the only pragmatic recourse.

Future work must address the limitations imposed by the Wasserstein distance itself. While theoretically sound, its computation remains a bottleneck, particularly in higher dimensions. Exploring alternative divergence measures – those offering a more computationally efficient, albeit potentially less theoretically pristine, path – warrants serious consideration. The pursuit of mathematical purity should not blind one to the practical necessity of scalability.

Ultimately, the value of this framework will be judged not by its theoretical completeness, but by its capacity to yield provably reliable quantile estimates in the face of data scarcity and noise. The current formulation provides a foundation; the true test lies in demonstrating its resilience beyond the idealized conditions so often favored by mathematical analysis.

Original article: https://arxiv.org/pdf/2602.11290.pdf

Contact the author: https://www.linkedin.com/in/avetisyan/