[RicardoSantos-99]

Closes #362.

Problem

Scholar.Linear has ordinary least squares, ridge, logistic, Bayesian ridge, isotonic and polynomial regression, but no $L_1$ regularization, so there is no way to fit a sparse linear model. Nothing in Scholar.Optimize applies either: bfgs, brent, golden_section and nelder_mead all assume a smooth objective, and the $L_1$ term is not differentiable at zero.

Solution

Two new models, both fitted with cyclic coordinate descent.

Scholar.Linear.ElasticNet minimizes

$$\frac{1}{2n}||y - Xw||^2_2 + \alpha\rho||w||_1 + \frac{\alpha(1 - \rho)}{2}||w||^2_2$$

where $\rho$ is :l1_ratio. Scholar.Linear.Lasso is the same model with l1_ratio: 1.0, so it reuses the solver, the way scikit-learn's Lasso subclasses its ElasticNet. It keeps its own options (without :l1_ratio) and its own struct, so predict/2 and the documentation stay specific to it.

The solver follows scikit-learn's enet_coordinate_descent: the same soft-thresholding update per coordinate, the duality gap computed once the largest update of an iteration falls below :tol relative to the largest coefficient, and the fit stopped once that gap is below :tol times $||y||^2_2$.

Options

• :alpha and :l1_ratio - regularization strength and the $L_1$ share of the penalty.
• :fit_intercept?, :max_iterations, :tol.
• :sample_weights - rescaled to sum to the number of samples, so the penalties keep their scale and only the ratios between weights matter, then the data is centered with the weighted means and multiplied by the square root of the weights. Same as scikit-learn.
• :positive? - constrains the coefficients to be non-negative, as scikit-learn's positive=True. Named with the ? suffix to follow the other boolean options of Scholar.Linear, such as :fit_intercept?.

Return values

Besides :coefficients and :intercept, the struct carries :dual_gap and :num_iterations, the equivalents of scikit-learn's dual_gap_ and n_iter_, since a defn cannot warn when the fit stops before converging.

Targets of shape {num_samples, num_targets} give coefficients of shape {num_targets, num_features}, plus one intercept, duality gap and iteration count per target, the same layout as Scholar.Linear.RidgeRegression. The coordinate descent solves all targets together, one column each. scikit-learn fits them one at a time, so each target is frozen once it converges: its coefficients, gap and iteration count stop changing while the other targets keep iterating. Each target therefore stops at the same iteration as when it is fitted alone.

Implementation notes

The scalar hyperparameters are passed as options rather than as numerical definition arguments, because a number given as an argument becomes an f32 tensor and an f64 fit would then see alpha: 0.1 as 0.10000000149. :sample_weights, which can be a tensor, is passed as an argument so it becomes a parameter of the compiled function instead of a constant inlined into it.

Validation

Checked against scikit-learn 1.6.1 (precompute=False) on 240+ randomly generated f64 problems, varying the number of samples and features, :alpha, :l1_ratio, :fit_intercept?, :positive?, :sample_weights, the number of targets, :max_iterations, :tol, and including constant, all-zero and nearly collinear columns. Coefficients and intercepts agree to about 1e-13 and the iteration counts are identical.

The tests cover this against the diabetes data and smaller fixtures, plus two checks that do not depend on scikit-learn: l1_ratio: 0.0 reproduces Scholar.Linear.RidgeRegression and alpha: 0.0 reproduces numpy.linalg.lstsq.

Known limitation

With l1_ratio: 0.0 the duality gap used as the stopping rule degenerates, so the fit runs until :max_iterations. This is inherited from the algorithm and happens in scikit-learn too; the :l1_ratio documentation points at Scholar.Linear.RidgeRegression for that case, the way the :alpha documentation already points at Scholar.Linear.LinearRegression for alpha: 0.0.

[josevalim]

Thank you!!!!!!