Figure 6.2: Penalty function approximation

% Section 6.1.2
% Boyd & Vandenberghe "Convex Optimization"
% Original by Lieven Vandenberghe
% Adapted for CVX Argyris Zymnis - 10/2005
%
% Comparison of the ell1, ell2, deadzone-linear and log-barrier
% penalty functions for the approximation problem:
%       minimize phi(A*x-b),
%
% where phi(x) is the penalty function
% Log-barrier will be implemented in the future version of CVX

% Generate input data
randn('state',0);
m=100; n=30;
A = randn(m,n);
b = randn(m,1);

% ell_1 approximation
% minimize   ||Ax+b||_1
disp('ell-one approximation');
cvx_begin
    variable x1(n)
    minimize(norm(A*x1+b,1))
cvx_end

% ell_2 approximation
% minimize ||Ax+b||_2
disp('ell-2');
x2=-A\b;

% deadzone penalty approximation
% minimize sum(deadzone(Ax+b,0.5))
% deadzone(y,z) = max(abs(y)-z,0)
dz = 0.5;
disp('deadzone penalty');
cvx_begin
    variable xdz(n)
    minimize(sum(max(abs(A*xdz+b)-dz,0)))
cvx_end


% log-barrier penalty approximation
%
% minimize -sum log(1-(ai'*x+bi)^2)

disp('log-barrier')

% parameters for Newton Method & line search
alpha=.01; beta=.5;

% minimize linfty norm to get starting point
cvx_begin
    variable xlb(n)
    minimize norm(A*xlb+b,Inf)
cvx_end
linf = cvx_optval;
A = A/(1.1*linf);
b = b/(1.1*linf);

for iters = 1:50

   yp = 1 - (A*xlb+b);  ym = (A*xlb+b) + 1;
   f = -sum(log(yp)) - sum(log(ym));
   g = A'*(1./yp) - A'*(1./ym);
   H = A'*diag(1./(yp.^2) + 1./(ym.^2))*A;
   v = -H\g;
   fprime = g'*v;
   ntdecr = sqrt(-fprime);
   if (ntdecr < 1e-5), break; end;

   t = 1;
   newx = xlb + t*v;
   while ((min(1-(A*newx +b)) < 0) | (min((A*newx +b)+1) < 0))
       t = beta*t;
       newx = xlb + t*v;
   end;
   newf = -sum(log(1 - (A*newx+b))) - sum(log(1+(A*newx+b)));
   while (newf > f + alpha*t*fprime)
       t = beta*t;
       newx = xlb + t*v;
       newf = -sum(log(1-(A*newx+b))) - sum(log(1+(A*newx+b)));
   end;
   xlb = xlb+t*v;
end


% Plot histogram of residuals

ss = max(abs([A*x1+b; A*x2+b; A*xdz+b;  A*xlb+b]));
tt = -ceil(ss):0.05:ceil(ss);  % sets center for each bin
[N1,hist1] = hist(A*x1+b,tt);
[N2,hist2] = hist(A*x2+b,tt);
[N3,hist3] = hist(A*xdz+b,tt);
[N4,hist4] = hist(A*xlb+b,tt);


range_max=2.0;  rr=-range_max:1e-2:range_max;

figure(1), clf, hold off
subplot(4,1,1),
bar(hist1,N1);
hold on
plot(rr, abs(rr)*40/3, '-');
ylabel('p=1')
axis([-range_max range_max 0 40]);
hold off

subplot(4,1,2),
bar(hist2,N2);
hold on;
plot(rr,2*rr.^2),
ylabel('p=2')
axis([-range_max range_max 0 11]);
hold off

subplot(4,1,3),
bar(hist3,N3);
hold on
plot(rr,30/3*max(0,abs(rr)-dz))
ylabel('Deadzone')
axis([-range_max range_max 0 25]);
hold off

subplot(4,1,4),
bar(hist4,N4);
rr_lb=linspace(-1+(1e-6),1-(1e-6),600);
hold on
plot(rr_lb, -3*log(1-rr_lb.^2),rr,2*rr.^2,'--')
axis([-range_max range_max 0 11]);
ylabel('Log barrier'),
xlabel('r')
hold off
ell-one approximation
 
Calling sedumi: 230 variables, 100 equality constraints
------------------------------------------------------------
SeDuMi 1.21 by AdvOL, 2005-2008 and Jos F. Sturm, 1998-2003.
Alg = 2: xz-corrector, Adaptive Step-Differentiation, theta = 0.250, beta = 0.500
eqs m = 100, order n = 203, dim = 232, blocks = 102
nnz(A) = 100 + 3000, nnz(ADA) = 100, nnz(L) = 100
Handling 31 + 1 dense columns.
 it :     b*y       gap    delta  rate   t/tP*  t/tD*   feas cg cg  prec
  0 :            1.53E+02 0.000
  1 :   4.19E+01 7.27E+01 0.000 0.4759 0.9000 0.9000   3.41  1  1  1.8E+00
  2 :   5.00E+01 2.82E+01 0.000 0.3883 0.9000 0.9000   1.71  1  1  6.5E-01
  3 :   5.33E+01 9.83E+00 0.000 0.3483 0.9000 0.9000   1.21  1  1  2.3E-01
  4 :   5.45E+01 3.25E+00 0.000 0.3302 0.9000 0.9000   1.07  1  1  7.9E-02
  5 :   5.49E+01 9.31E-01 0.000 0.2868 0.9000 0.9000   1.02  1  1  2.3E-02
  6 :   5.51E+01 2.79E-01 0.000 0.2998 0.9000 0.9039   1.01  1  1  6.7E-03
  7 :   5.51E+01 5.33E-02 0.000 0.1909 0.9230 0.9000   1.00  1  1  1.7E-03
  8 :   5.51E+01 9.92E-03 0.000 0.1862 0.9000 0.9101   1.00  1  1  2.5E-04
  9 :   5.51E+01 2.00E-04 0.000 0.0202 0.9900 0.9901   1.00  1  1  4.1E-06
 10 :   5.51E+01 5.64E-07 0.000 0.0028 0.9990 0.9990   1.00  4  4  9.6E-09

iter seconds digits       c*x               b*y
 10      0.1   7.9  5.5128922241e+01  5.5128921562e+01
|Ax-b| =   1.7e-08, [Ay-c]_+ =   1.9E-10, |x|=  1.2e+01, |y|=  8.8e+00

Detailed timing (sec)
   Pre          IPM          Post
1.000E-02    6.000E-02    0.000E+00    
Max-norms: ||b||=1.957607e+00, ||c|| = 1,
Cholesky |add|=16, |skip| = 0, ||L.L|| = 1.
------------------------------------------------------------
Status: Solved
Optimal value (cvx_optval): +55.1289
ell-2
deadzone penalty
 
Calling sedumi: 300 variables, 130 equality constraints
   For improved efficiency, sedumi is solving the dual problem.
------------------------------------------------------------
SeDuMi 1.21 by AdvOL, 2005-2008 and Jos F. Sturm, 1998-2003.
Alg = 2: xz-corrector, Adaptive Step-Differentiation, theta = 0.250, beta = 0.500
eqs m = 130, order n = 301, dim = 301, blocks = 101
nnz(A) = 3200 + 0, nnz(ADA) = 7000, nnz(L) = 3565
 it :     b*y       gap    delta  rate   t/tP*  t/tD*   feas cg cg  prec
  0 :            1.33E+01 0.000
  1 :  -5.72E+00 4.39E+00 0.000 0.3312 0.9000 0.9000   5.07  1  1  1.5E+00
  2 :  -1.61E+01 1.39E+00 0.000 0.3176 0.9000 0.9000   1.35  1  1  3.2E-01
  3 :  -1.99E+01 4.23E-01 0.000 0.3032 0.9000 0.9000   1.09  1  1  9.0E-02
  4 :  -2.10E+01 1.45E-01 0.000 0.3426 0.9000 0.9000   1.02  1  1  3.1E-02
  5 :  -2.13E+01 4.69E-02 0.000 0.3241 0.9000 0.9000   1.01  1  1  9.9E-03
  6 :  -2.13E+01 4.22E-05 0.000 0.0009 0.9000 0.0000   1.00  1  1  4.3E-03
  7 :  -2.14E+01 7.31E-06 0.000 0.1730 0.9133 0.9000   1.00  1  1  7.8E-04
  8 :  -2.15E+01 4.83E-07 0.000 0.0661 0.9902 0.9900   1.00  1  1  5.1E-05
  9 :  -2.15E+01 1.32E-08 0.000 0.0273 0.9900 0.9829   1.00  1  1  1.5E-06
 10 :  -2.15E+01 4.27E-13 0.000 0.0000 1.0000 1.0000   1.00  1  1  4.9E-11

iter seconds digits       c*x               b*y
 10      0.1   Inf -2.1468211790e+01 -2.1468211790e+01
|Ax-b| =   1.4e-10, [Ay-c]_+ =   4.0E-11, |x|=  1.1e+01, |y|=  4.4e+00

Detailed timing (sec)
   Pre          IPM          Post
1.000E-02    6.000E-02    0.000E+00    
Max-norms: ||b||=1, ||c|| = 1.957607e+00,
Cholesky |add|=0, |skip| = 0, ||L.L|| = 3.41276.
------------------------------------------------------------
Status: Solved
Optimal value (cvx_optval): +21.4682
log-barrier
 
Calling sedumi: 200 variables, 31 equality constraints
   For improved efficiency, sedumi is solving the dual problem.
------------------------------------------------------------
SeDuMi 1.21 by AdvOL, 2005-2008 and Jos F. Sturm, 1998-2003.
Alg = 2: xz-corrector, Adaptive Step-Differentiation, theta = 0.250, beta = 0.500
eqs m = 31, order n = 201, dim = 201, blocks = 101
nnz(A) = 3100 + 0, nnz(ADA) = 961, nnz(L) = 496
 it :     b*y       gap    delta  rate   t/tP*  t/tD*   feas cg cg  prec
  0 :            7.10E+01 0.000
  1 :  -8.67E+00 5.16E+00 0.000 0.0727 0.9900 0.9900  -0.54  1  1  4.6E+01
  2 :  -3.94E+00 2.60E+00 0.000 0.5034 0.9000 0.9000   2.91  1  1  1.0E+01
  3 :  -1.63E+00 1.22E+00 0.000 0.4698 0.9000 0.9000   4.10  1  1  2.0E+00
  4 :  -1.33E+00 4.53E-01 0.000 0.3710 0.9000 0.9000   1.61  1  1  5.9E-01
  5 :  -1.24E+00 1.62E-01 0.000 0.3586 0.9000 0.9000   1.21  1  1  2.0E-01
  6 :  -1.22E+00 6.27E-02 0.000 0.3860 0.9000 0.9000   1.07  1  1  7.5E-02
  7 :  -1.21E+00 2.25E-02 0.000 0.3588 0.9000 0.9000   1.03  1  1  2.7E-02
  8 :  -1.20E+00 9.34E-03 0.000 0.4154 0.9000 0.6609   1.01  1  1  1.1E-02
  9 :  -1.20E+00 1.44E-03 0.000 0.1545 0.9243 0.9000   1.00  1  1  1.2E-03
 10 :  -1.20E+00 5.55E-05 0.000 0.0384 0.9904 0.9900   1.00  1  1  3.0E-05
 11 :  -1.20E+00 8.97E-06 0.000 0.1617 0.9102 0.9000   1.00  1  1  3.8E-06
 12 :  -1.20E+00 1.56E-06 0.000 0.1737 0.9092 0.9000   1.00  1  1  5.5E-07
 13 :  -1.20E+00 5.56E-09 0.000 0.0036 0.9990 0.9990   1.00  1  1  2.9E-09

iter seconds digits       c*x               b*y
 13      0.1   8.5 -1.2012704624e+00 -1.2012704662e+00
|Ax-b| =   1.9e-09, [Ay-c]_+ =   0.0E+00, |x|=  3.1e-01, |y|=  1.5e+00

Detailed timing (sec)
   Pre          IPM          Post
1.000E-02    6.000E-02    0.000E+00    
Max-norms: ||b||=1, ||c|| = 1.957607e+00,
Cholesky |add|=0, |skip| = 0, ||L.L|| = 1.35595.
------------------------------------------------------------
Status: Solved
Optimal value (cvx_optval): +1.20127