Python library for large scale convex optimization

Question

I'm trying to find a library for convex optimization in python which can handle a large amount of variables and constraints.

My problem is in the form

min pT*q + |c|

lower bound < q < upper bound
lower bound < c < upper bound
A* q + B*c = b
lower bound < Rq < upper bound

from cvxopt.modeling import variable, op, sum
q = variable(numb_prod, 'q')
c = variable(numb_capacity, 'c')
prod_price = p.trans() * q

problem = op(sum(prod_price) + 1e-6*sum(abs(c)), [constraints])
problem.solve(format='sparse', solver='glpk', options={'glpk':{'msg_lev':'GLP_MSG_OFF'}})

Where p, b are vectors and A,B,R are sparse matrices and q,c variables For more info about the matrices one can look here https://math.stackexchange.com/questions/4018792/python-library-for-large-scale-convex-optimization?noredirect=1#comment8297384_4018792

Currently I'm having 108575 variables which is too large for cvxopt. I was hopping running an optimization which can handle the amount of variables in the millions...

Is there any library that can handle that? Or does one need to look into another language?

The problems seems to be inside cvxopt and not necessarily the solver Error message:

Traceback (most recent call last): File "C:\Users\OsRas\AppData\Local\Continuum\anaconda3\lib\site-packages\IPython\core\interactiveshell.py", line 3325, in run_code exec(code_obj, self.user_global_ns, self.user_ns) File "", line 489, in q, c, total_cost_x, water_values, list_numb_of_capacity_variables = solution(demand, price_purchase_power, supply) File "", line 391, in solution test = problem.solve(format='sparse', solver='glpk', options={'glpk':{'msg_lev':'GLP_MSG_OFF'}}) File "C:\Users\OsRas\AppData\Local\Continuum\anaconda3\lib\site-packages\cvxopt\modeling.py", line 2600, in solve t = self._inmatrixform(format) File "C:\Users\OsRas\AppData\Local\Continuum\anaconda3\lib\site-packages\cvxopt\modeling.py", line 2479, in _inmatrixform for v in variables: vmap[v] = x[vslc[v]] File "C:\Users\OsRas\AppData\Local\Continuum\anaconda3\lib\site-packages\cvxopt\modeling.py", line 240, in getitem return (+self).getitem(key) File "C:\Users\OsRas\AppData\Local\Continuum\anaconda3\lib\site-packages\cvxopt\modeling.py", line 931, in getitem f._linear = self._linear[l] File "C:\Users\OsRas\AppData\Local\Continuum\anaconda3\lib\site-packages\cvxopt\modeling.py", line 1385, in getitem for v,c in iter(self._coeff.items()): OverflowError: Python int too large to convert to C long

Maybe usefull things on here plato.asu.edu/sub/nlores.html or leaves.shufe.edu.cn/static/docs/lc/manual.pdf — Willem Hendriks
osqp.org/docs/index.html from the same list is also something I would like to throw the problem at — Willem Hendriks

Erwin Kalvelagen Erwin Kalvelagen · Accepted Answer · 2021-02-09T17:33:43

This is really an LP (Linear Programming) problem:

  min sum(i, p(i)*q(i)) + sum(i,cabs(i))
  cabs(i) >= c(i)
  cabs(i) >= -c(i)
   other constraints

where cabs is an additional positive variable.

A model with 108,575 variables is not extremely large for an LP, so any good LP solver will do.

Python library for large scale convex optimization

1 Answers