why is the ImFil optimizer history not completely printed?

Question

I use the [scikit-quant] (https://scikit-quant.readthedocs.io/en/latest/starting.html) implementation of the ImFil optimizer. I want to plot the objective function values vs. iteration number and thus need the full history of values. It returns a history object, but when I print it, it is filled with " ..." between the values. The output is not complete. I would expect 200 rows with arrays with 13 elements. What I get instead is:

history:  [[-0.83333333  0.1         0.         ...  0.1         0.
   0.        ]
 [-0.83333333  3.24159265  0.         ...  0.1         0.
   0.        ]
...
   0.        ]]

I think this is not a problem of size limits to the output in my IDE because it also happens when writing it to a text file.

Details: I made no changes in the code, so it is not used as standalone. I this case SQImFil/_imfil.py minimize returns objfunc.get_history() (which returns self.stats.full_history()) which returns self._history as numpy array. Here I find the documentation """Return the full call history. Each row represents one call, with the first column the result from the objective function call, the other columns the parameters used."""

Answer 1

I am not familiar with that package but I have checked source code quickly:

\scikit-quant\opt\imfil\python\SQImFil\_optset.py

This seems to be confusing description:

def optset(optin=None, **optkwds):

...

  complete_history: 1 or 'on' to store complete history of all calls to f. 0
         or 'off' otherwise. The default is 'off'. Do not turn this off unless
         you are having severe problems with storage.

When default is 'off' already then why user should not 'turn that off'?

And then we have:

def _create_defaults():
    """Default values"""

    class OptSet(object):
        __slots__ = {
            'add_new_directions' : [],  # function to add new directions.
            #
            'armijo_reduction' : .5,    # Step size reduction factor for Armijo rule.
            #
            'complete_history' : 0,     # Store complete history.

So is that the case that we have complete_history switched off by default?

Try to use this code and check if problem is fixed:

opt_common = {'complete_history' : True}
optset = SQImFil.optset(**opt_common)

There is as well valuable comment here:

#-----
def minimize(f, x0, bounds, budget=10000, optin=None, **optkwds):
    """
  Minimization of noisy functions subject to explicit bound
  constraints + hidden constraints
...
  You may want to use this to build surrogates, decide to add new points to
  the stencil, or for troubleshooting. The complete history can take up a
  lot of room. I will only return it as output if you ask for it, and will
  not store it at all if you set the complete_history option to '0'.
  If I don't store it at all, then your functions to add directions to the
  stencil can't use that data.
...
    if options.standalone:
        if options.complete_history:
            return result, histout, complete_history # <- have to be requested in method options
        return result, histout
    return result, objfunc.get_history()

It looks like in this example method above complete history is only returned as output when requested explicitely.
You need to check your code to be sure you request it and use whenever it should be.