Discussion:
[math] How to use LevenbergMarquardtOptimizer for finding the optimal dampening factor for exponential smoothing
Manish Java
2018-05-16 13:28:25 UTC
I am trying to write a Java program for generating a forecast using
exponential smoothing as described here:
https://www.itl.nist.gov/div898/handbook/pmc/section4/pmc431.htm. As
described at the linked document, exponential smoothing uses a dampening
factor "alpha". The document further goes on to say that an optimal value
for "alpha" can be found using the "Marquardt procedure", which I take as
referring to the Levenberg-Marquardt algorithm. From the linked document,
it seems that the problem of finding the optimal "alpha" is treated as a
least-squares problem and fed into the optimizer, with an initial guess for
"alpha".

After extensive web search I could not find any ready example of using the
Levenberg-Marquardt algorithm to find "alpha" for this kind of a problem,
with any programming language. So, I dug into the Javadocs and test cases
for the class *LevenbergMarquardtOptimizer* to see if I could come up with
a solution of my own. My program is given an array of values, say *[9, 8,
9, 12, 10, 12, 11, 7, 13, 9, 11, 10]*, and an initial guess for "alpha",
say *0.2*. I have been able to determine that this information needs to be
converted into a *LeastSquaresProblem*, for which I have done the following
so far:

1. Set the input array as the *target*;
2. Set the starting point *start *as the initial value of alpha (*{ 0.2
}*);
3. Set *weight* to *[1, 1 - alpha, (1 - alpha)^2, ...]*; and
4. Set the optimization function of the model to return smooth values
for each of the input values.

I am now unsure how the Jacobian should be calculated. I would like to know
if I have approached the problem correctly so far, and how to calculate the
Jacobian. I have not been able to find any material on the web or printed
form that describes the procedure for finding the Jacobian for a problem
like this.

Any help or pointers will be greatly appreciated.
j***@uni-ulm.de
2018-05-16 16:24:02 UTC
Dear Manish,

There are a few issues with your approach:

first: as far as I understand, the LevenbergMarquardtOptimizer is
designed to optimize more than one parameter. In the present case
alpha is the sole parameter to be optimized. There is no Jacobian for
the one-dimensional case.

In the web-side that you mentioned, they recommend to use a proper
starting value for the smoothed sequence as additional unknown
parameter. In your case it would be something around 9. With a second
parameter to be optimized, you could formally use the
LevenbergMarquardtOptimizer.

second: you alpha-value should be restrained to stay in the range of
0<alpha<1.0.
You can achieve this by using a fit parameter x, and in the routine
that calculates the predicted sequence you have to first convert x to
alpha using for example the formula alpha= exp(x)/(1+exp(x)).

third: your calculation for the weights does not reflect the
weight-formula they recommend on the web-side.

fourth: The jacobian is the derivative of predicted sequence values
against the parameters,i.e against alpha and the startvalue mentioned
above. However, in your case the 'weights', which correspond to the
inverse of the variance or standard deviation of the observations (aka
measurement error) also depend on alpha, and hence you do not have a
straightforward least squares problem, were the weights (or
measurement errors or observational errors) are assumed to be constant.

For your case, I would use a standard optimizer like 'Powell' or
'BobyQA' and in the 'value'-function I would calculate the 'Residual
sum of squares' based on observations, sequence- and weight formulas.

Good luck, Josef
Post by Manish Java
I am trying to write a Java program for generating a forecast using
https://www.itl.nist.gov/div898/handbook/pmc/section4/pmc431.htm. As
described at the linked document, exponential smoothing uses a dampening
factor "alpha". The document further goes on to say that an optimal value
for "alpha" can be found using the "Marquardt procedure", which I take as
referring to the Levenberg-Marquardt algorithm. From the linked document,
it seems that the problem of finding the optimal "alpha" is treated as a
least-squares problem and fed into the optimizer, with an initial guess for
"alpha".
After extensive web search I could not find any ready example of using the
Levenberg-Marquardt algorithm to find "alpha" for this kind of a problem,
with any programming language. So, I dug into the Javadocs and test cases
for the class *LevenbergMarquardtOptimizer* to see if I could come up with
a solution of my own. My program is given an array of values, say *[9, 8,
9, 12, 10, 12, 11, 7, 13, 9, 11, 10]*, and an initial guess for "alpha",
say *0.2*. I have been able to determine that this information needs to be
converted into a *LeastSquaresProblem*, for which I have done the following
1. Set the input array as the *target*;
2. Set the starting point *start *as the initial value of alpha (*{ 0.2
}*);
3. Set *weight* to *[1, 1 - alpha, (1 - alpha)^2, ...]*; and
4. Set the optimization function of the model to return smooth values
for each of the input values.
I am now unsure how the Jacobian should be calculated. I would like to know
if I have approached the problem correctly so far, and how to calculate the
Jacobian. I have not been able to find any material on the web or printed
form that describes the procedure for finding the Jacobian for a problem
like this.
Any help or pointers will be greatly appreciated.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-***@commons.apache.org
j***@uni-ulm.de
2018-05-16 16:45:56 UTC
Sorry, i got confused with the term 'weight', which in the present
case is not related to measurement errors. Please ignore my fourth
point. Nevertheless I would use one of standard optimizers.

Regards, Josef
Post by j***@uni-ulm.de
Dear Manish,
first: as far as I understand, the LevenbergMarquardtOptimizer is
designed to optimize more than one parameter. In the present case alpha
is the sole parameter to be optimized. There is no Jacobian for the
one-dimensional case.
In the web-side that you mentioned, they recommend to use a proper
starting value for the smoothed sequence as additional unknown
parameter. In your case it would be something around 9. With a second
parameter to be optimized, you could formally use the
LevenbergMarquardtOptimizer.
second: you alpha-value should be restrained to stay in the range of
0<alpha<1.0.
You can achieve this by using a fit parameter x, and in the routine
that calculates the predicted sequence you have to first convert x to
alpha using for example the formula alpha= exp(x)/(1+exp(x)).
third: your calculation for the weights does not reflect the
weight-formula they recommend on the web-side.
fourth: The jacobian is the derivative of predicted sequence values
against the parameters,i.e against alpha and the startvalue mentioned
above. However, in your case the 'weights', which correspond to the
inverse of the variance or standard deviation of the observations (aka
measurement error) also depend on alpha, and hence you do not have a
straightforward least squares problem, were the weights (or measurement
errors or observational errors) are assumed to be constant.
For your case, I would use a standard optimizer like 'Powell' or
'BobyQA' and in the 'value'-function I would calculate the 'Residual
sum of squares' based on observations, sequence- and weight formulas.
Good luck, Josef
Post by Manish Java
I am trying to write a Java program for generating a forecast using
https://www.itl.nist.gov/div898/handbook/pmc/section4/pmc431.htm. As
described at the linked document, exponential smoothing uses a dampening
factor "alpha". The document further goes on to say that an optimal value
for "alpha" can be found using the "Marquardt procedure", which I take as
referring to the Levenberg-Marquardt algorithm. From the linked document,
it seems that the problem of finding the optimal "alpha" is treated as a
least-squares problem and fed into the optimizer, with an initial guess for
"alpha".
After extensive web search I could not find any ready example of using the
Levenberg-Marquardt algorithm to find "alpha" for this kind of a problem,
with any programming language. So, I dug into the Javadocs and test cases
for the class *LevenbergMarquardtOptimizer* to see if I could come up with
a solution of my own. My program is given an array of values, say *[9, 8,
9, 12, 10, 12, 11, 7, 13, 9, 11, 10]*, and an initial guess for "alpha",
say *0.2*. I have been able to determine that this information needs to be
converted into a *LeastSquaresProblem*, for which I have done the following
1. Set the input array as the *target*;
2. Set the starting point *start *as the initial value of alpha (*{ 0.2
}*);
3. Set *weight* to *[1, 1 - alpha, (1 - alpha)^2, ...]*; and
4. Set the optimization function of the model to return smooth values
for each of the input values.
I am now unsure how the Jacobian should be calculated. I would like to know
if I have approached the problem correctly so far, and how to calculate the
Jacobian. I have not been able to find any material on the web or printed
form that describes the procedure for finding the Jacobian for a problem
like this.
Any help or pointers will be greatly appreciated.
---------------------------------------------------------------------
---------------------------------------------------------------------
To unsubscribe, e-mail: user-***@commons.apache.org