Repository navigation
PMStandardScaler fails when scale = 0 #124
Description
A relevant example:
x := PMMatrix rows: #(#(1 0) #(1 0) #(1 1) #(1 1)).
scaler := PMStandardizationScaler new.
scaler fit: x.
scaler scale. "0"
scaler transform: x. "ZeroDivideError"Possible fix: add a "handleZeroScale" method which changes scale to 1 when it is 0.
Question: Can this be method be added in PMDataTransformer? If we add more scalers in future, all would need this message.
Activity
SergeStinckwich commentedon May 27, 2019
Why we should modify the scale to 1 when this is 0 ? 😄
SergeStinckwich commentedon May 27, 2019
we should have a look how this is done in scikit-learn : https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html
AtharvaKhare commentedon May 27, 2019
We are doing
(element - mean) / scale. Whenscale=0, we can't divide by zero. Since all elements are same, theelement-meanwill be 0 as it should be. So any number in the denominator will be fine.1specifically because any number divided by it is itself, but again, any number should be ok.sklearn is also setting scale to 1.
SergeStinckwich commentedon May 28, 2019
ok great if they use the same trick. I find the Python code here: https://github.com/scikit-learn/scikit-learn/blob/7813f7efb/sklearn/preprocessing/data.py#L62
SergeStinckwich commentedon May 29, 2019
Can you propose a fix, should be easy ?
AtharvaKhare commentedon May 29, 2019
One of the fix is to modify this method:
PolyMath/src/Math-PrincipalComponentAnalysis/PMStandardizationScaler.class.st
Lines 24 to 26 in aaad4b2
To calculate sqrt for every element, compare to 0, and return sqrt or 1.
Other way is to add a
handleZerosmethod, but I do not think we should go there unless we add more types of scaling.Do you want me to create PR using first suggested fix?
SergeStinckwich commentedon May 29, 2019
Maybe later will have multiple type of scalin, but at the moment, we have only one.
So keep it simple and use first method. Maybe add a test when scale is 0 that it returns 1.
[-]PMStandardScaler fails when scale = 0[/-][+]``PMStandardScaler`` fails when scale = 0[/+]