sklearn.neural_networks.MLPRegressor - unable to calculate accuracy score

Question

This is my first post on StackOverflow! I am using the MLPRegressor to generate a binary class multioutput prediction for my problem. Once I get my prediction, I round all the values using numpy.round(), so that I can use accuracy_score(since accuracy score only works for classification problems). After this, I try to use sklearn.metrics.accuracy_score when I get the following error:

ValueError: Classification metrics can't handle a mix of multilabel-indicator and multiclass-multioutput targets

This error only occurs when I manually set the max_iter keyword argument in the MLPRegressor. When I do not set it manually, the regressor does not converge but the error does not occur.

from sklearn.neural_network import MLPRegressor
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
import numpy as np
from joblib import dump, load
data = np.loadtxt('tictac_multi.txt')
X = data[:,:9]
y = data[:,9:]
X_train,X_test,y_train,y_test = train_test_split(X,y,test_size = 0.20,random_state=7)
regr = MLPRegressor(random_state=7,hidden_layer_sizes=(9,81,729,81,81,9),activation='tanh',learning_rate='invscaling',solver='adam',max_iter = 400).fit(X_train, y_train)
preds = regr.predict(X_test)
preds = np.round(preds)
print(accuracy_score(y_test,preds))

Here's the link to the dataset: http://www.connellybarnes.com/work/class/2016/deep_learning_graphics/proj1/tictac_multi.txt

Stack Trace:

Traceback (most recent call last):
  File "mlp.py", line 21, in <module>
    scores.append(accuracy_score(y_test,preds))
  File "C:\Users\animu\AppData\Local\Programs\Python\Python37\lib\site-packages\sklearn\utils\validation.py", line 73, in inner_f
    return f(**kwargs)
  File "C:\Users\animu\AppData\Local\Programs\Python\Python37\lib\site-packages\sklearn\metrics\_classification.py", line 187, in accuracy_score
    y_type, y_true, y_pred = _check_targets(y_true, y_pred)
  File "C:\Users\animu\AppData\Local\Programs\Python\Python37\lib\site-packages\sklearn\metrics\_classification.py", line 91, in _check_targets
    "and {1} targets".format(type_true, type_pred))
ValueError: Classification metrics can't handle a mix of multilabel-indicator and multiclass-multioutput targets

Answer 1

As the error message states, this happens because the

Classification metrics can't handle a mix of multilabel-indicator and multiclass-multioutput targets

What this means, is that accuracy_score() can work in a multilabel case such as yours, but not if the class labels are not binary.

You are stating, that you have a binary class multioutput prediction, but in your predictions, the preds[89] contains a value of 2, besides the binary output of 0 and 1

preds[89]

returns

array([ 0., -0., -0., -0.,  1.,  2., -0., -0., -0.])

Other entries besides 89 in your predictions array, with values that are not binary can be found in:

• preds[139]
• preds[501]
• preds[503]
• preds[770]
• preds[1039]
• preds[1107]

So you have to make sure now, that these entries (all of them have the value 2) are turned into a binary label (0 or 1) in order for accuracy_score() to work.

Possible solution:

You could replace all occurences of the target value 2 by the value 1:

for outer_index in range(preds.shape[0]):
  for index in range(preds[outer_index].shape[0]):
    if(np.abs(preds[outer_index][index]) != 0 and np.abs(preds[outer_index][index]) != 1):
      preds[outer_index][index]=1

Then you can call your accuracy_score() method:

print(accuracy_score(y_test,preds))

which returns

0.8367658276125095