如何计算polyfit对数据的拟合线的误差部分？

Question

我对 polyfit 的错误部分感到困惑。我有以下代码：

def polyfit(df,columns, degree):
    coef=[]
    error=[]
    x = np.array(list(range(0,df.shape[0])))
    for skill in columns:
        y=df[skill]
        y=pd.to_numeric(y)
        coeffs = numpy.polyfit(x, y, degree)

        # Polynomial Coefficients
        coef.append(coeffs.tolist()[0])

        # r-squared
        p = numpy.poly1d(coeffs)
        # fit values, and mean
        yhat = p(x)                         # or [p(z) for z in x]
        ybar = numpy.sum(y)/len(y)          # or sum(y)/len(y)
        ssreg = numpy.sum((yhat-ybar)**2)   # or sum([ (yihat - ybar)**2 for yihat in yhat])
        sstot = numpy.sum((y - yhat)**2)    # or sum([ (yi - ybar)**2 for yi in y])
        error.append(ssreg / sstot)
        
    results = pd.DataFrame({'skills':columns, 'coef': coef, 'error':error, 'error2':sstot})
    return results

其中 df 的样本是：

new_list    administrative coordination administrative law  administrative support
0   0.0465116   0.0232558   0.0581395
1   0.0714286   0   0.0285714
2   0.0210526   0   0.0421053
3   0.0288462   0.00961538  0.0961538
4   0.0714286   0.0238095   0.107143
5   0.00952381  0   0.0666667
6   0.0285714   0.00952381  0.0666667
7   0.0428571   0   0.0428571
8   0.111111    0.0277778   0.138889
9   0   0.0136986   0.0273973

结果如下：

polyfit(df,['administrative coordination',  'administrative law',   'administrative support'], 1)

skills  coef    error   error2
0   administrative coordination -0.000573   0.002681    0.011538
1   administrative law  0.000511    0.020165    0.011538
2   administrative support  0.002245    0.036025    0.011538

但是为什么所有列的 error2 都相同？我在计算误差部分时犯了错误？我想找到错误最小的列。 我所说的错误是指拟合线到数据点的最小距离。

Answer 1

您的变量

sstot

是一个标量，将在循环的每次迭代中重置。这意味着当您运行以下行时：

results = pd.DataFrame({'skills':columns, 'coef': coef, 'error':error, 'error2':sstot}) 

列

error2

将在 for 循环的最近一次迭代中设置为 sstot 的标量值，这就是 error2 具有相同值的原因。

我猜测您的意思是跟踪每个

skill

的 sstot，因此您可以创建一个名为

error2, then set the column 

error2

equal to this list (like the lists you created for

coef

and

error` 的列表。例如：

def polyfit(df,columns, degree):
    coef=[]
    error=[]
    error2=[]
    x = np.array(list(range(0,df.shape[0])))
    for skill in columns:
        y=df[skill]
        y=pd.to_numeric(y)
        coeffs = np.polyfit(x, y, degree)

        # Polynomial Coefficients
        coef.append(coeffs.tolist()[0])

        # r-squared
        p = np.poly1d(coeffs)

        # fit values, and mean
        yhat = p(x)                         # or [p(z) for z in x]
        ybar = np.sum(y)/len(y)          # or sum(y)/len(y)
        ssreg = np.sum((yhat-ybar)**2)   # or sum([ (yihat - ybar)**2 for yihat in yhat])
        sstot = np.sum((y - yhat)**2)    # or sum([ (yi - ybar)**2 for yi in y])
        error.append(ssreg / sstot)
        error2.append(sstot)

    results = pd.DataFrame({'skills':columns, 'coef': coef, 'error':error, 'error2':error2})
    return results

使用样品的结果

df

:

>>> polyfit(df,['administrative coordination',  'administrative law',   'administrative support'], 1)
                        skills      coef     error    error2
0  administrative coordination -0.000573  0.002681  0.010100
1           administrative law  0.000511  0.020165  0.001069
2       administrative support  0.002245  0.036025  0.011538