在使用astype() 将列的数据类型从 'int64' 转换为 'float64' 时,触发LossySetItemError
我一直在尝试在Python/Pandas中对数据进行归一化,但遇到了一个有点棘手的问题。归一化数据需要把每列的值除以该列的最大值,从而得到0 到1 的区间。Pandas把我的DataFrame的所有列值都设为了int64数据类型,因此这样的操作不可行。为了解决这个问题,我尝试使用下面的代码将列值转换为float64:
import pandas as pd
import numpy as np
data = pd.read_excel("file_name.xlsx", header = [10, 11]) # read in data; values are int
for i in range(0, len(data.columns), 1):
if i == 0: # the first column contains time values that are already dtype float64
pass
else:
data.iloc[:, i] = data.iloc[:, i].astype(np.float64) # this line throws an error
然而,这会抛出一个 LossySetItemError: Invalid value 'value' Name: 'column_name', length 'number', dytype: 'float64' for dtype: 'int64。
我的第一种做法是用 dtype = float64 强制Pandas将我的Excel文件中的所有数值读作float64,但这同样会报错:Unable to convert column ('First_multi_index', 'second_multi_index') to type float64。我相信这是因为我使用了多级列名(multi-indexed column names),这是为了方便处理数据而需要的。
一些Google搜索表明,使用 astype() 才是执行此转换的正确方式,所以我不知道为什么它不起作用。坦白说,我对编程并不太熟悉。如果要猜测,可能我错过了一些显而易见的东西。任何帮助将不胜感激。我也在下面提供了我正在使用的库版本:
Python = 3.13.12
Pandas = 3.0.0
Numpy = 2.4.2
以下是确切的错误输出(我已对其中的部分个人信息进行了处理):
LossySetitemError Traceback (most recent call last)
File ~\.conda\envs\stats\Lib\site-packages\pandas\core\indexing.py:2144, in _iLocIndexer._setitem_single_column(self, loc, value, plane_indexer)
2143 try:
-> 2144 self.obj._mgr.column_setitem(
2145 loc, plane_indexer, value, inplace_only=True
2146 )
2147 except (ValueError, TypeError, LossySetitemError) as exc:
2148 # If we're setting an entire column and we can't do it inplace,
2149 # then we can use value's dtype (or inferred dtype)
2150 # instead of object
File ~\.conda\envs\stats\Lib\site-packages\pandas\core\internals\managers.py:1518, in BlockManager.column_setitem(self, loc, idx, value, inplace_only)
1517 if inplace_only:
-> 1518 col_mgr.setitem_inplace(idx, value)
1519 else:
File ~\.conda\envs\stats\Lib\site-packages\pandas\core\internals\managers.py:2220, in SingleBlockManager.setitem_inplace(self, indexer, value)
2217 if isinstance(arr, np.ndarray):
2218 # Note: checking for ndarray instead of np.dtype means we exclude
2219 # dt64/td64, which do their own validation.
-> 2220 value = np_can_hold_element(arr.dtype, value)
2222 if isinstance(value, np.ndarray) and value.ndim == 1 and len(value) == 1:
2223 # NumPy 1.25 deprecation: https://github.com/numpy/numpy/pull/10615
File ~\.conda\envs\stats\Lib\site-packages\pandas\core\dtypes\cast.py:1725, in np_can_hold_element(dtype, element)
1724 # Anything other than integer we cannot hold
-> 1725 raise LossySetitemError
1726 if (
1727 dtype.kind == "u"
1728 and isinstance(element, np.ndarray)
1729 and element.dtype.kind == "i"
1730 ):
1731 # see test_where_uint64
LossySetitemError:
The above exception was the direct cause of the following exception:
TypeError Traceback (most recent call last)
File c:\users\user\onedrive - library\projects\data analysis\general data analysis program\easy_data.py:58
56 tht_norm = dc.Tht_norm(directory)
57 case 3: # non functional
---> 58 replicate = dc.ThT_rep(directory)
59 case 4:
60 cd = dc.cd_plot(directory)
File ~\OneDrive - library\Projects\Data Analysis\General Data Analysis Program\data_classes.py:189, in ThT_rep.__init__(self, directory)
187 self.data, self.headers, self.concentrations = super().determine_dataset(self.data)
188 self.replicate = self.select_rep(self.headers)
--> 189 self.data = self.norm_data(self.data)
190 self.figure = super().initialize_plot()
191 self.plot_data(self.data, self.headers, self.concentrations, self.figure, self.replicate)
File ~\OneDrive - library\Projects\Data Analysis\General Data Analysis Program\data_classes.py:217, in ThT_rep.norm_data(self, data)
215 data.iloc[:,i] = data.iloc[:,i] - minimum
216 maximum = data.iloc[:,i].max()
--> 217 data.iloc[:, i] = data.iloc[:,i] / maximum
218 break
219 elif response == 'n' or response == 'N':
File ~\.conda\envs\stats\Lib\site-packages\pandas\core\indexing.py:938, in _LocationIndexer.__setitem__(self, key, value)
933 self._has_valid_setitem_indexer(key)
935 iloc: _iLocIndexer = (
936 cast("_iLocIndexer", self) if self.name == "iloc" else self.obj.iloc
937 )
--> 938 iloc._setitem_with_indexer(indexer, value, self.name)
File ~\.conda\envs\stats\Lib\site-packages\pandas\core\indexing.py:1953, in _iLocIndexer._setitem_with_indexer(self, indexer, value, name)
1950 # align and set the values
1951 if take_split_path:
1952 # We have to operate column-wise
-> 1953 self._setitem_with_indexer_split_path(indexer, value, name)
1954 else:
1955 self._setitem_single_block(indexer, value, name)
File ~\.conda\envs\stats\Lib\site-packages\pandas\core\indexing.py:1997, in _iLocIndexer._setitem_with_indexer_split_path(self, indexer, value, name)
1993 self._setitem_with_indexer_2d_value(indexer, value)
1995 elif len(ilocs) == 1 and lplane_indexer == len(value) and not is_scalar(pi):
1996 # We are setting multiple rows in a single column.
-> 1997 self._setitem_single_column(ilocs[0], value, pi)
1999 elif len(ilocs) == 1 and 0 != lplane_indexer != len(value):
2000 # We are trying to set N values into M entries of a single
2001 # column, which is invalid for N != M
2002 # Exclude zero-len for e.g. boolean masking that is all-false
2004 if len(value) == 1 and not is_integer(info_axis):
2005 # This is a case like df.iloc[:3, [1]] = [0]
2006 # where we treat as df.iloc[:3, 1] = 0
File ~\.conda\envs\stats\Lib\site-packages\pandas\core\indexing.py:2163, in _iLocIndexer._setitem_single_column(self, loc, value, plane_indexer)
2151 dtype = self.obj.dtypes.iloc[loc]
2152 if dtype not in (np.void, object) and not self.obj.empty:
2153 # - Exclude np.void, as that is a special case for expansion.
2154 # We want to raise for
(...) 2161 # - Exclude empty initial object with enlargement,
2162 # as then there's nothing to be inconsistent with.
-> 2163 raise TypeError(
2164 f"Invalid value '{value}' for dtype '{dtype}'"
2165 ) from exc
2166 self.obj.isetitem(loc, value)
2167 else:
2168 # set value into the column (first attempting to operate inplace, then
2169 # falling back to casting if necessary)
TypeError: Invalid value '0 0.014300
1 0.008976
2 0.005148
3 0.009020
4 0.005676
1238 0.866068
1239 0.863472
1240 0.876716
1241 0.888596
1242 0.875308
Name: Sample X1, Length: 1243, dtype: float64' for dtype 'int64'
此外,这里有一段我正在处理的数据片段。它应该能够稳定地产生该错误:
| 时间 | 信号 |
|---|---|
| 0.00 | 1000.0 |
| 0.01 | 2000.0 |
| 0.02 | 3000.0 |
解决方案
错误发生的原因是你的DataFrame列是 int64。但归一化会产生介于0 和1 之间的浮点数。
Pandas不允许将浮点值放入一个 int64 列,因此会报错。
一个简单的解决方案是先把DataFrame转换成浮点数类型。
data = pd.read_excel("file_name.xlsx", header=[10,11])
data = data.astype(float)
然后归一化:
data = (data - data.min()) / (data.max() - data.min())