pandas大数据处理技巧

1.读取数据

2.查看内存占用情况

查看各种数据类型内存占用情况：

for dtype in ['float64','object','int64']:     selected_dtype = gl.select_dtypes(include=[dtype])     mean_usage_b = selected_dtype.memory_usage(deep=True).mean()     mean_usage_mb = mean_usage_b / 1024 ** 2     print("Average memory usage for {} columns: {:03.2f} MB".format(dtype,mean_usage_mb))

Average memory usage for float64 columns: 1.29 MB Average memory usage for object columns: 9.51 MB Average memory usage for int64 columns: 1.12 MB

 3.查看各种数据类型可取值的范围

import numpy as np int_types = ["uint8", "int8", "int16","int32","int64"] for it in int_types:     print(np.iinfo(it))

4.计算各种数据类型的内存占用量

def mem_usage(pandas_obj):     if isinstance(pandas_obj,pd.DataFrame):         usage_b = pandas_obj.memory_usage(deep=True).sum()     else: # we assume if not a df it's a series         usage_b = pandas_obj.memory_usage(deep=True)     usage_mb = usage_b / 1024 ** 2 # convert bytes to megabytes     return "{:03.2f} MB".format(usage_mb)  gl_int = gl.select_dtypes(include=['int64']) # http://pandas.pydata.org/pandas-docs/stable/generated/pandas.to_numeric.html converted_int = gl_int.apply(pd.to_numeric,downcast='unsigned')  print(mem_usage(gl_int)) print(mem_usage(converted_int))

7.87 MB 1.48 MB 

将dataframe进行数据转换

optimized_gl = gl.copy()  optimized_gl[converted_int.columns] = converted_int optimized_gl[converted_float.columns] = converted_float  print(mem_usage(gl)) print(mem_usage(optimized_gl))

860.50 MB 803.61 MB

将星期数据进行编码 

将整形数据转换为无符号整形 

 对重复数据<0.5的数据进行编码

converted_obj = pd.DataFrame()  for col in gl_obj.columns:     num_unique_values = len(gl_obj[col].unique())     num_total_values = len(gl_obj[col])     if num_unique_values / num_total_values < 0.5:         converted_obj.loc[:,col] = gl_obj[col].astype('category')     else:         converted_obj.loc[:,col] = gl_obj[col]   

此时的内存： 

 日期数据转换