| CARVIEW |
?mpandas??????nDiabetes?f?[?^?Z?b?g???g???ĉ?A???͂??Ă݂悤?i?O?????ҁj?FPython?f?[?^????????
???Ђ??}??l???ʔ͈͂?p?????O??l?̌??o?A???????x???̕??z?̒????A?f?[?^?Z?b?g??Z?X?R?A?W???????s???āADiabetes?f?[?^?Z?b?g?ʼn?A???͂????鏀???????Ă????܂??B
???̋L???͉??????ł??B????o?^?i?????j????ƑS?Ă????????????܂??B
?{?V???[?Y?Ɩ{?A?ڂɂ???
?@?{?V???[?Y?uPython?f?[?^????????v?́APython?̊?b???}?X?^?[?????l??ΏۂɈȉ??̂悤?ȁAPython???g???ăf?[?^?????????悤?Ƃ????Ƃ??ɕ֗??Ɏg????c?[????C?u?????A?t???[?????[?N?̎g?????̊?b???????????̂ł??B
- NumPy?i?uNumPy??????v?̖ڎ??????????j
- pandas?i?{?A?ځj
- Matplotlib
?@?Ȃ??A?{?A?ڂł͈ȉ??̃o?[?W???????g?p???Ă??܂??B
- Python 3.12
- pandas 2.2.1
?O??̂܂Ƃ?
?@?O???́Ascikit-learn??????Diabetes?f?[?^?Z?b?g???Ƃ??āA???̊T?v?ׂ???A???W?????v?Z???ăq?[?g?}?b?v???v???b?g??????A?U?z?}???v???b?g?????肵?Ȃ???Atarget??Ƃ̑??ւ????????ȗǂꂩ???l???Ă??܂????B
?@?ȉ??͂??̃q?[?g?}?b?v?ł??B
?@???̃q?[?g?}?b?v????͈ȉ??̗?target??Ƃ̑??ւ?0.3?ȏ???????́|0.3?ȉ??ł??邱?Ƃ??m?F????܂????B
- bmi?FBMI
- bp?F????
- s3?F?P?ʃR???X?e???[??
- s4?F?q?Ƀ??X?g?????l?^?P?ʃR???X?e???[??
- s5?F???????b
- s6?F?????l
?@?O??̋L???ł́A???ł?bmi???s3???2??A???͂??s???ۂ̃??f???̓????ʂƂ??āA?\?????x?ւ̊?^?x???????\????????ƍl???܂????B???̗?ɂ??ẮA?\?????x?̌????????????ۂɒlj????邩?ۂ????l???邱?Ƃɂ??Ă??܂??B
?@?܂??A?????̗?̒l???????ɁAtarget??̒l???c???ɂ????U?z?}???v???b?g???܂????B???ꂪ?ȉ??ł??B
?@?????̎U?z?}????͊e???target??̊ԂɈ??̑??ւ??????܂??i???????As4?ɂ??Ă͂?◣?U?I?ł????j?B
?@????͌??̃f?[?^?Z?b?g???炱???̗?i??target??j???o???āA???̑???𑱂??Ă??????Ƃɂ??܂??B?Ƃ????킯?ŁA?ȉ??̃??W???[?????C???|?[?g???Ă????܂??傤?B
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from sklearn import datasets
?@?????āA?ȉ??̃R?[?h??DataFrame?I?u?W?F?N?g???쐬???Ă????܂??傤?i???̃f?[?^?Z?b?g??df_org?ɕۑ????Ă????܂??j?B
diabetes = datasets.load_diabetes()
X = diabetes.data
y = diabetes.target
feature_names = diabetes.feature_names
df_org = pd.DataFrame(X, columns=feature_names)
df = df_org.loc[:, ['bmi', 'bp', 's3', 's4', 's5', 's6']]
df['target'] = y
?@?????ł͏?Ńs?b?N?A?b?v??????i??target??j???????܂?DataFrame?I?u?W?F?N?g???쐬???A?ȉ??ł͂???????????Ƃɂ??܂??i???̃f?[?^?Z?b?g??df_org?Ƃ??????O?Ŏ???Ă????܂??j?B
?@????????͎??̂悤?ȏ????????ɂ??Ă??????Ƃɂ??܂??傤?B
- ?O??l???`?F?b?N???āADataFrame?I?u?W?F?N?g????폜????
- target??̕??z?ɕ肪?Ȃ????ǂ??????m?F????
- ?f?[?^?Z?b?g??W????????iZ?X?R?A?W?????j
?O??l???`?F?b?N???āADataFrame?I?u?W?F?N?g????폜
?@?O??l???f?[?^?Z?b?g?Ɋ܂܂?Ă???ƁA???f???̊w?K???s????ɂȂ?A?\?????x???ቺ????\????????܂??B?????ŁA?܂??͊O??l?????邩?ǂ??????`?F?b?N???āA?i???炩?̒l?ɒu????????Ȃǂ̑Ή????l?????܂????j????͒P???ɂ??????܂ލs???폜???邱?Ƃɂ??܂??B
?@?܂??͊O??l?????邩?ǂ????????????Ă݂܂??傤?B
features = df.drop('target', axis=1) # ???Ђ??}??target??ȊO??`??
plt.figure(figsize=(12, 6))
plt.boxplot(features.values, tick_labels=features.columns)
plt.title("Boxplot of Features")
plt.xlabel("Features")
plt.ylabel("Values")
plt.show()
?@?????Visual Studio Code?Ŏ??s???????ʂ??ȉ??ł??B
?@???Ђ??}?ł͎l???ʔ͈́i??1?l???ʐ???Q1?????3?l???ʐ???Q3?̊ԁj????`?s???????t?i???????`?j?Ƃ??ĕ`????A???̏㉺?[????͎l???ʔ͈́iQ3?|Q1?j??1.5?{???????????㉺?ɐL???ĕE?i?Ђ??j?̂悤?ɕ`?悵?Ă??܂??B???͈̔͂???O?ꂽ?l???O??l?Ƃ??Č??Ȃ??̂???ʓI?ł??B???̌??ʂ?????ƁAbp????????S?Ă̓????ʂŊO??l?????݂??Ă???悤?ł??B
?@target??ɂ??Ă????????Ƃ????Ă݂܂??itarget??ɂ??Ă̓X?P?[?????O????Ă??Ȃ????Ƃ???A?c???̒l???????ʂ̗?Ƃ͑傫???قȂ??Ă??邽?߁A?ʈ????Ƃ??Ă??܂??j?B
plt.boxplot(df['target'], tick_labels=['target'])
plt.show()
?@???̌??ʂ͈ȉ??̂悤?ɂȂ?܂????B
?@target??ɂ͊O??l???m?F????܂???ł??????A?e?????ʂɊO??l?????邱?Ƃ͎??o?I?Ɋm?F?ł??܂????B?????ŁA?O??l???܂ލs?????ō\???????DataFrame?I?u?W?F?N?g???쐬???Ă݂܂??B?ȉ??͂??̂??߂̃R?[?h?ł??B
Q1 = df.quantile(0.25)
Q3 = df.quantile(0.75)
IQR = Q3 - Q1
lower_bound = Q1 - 1.5 * IQR
upper_bound = Q3 + 1.5 * IQR
mask = (df < lower_bound) | (df > upper_bound)
outliers = df[mask.any(axis=1)]
print(f'outliers: {outliers.shape[0]} rows')
outliers
?@?????ł͑?1?l???ʐ??iQ1?j?Ƒ?3?l???ʐ??iQ3?j?A?l???ʔ͈́iIQR?j?????߂āA????????O??l???ǂ????肷?邵?????l?Ƃ???lower_bound?Aupper_bound???v?Z???Ă??܂??B
?@???̐}?̂悤??IQR?̓??e???o?͂??Ă݂܂??傤?B?????͒P?Ƃ̒l?i?X?J???[?l?j?ł͂Ȃ??A?Ƃɓ???̒l?ɂȂ??Ă??邱?Ƃɂ͒??ӂ??Ă????????B
IQR?͗Ƃ̎l???ʔ͈͂??܂?ł????@?????āA?????̂??????l???????????l?A???????͏???̂??????l?????傫???l???܂ލs???????}?X?N???umask = (df < lower_bound) | (df > upper_bound)?v?̂悤?ɂ??Čv?Z???Ă??܂??B???̐}?̂悤??mask?̓??e???o?͂??Ă݂܂??傤?B
s3???441?s?ڂ?True?ɂȂ??Ă???_?ɒ????@???ɂ??̃}?X?N???g???āudf[mask.any(axis=1)]?v?Ƃ??Ă??܂??B????́Aany???\?b?h?Łu?????ꂩ?̗?Ń}?X?N?̒l??True?ƂȂ??Ă???s?v??DataFrame?I?u?W?F?N?g??????o?????Ƃ??Ӗ????Ă??܂??B
?@???s???ʂ??ȉ??Ɏ????܂??B
?@???łɖ`???Ō??Ă??????????U?z?}?ŊO??l???ԂŃv???b?g?????悤?ɂ??Ă݂܂????B
cols = ['bmi', 'bp', 's3', 's4', 's5', 's6']
fig, axes = plt.subplots(nrows=2, ncols=3, figsize=(15, 10))
axes = axes.flatten()
for n, col in enumerate(cols):
ax = axes[n]
ax.scatter(df.loc[~mask[col], col], df.loc[~mask[col], 'target'], color='blue')
ax.scatter(df.loc[mask[col], col], df.loc[mask[col], 'target'], color='red')
ax.set_title(col)
ax.set_xlabel(col)
ax.set_ylabel('target')
plt.show()
?@???s???ʂ??ȉ??Ɏ????܂??B
?@???Ђ??}?ł͊O??l??bmi???s3?`s6??Ɍ????܂????B?U?z?}?ł????̂悤?ɂȂ??Ă??邱?Ƃ??m?F?ł??܂????B
?@?ł́A?O??l???܂ލs???폜???܂??傤?B????ɂ͊???̂?????????܂??B?ȉ??͐?قǍ쐬????mask???g?????̂ł??B
df_cleaned = df[~mask.any(axis=1)]
print(f'cleaned DataFrame: {df_cleaned.shape[0]} lines')
?@??قǂ́umask.any(axis=1)?v?Ƃ??āA?O??l???܂ލs?????o???Ă??܂????B???x?́u~mask.any(axis=1)?v?Ƃ??āA???̐^?U?l?]???Ă??܂??B????ɂ??A?O??l???܂܂Ȃ??s?????????o???悤?ɂ??Ă??܂??B
?@????1?̂??????Љ?܂??傤?B???????DataFrame?I?u?W?F?N?g??drop???\?b?h???g???܂??B
df_cleaned_2 = df.drop(outliers.index)
print(f'cleaned DataFrame: {df_cleaned_2.shape[0]} lines')
?@outliers?Ƃ???DataFrame?I?u?W?F?N?g?ɂ́A?O??l???܂ލs???????i?[????Ă??܂??Bdrop???\?b?h?ɂ????̃C???f?b?N?X??^???邱?ƂŁA????DataFrame?I?u?W?F?N?g???炻?????폜?????A?V????DataFrame?I?u?W?F?N?g???쐬???Ă??܂??B
?@?ǂ??????????̂ł??傤???H?@?Ɩ???Ă??M?҂ɂ͔??f?????܂???B?O?҂̓}?X?N???g???Ă??܂????A?u?^?U?l?]???Ă???Ƃ??낪??????ɂ????v?Ƃ??u?O??l???폜???遁?O??l?łȂ????̂?I??????_????????ɂ????v?Ƃ????l?????邩??????܂???B??҂͍폜????Ώۂ??_?C???N?g?Ɏw?肵?Ă???Ƃ????_?ł͕?????₷???Ƃ?????ł??傤?B?܂??A?ォ??O??l?ɂ??Č??????????ꍇ?ɂ́A???????܂?DataFrame?I?u?W?F?N?g???쐬????ł??傤???炻?̊??p?Ƃ??????܂??B
?@?Ƃ??????ƂŁA?ȍ~?ł͊O??l???폜????DataFrame?I?u?W?F?N?g?ɑ??đ??삵?Ă??????Ƃɂ??܂??B
df = df_cleaned
target??̕??z?ɕ肪?Ȃ????ǂ??????`?F?b?N
?@????target??̕??z?ɕ肪?Ȃ????ǂ??????m?F???܂??傤?B?^?[?Q?b?g?????K???z?ɏ]???Ă??Ȃ??Ă???A???͓͂K?p?\?ł????A???z?̕肪?Ȃ????ǂ??????m?F???Ă????̂͗L?p?ł??B?????ŁAtarget??̃q?X?g?O???????v???b?g???Ď??o?I?Ɋm?F???Ă݂܂??傤?B
plt.hist(df["target"], bins=20, edgecolor="black")
plt.title("Distribution of target")
plt.xlabel("Target Value")
plt.ylabel("Frequency")
plt.show()
?@???s???ʂ͈ȉ??̂悤?ɂȂ?܂????B
?@?ꌩ?????Ƃ???ł́Atarget??̕??z?͐??K???z?Ƃ͈قȂ?`?????Ă??܂??B?????ŁA???l?I?ɂǂ̒??x?̕肪???邩???m?F???Ă݂܂??傤?B????ɂ?DataFrame.skew???\?b?h??DataFrame.kurtosis???\?b?h???g???܂??B?O?҂̓f?[?^???z?̘c?x?i?킢?ǁAskewness?j?ׂ邽?߂Ɏg???A??҂̓f?[?^?̐?x?i????ǁAkurtosis?j?ׂ邽?߂Ɏg???܂??B
?@?c?x?Ɛ?x?͂Ƃ??Ƀf?[?^?Z?b?g?̕??z?̕??`????????w?W?Ƃ??Ďg???܂??B?????̂????A?c?x?͕??z?ɑ??鍶?E?̔?Ώ̐??i???䂪?݁j??\???܂??B?Ⴆ?ΐ??K???z?̂悤?ɍ??E?Ώ̂ł???i???o?I?ɂ͍??E?ɓ????????삪?L?????Ă???j?Askew???\?b?h?̖߂?l??0?i?ɋ߂??l?j?ɂȂ?܂??B?E?ɐ??삪?L?????Ă??镪?z?ł???Ζ߂?l?͐????ɁA???ɐ??삪?L?????Ă???悤?ȕ??z?ł???Ζ߂?l?͕????ɂȂ?܂??B
?@??x?͕??z?̂Ƃ??????????܂??B??l??0?̏ꍇ?A???z?̌`???K???z?ɋ߂??\????????܂??i??l??3?Ƃ?????߂?????܂??Bpandas??DataFrame.kurtosis???\?b?h?͐??K???z?ɑ??Ă?0??Ԃ??܂??j?B??????傫?Ȓl?ł?????Ƃ????????z?ɂȂ??Ă??邱?Ƃ??????A?????菬???ȕ??z?ł͕??ׂ????????z?ł??邱?Ƃ??????܂??B
?@?????Ă݂܂??傤?B
print('skewness:', df['target'].skew())
print('kurtosis:', df['target'].kurt())
?@???s???ʂ??ȉ??Ɏ????܂??B
?c?x??0.435?͉E?ɐ??삪?L???邱?Ƃ??A??x??-0.88?͂???قǐ???Ă??Ȃ????Ƃ??????Ă????@?c?x??0.435???x?ƂȂ?܂????B????͉E???ɐ??삪?L????悤?ȕ??z?ł??邱?Ƃ??????Ă??܂??i??قǂ̉摜??????Ƃ??̂悤?ɂȂ??Ă??܂??ˁj?B??x??-0.876???x?ł??B????́A?ǂ??炩?Ƃ????Ε??ׂ????????z?ɂȂ??Ă??邱?Ƃ??????Ă??܂??B
?@?c?x?Ɛ?x?͕????????̂ł????A????target??̕??z??肪???Ȃ????̂ƍl???Ă??̂܂g???Ă??悢???A????Ƃ????炩?̏C?????K?v???ł??B??ʓI?ɂ͘c?x????x???u???̐?Βl??0.5?ȓ??ł???Ε肪???????v?ƌ??Ȃ??܂??B?u???̐?Βl??0.5?`1.0?͈̔͂ł???A?K?v?ɉ????ďC????????????v???ƂɂȂ?܂??B????ȏ?ł???ΏC?????K?v?ɂȂ?\?????????ł??B
?@?c?x??0.435?Ƃ????l?͕肪???????ƌ??Ȃ??????ł??B??x??-0.876?Ƃ????l?ɂ??Ă͏ɂ???Ă͏C?????K?v?ƂȂ?܂??B???A????͂??̂܂܂ɂ??Ă??????Ƃɂ??܂????B
????0?A?W????1?ƂȂ?悤?ɃX?P?[?????O
?@?O??ɂ??q?ׂ܂??????ADiabetes?f?[?^?Z?b?g?ł́u???ς?0?A???[?N???b?h?m?????iL2?m?????j??1?A?܂?sum(x2)??1?j?ɐ??K?????ꂽ?l?ƂȂ??āv???܂??B?????ł́A??????u????0?A?W????1?v?ƂȂ?悤?ɃX?P?[?????O???Ă݂܂??i???̂悤?ȃX?P?[?????O?̂??Ƃ?Z?X?R?A?W?????ȂǂƌĂԂ??Ƃ?????܂??j?B
?@?????ł͈ȉ??̕??@?ł???????߂܂??B
- ?e??̕W?????????߂āA?e?l??W?????ŏ??Z????
?@?ڍׂ́uAI?E?@?B?w?K?̗p?ꎫ?T?v?́u???K???^?W?????Ƃ́H?v???Q?Ƃ??Ă??????????̂Ƃ??āA?ȉ??̎??????Ă????????B
?@?????ŁAx'?̓X?P?[?????O??̊e??̒l?Ax?̓X?P?[?????O?O?̊e??̒l?A?n?b?g?t????x?͕??ϒl?Aσ?͕W??????\???Ă??܂??B?v????ɁA?e??̊e?l?ƕ??ϒl?̍?????W?????ŏ??Z????A???ϒl??0?ŕW??????1?̃f?[?^?Z?b?g?ɕW?????ł???Ƃ??????Ƃł??B
?@?Ƃ????킯?ŁA???q?ׂ??悤?Ɂu?e??̊e?l?ƕ??ϒl?̍???W?????ŏ??Z????v???ƂŁA???ϒl??0?A?W??????1?ƂȂ?悤?ɂ??Ă݂܂??傤?B???̑O?Ɍ??݂̕W???????m?F???Ă????܂??B
print(df_cleaned.std(ddof=0))
?@???s???ʂ??ȉ??Ɏ????܂??B
?W??????1?ł͂Ȃ????Ƃ??m?F?@????0?A?W????1?̃f?[?^?Z?b?g?ɂȂ?悤?ɃX?P?[?????O????R?[?h???ȉ??Ɏ????܂??i?????ł?target????X?P?[?????O?̑Ώۂ???O???Ă??܂??j?B
df_standardized = df.copy()
df_standardized = df_standardized.drop(['target'], axis=1)
std = df_standardized.std(ddof=0)
df_standardized = (df_standardized - df_standardized.mean()) / std
?@????ŕW??????1?ɂȂ??????ǂ??????m?F???܂??B
print(df_standardized.std(ddof=0))
?@???s???ʂ??ȉ??Ɏ????܂??B
?W??????1?ɂȂ????@?m???ɕW??????1?ɂȂ?܂????B
?@????͊O??l?̔????ƍ폜?Atarget??̕??z?ɕ肪?Ȃ????ǂ????̊m?F?AZ?X?R?A?W?????Ƃ??????????????Ă??܂????B????́A???̃f?[?^?Z?b?g???g???ĉ?A???͂??s???A???̕]???????Ă݂܂??B
?@?Ō?ɂ??낢??ƏC???????????f?[?^?Z?b?g??CSV?t?@?C???ɕۑ????Ă????܂??傤?i?W?????̑O?ɗ??Ƃ??Ă???target??𑫂??Ă??܂??j?B
df_standardized['target'] = df['target']
df.to_csv('diabetes.csv', index=False)
AI?E?f?[?^?T?C?G???X?̊w?т?????????
???S?Ҍ????A?f?[?^???́EAI?E?@?B?w?K?EPython?̕????@?@??IT??Deep Insider?Ŋw?ڂ?
Copyright© Digital Advantage Corp. All Rights Reserved.
?A?C?e?B???f?B?A????̂??m?点
??IT eBook
???ڂ̃e?[?}
?ҏW??????̂??m?点
RSS?ɂ???
?A?C?e?B???f?B?AID?ɂ???
???[???}?K?W???o?^
??IT?̃??[???}?K?W???́A ???????A???ׂĖ????ł??B???Ѓ??[???}?K?W???????w?ǂ????????B
ITmedia?̓A?C?e?B???f?B?A??????Ђ̓o?^???W?ł??B
???f?B?A?ꗗ | ????SNS | ?L???ē? | ???₢???킹 | ?v???C?o?V?[?|???V?[ | RSS | ?^?c??? | ?̗p??? | ??????