CARVIEW |
Select Language
HTTP/2 200
date: Thu, 24 Jul 2025 21:13:32 GMT
content-type: text/html; charset=utf-8
vary: X-PJAX, X-PJAX-Container, Turbo-Visit, Turbo-Frame, X-Requested-With,Accept-Encoding, Accept, X-Requested-With
etag: W/"a86fa072760106655111b71fd3b6fe1a"
cache-control: max-age=0, private, must-revalidate
strict-transport-security: max-age=31536000; includeSubdomains; preload
x-frame-options: deny
x-content-type-options: nosniff
x-xss-protection: 0
referrer-policy: origin-when-cross-origin, strict-origin-when-cross-origin
content-security-policy: default-src 'none'; base-uri 'self'; child-src github.githubassets.com github.com/assets-cdn/worker/ github.com/assets/ gist.github.com/assets-cdn/worker/; connect-src 'self' uploads.github.com www.githubstatus.com collector.github.com raw.githubusercontent.com api.github.com github-cloud.s3.amazonaws.com github-production-repository-file-5c1aeb.s3.amazonaws.com github-production-upload-manifest-file-7fdce7.s3.amazonaws.com github-production-user-asset-6210df.s3.amazonaws.com *.rel.tunnels.api.visualstudio.com wss://*.rel.tunnels.api.visualstudio.com objects-origin.githubusercontent.com copilot-proxy.githubusercontent.com proxy.individual.githubcopilot.com proxy.business.githubcopilot.com proxy.enterprise.githubcopilot.com *.actions.githubusercontent.com wss://*.actions.githubusercontent.com productionresultssa0.blob.core.windows.net/ productionresultssa1.blob.core.windows.net/ productionresultssa2.blob.core.windows.net/ productionresultssa3.blob.core.windows.net/ productionresultssa4.blob.core.windows.net/ productionresultssa5.blob.core.windows.net/ productionresultssa6.blob.core.windows.net/ productionresultssa7.blob.core.windows.net/ productionresultssa8.blob.core.windows.net/ productionresultssa9.blob.core.windows.net/ productionresultssa10.blob.core.windows.net/ productionresultssa11.blob.core.windows.net/ productionresultssa12.blob.core.windows.net/ productionresultssa13.blob.core.windows.net/ productionresultssa14.blob.core.windows.net/ productionresultssa15.blob.core.windows.net/ productionresultssa16.blob.core.windows.net/ productionresultssa17.blob.core.windows.net/ productionresultssa18.blob.core.windows.net/ productionresultssa19.blob.core.windows.net/ github-production-repository-image-32fea6.s3.amazonaws.com github-production-release-asset-2e65be.s3.amazonaws.com insights.github.com wss://alive.github.com api.githubcopilot.com api.individual.githubcopilot.com api.business.githubcopilot.com api.enterprise.githubcopilot.com; font-src github.githubassets.com; form-action 'self' github.com gist.github.com copilot-workspace.githubnext.com objects-origin.githubusercontent.com; frame-ancestors 'none'; frame-src viewscreen.githubusercontent.com notebooks.githubusercontent.com; img-src 'self' data: blob: github.githubassets.com media.githubusercontent.com camo.githubusercontent.com identicons.github.com avatars.githubusercontent.com private-avatars.githubusercontent.com github-cloud.s3.amazonaws.com objects.githubusercontent.com release-assets.githubusercontent.com secured-user-images.githubusercontent.com/ user-images.githubusercontent.com/ private-user-images.githubusercontent.com opengraph.githubassets.com copilotprodattachments.blob.core.windows.net/github-production-copilot-attachments/ github-production-user-asset-6210df.s3.amazonaws.com customer-stories-feed.github.com spotlights-feed.github.com objects-origin.githubusercontent.com *.githubusercontent.com; manifest-src 'self'; media-src github.com user-images.githubusercontent.com/ secured-user-images.githubusercontent.com/ private-user-images.githubusercontent.com github-production-user-asset-6210df.s3.amazonaws.com gist.github.com; script-src github.githubassets.com; style-src 'unsafe-inline' github.githubassets.com; upgrade-insecure-requests; worker-src github.githubassets.com github.com/assets-cdn/worker/ github.com/assets/ gist.github.com/assets-cdn/worker/
server: github.com
content-encoding: gzip
accept-ranges: bytes
set-cookie: _gh_sess=t9YtuF9b5Ot6g48Q2EZP5L3UyXMJaL3p0TNjcWT3RFBcIOX3Myu28Xw2L6BuTeXNLNDKh4BUz0hFLvH7oHB%2BAIGdJp8IZBQfc8ZEBwaKBuaS2rJowb5QTuRMJDsumxkf63tJsHPsPWYiOHwOYKSRs%2Buhgn1RmuCMiZqO5G7z8LeptFTWcS2yr0cTf8MC%2BK7xCs4klwabp3mQdp4MpRa0pyfwiEXYWhgDiJkV1FlvpmfvlpPQdsWZJSIq24NSk2mXTbd%2BVjSwHMjt3RsDWJByvQ%3D%3D--oTH%2BN0V2NtCj1KhI--%2FIZcfXtWS%2BLnNqkEZ0MeTg%3D%3D; Path=/; HttpOnly; Secure; SameSite=Lax
set-cookie: _octo=GH1.1.1024905945.1753391611; Path=/; Domain=github.com; Expires=Fri, 24 Jul 2026 21:13:31 GMT; Secure; SameSite=Lax
set-cookie: logged_in=no; Path=/; Domain=github.com; Expires=Fri, 24 Jul 2026 21:13:31 GMT; HttpOnly; Secure; SameSite=Lax
x-github-request-id: ADEC:3AC893:7EF72:AED3A:6882A1FB
A simple example of how to integrate the Spark parallel computing framework and the scikit-learn machine learning toolkit. This script randomly generates test and train data sets, trains an ensemble of decision trees using boosting, and applies the ensemble to the test set. The ensemble training is done in parallel. · GitHub
Show Gist options
Save wpm/6454814 to your computer and use it in GitHub Desktop.
{{ message }}
Instantly share code, notes, and snippets.
Last active
December 3, 2018 02:56
-
Star
32
(32)
You must be signed in to star a gist -
Fork
10
(10)
You must be signed in to fork a gist
-
Save wpm/6454814 to your computer and use it in GitHub Desktop.
A simple example of how to integrate the Spark parallel computing framework and the scikit-learn machine learning toolkit. This script randomly generates test and train data sets, trains an ensemble of decision trees using boosting, and applies the ensemble to the test set. The ensemble training is done in parallel.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
from pyspark import SparkContext | |
import numpy as np | |
from sklearn.cross_validation import train_test_split, Bootstrap | |
from sklearn.datasets import make_classification | |
from sklearn.metrics import accuracy_score | |
from sklearn.tree import DecisionTreeClassifier | |
def run(sc): | |
def zero_matrix(n, m): | |
return np.zeros(n*m, dtype = int).reshape(n, m) | |
def vote_increment(y_est): | |
increment = zero_matrix(y_est.size, n_ys) | |
increment[np.arange(y_est.size), y_est] = 1 | |
return increment # test point x class matrix with 1s marking the estimator prediction | |
X, y = make_classification() | |
X_train, X_test, y_train, y_test = train_test_split(X, y) | |
n_test = X_test.shape[0] | |
n_ys = np.unique(y_train).size | |
model = DecisionTreeClassifier() | |
# Partition the training data into random sub-samples with replacement. | |
samples = sc.parallelize(Bootstrap(y.size)) | |
# Train a model for each sub-sample and apply it to the test data. | |
vote_tally = samples.map(lambda (index, _): | |
model.fit(X[index], y[index]).predict(X_test) | |
).map(vote_increment).fold(zero_matrix(n_test, n_ys), np.add) # Take the learner majority vote. | |
y_estimate_vote = np.argmax(vote_tally, axis = 1) | |
return accuracy_score(y_test, y_estimate_vote) | |
if __name__ == '__main__': | |
print run(SparkContext("local", "Boost")) |
Its bagging, though,
Nevertheless, this code is awesome
cross_validation.Bootstrap is deprecated. cross_validation.KFold or cross_validation.ShuffleSplit are recommended instead.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
You can’t perform that action at this time.
It looks more like bagging than boosting to me... Am I missing something here?