distributed data parallel coding in spark











up vote
-1
down vote

favorite












Hi I have a master slave data parallel problem. On the master I create N decision trees and have a large data set that is too big for a single machine. I would like to copy the N trees and divide the data set between the slaves (map). Then apply the N trees to the data on each of the slaves. This would return a value which would be sent back to the master machine (reduced). The decision trees are created in a custom way so I can't use MLIB.



Could someone please point me to some literature on how this could be done. Thanks.






apache-spark pyspark apache-spark-mllib







share|improve this question




























    up vote
    -1
    down vote

    favorite












    Hi I have a master slave data parallel problem. On the master I create N decision trees and have a large data set that is too big for a single machine. I would like to copy the N trees and divide the data set between the slaves (map). Then apply the N trees to the data on each of the slaves. This would return a value which would be sent back to the master machine (reduced). The decision trees are created in a custom way so I can't use MLIB.



    Could someone please point me to some literature on how this could be done. Thanks.






    apache-spark pyspark apache-spark-mllib







    share|improve this question


























      up vote
      -1
      down vote

      favorite









      up vote
      -1
      down vote

      favorite











      Hi I have a master slave data parallel problem. On the master I create N decision trees and have a large data set that is too big for a single machine. I would like to copy the N trees and divide the data set between the slaves (map). Then apply the N trees to the data on each of the slaves. This would return a value which would be sent back to the master machine (reduced). The decision trees are created in a custom way so I can't use MLIB.



      Could someone please point me to some literature on how this could be done. Thanks.






      apache-spark pyspark apache-spark-mllib







      share|improve this question















      Hi I have a master slave data parallel problem. On the master I create N decision trees and have a large data set that is too big for a single machine. I would like to copy the N trees and divide the data set between the slaves (map). Then apply the N trees to the data on each of the slaves. This would return a value which would be sent back to the master machine (reduced). The decision trees are created in a custom way so I can't use MLIB.



      Could someone please point me to some literature on how this could be done. Thanks.






      apache-spark pyspark apache-spark-mllib




      apache-spark pyspark apache-spark-mllib






      share|improve this question















      share|improve this question













      share|improve this question




      share|improve this question








      edited 3 hours ago









      Ram Ghadiyaram

      15.4k53972




      15.4k53972










      asked 3 hours ago









      Moses S

      394




      394





























          active

          oldest

          votes











          Your Answer






          StackExchange.ifUsing("editor", function () {
          StackExchange.using("externalEditor", function () {
          StackExchange.using("snippets", function () {
          StackExchange.snippets.init();
          });
          });
          }, "code-snippets");

          StackExchange.ready(function() {
          var channelOptions = {
          tags: "".split(" "),
          id: "1"
          };
          initTagRenderer("".split(" "), "".split(" "), channelOptions);

          StackExchange.using("externalEditor", function() {
          // Have to fire editor after snippets, if snippets enabled
          if (StackExchange.settings.snippets.snippetsEnabled) {
          StackExchange.using("snippets", function() {
          createEditor();
          });
          }
          else {
          createEditor();
          }
          });

          function createEditor() {
          StackExchange.prepareEditor({
          heartbeatType: 'answer',
          convertImagesToLinks: true,
          noModals: true,
          showLowRepImageUploadWarning: true,
          reputationToPostImages: 10,
          bindNavPrevention: true,
          postfix: "",
          imageUploader: {
          brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
          contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
          allowUrls: true
          },
          onDemand: true,
          discardSelector: ".discard-answer"
          ,immediatelyShowMarkdownHelp:true
          });


          }
          });














           

          draft saved


          draft discarded


















          StackExchange.ready(
          function () {
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53400793%2fdistributed-data-parallel-coding-in-spark%23new-answer', 'question_page');
          }
          );

          Post as a guest















          Required, but never shown






























          active

          oldest

          votes













          active

          oldest

          votes









          active

          oldest

          votes






          active

          oldest

          votes
















           

          draft saved


          draft discarded



















































           


          draft saved


          draft discarded














          StackExchange.ready(
          function () {
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53400793%2fdistributed-data-parallel-coding-in-spark%23new-answer', 'question_page');
          }
          );

          Post as a guest















          Required, but never shown





















































          Required, but never shown














          Required, but never shown












          Required, but never shown







          Required, but never shown

































          Required, but never shown














          Required, but never shown












          Required, but never shown







          Required, but never shown







          -

          Popular posts from this blog

          Sphinx de Gizeh

          Image
          Pour les articles homonymes, voir Sphinx. Sphinx de Gizeh.mw-parser-output .entete.egypte-antique{background-image:url("//upload.wikimedia.org/wikipedia/commons/c/cc/Picto_infobox_ancient_Egypt.png")} Site d'Égypte antique Le visage du sphinx de Gizeh. Noms en arabe أبو الهول , Abou al-Hôl Localisation Région Basse-Égypte Coordonnées 29° 58′ 31″ nord, 31° 08′ 16″ est Géolocalisation sur la carte : Égypte Sphinx de Gizeh modifier   Le sphinx de Gizeh est la statue thérianthrope qui se dresse devant les grandes pyramides du plateau de Gizeh, en Basse-Égypte. Sculpture monumentale monolithique la plus grande du monde [ 1 ] avec 73,5 mètres de longueur, 14 mètres de largeur et 20,22 mètres de hauteur, elle représente un sphinx couchant. Réalisée vers -2500, elle serait attribuée à Djédefrê, l'un des pharaons de la IV e  dynastie, représentant son père le pharaon Khéops. Sommaire 1 Étymologie 2 Desc...
          Read more

          Dijon

          Image
          Dijon.mw-parser-output .entete.map{background-image:url("//upload.wikimedia.org/wikipedia/commons/7/7a/Picto_infobox_map.png")} De haut en bas, de gauche à droite : Panorama du centre historique vu depuis la Cathédrale Saint-Bénigne, le tramway passant sous l'Auditorium , la place François Rude, les Tuiles vernissées de l'Hôtel Aubriot, le centre commercial Toison d'or et le jardin Darcy. Blason Logo Administration Pays   France Région Bourgogne-Franche-Comté (préfecture) Département Côte-d'Or (préfecture) Arrondissement Dijon (chef-lieu) Canton bureau centralisateur de 6 cantons : Dijon-1, 2, 3, 4, 5, 6 Intercommunalité Dijon Métropole Maire Mandat François Rebsamen 2015-2020 Code postal 21000 (comportait aussi le code 21100 quand la ville possédait deux bureaux distributeurs) Code commune 21231 Démographie Gentilé ...
          Read more

          Langue

          Image
          Pour les articles homonymes, voir Langue (homonymie). Une langue est un système évolutif de signes linguistiques, vocaux, graphiques ou gestuels, qui permet la communication entre les individus. Langues du monde par familles :       Afro-asiatiques       Nigéro-congolaises       Nilo-sahariennes       Khoïsan       Indo-européennes       Caucasiennes       Altaïques       Ouraliennes       Dravidiennes       Sino-tibétaines       Austroasiatiques       Austronésiennes       Pama-nyungan       Papoues       Tai-Kadai       Amérindiennes       Na-dené       Eskimo-aléoutes       Isolat Sommaire 1 Définition : langue et langage 1.1 Langue et dialecte 1.2 Langue naturelle et langue construite 1.3 Langue vivante et langue morte 1.4 Autre 2 Histoire : origine des langues 3 Classification 3.1 Nombre de langues 3.2 Nombre de locuteurs par langue 3.3 Dénominations 4 Év...
          Read more