## Copyright (C) 2004-2007 Javier Fernández Baldomero, Mancia Anguita ## ## This program is free software ## ## You should have received a copy of the GNU General Public ## License along with MPITB/Octave; see the file COPYING. If not, ## write to the Free Software Foundation, Inc., 51 Franklin Street, ## Fifth Floor, Boston, MA 02110-1301, USA. ## ## (or see URL http://www.fsf.org/licensing/licenses/gpl.html) *************** * MPI TOOLBOX * *************** OCT bindings for LAM 7.1.3 calls (MPI 1.2 - MPI 2.0) under Octave 2.9.10 Linux (tested with Kernel 2.6.20, Fedora Core 6) LAM/MPI License also included f/reference, see file LAM_license.txt (or see http://www.lam-mpi.org/community/license.php) *********** * INSTALL * *********** The preferred setup is a user's own local copy of LAM & MPITB, since - usually not many cluster users will use MPITB (usually just one) - MPITB requires a dynamically linked LAM, which _was_ not usual rpm packages used to be configured for only static libmpi.a recently, FC6 comes with a dynlibs lam-libs rpm package - you will not depend on lazy sysadmins who do not care I have seen clusters where LAM existed only in the headed node or where the headed-node LAM was newer than the headless one that can cause difficult-to-diagnose malfunction BTW, your octave must support DLDs, since the whole MPITB consists of DLD (.oct) files. Type "octave_config_info" at the Octave prompt and look for the "DLD" flag. ------------------- - Install summary - ------------------- % cd ~/octave # choose normal ubication % tar zxvf .tgz # Decompress and untar % cd mpitb # enter the new subdir % cat BASHRC_stub # study the suggested .bashrc stubs # configure your account so that both Octave & LAM/MPI # are correctly included in PATH and LD_LIBRARY_PATH # cyclelogin (logout/login) for changes to take effect % rm DLD # ignore the pre-compiled version % mkdir DLD # most probably your setup won't match % cd src # rebuild MPITB for your system % make # this will generate a new DLD subdir The "make" procedure assumes your mkoctfile/octave-config and laminfo/mpicc commands are correctly configured and working, and uses them to deduce the libraries and setup required to recompile MPITB. See Makefile.env for details on: # # octave-config --print INCLUDEDIR OCTINCLUDEDIR # octave-config --print OCTLIBDIR # # mkoctfile --print INCFLAGS CXXFLAGS CXXPICFLAG XTRA_CXXFLAGS # mkoctfile --print DL_LDFLAGS LFLAGS # mkoctfile --print LIBS FLIBS OCTAVE_LIBS BLAS_LIBS FFTW_LIBS # # laminfo -path incdir -parsable | cut -d: -f3 # laminfo -path libdir -parsable | cut -d: -f3 # # mpiCC --showme:compile # mpiCC --showme:link -------------------------------- - Configuring Octave's startup - -------------------------------- MPITB comes with an .octaverc file linked to startups/startup_MPITB.m. If you run octave from ~/octave/mpitb, it will correctly source the provided startup run-command .octaverc. -------------------------------- % cd ~/octave/mpitb % ls -la .octaverc # make sure the link is there % octave ... Set SSI rpi to tcp with the command: putenv('LAM_MPI_SSI_rpi','tcp'), MPI_Init Help on MPI: help mpi octave:1> -------------------------------- If you run octave from somewhere else, you can simply create a similar link there. For instance, if you run octave from ~/octave, simply do: -------------------------------- % cd ~/octave % ln -s mpitb/startups/startup_MPITB.m .octaverc % ls -la .octaverc # make sure you did it right -------------------------------- If you already had a .octaverc which you must keep, source the provided startup code from yours, like this: -------------------------------- % cd ~/octave % ls -la .octaverc # make sure you have one % ln -s mpitb/startups/startup_MPITB.m . # link it here, same name % vi .octaverc # add these lines to yours % MPI Toolbox startup M-file, if it exists. if exist('startup_MPITB.m','file') startup_MPITB end -------------------------------- Now run octave. If everything is right you will read these help lines about MPI: -------------------------------- % octave ... Set SSI rpi to tcp with the command: putenv('LAM_MPI_SSI_rpi','tcp'), MPI_Init Help on MPI: help mpi octave:1> -------------------------------- ----------------------------------- - Configuring users' shell script - ----------------------------------- The provided run-command startup_MPITB.m uses an environment variable ($MPITB_HOME) to discover where MPITB has been installed. If it is not defined, it tries the suggested ~/octave/mpitb location, so if you install MPITB there, you will not need any .bashrc customization (apart from the previous LAM and Octave setup you might have). Please see startup_MPITB.m for details. If you choose a different location, or want to set MPITB_HOME anyways, you can edit your shell-run-command to include the definition. MPITB comes with a BASHRC_stub file with suggestions on how you could setup your .bashrc file, just in case LAM/MPI and/or Octave were not already included in your PATH and LD_LIBRARY_PATH. tcsh users should remember their syntax is different # export MPITB_HOME=~/octave/mpitb # for bash # setenv MPITB_HOME ~/octave/mpitb # for tcsh --------------------- - About recompiling - --------------------- Notice that several precompiled versions might be included in the compressed tarball, re-made in separate subdirs customarily named DLD---gcc-. If some precompiled version matches your setup, you might want to relink DLD there and give it a try, saving you from recompiling: -------------------------------- % cd ~/octave/mpitb % rm DLD # ignore suggested version % ln -s DLD-2.1.73-7.1.2-gcc3.2-i386 DLD # choose preferred one % ls -lad DLD* # make sure you did it right % octave # .octaverc will stick to it Set SSI rpi to tcp with the command: putenv('LAM_MPI_SSI_rpi','tcp'), MPI_Init Help on MPI: help mpi octave:1> MPI_Initialized # working? ans = 0 # great! octave:2> -------------------------------- If no precompiled version works in your setup, you must recompile. The Makefile is in $MPITB_HOME/src. You simply cd there and type "make". It recompiles the .cc files found there to .oct files in $MPITB_HOME/DLD. It creates the DLD subdir if required. Existence of the DLD subdir is an "order-only" prerequisite (see GNU Make manual, sect.4.3 "Types of prerequisites"), and as such it is added after | in the prerequisite list: -------------------------------- #$(INSTDLDS): $(INSTDIR)/%: % $(INSTDIR) $(INSTDLDS): $(INSTDIR)/%: % | $(INSTDIR) -------------------------------- Some users have remarked that | is a too recent feature of GNU Make and their make fails to build MPITB. The older-style, normal prerequisite version is commented above the newer order-only syntax. Remove the newer line and uncomment the older if your make won't support order-only prerequisites. Thanks to Dennis van Dok for pointing this out. Michael Creel and Thomas Weber also contributed to previous versions of the Makefile. Breanndán Ó Nualláin also helped solving bugs in the sources (see acknowledgement in MPI_Unpack.cc). As told above, the "make" procedure assumes your mkoctfile/octave- config and laminfo/mpicc commands are correctly configured and working. You might need to edit/fix your .bashrc run-command as suggested in BASHRC_stub and logout/login for changes to take effect, so that all those commands work correctly before invoking make. -------------------------------------- - LAM default hostfile (boot schema) - -------------------------------------- You need a bhost file to boot a LAM (Local Area Multicomputer). Recent LAM versions will pick a ./lam-bhost.def file if it exists in your current directory, instead of just relying on a $LAMBHOST environment variable pointing to the file. This greatly simplifies configuration, since most users will never use $LAMBHOST. Simply create ./lam-bhost.def and add some hostnames on which you have an account. See "man bhost" if you require special syntax for multiprocessors or for different username on those hosts. Then boot the LAM: -------------------------------- % cd ~/octave/mpitb % mkdir myproject % cd myproject % cp ../startups/startup_MPITB.m .octaverc # copy-edit startup % vi .octaverc # & customize it if you want # ... # % customize the -marked sections # ... # % q = [p '/']; # % if ~exist(q, 'dir'), clear p q, error(' not found'), end # % addpath(q); % vi lam-bhost.def # define your LAM # host1 # host2 user=guest # host3 cpu=2 % lamboot # boot it % lamnodes # check it works ok % octave # now from Octave Set SSI rpi to tcp with the command: putenv('LAM_MPI_SSI_rpi','tcp'), MPI_Init Help on MPI: help mpi octave:1> MPI_Init # working? ans = 0 # great! octave:2> -------------------------------- If you don't get error messages, there you go. If you get errors, you must solve them before using MPITB under Octave. For LAM-related errors, read the LAM FAQ since most probably somebody already had exactly the same problem as you, and the answer is already there. LAM must be already booted when you use MPI_Init from inside Octave (ok, system("lamboot") would have worked too, agreed :-) Depending on how you plan to start your parallel application, you must place your headed node (where you start Octave) first or last in your bhost file. If you will use mpirun to start several non-interactive Octave commands, you probably want your headed node first in list. If you will use MPI_Comm_spawn() from your master Octave session, you want your headed node last in the list. That way, spawned octave processes will go to the other cluster nodes before starting to oversubscribe the headed node where you are already running an octave process. --------------------------------------- - Utility scripts ($MPITB_HOME/utils) - --------------------------------------- Although the user could conceivably have a different bhost file in each project subdir, in normal use you will be always using the same multicomputer. Conceivably, an utility script could be customized to boot a new multicomputer on those always-the-same computers. Or take advantage of an already booted LAM, if it meets the requirements expressed in a list or arguments (number of hosts, RPI, etc). That is the role of $MPITB_HOME/utils/LAM_Init.m. If you are planning to use the utilities provided in the $MPITB_HOME/utils subdir, you might want to edit the LAM_Init.m file so that the builtin HOSTS variable describes your cluster. Or perhaps get used to define a global HOSTS in your code before invoking LAM_Init. Octave_Spawn is another interesting utility to simplify the tasks involved in using MPI_Comm_spawn() to start slave Octave instances from a master Octave session. It assumes that slave Octaves will source the provided startup_mergeParent.m (or they will merge with the parent communicator somehow else). Another startup_bcastSlv.m is provided for applications in which the command to execute will be decided at runtime (so you cannot use mpirun), but all slaves will run the same command. A much more elaborated protocol is available with the startup_ncmdsSlv.m, together with the NumCmds_Init and NumCmds_Send utilities. According to this "protocol", the master instance sends an initial message indicating the number of commands that will be sent to slave instances. The slaves then enter a loop in which they repeatedly wait for a new command and evaluate it. If the number of commands is "0", slaves repeat the loop forever... until a "break", "quit" or "exit" command is sent, of course. -------------------------------- - What now? TUTORIAL or demos? - -------------------------------- Do skip this indented section ||| To start getting acquainted to MPI, one should open an xterm window, ||| "lamboot" a multicomputer and then start "octave". Alternatively, ||| "lamboot" can be run from another xterm if one forgets to call it ||| before starting octave. Then, one might want to open another xterm/ ||| editor from where to copy/paste lines from the tutorial file to the ||| octave window. In the past I have expressed my views that this is the ||| fastest way to learn MPI, and that you won't regret reproducing the ||| whole tutorial, or at least the sections dealing on MPI routines you ||| are interested in. ||| ||| But the fact is most users will benefit from the basic six, i.e., they ||| will just use MPI_Init/_Finalize, MPI_Send/_Recv, MPI_Comm_rank/_size. ||| No tutorial is required to learn those. In fact most users will have ||| already figured out the utility and use of the basic six just after we ||| named it. ||| ||| So now... we are recommending the use of demos to start learning MPITB. In increasing difficulty order, they are (for now): - Hello - Pi - PingPong - Mandelbrot - NPB/EP - Spawn -> not mpirun-nable - Wavelets -> still uses utils subdir Simply change into the respective subdirs, read the README files, the code (specially the help sections) and try to run them. Most of the demos have been re-designed to be mpirun'nable, instead of using MPI_Comm_spawn from a master Octave to start slave Octaves. The older version may still hang around for some demos, too. If you suceed with the demos, you can safely ignore the TUTORIAL file and the rest of this file. With respect to this file, it finishes with some explanations about the tutorial and the last old-style demos, Spawn and Wavelets. ---------------------- - The tutorial story - ---------------------- If in a future parallel application of yours you have doubts about howto use a particular MPITB command, search in the TUTORIAL for that command's name to see its typical usage, even if you do not plan to try it out interactively. That is the only reason that the TUTORIAL file has not been removed from MPITB yet. Due to its _tutorial_, interactive nature, the tutorial uses MPI_Comm_spawn to start and finish new slaves as required for each exercise. Not only Octave slaves are dynamically spawned, but they are run on top of an xterm window, so the user can see the xterms pop-up, and write MPITB commands on them... and see how MPI_Recv gets hung until a matching MPI_Send is issued, and cause a fatal error and see the slave xterm die and the master Octave hung on an awaiting recv... and by the way, learn how to debug a parallel Octave application. Experience has shown us that this approach is an error, since the interest of most users on MPITB dies when they experience they cannot get an xterm back in their GUI. Reasons for that are usually: - std firewall rules on personal PC/ws (blocking all X traffic back) learning iptables just to be able to use MPITB is an overkill - standard SSH tunneling ability which obscures the problem. xterm problem? Which problem? I've been sending xterms back to my PC since I was a child... - standard SSH configuration on headed nodes (using localhost:10) the skills to convince sysadmins are yet harder than iptables - no direct connectivity from headless nodes (usually in 192.168.x.y) back to user's PC, since nobody really sits at the headed node. Efforts to include explanations and workarounds for these problems has proven to be not only a lost battle, but worse a tactical error, since users will forever remember that they had problems _with_MPITB_, not with X11, sshd, firewalls, 192.168.x.y routing or lack of sysadmin advice. The "lamprep" and "proxy" shell-scripts are kept in the "$MPITB_HOME/stdlones" subdir as romantic, silent milestones (statues to the unknown soldier if we continue the analogy), to never forget this was a lost battle. -------------------------------- - The Spawn and Wavelets demos - -------------------------------- The Spawn demo makes sense with DEBUG set to true, in order to see the xterms running Octave on each slave computer. It is an exercise on howto use the LAM_Init and Octave_Spawn utilities, together with the "info" startup hint to choose a particular slave host. The Wavelets demo is really complex. Read the Contents.m file to get a "roadmap" of the different files included. It's a step-by-step whole example of how would one go through parallelizing a sequential octave application (yup, I must go on developing it to get better speedups, but I wanted to keep it didactical, so I've spent much more time in documenting the --wrong,initial--- design decisions than in obtaining good speedups). The data subdir stores the instrumentation data collected using the annotation/timestamp utilities init_stamp, time_stamp, last_stamp, clas_instr, save_instr, load_instr, look_instr, lkup_label You might find them useful to timestamp your own parallel applications and easily discover where the time is mostly spent. -------------- - What then? - -------------- If you want me to add some application of yours to the MPITB web page, or mention a paper or publication in a congress which uses MPITB, or link to some document you prepared for your research fellows/students dealing on MPITB, drop me an e-mail. I'd certainly like to see the "MPITB papers" list growing :-) ************************************* * Thanks for your interest in MPITB * ************************************* Granada, SPAIN, 04/May/2007