%%%%%%%%% % Tests % %%%%%%%%% % % This file is to be displayed in an xterm/kvt/gnome-terminal % while Octave is running in another % Use Copy/Paste and check Octave's answers % % LAM/MPI should have been started with lamboot (in the shell) % before issuing MPI_Init from Octave % Later a "lamprep" command will be introduced, % and the sequence will be: lamprep, lamboot, octave, MPI_Init % % Please read MPI man pages/std doc. for more info % The tests (loosely) follow the std. and "MPI Primer" order % of introduction. % % Use with "a grain of salt" % You probably want to "lamclean" after every SIGSEGV or octave_core % MPITB is rather rude to Octave sometimes (abusing from scalars etc) % and this tutorial exercises some "bad techniques" premeditately, % such as calling MPI_Init twice without clearing MPITB, % or Packing/Unpacking past end of buffer... % ie. the kind of things that produce a SIGSEGV or octave_core % %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % Enter/quit MPI (xterm may need "reset" after Abort) % (we won't send - you can lamboot just your localhost) % MPI_Init, _Initialized, _Abort, _Comm_rank, _Comm_size % MPI_Get_processor_name, _Get_version, _Wtime, _Wtick % MPI_COMM_WORLD, _SELF, _NULL % MPI_[SUB]VERSION %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% help mpi help misc [info flag]=MPI_Initialized % info=0, flag=0 MPI_Init [info flag]=MPI_Initialized % flag=1 help MPI_Init help MPI_Initialized [info rank]=MPI_Comm_rank(MPI_COMM_WORLD) % rank=0 [info rank]=MPI_Comm_rank(MPI_COMM_SELF) [info size]=MPI_Comm_size(MPI_COMM_SELF) % size=1 help MPI_Comm_rank help MPI_Comm_size help MPI_COMM_WORLD help MPI_COMM_SELF help MPI_COMM_NULL [info name]=MPI_Get_processor_name % name='myhost.mydomain.mytop' [info major minor]=MPI_Get_version % major.minor=1.2 MPI_VERSION MPI_SUBVERSION help MPI_VERSION tmorg=MPI_Wtime % arbitrary resol=MPI_Wtick % clock resolution in seconds elaps=MPI_Wtime-tmorg % elapsed time in seconds help MPI_Wtime help MPI_Abort MPI_Abort(MPI_COMM_WORLD,7) % octave aborted %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % Packing data % (we won't send - you can lamboot just your localhost) % MPI_Pack, Unpack, Pack_size % MPI_Errhandler_set, _get % MPI_ERRORS_ARE_FATAL, _ERRORS_RETURN, _ERRHANDLER_NULL %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %------------------------------------------------------- % Can Pack Octave vars on contiguous memory (from another var's storage) % Preferred buffer storage: strings (as many bytes as chars in string) %------------------------------------------------------- MPI_Init A=[1 2;3 4] help MPI_Pack help MPI_Pack_size [info s]=MPI_Pack_size(A,MPI_COMM_WORLD) % s=58 buf='0123456789'; buf=[buf buf buf]; buf=[buf buf]; buf(s+1:end)='' s, l=length(buf), whos buf % l=58 pos=0 % pack starting from 0 info=MPI_Pack(A,buf,pos,MPI_COMM_WORLD) pos % pack leaves ptr @ 58 %------------------------------------------------------- % Can Unpack, use other index var (pos2 here) % Can Unpack with a different varname %------------------------------------------------------- help MPI_Unpack pos2=0 % start unpacking from 0 [info retnams]=MPI_Unpack(buf,pos2,MPI_COMM_WORLD,'B') % retnams='A' pos2 % unpack leaves ptr at 58 whos -variables, A, B % unpacked with name 'B' equal=all(all(A==B)) clear A B % original var 'A' deleted pos2=0 % unpack it again [info retnams]=MPI_Unpack(buf,pos2,MPI_COMM_WORLD) % same name this time pos2 whos -variables, A % there it is %------------------------------------------------------- % By not specifying names, you will Unpack as many vars % as were packed along with their original names % caveat: buffer must contain complete vars, no trailing garbage/empty room %------------------------------------------------------- B=[A A A]; B(2,6)=0 % build some different var [info s]=MPI_Pack_size(A,MPI_COMM_WORLD) % add sizes: s=58 [info t]=MPI_Pack_size(B,MPI_COMM_WORLD) % t=122 s=s+t % s+t=180 buf=repmat(' ',1,s); whos buf, pos=0 % room enough info=MPI_Pack(A,buf,pos,MPI_COMM_WORLD), pos % pos=58 info=MPI_Pack(B,buf,pos,MPI_COMM_WORLD), pos % pos=180 clear A B % A, B, removed WORLD=MPI_COMM_WORLD, pos2=0 % tired of writing it all [info retnams]=MPI_Unpack(buf,pos2,WORLD) % A, B, created again pos2 % pos2=180 whos A B, A, B B(2,6)=4 % retouching to detect unpack pos2=58 % index startpoint for B [info retnams]=MPI_Unpack(buf,pos2,WORLD) % B overwritten B % there, B(2,6)=0 again pos2=58 [info retnams]=MPI_Unpack(buf,pos2,WORLD,'A') % A overwritten A %------------------------------------------------------- % Can Unpack again using new index and names % Can't unpack past end of buffer % Must change Errhandler to avoid Abort %------------------------------------------------------- pos2=0 [info retnams]=MPI_Unpack(buf,pos2,WORLD,'C'), pos2 % pos2=58 whos C, C [info retnams]=MPI_Unpack(buf,pos2,WORLD,'D'), pos2 % pos2=180 whos D, D help MPI_Errhandler_get [info eh]=MPI_Errhandler_get(MPI_COMM_WORLD) MPI_ERRHANDLER_NULL MPI_ERRORS_RETURN MPI_ERRORS_ARE_FATAL isfatal = eh==MPI_ERRORS_ARE_FATAL % default Errhandler MPI_Errhandler_set(WORLD,MPI_ERRORS_RETURN) % staying alive help MPI_Errhandler_set help MPI_ERRORS_ARE_FATAL help MPI_ERRORS_RETURN pos2 % pos2=180 now!!! [info retnams]=MPI_Unpack(buf,pos2,WORLD,'E'), pos2 % info=13 [info msg] = MPI_Error_string (info) % invalid argument help MPI_Error_string help errors help MPI_ERRCODES %------------------------------------------------------- % What happens without changing default error handler? %------------------------------------------------------- MPI_Errhandler_set(WORLD,MPI_ERRORS_ARE_FATAL) % any of these MPI_Errhandler_set(MPI_COMM_WORLD,MPI_ERRORS_ARE_FATAL) % will do it [info retnams]=MPI_Unpack(buf,pos2,WORLD,'E'), pos2 % ouch! aborted %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % Basic Send/Recv % (we'll send to ourselves - you can lamboot just your localhost) % MPI_Send, Recv, [I]Probe, Get_[count|elements] % MPI_ANY_SOURCE, _ANY_TAG % MPI_Finalize, _Finalized %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% MPI_Init A=[1 2;3 4] %------------------------------------------------------- % Can send to self %------------------------------------------------------- [info rank]=MPI_Comm_rank(MPI_COMM_WORLD) % rank=0 TAG=7 % nothing pending, flag==0 [info flag stat]=MPI_Iprobe(MPI_ANY_SOURCE,MPI_ANY_TAG,MPI_COMM_WORLD) % stat meaningless when !flag minus1 = MPI_ANY_SOURCE==-1 % wildcards are -1 minus1 = MPI_ANY_TAG ==-1 help MPI_Iprobe % Non-blocking help MPI_Probe % blocking help MPI_Send WORLD=MPI_COMM_WORLD % sending to ourselves (rnk 0) info=MPI_Send(A,rank,TAG,WORLD) % notice: no packing [info flag stat]=MPI_Iprobe(-1,-1,WORLD) % stat valid - flag 1 [info stat]=MPI_Probe (-1,-1,WORLD) % would have blocked if !flag whos A stat % 32-bytes (8B/double) [info count]=MPI_Get_count(stat) % 32-bytes message (4 doubles) [info elems]=MPI_Get_elements(stat, 'chars') % room for 32 chars [info elems]=MPI_Get_elements(stat, '') % it only matters type [info elems]=MPI_Get_elements(stat, 0) % room for 4 doubles [info elems]=MPI_Get_elements(stat, []) [info elems]=MPI_Get_elements(stat, stat) % uncompatible->MPI_PACKED [info elems]=MPI_Get_elements(stat, {0}) % (count in bytes) [info elems]=MPI_Get_elements(stat, {'a'}) % 32 bytes always [info elems]=MPI_Get_elements(stat, {}) [info elems]=MPI_Get_elements(stat, true) % room for 32 logicals help MPI_Get_count help MPI_Get_elements help MPI_Recv [info elems]=MPI_Get_elements(stat, []) % should prepare room for 4dbls B=zeros(1,elems) % same size, same type [info stat]=MPI_Recv(B,rank,TAG,WORLD) A, B % notice row-major-order [info flag]=MPI_Iprobe(-1,-1,WORLD) % no more messages % notice we didn't ask for stat %------------------------------------------------------- [info flag]=MPI_Initialized % flag==1, we already _Init [info flag]=MPI_Finalized % flag==0, haven't _Finalized info=MPI_Finalize [info flag]=MPI_Finalized % flag==1, right, have done it [info flag]=MPI_Initialized % flag==1, it remembers well help MPI_Init % warned not to Init twice MPI_Init % any of these two will do it MPI_Finalize % see it? I told you :-) %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % Compatible Octave<->MPI types can be sent/recvd % using destvar with same type and size - no need to pack % % Other Octave types can only be sent by first MPI_Pack'ing them % When packing, msgcount is in bytes, not in Octave #elements % % id, Octave type_name, and corresponding MPI type % 1 cell use MPI_Pack % 2 scalar MPI_DOUBLE % 3 complex scalar MPI_DOUBLE * 2 % 4 matrix MPI_DOUBLE * n % 5 complex matrix MPI_DOUBLE *2n % 6 range MPI_DOUBLE * 3 / DBL*3+INT using MPI_Pack() % 7 bool MPI_CXX_BOOL % 8 bool matrix MPI_CXX_BOOL*n % 9 char matrix MPI_CHAR * n % 10 string MPI_CHAR * n % % 11 int8 MPI_BYTE % 12 int16 MPI_SHORT % 13 int32 MPI_INT % 14 int64 MPI_LONG_LONG % 15 uint8 MPI_UNSIGNED_CHAR % 16 uint16 MPI_UNSIGNED_SHORT % 17 uint32 MPI_UNSIGNED % 18 uint64 MPI_UNSIGNED_LONG_LONG % % 19 int8 array MPI_BYTE * n % 20 int16 array MPI_SHORT * n % 21 int32 array MPI_INT * n % 22 int64 array MPI_LONG_LONG * n % 23 uint8 array MPI_UNSIGNED_CHAR * n % 24 uint16 array MPI_UNSIGNED_SHORT * n % 25 uint32 array MPI_UNSIGNED * n % 26 uint64 array MPI_UNSIGNED_LONG_LONG * n % % 27 struct use MPI_Pack % % 28 file unsupported % 29 list unsupported % 30 cs-list unsupported % 31 va-arg unsupported % 32 magic-colon unsupported % 33 built-in function unsupported % 34 built-in mapper function unsupported % 35 user-defined function unsupported % 36 dynamically-linked functionunsupported % 37 function handle unsupported % 38 inline function unsupported % 39 streamoff unsupported % %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % Sending variables w/ and w/o packing % (we'll send to ourselves - you can lamboot just your localhost) %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% MPI_Init WORLD = MPI_COMM_WORLD %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % 1 cell use MPI_Pack %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% A={1, [2 3]; 4i, [5 6i]; % 4x2 cell array true, logical([false;true]); "hello", 1:3:50} whos A [info s]=MPI_Pack_size(A,WORLD), pos=0 % s=318 bytes buf=repmat(' ',1,s); whos buf % make room!! info=MPI_Pack(A,buf,pos,WORLD), pos % pos=318 info=MPI_Send( buf,0,0,WORLD) % to rank 0 (ourselves) with tag 0 [info stat]=MPI_Probe(-1,-1,WORLD) % stat.len=318 [info count]=MPI_Get_count(stat) % count=318 bf2=repmat(' ',1,count); whos bf2 % make room!! [info stat]=MPI_Recv(bf2,0,0,WORLD) equal = all(buf==bf2) % ok, well received pos2=0 [info retnams]=MPI_Unpack(bf2,pos2,WORLD,'B'), pos2 % pos2=318 A, B, whos A B comp=zeros(prod(size(A)),1); idx=1; [nr,nc]=size(A); for r=1:nr, for c=1:nc, comp(idx++)=all(all(A{r,c}==B{r,c}));end,end equal=all(comp) %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % 2 scalar MPI_DOUBLE %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% A=3.1416 info =MPI_Send(A,0, 0,WORLD) % to rank 0 (ourselves) with tag 0 [info stat]=MPI_Probe(-1,-1,WORLD) % stat.len=8 [info count]=MPI_Get_count(stat) % count=8B [info elems]=MPI_Get_elements(stat,[]) % elems=1 double whos A % A= simple scalar B=0, whos B [info stat]=MPI_Recv(B,0,0,WORLD) B, equal = B==A % there it is, B==3.1416 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % 3 complex scalar MPI_DOUBLE * 2 % TODO: support octave complex using LAM-specific MPI_CXX_DOUBLE_COMPLEX %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% A = 1.4142 + 2.7183i info =MPI_Send(A,0, 0,WORLD) % to rank 0 (ourselves) with tag 0 [info stat]=MPI_Probe(-1,-1,WORLD) % stat.len=16 [info count]=MPI_Get_count(stat) % count=16B [info elems]=MPI_Get_elements(stat,i) % elems=2 doubles, ain't no MPI_COMPLEX whos A % A= complex scalar B=i, whos B [info stat]=MPI_Recv(B,0,0,WORLD) % _ B, equal = B==A % there it is, B==V2 + ei %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % 4 matrix MPI_DOUBLE * n %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% A=[1 2 3;4 5 6] info =MPI_Send(A,0, 0,WORLD) % to rank 0 (ourselves) with tag 0 [info stat]=MPI_Probe(-1,-1,WORLD) % stat.len=48 [info count]=MPI_Get_count(stat) % count=48B [info elems]=MPI_Get_elements(stat,[]) % elems=6 doubles, ain't no MPI_ARRAY whos A % A= 2x3, 48B double array B=zeros(2,3), whos B [info stat]=MPI_Recv(B,0,0,WORLD) B, equal = all(all(A==B)) % there it is B=[9 8;7 6;5 4], whos B info =MPI_Send(A,0, 0,WORLD) % send/recv again [info stat]=MPI_Recv(B,0,0,WORLD) B % notice FORTRAN row-major order equal = all(all( A==reshape(B, size(A)) )) % tricky, huh? %------------------------------- % NDArray %------------------------------- B=A+6 C=cat(3,A,B) % 3D array 2x3x2 12 doubles 96B whos C info =MPI_Send(C,0, 0,WORLD) % to rank 0 (ourselves) with tag 0 D=zeros(2,3,2), whos D [info stat]=MPI_Recv(D,0,0,WORLD) % recvd 96B D, equal = all(all(all(C==D))) % there it is D=zeros(4,3), whos D % easier to see this way info =MPI_Send(C,0, 0,WORLD) [info stat]=MPI_Recv(D,0,0,WORLD) % recvd 96B equal = all(all( D==reshape(C, size(D)) )) % tricky, huh? D % real mess ND fortran order %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % 5 complex matrix MPI_DOUBLE *2n %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% A=[1 2 3;4 5 6i] info =MPI_Send(A,0, 0,WORLD) % to rank 0 (ourselves) with tag 0 [info stat]=MPI_Probe(-1,-1,WORLD) % stat.len=96 [info count]=MPI_Get_count(stat) % count=96B [info elems]=MPI_Get_elements(stat,[]) % elems=12 doubles, ain't no MPI_CX_ARR whos A % A= 2x3, 96B complex array B=zeros(size(A)), B(1,1)=i, whos B [info stat]=MPI_Recv(B,0,0,WORLD) B, equal = all(all( A==B)) % there it is B=zeros(fliplr(size(A))), B(1,1)=i, whos B % playing with shape info =MPI_Send(A,0, 0,WORLD) % send/recv again [info stat]=MPI_Recv(B,0,0,WORLD) B % notice FORTRAN rowmajor order equal = all(all( A==reshape(B, size(A)) )) % tricky, huh? B=zeros(2,6), whos B % playing again with dims, now with types too info =MPI_Send(A,0, 0,WORLD) % send/recv again [info stat]=MPI_Recv(B,0,0,WORLD) B % there it is, can you see the 6 down? equalreal = all(all( real(A)== reshape(B(1,:), size(A)) )) equalimag = all(all( imag(A)== reshape(B(2,:), size(A)) )) % yet more tricky, huh? %------------------------------- % NDArray %------------------------------- B=A+6 C=cat(3,A,B) % 3D array 2x3x2 12 complex 192B whos C info =MPI_Send(C,0, 0,WORLD) % to rank 0 (ourselves) with tag 0 D=zeros(2,3,2); D(1,1,1)=i, whos D % room for recv [info stat]=MPI_Recv(D,0,0,WORLD) % recvd 192B D, equal = all(all(all(C==D))) % there it is %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % 6 range MPI_DOUBLE * 3 / DBL*3+INT using MPI_Pack() %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% A=0:3:30, whos A % ranges are really 3 doubles + 1 int info=MPI_Send(A,0,0,WORLD) % when used as buffers, only 3 doubles [info stat]=MPI_Probe(-1,-1,WORLD) [info count]=MPI_Get_count(stat) % count=24B [info elems]=MPI_Get_elements(stat,1:2:3)% elems=3dbl, ain't no MPI_RANGE B=1:11, A, whos A B % B is also 3 doubles [info stat]=MPI_Recv(B,-1,-1,WORLD) % receive A in-place over B B, equal = all(A==B) % B overwritten B=1:3, A, whos A B % the additional int is nelems in range info=MPI_Send(A,0,0,WORLD) % only 3 doubles are sent/recvd [info stat]=MPI_Recv(B,-1,-1,WORLD) % when received in-place... B % base/incr/etc are changed equal = all(A==B) % but nelem remains unchanged equal = all(A(1:3)==B(1:3)) % use MPI_Pack/_Unpack to include nelms %------------------------------- % Packing ranges %------------------------------- [info s]=MPI_Pack_size(A,WORLD), pos=0 % s=50 bytes buf=repmat(' ',1,s); whos buf % make room!! info=MPI_Pack(A,buf,pos,WORLD), pos % pos=50 info=MPI_Send( buf,0,0,WORLD) % to rank 0 (ourselves) with tag 0 [info stat]=MPI_Probe(-1,-1,WORLD) % stat.len=50 [info count]=MPI_Get_count(stat) % count=50 bf2=repmat(' ',1,count); whos bf2 % make room!! [info stat]=MPI_Recv(bf2,0,0,WORLD) equal = all(buf==bf2) % ok, well received pos2=0 [info retnams]=MPI_Unpack(bf2,pos2,WORLD,'B'), pos2 % pos2=50 A, B, whos A B %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % Too long session !!! %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% MPI_Finalize % sorry for such a lengthy session quit %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % Sending variables w/ and w/o packing (continued) % (we'll send to ourselves - you can lamboot just your localhost) %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% MPI_Init WORLD = MPI_COMM_WORLD %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % 7 bool MPI_CXX_BOOL %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% A=true, whos A % bool, 1B info=MPI_Send(A,0,0,WORLD) [info stat]=MPI_Probe(-1,-1,WORLD) [info count]=MPI_Get_count(stat) % yup, 1B [info elems]=MPI_Get_elements(stat,true)% elems=1 logicals B=false, whos B [info stat]=MPI_Recv(B,-1,-1,WORLD) B % there it is %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % 8 bool matrix MPI_CXX_BOOL*n %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% A(3,3)=true, whos A % this now works well (octave-2.1.60) info=MPI_Send(A,0,0,WORLD) [info stat]=MPI_Probe(-1,-1,WORLD) [info count]=MPI_Get_count(stat) % yup, 9B [info elems]=MPI_Get_elements(stat,true)% elems=9 logicals clear B, B(5,2)=true, whos B % logical array big enough to hold A [info stat]=MPI_Recv(B,-1,-1,WORLD) % receive A in-place over B B % B overwritten, notice row-major order %------------------------------- % NDArray %------------------------------- C=cat(3,A,not(A)) % 3D array 3x3x2 18 bools (18B) whos C info =MPI_Send(C,0, 0,WORLD) % to rank 0 (ourselves) with tag 0 clear D, D(3,3,2)=true, whos D % room for recv [info stat]=MPI_Recv(D,0,0,WORLD) % recvd 192B D, equal = all(all(all(C==D))) % there it is %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % 9 char matrix MPI_CHAR * n %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% A=['A','B';'C','D'], whos A % charMatrix -> string info=MPI_Send(A,0,0,WORLD) [info stat]=MPI_Probe(-1,-1,WORLD) % 4B, 1B each char [info count]=MPI_Get_count(stat) [info elems]=MPI_Get_elements(stat,'') whos A % A = 2x2 string B="01234", whos B % 1 spare char [info stat]=MPI_Recv(B,-1,-1,WORLD) B, whos B % there it is %------------------------------- % NDArray %------------------------------- char(A+4) C=cat(3,A,char(A+4)) % not printable whos C % 3D chararray 2x2x2 8B info =MPI_Send(C,0, 0,WORLD) % to rank 0 (ourselves) with tag 0 [info stat]=MPI_Probe(-1,-1,WORLD) % 8B, 1B each char [info count]=MPI_Get_count(stat) D=repmat(' ',1,count); whos D % a string we can see [info stat]=MPI_Recv(D,0,0,WORLD) % recvd 8B D, equal = all(D==reshape(C,[1,count])) % equal except fortran order %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % 10 string MPI_CHAR * n %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% A='hello, world' info=MPI_Send(A,0,0,WORLD) [info stat]=MPI_Probe(-1,-1,WORLD) % 12B, 1B each char [info count]=MPI_Get_count(stat) [info elems]=MPI_Get_elements(stat,'') whos A % A = 1x12 char array B='string long enough to hold A', whos B [info stat]=MPI_Recv(B,-1,-1,WORLD) B, whos B % overwritten 'hello, worldenough' B=[B ; "padding"], whos B % string (multiline) info=MPI_Send(A,0,0,WORLD) [info stat]=MPI_Recv(B,-1,-1,WORLD) B % funny fortan row major order %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % 11 int8 MPI_BYTE % 12 int16 MPI_SHORT % 13 int32 MPI_INT % 14 int64 MPI_LONG_LONG % 15 uint8 MPI_UNSIGNED_CHAR % 16 uint16 MPI_UNSIGNED_SHORT % 17 uint32 MPI_UNSIGNED % 18 uint64 MPI_UNSIGNED_LONG_LONG %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% intmin( 'int8'),intmax( 'int8'),-2^7,2^7-1 intmin('uint8'),intmax('uint8'), 0,2^8-1 A=int8(-129.5), B=int8(0), whos A B info =MPI_Send(A,0, 0,WORLD) [info stat]=MPI_Probe(-1,-1,WORLD) % stat.len=1B [info count]=MPI_Get_count(stat) % count=1B [info elems]=MPI_Get_elements(stat,int8([])) % 1 int8 [info stat]=MPI_Recv(B,0,0,WORLD) B, equal = A==B % there it is B=int16(0) % little-endian machine info =MPI_Send(A,0, 0,WORLD) % stores LSB first [info stat]=MPI_Recv(B,0,0,WORLD) % in a 2B word -> B % 80h -> 0080h => -128 -> 128 B=uint8(0) % removing the sign info =MPI_Send(A,0, 0,WORLD) % by receiving over an [info stat]=MPI_Recv(B,0,0,WORLD) % unsigned variable B % -128 -> 128 A=int32(1E6), B=uint64(0), whos A B info =MPI_Send(A,0, 0,WORLD) [info stat]=MPI_Recv(B,0,0,WORLD) B, equal = A==B % can recv 4B over 8B %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % 19 int8 array MPI_BYTE * n % 20 int16 array MPI_SHORT * n % 21 int32 array MPI_INT * n % 22 int64 array MPI_LONG_LONG * n % 23 uint8 array MPI_UNSIGNED_CHAR * n % 24 uint16 array MPI_UNSIGNED_SHORT * n % 25 uint32 array MPI_UNSIGNED * n % 26 uint64 array MPI_UNSIGNED_LONG_LONG * n %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% clear A, A(2,3)=int8(1) A=cat(3,A,A), B=A; B(2,3,:)=0 whos A B info =MPI_Send(A,0, 0,WORLD) [info stat]=MPI_Recv(B,0,0,WORLD) B, equal = all(all(all(A==B))) B=int16(ones(3,2)) % recv'ing 12 int8 over 6 int16 info =MPI_Send(A,0, 0,WORLD) % notice little-endian [info stat]=MPI_Recv(B,0,0,WORLD) % and fortran order B % 00 01 -> 0100 = 2^8 equal = all(all(B'==repmat([0 0 2^8],2,1))) B=int32([0 0 0]') % recv'ing 12 int8 over 3 int32 info =MPI_Send(A,0, 0,WORLD) [info stat]=MPI_Recv(B,0,0,WORLD) % 0000 0100 -> 0100_0000 = B, equal = all(B'==[0 2^8 2^24]) % 2^24 = 16M A=int64([-1 1;10 100]), B=int64(zeros(2)) info =MPI_Send(A,0, 0,WORLD) [info stat]=MPI_Recv(B,0,0,WORLD) B, equal = all(all(A==B)) %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % 27 struct use MPI_Pack %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% clear A A.one=1; A.two=2; A.three=3; A(3)=A whos A % 1x3 struct array [info s]=MPI_Pack_size(A,WORLD), pos=0 % s=376 bytes octave-2.1.60 buf=repmat(' ',1,s); whos buf % make room!! info=MPI_Pack(A,buf,pos,WORLD), pos % pos=376 info=MPI_Send( buf,0,0,WORLD) % to rank 0 (ourselves) with tag 0 [info stat]=MPI_Probe(-1,-1,WORLD) % stat.len=376 [info count]=MPI_Get_count(stat) % count=376 bf2=repmat(' ',1,count); whos bf2 % make room!! [info stat]=MPI_Recv(bf2,0,0,WORLD) equal = all(buf==bf2) % ok, well received pos2=0 [info retnams]=MPI_Unpack(bf2,pos2,WORLD,'B'), pos2 % pos2=376 A, B, whos A B fnA=fieldnames(A); fnB=fieldnames(B) idx=1; for m=1:3, comp(idx++)=all(fnA{m}==fnB{m});end equalfnames=all(comp) comp=zeros(size(A)); idx=1; nc=length(A); idx=1; for [valA keyA]=A, eval(["valB = B." keyA ";"]); flag=length(valA)==length(valB); for l=1:length(A), flag &= all(all(valA{l}==valB{l})); end comp(idx++) = flag; % this serves just for this struct end % it is not a general... equal=all(comp) % ..."struct compare" function %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % Too long session !!! %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% MPI_Finalize % sorry for such a lengthy session quit %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % Sending variables w/ and w/o packing (continued) % (we'll send to ourselves - you can lamboot just your localhost) %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% MPI_Init WORLD = MPI_COMM_WORLD %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % UNSUPPORTED CLASSES: % 28 file unsupported % 29 list unsupported % 30 cs-list unsupported % 31 va-arg unsupported % 32 magic-colon unsupported % 33 built-in function unsupported % 34 built-in mapper function unsupported % 35 user-defined function unsupported % 36 dynamically-linked functionunsupported % 37 function handle unsupported % 38 inline function unsupported % 39 streamoff unsupported %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% f = @sin, whos f % function handle ?!? what's that? MPI_Send(f,0,0,WORLD) % error: "unsupported datatype" buf=repmat(' ',1,500); whos buf, pos=0 info=MPI_Pack(f,buf,pos,WORLD) % unsupported class, info=32 [nfo msg]=MPI_Error_string(info) % "operation not supported on type" %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % PACK-ABLE CLASSES: % 1 cell use MPI_Pack %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% A={} % trivial and normal cells B={'foo',[1:4],'bar',[1 3;5 7]} [i sa]=MPI_Pack_size(A,WORLD) % sa== 26 [i sb]=MPI_Pack_size(B,WORLD) % sb== 196 sz=sa+sb % sz== 222 buf=repmat(' ',1,sz); whos buf, pos=0 info=MPI_Pack(A,buf,pos,WORLD), pos % pos= 26 info=MPI_Pack(B,buf,pos,WORLD), pos % pos=222 info=MPI_Send (buf,0,0,WORLD) info=MPI_Send(B,0,0,WORLD) % can't send cells directly clear A B buf sa sb sz % "invalid buffer datatype" [info stat]=MPI_Probe(-1,-1,WORLD) % stat.len=222 bytes [info elems]=MPI_Get_elements(stat,'') % or 222 chars bf2=repmat(' ',1,elems); whos bf2 [info stat]=MPI_Recv(bf2,0,0,WORLD) % stat.len=222 A % A / B undefined B pos=0, [info retnams]=MPI_Unpack(bf2,pos,WORLD) whos A B, pos, A, B %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % PACK-ABLE CLASSES: % 27 struct use MPI_Pack %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% clear A A.name='foo'; A.ID=0 A(2).name='bar'; A(2).ID=1 whos A % 22B w/o fieldnames, dims, lengths... [i sz]=MPI_Pack_size(A,WORLD) % sz=206 including all info buf=repmat(' ',1,sz); whos buf, pos=0 info=MPI_Pack(A,buf,pos,WORLD), pos % pos=206 info=MPI_Send (buf,0,0,WORLD) info=MPI_Send(A,0,0,WORLD) % can't send structs directly clear A buf % "invalid buffer datatype" [info stat]=MPI_Probe(-1,-1,WORLD) % stat.len=206 buf=repmat(' ',1,stat.len); whos buf [info stat]=MPI_Recv(buf,0,0,WORLD) % stat.len=206 pos=0 [info retnams]=MPI_Unpack(buf,pos,WORLD) A, whos A, pos % pos=206 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % PACK-ABLE CLASSES: % complex struct example %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% clear A % complex example A.one=1; A.two={"hello", [1 2], {3,4}}; A.three.a=1:2:5; A.three.b=4-5i whos A % contents: 85B [info s]=MPI_Pack_size(A,WORLD), pos=0 % s=508 octave-2.1.60 buf=repmat(' ',1,s); whos buf info=MPI_Pack(A,buf,pos,WORLD), pos % pos=508 info=MPI_Send(buf,0,0,WORLD) clear buf [info stat]=MPI_Probe(-1,-1,WORLD) % stat.len=508 buf=repmat(' ',1,stat.len); whos buf [info stat]=MPI_Recv(buf,0,0,WORLD) % stat.len=508 pos2=0 [info retnams]=MPI_Unpack(buf,pos2,WORLD,'B') pos2 % 508 A, B, whos A B fnA=fieldnames(A); fnB=fieldnames(B) idx=1; for m=1:3, comp(idx++)=all(fnA{m}==fnB{m});end, equalfnames=all(comp) %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % enough %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% MPI_Finalize quit %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % BUFFER CLASSES: % packing _ON_ other datatypes, tired of using always strings %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % not all supported Octave datatypes can work as buffers for MPI_Pack %------------------------------------------------------- MPI_Init % again at it WORLD=MPI_COMM_WORLD % save typing MPI_Errhandler_set(WORLD,MPI_ERRORS_RETURN) % try to survive to errors data = 3.1416 % something to pack [info sd]=MPI_Pack_size(data, WORLD) % siz(data)=33 bytes tiny = true % smallest thing in Octave [info st]=MPI_Pack_size(tiny ,WORLD) % siz(tiny)=26 bytes t = true % smallest thing even in name [info st]=MPI_Pack_size(t ,WORLD) % siz(t )=23 bytes whos -variables %------------------------------------------------------- % cell: unsupported as buffer %------------------------------------------------------- buff={1,2,3}, whos buff pos=0 info =MPI_Pack(data,buff,pos,WORLD), pos % error message %------------------------------------------------------- % error: get_MPI_Buff: invalid buffer datatype: cell %------------------------------------------------------- %------------------------------------------------------- % scalar: too small even for the smallest Pack %------------------------------------------------------- buff=1.4142, whos buff sd, pos=0 % 33B won't fit in a double info =MPI_Pack(data,buff,pos,WORLD), pos % info==13, pos==8B / double st, pos=0 % 23B won't fit either info =MPI_Pack(t ,buff,pos,WORLD), pos % info==13, pos==8 [i2 msg]=MPI_Error_string(info) % invalid argument info ==MPI_ERR_ARG %------------------------------------------------------- complex scalar: too small even for the smallest Pack %------------------------------------------------------- buff=1.4142i, whos buff sd, pos=0 % 33B won't fit in 2 doubles info =MPI_Pack(data,buff,pos,WORLD), pos % info==15, pos==13B / cplx 16B st, pos=0 % 23B won't fit either info =MPI_Pack(t ,buff,pos,WORLD), pos % info==15, pos==14B [i2 msg]=MPI_Error_string(info) % message truncated info ==MPI_ERR_TRUNCATE % -------------------------------------- % you might want to read MPI_Pack.cc code % and analyze the info stored when packing: % t_id (int) (1int is 4B) % name.length (int) (another 4B) % -> Pack(on-doub) fails exactly here (after 8B) % name[length+1](chars) (that is 2B for "t\0") % ndim==2 (int) (another 4B) % -> upto 14B fit well in cplx=2dbl=16B % dims[ 2 ](ints) (another 8B) % -> Pack(on-cplx) fails in middle of this, % in the 1st int (14+4=18B > 16B) % -------------------------------------- %------------------------------------------------------- matrix: ok %------------------------------------------------------- buff=[0 1 2 3], whos buff % 4dbl * 8B == 32B sd, pos=0 % 33B won't fit there info =MPI_Pack(data,buff,pos,WORLD), pos % info==15, pos==25 == 33-8 [i2 msg]=MPI_Error_string(info) % message truncated info ==MPI_ERR_TRUNCATE st, pos=0 % 23B will info =MPI_Pack(t ,buff,pos,WORLD), pos % info==0 , pos==23 buff % of course, totally wrecked buff(end+1)=0 % adding room whos buff % 0's are a question of scale sd, pos=0 % 33B will fit now info =MPI_Pack(data,buff,pos,WORLD), pos % info==0 , pos==33 clear buff, buff(2,2,2)=1, whos buff % also for ND arrays 8*8=64B sd, pos=0 % 33B will fit now info =MPI_Pack(data,buff,pos,WORLD), pos % info==0 , pos==33 st info =MPI_Pack(t ,buff,pos,WORLD), pos % info==0 , pos==33+23=56 %------------------------------------------------------- complex matrix: ok %------------------------------------------------------- buff=[0 i], whos buff % 2cpl *16B == 32B sd, pos=0 % 33B won't fit there info =MPI_Pack(data,buff,pos,WORLD), pos % info==15, pos==25 == 33-8 [i2 msg]=MPI_Error_string(info) % message truncated info ==MPI_ERR_TRUNCATE st, pos=0 % 23B will info =MPI_Pack(t ,buff,pos,WORLD), pos % info==0 , pos==23 buff % of course, totally wrecked buff(end+1)=0 % adding room whos buff % 0's are a question of scale sd, pos=0 % 33B will fit now info =MPI_Pack(data,buff,pos,WORLD), pos % info==0 , pos==33 clear buff, buff(2,2,2)=i, whos buff % also for ND arrays 8*16=128B sd, pos=0 % 33B will fit now info =MPI_Pack(data,buff,pos,WORLD), pos % info==0 , pos==33 st info =MPI_Pack(t ,buff,pos,WORLD), pos % info==0 , pos==33+23=56 %------------------------------------------------------- range: too small but for the smallest Pack %------------------------------------------------------- buff=0:3, whos buff % 3dbl+1int = 28B sd, pos=0 % 33B won't fit there info =MPI_Pack(data,buff,pos,WORLD), pos % info==15, pos==25 == 33-8 [i2 msg]=MPI_Error_string(info) % message truncated info ==MPI_ERR_TRUNCATE st, pos=0 % 23B will info =MPI_Pack(t ,buff,pos,WORLD), pos % info==0 , pos==23 buff % of course, totally wrecked whos buff % the last int was "nelem" pos=0 [i2 rnm]=MPI_Unpack(buff, pos, WORLD ,'tNewNm') % but t's packdata are there whos tNewNm, tNewNm, t, arequal = tNewNm==t buff=0:4 % larger range doesnot help whos buff % 3dbl+1int = 28B always sd, pos=0 % 33B won't fit again info =MPI_Pack(data,buff,pos,WORLD), pos % info==15, pos==25 == 33-8 [info sn]=MPI_Pack_size(tNewNm ,WORLD) % siz(tNewNm)=28 bytes sn, pos=0 % using tNewNam wouldn't fit info =MPI_Pack(tNewNm,buff,pos,WORLD), pos % info==0 , pos==28 %------------------------------------------------------- bool: too small even for the smallest Pack %------------------------------------------------------- buff=false, whos buff % bool=1B sd, pos=0 % 33B won't fit in a bool info =MPI_Pack(data,buff,pos,WORLD), pos % info==15, pos==0B st, pos=0 % 23B won't fit either info =MPI_Pack(t ,buff,pos,WORLD), pos % not even t_id (1int) fits [i2 msg]=MPI_Error_string(info) % message truncated info ==MPI_ERR_TRUNCATE %------------------------------------------------------- bool matrix: ok %------------------------------------------------------- buff=repmat(false, [1,sd+st]), whos buff % room for data+t sd, st, sd+st, pos=0 % 33 + 23 == 56B info =MPI_Pack(data,buff,pos,WORLD), pos % pos==33 info =MPI_Pack(t ,buff,pos,WORLD), pos % pos==56 clear buff, pos=0 % and also for ND arrays buff(2,2,2,2,2)=true % 2^5=32, won't fit data info =MPI_Pack(data,buff,pos,WORLD), pos % info==15, pos==25 pos=0 % retry info =MPI_Pack(t ,buff,pos,WORLD), pos % info== 0, pos==23 %------------------------------------------------------- char matrix: unavailable under Octave %------------------------------------------------------- buff=['A' 'B';'C' 'D'], whos buff % see? it's a string buff=[buff buff; buff buff], whos buff % 4x4=16B buff=cat(3,buff,buff) % 32B, can't print it sd, st, sd+st, pos=0 % 33 + 23 == 56B info =MPI_Pack(data,buff,pos,WORLD), pos % info==15, pos==25 pos=0 % retry info =MPI_Pack(t ,buff,pos,WORLD), pos % info== 0, pos==23 %------------------------------------------------------- string: ok %------------------------------------------------------- buff=repmat(' ', [1,sd+st]), whos buff % room for data+t sd, st, sd+st, pos=0 % 33 + 23 == 56B info =MPI_Pack(data,buff,pos,WORLD), pos % pos==33 info =MPI_Pack(t ,buff,pos,WORLD), pos % pos==56 %------------------------------------------------------- int scalars: too small even for the smallest Pack %------------------------------------------------------- buff1=int8 ( -8), buff2=uint8 ( 8) buff3=int16(-16), buff4=uint16(16) buff5=int32(-32), buff6=uint32(32) buff7=int64(-64), buff8=uint64(64) whos buff* sd, pos=0 % 33B won't fit in any int info =MPI_Pack(data,buff1,pos,WORLD), pos % info==15, pos==0B info =MPI_Pack(data,buff3,pos,WORLD), pos % info==15, pos==0B info =MPI_Pack(data,buff5,pos,WORLD), pos % info==13, pos==4B (int32=int) pos=0 % retry info =MPI_Pack(data,buff7,pos,WORLD), pos % info==13, pos==8B (int64=2int) [i2 msg]=MPI_Error_string(info) % invalid argument info ==MPI_ERR_ARG pos=0 info =MPI_Pack(data,buff1,pos,WORLD), pos % info==15, pos==0B [i2 msg]=MPI_Error_string(info) % message truncated info ==MPI_ERR_TRUNCATE clear buff* %------------------------------------------------------- int arrays: ok %------------------------------------------------------- buff=int64(zeros(2)); buff=cat(3,buff,buff); buff(2,2,2)=1, whos buff % 8x8=64B sd, st, sd+st, pos=0 % 33 + 23 == 56B info =MPI_Pack(data,buff,pos,WORLD), pos % pos==33 info =MPI_Pack(t ,buff,pos,WORLD), pos % pos==56 %------------------------------------------------------- struct: unsupported as buffer %------------------------------------------------------- buff.a=0, whos buff info =MPI_Pack(data,buff,pos,WORLD), pos % error message %------------------------------------------------------- % error: get_MPI_Buff: invalid buffer datatype: struct %------------------------------------------------------- MPI_Finalize % done quit %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % PACK-ABLE CLASSES: % all normal classes can be packed as well %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% MPI_Init % again at it WORLD=MPI_COMM_WORLD %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % Pack'em'all at once %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% buf=repmat(' ',1,1250); whos buf % 1250B buffer A=2 % scalar [info s]=MPI_Pack_size(A,WORLD), pos=0 % s=30 bytes info=MPI_Pack(A,buf,pos,WORLD), pos % pos=30 B=A+i % complex scalar [info s]=MPI_Pack_size(B,WORLD) % s=38 bytes info=MPI_Pack(B,buf,pos,WORLD), pos % pos=30+38=68 C=[1 2;3 4]; C=cat(3,C,C) % matrix (ND) [info s]=MPI_Pack_size(C,WORLD) % s=94 bytes info=MPI_Pack(C,buf,pos,WORLD), pos % pos=68+94=162 D=C; D(2,2,2)=i % complex matrix (ND) [info s]=MPI_Pack_size(D,WORLD) % s=158 bytes info=MPI_Pack(D,buf,pos,WORLD), pos % pos=162+158=320 E=1:2:30 % range [info s]=MPI_Pack_size(E,WORLD) % s=50 bytes info=MPI_Pack(E,buf,pos,WORLD), pos % pos=320+50=370 F=true, whos F % bool [info s]=MPI_Pack_size(F,WORLD) % s=23 bytes info=MPI_Pack(F,buf,pos,WORLD), pos % pos=370+23=393 G(2,2,2)=true, whos G % bool matrix (ND) [info s]=MPI_Pack_size(G,WORLD) % s=38 bytes info=MPI_Pack(G,buf,pos,WORLD), pos % pos=393+38=431 H=['A','B';'C','D'], whos H % charMatrix -> string H=cat(3,H,H), whos H % problem printing ND char arrays [info s]=MPI_Pack_size(H,WORLD) % s=38 bytes info=MPI_Pack(H,buf,pos,WORLD), pos % pos=431+38=469 I="Hello, World!", whos I % string [info s]=MPI_Pack_size(I,WORLD) % s=39 bytes info=MPI_Pack(I,buf,pos,WORLD), pos % pos=469+39=508 J=[I ; "padding"], whos J % string (multiline) [info s]=MPI_Pack_size(J,WORLD) % s=52 bytes info=MPI_Pack(J,buf,pos,WORLD), pos % pos=508+52=560 K= int8 (-1), L(2,2,2)= int8 (-2) M=uint8 ( 1), N(2,2,2)=uint8 ( 2) O= int16(-1), P(2,2,2)= int16(-2) Q=uint16( 1), R(2,2,2)=uint16( 2) S= int32(-1), T(2,2,2)= int32(-2) U=uint32( 1), V(2,2,2)=uint32( 2) W= int64(-1), X(2,2,2)= int64(-2) % problem with u/int64array oct-2.1.60 Y=uint64( 1), Z(2,2,2)=uint64( 2) % can't do that ?!? X= int64(zeros(2,2,2)); X(2,2,2)=-2 % manually fixed Z=uint64(zeros(2,2,2)); Z(2,2,2)= 2 whos -variables [info s]=MPI_Pack_size(K,WORLD), t =s % s=23, t=23 [info s]=MPI_Pack_size(L,WORLD), t+=s % s=38, t=61 [info s]=MPI_Pack_size(M,WORLD), t+=s % s=23, t=84 [info s]=MPI_Pack_size(N,WORLD), t+=s % s=38, t=122 [info s]=MPI_Pack_size(O,WORLD), t+=s % s=24, t=146 [info s]=MPI_Pack_size(P,WORLD), t+=s % s=46, t=192 [info s]=MPI_Pack_size(Q,WORLD), t+=s % s=24, t=216 [info s]=MPI_Pack_size(R,WORLD), t+=s % s=46, t=262 [info s]=MPI_Pack_size(S,WORLD), t+=s % s=26, t=288 [info s]=MPI_Pack_size(T,WORLD), t+=s % s=62, t=350 [info s]=MPI_Pack_size(U,WORLD), t+=s % s=26, t=376 [info s]=MPI_Pack_size(V,WORLD), t+=s % s=62, t=438 [info s]=MPI_Pack_size(W,WORLD), t+=s % s=30, t=468 [info s]=MPI_Pack_size(X,WORLD), t+=s % s=94, t=562 [info s]=MPI_Pack_size(Y,WORLD), t+=s % s=30, t=592 [info s]=MPI_Pack_size(Z,WORLD), t+=s % s=94, t=686 pos+t % size after packing K-Z: 1246 prod(size(buf)), sizeof(buf) % total buffer size: 1250 it_fits = pos+t % and the FAQ at http://www.openssh.org % Really, really... ask _her_, not me %------------------------------------------------------- %------------------------------------------------------- % rsh/ssh and the PATH problem % Run these in the xterm/kvt/gnome-terminal shell %------------------------------------------------------- hostname # this is called ox0, my cluster headed node ssh ox1 # log into the remote host, some other node hostname # double-check if we got to ox1 echo $PATH # check for /usr/X11R6/bin in $PATH in ox1 which xterm # double-check that you can call xterm exit # come back to ox0, your login computer ssh ox1 xterm # not found? ssh ox1 which xterm # probably /usr/X11R6/bin added to PATH ssh ox1 'echo $PATH' # at login-time, not at rsh-time ssh ox1 /usr/X11R6/bin/xterm # quick-n-dirty workaround %------------------------------------------------------- % You can fix the "no xterm in path" error by: % adding /usr/X11R6/bin to your $PATH in .bashrc (sourced at rsh-time) % specifying a full pathname to xterm (our preferred option) % asking your sysadmin for a fix (adding it to /etc/bashrc ???) %------------------------------------------------------- %------------------------------------------------------- % rsh/ssh and the DISPLAY problem % usually no DISPLAY problem with ssh, it automagically forwards X11 % run these in an X11 window environment, of course %------------------------------------------------------- ssh ox1 /usr/X11R6/bin/xterm # type this in the pop-up xterm window hostname # double-check this xterm comes from ox1 echo $PATH # this is your rsh-time PATH echo $DISPLAY # localhost:0.0 (in ox1) to localhost:11.0 (in ox0) exit # allright, finish the pop-up xterm window echo $DISPLAY # ssh automagically forwards X11 graphics here %------------------------------------------------------- % You can fix the "cannot open display" error by: % possibly running 'xhost +', xauth, Xnest, lbxproxy, whatever % specifying a full display to xterm (xterm -display ) % retouching your DISPLAY before running octave (BIG problem if using ssh) % and/or asking your sysadmin for a fix (really, really... ask her, not me) %------------------------------------------------------- echo $DISPLAY # localhost:11.0 xterm # localhost:11 in ox0 reaches us exit # wipe the xterm, just testing xterm -display ox0:11 # does this xterm reaches you? # if not, read the next section %------------------------------------------------------- % Now, the DISPLAY problem revisited (this is the BIG one) % Why ox0:11 is not the same thing as localhost:11.0 in ox0? % ------- % ----------- ------- |---| ox1 | % | myRHbox |----------| ox0 |-----| ------- % ----------- ------- |---| ox2 | % ------- % I connect from myRHbox (in my office) to the cluster visible node (ox0) % the other cluster nodes are not visible from my network and vice-versa % in this situation, I would need a proxy manager, such as lbxproxy % really, you don't want to read xfindproxy man page and such % (neither your sysadmin does, I bet) % % it would be much better if the ssh X11forwarding could be reutilized % two problems here: the "localhost" trap and lamboot finishing ssh % % the bad news first: lamboot typically uses 'ssh -x' to boot lamd % in cluster nodes (see LAMRSH in /etc/profile.d/lam.sh, or % LAM_MPI_SSI_boot_rsh_agent description in lamboot and lamssi_boot % man pages, or the --with-rsh="ssh -x" LAM configure switch probably % used to compile your LAM installation), ie, it won't forward X11 % (you could override the "ssh -x" with the LAM_MPI_SSI envvar) % % then, the very bad news: even if you did, lamboot finishes the % ssh sessions after booting, so the inherited $DISPLAY for the % lamd process in ox1/ox2 (if you used ssh -X instead of ssh -x) % is lost anyways. In other words, it would be useless that the % lamd process in ox1 had a $DISPLAY=ox1:10 var in its environment, % since that DISPLAY worked just because of the ssh forwarding, % and just that ssh was finished long ago, when lamd started and lamboot % returned to the shell prompt. What we need is some kind of $DISPLAY % in ox1 where we could send windows to, something like ox0:10. % But ox0:10 won't work, as we have just checked. That's the % "localhost trap" mentioned above. It can be workarounded %------------------------------------------------------- hostname # this is an xterm in my myRHbox echo $DISPLAY # :0.0 default local value ssh ox0 # get to the cluster hostname # this is ox0 echo $DISPLAY # localhost:11.0 typical ssh tunneling/forwarding xterm # it works due to ssh hostname echo $DISPLAY exit # done with it ssh ox1 # get to cluster node xterm # it works due to ssh hostname echo $DISPLAY # localhost:10.0 typ ssh fwd exit # done with it exit # back to ox0 xterm -display myRHbox:0 # can't open, certainly xterm -display ox0:11 # can't open, either xterm -display localhost:11 # this is the localhost trap exit # finish the xterm %------------------------------------------------------- % see the point? % no matter how far can I reach down the ssh's and it gets all fwded % but only as long as I don't touch the "localhost" % _and_ I need somehow to refer to "localhost:11" at ox0 from ox1 % (it can't be localhost, it must be ox0:) % % moreover, myRHbox was installed with paranoid security level % so I would have had to add a new iptables rule to let X11 traffic in % (iptables -I RH-...-INPUT 4 -p tcp -s --dport x11 -j ACCEPT) % if not for ssh X11forwarding %------------------------------------------------------- %------------------------------------------------------- % finally, the good news: ~/.ssh/environment and sshd_config:X11UseLocalhost % % ssh allows specifying environment vars % to be inherited from a file in the destination node (see man page), % so, if only we had a good ox0: display at ox0, % since we could create ox1:~/.ssh/environment containing % DISPLAY=ox0: % % if your home ox0:~ is shared in ox1:~, % it's too easy: echo DISPLAY=$DISPLAY > ~/.ssh/environment % but do it _after_ loggin in into ox0 and _before_ lamboot from ox0 % and remove the file at logout % (you probably want to add "rm -f ~/.ssh/environment" to ~/.bashrc) % do not add that echo line to your .bashrc in ox0, since it will % not only affect the next logins to ox1 (and hence the lamd environment), % but also adversely affect future ssh logins to ox0 % % That was assuming that your home ox0:~ is shared to ox1:~ % if not you would use ssh ox1 echo DISPLAY=$DISPLAY \> ~/.ssh/environment % notice that "$DISPLAY" was not escaped - it is interpreted in ox0 % notice that \> _was_ indeed escaped - stdout is redirected in ox1 % % we'll study the localhost trap workaround now % please read the X11UseLocalhost entry in sshd_config man page % and the kill -HUP `pidof sshd` remark in sshd man page % you need that root@ox0 is your friend and doesn't mind editing/signalling % have her edit ox0:/etc/ssh/sshd_config to include an uncommented line: % X11UseLocalhost no % and then kill -HUP `pidof sshd` so that sshd re-reads its config file % (well, she could first ps axu|grep sshd|grep root and signal only those) % (ok, she could also simply reboot, but that's brute force) %------------------------------------------------------- hostname # this is an xterm in my myRHbox echo $DISPLAY # :0.0 default local value ssh ox0 # get to the cluster hostname # this is ox0 echo $DISPLAY # great!!! ox0:11 or whatever ssh ox1 # get to cluster node hostname # this is ox1 echo $DISPLAY # localhost:10, not interested in this one xterm # it works now, but not when lamboot exits exit xterm -display ox0:11 # great!!! exit exit lamboot -s -v # edit $LAMBHOST to include ox1 & ox0 in that order mpirun -c 1 xterm # so that 1st process goes to ox1 mpirun -c 1 /usr/X11R6/bin/xterm #not found? lamhalt # can't open display? echo $DISPLAY echo DISPLAY=$DISPLAY > ~/.ssh/environment cat !$ # dbl-chk future lamboot environment lamboot -s -v # this time lamd will inherit ox0:11 mpirun -c 1 /usr/X11R6/bin/xterm #there you have it exit # notice annoying mpirun warning lamclean # about rude processes not MPI_Init'ing mpirun -c 1 /usr/X11R6/bin/xterm #again, this time we want perfection cd $MPITB_HOME # notice that mpirun blocked ls # $MPITB_HOME typically ~/octave/mpitb cat MPI_Init.c # this C program simply initializes MPI mpicc -o MPI_Init MPI_Init.c ./MPI_Init # compile & run it exit # mpirun unblocks now lamhalt # flying colors!!! you can use MPITB %------------------------------------------------------- % You could even create a "lamprep" comand to be issued after ssh ox0 % so for all that session you can forget about DISPLAYs and so on % for instance, if your LAM installation is personal you can write in % cd $LAMHOME/bin % cat > lamprep % echo DISPLAY=$DISPLAY > ~/.ssh/environment % ^D % chmod 755 lamprep %------------------------------------------------------- % please do notice the amount of effort I have done (this subtutorial), % trying to help beginners not acquainted to ssh, X11 and LAM/MPI % I'm not the maker of any of these % your sysadmin, or your teacher, should be the ones helping you % if you're alone and you can't get past this point, % MPITB is not for you (too complex for your help environment) %------------------------------------------------------- %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % double-checking the Octave startup "else slave branch" % this is to make sure the children Octaves won't run anything else %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % Edit your .octaverc in the slave computer (if you have shared home, you can % edit your .octaverc in the master computer, since it's the same file), % which should be linked to (or include the code at) % $MPITB_HOME/startups/octaverc_MPI.m % and make sure that in the final else branch the command is commented out: % else % % startup_mergeParent % yup, keep it commented out % end %------------------------------------------------------- % do this in the UNIX shell %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% pushd $MPITB_HOME; ls # usually ~/octave/mpitb ls -la startups # there is a startups subdir ls -la .oct* # there should be a link .octaverc->startups/ less .octaverc # this is the normal MPITB startup octaverc_MPI popd # or wherever, as long as .octaverc uses MPITB vi .octaverc # make sure last 3 lines look as shown above octave # again back to work %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % Spawning children processes % MPI_Comm_spawn, _Comm_get_parent, _Intercomm_merge %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% (remember we assume $LAMBHOST includes ox1 & ox0, master node ox0 last in list) (remember we assume you can ssh to the other computer ox1 w/out passwd/output) (remember we assume you have octave installed/available on both hosts) (remember to lamprep/lamboot before starting octave in your master ox0 node) %------------------------------------------------------- % spawning a simple xterm % requires some MPI_Init program to unblock the calling MPI_Comm_spawn % read the MPI_Init man page (it's a collective MPI call) % related to this, you can study the differences between lamboot / lamexec %------------------------------------------------------- MPI_Init [info children errs] = MPI_Comm_spawn ('/usr/X11R6/bin/xterm',... {},1,MPI_INFO_NULL,0,MPI_COMM_SELF) %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % type this in the spawned xterm %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% pwd # notice starting subdir cd $MPITB_HOME # typically ~/octave/mpitb ls cd stdlones; ls less MPI_Init.c # it simply MPI_Inits / print rank/size mpicc -o MPI_Init MPI_Init.c ./MPI_Init # notice how MPI_Comm_spawn unblocks less MPI_Finalize.c # can't Finalize before Init mpicc -o MPI_Finalize MPI_Finalize.c ./MPI_Finalize # error, can't _Finalize w/out previous _Init exit # finished with the child xterm example %------------------------------------------------------- % spawning a child Octave % requires Octave installed or available on the 1st slave node, of course % processes are spawned round-robin in the $LAMBHOST order % ie, if your lamnodes are ox1, ox2, ox0, and you spawn 2 Octaves, % the 1st goes to ox1, the 2nd to ox2, and you need Octave available on each %------------------------------------------------------- % we assume you have double-checked (and commented out) the else startup branch %------------------------------------------------------- help MPI_Comm_spawn [info children errs] = MPI_Comm_spawn ('/usr/X11R6/bin/xterm',... {'-e','octave'},1,MPI_INFO_NULL,0,MPI_COMM_SELF) %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % type this in the new Octave % exciting, isn't it? :-) %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% MPI_Init % notice how MPI_Comm_spawn unblocks [info rank]=MPI_Comm_rank(MPI_COMM_SELF)% rank=0 [info size]=MPI_Comm_size(MPI_COMM_SELF)% size=1 !! [info rank]=MPI_Comm_rank(MPI_COMM_WORLD) [info size]=MPI_Comm_size(MPI_COMM_WORLD) %------------------------------------------------------- % being size=1, how am I supposed to send messages to the parent? % ...merging the intercomm's local&remote groups before % after that, you can Send to spawned %------------------------------------------------------- %%%%%%%%% under child Octave %%%%%%%%%%%%%%%%%%%% help MPI_Comm_get_parent % don't use the 'q'key help MPI_Intercomm_merge % press to end [info parent]=MPI_Comm_get_parent [info NEWORLD] = MPI_Intercomm_merge (parent, 1) % blck %%%% parent Octave %%%%%%%%%%%%%% [info NEWORLD] = MPI_Intercomm_merge (children, 0) % Intercomm unblocks [info rank]=MPI_Comm_rank(NEWORLD) % rank=0 [info size]=MPI_Comm_size(NEWORLD) % size=2 !!! [info rank]=MPI_Comm_rank(NEWORLD) % rank=1 [info stat]=MPI_Probe(-1,-1,NEWORLD) % blocks A="Hello, New World!" info=MPI_Send(A,1,7,NEWORLD) % dst=1, tag=7 %%%%%%%% Probe unblocks now %%%%%%%%%%%%%%%%%%%%% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% [info chars]=MPI_Get_elements(stat,'') % 17 chars A=repmat(' ',1,chars); whos A [info stat]=MPI_Probe(-1,-1,NEWORLD) % still there [info stat]=MPI_Recv (A,0,7,NEWORLD) % get it A % there it is %------------------------------------------------------- % Can Recv from spawned %------------------------------------------------------- B=[1 2; 3 4] info=MPI_Send(B,0,7,NEWORLD) % dst=0, tag=7 [info stat]=MPI_Probe(-1,-1,NEWORLD) % doesn't block, learn size [info dbls]=MPI_Get_elements(stat,[]) % 32B == 4 doubles B=zeros(1,dbls+1) % (spare) room for them [info stat]=MPI_Recv(B,1,7,NEWORLD) % src=1, tag=7, stat.len=32 B % there it is %------------------------------------------------------- % bright finale: Forcing a SIGSEGV %------------------------------------------------------- MPI_Errhandler_set(MPI_COMM_WORLD,MPI_ERRORS_RETURN) % trying to survive [info eh]=MPI_Errhandler_get(MPI_COMM_WORLD) world_err_ret = eh==MPI_ERRORS_RETURN % yup, changed MPI_Errhandler_set(NEWORLD,MPI_ERRORS_RETURN) % trying hard to survive [info eh]=MPI_Errhandler_get(NEWORLD) neworld_err_ret = eh==MPI_ERRORS_RETURN % yup, changed %%%%%%%%% under child Octave %%%%%%%%%%%%%%%%%%%% [info eh]=MPI_Errhandler_get(MPI_COMM_WORLD) world_err_ret = eh==MPI_ERRORS_RETURN % not changed [info eh]=MPI_Errhandler_get(NEWORLD) neworld_err_ret = eh==MPI_ERRORS_RETURN % not here info=MPI_Send(B,0,7,NEWORLD) % send again C=zeros(1,dbls-1) % (not enough) room for them [info stat]=MPI_Recv(C,1,7,NEWORLD) % info==15, len=24, err=21775 [inf2 msg ]=MPI_Error_string(info) % message truncated [inf2 msg ]=MPI_Error_string(stat.err) % message truncated [inf2 msg ]=MPI_Error_string(stat.err+1) % not valid errcode [inf2 cls ]=MPI_Error_class (stat.err) % cls=15 [inf2 msg ]=MPI_Error_string(cls) % message truncated %%%%%%%%% under child Octave %%%%%%%%%%%%%%%%%%%% info=MPI_Send(B,0,7,NEWORLD) % send again D=1:10 [info stat]=MPI_Recv(D,1,7,NEWORLD) % info==15, len=24, err=21775 D %%%%%%%%% under child Octave %%%%%%%%%%%%%%%%%%%% A.field=0, pos=0 [info s]=MPI_Pack_size(A,NEWORLD) % s=90 buf=repmat(' ',1,s); whos buf info=MPI_Pack(A,buf,pos,NEWORLD), pos % pos=90 info=MPI_Send( buf,0,7,NEWORLD) buf='hello', pos=0 [info stat]=MPI_Recv(buf,1,7,NEWORLD) % len=5!!!, msg truncated [inf2 msg ]=MPI_Error_string(info) % message truncated [info retnams]=MPI_Unpack(buf,pos,NEWORLD) % unpack all, reckless %------------------------------------------------------- % Segmentation fault %------------------------------------------------------- lamclean # if you lamclean, the child will die %------------------------------------------------------- % ouch! forgot about child Octave %------------------------------------------------------- %%%%%%%% Orphan child... %%%%%%%%%%%%%%%%%%%%%%%% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% info=MPI_Finalize % if you didn't lamclean quit % clean manually %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % Next session is better understood with rpi tcp instead of lamd. % Since default rpi changes with version (7.1.1 defaults to tcp) % we better make sure about which one are we using. % lamd may buffer sends, making MPI_Send/Ssend/Rsend indistingishable % or may block messages smaller than tcp_short (why?!?) % type this in your shell, before starting Octave again %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% pushd $MPITB_HOME; ls # usually ~/octave/mpitb cd stdlones; ls less BW.c # sample BandWidth measurement program man lamssi man lamssi_rpi # check the "available modules" section env | egrep "LAM|MPI" # where LAM_MPI_SSI_rpi envvar is introduced mpicc -o BW BW.c # compile the sample program ./BW # nope, 2 computers for ping-pong mpirun -c 2 BW # good performance (11MB/s my FastEthernet) mpirun -c 2 -ssi rpi tcp BW # notice LAM_MPI_SSI_rpi=tcp, same 11MB/s mpirun -c 2 -ssi rpi lamd BW # notice LAM_MPI_SSI_rpi=lamd, just 8.5MB/s mpirun -c 2 BW # defaults to 11MB/s export LAM_MPI_SSI_rpi=lamd # forcing lamd at hand (without -ssi switch) mpirun -c 2 BW # we get 8.5MB/s mpirun -c 2 -ssi rpi tcp BW # notice LAM_MPI_SSI_rpi=tcp, overridden export LAM_MPI_SSI_rpi=tcp mpirun -c 2 BW # again forced default 11MB/s mpirun -c 2 -ssi rpi lamd BW # notice LAM_MPI_SSI_rpi=lamd, overridden popd # ready to run Octave with rpi tcp export LAM_MPI_SSI_rpi=tcp # make sure octave %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % Point-to-point send modes % MPI_Bsend, _Ssend, _Rsend, _Buffer_attach, _Buffer_detach %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% (remember we assume $LAMBHOST includes ox1 & ox0, master node last in list) (remember we assume you can ssh to the other computer ox1 w/out passwd/output) (remember we assume you have Octave installed at both hosts) (remember we assume you have edited .octaverc / commented out "mergeParent") (lamprep/lamboot before starting octave if this is a new ssh session) MPI_Init [info children errs] = MPI_Comm_spawn ('/usr/X11R6/bin/xterm',... {'-e','octave'},1,MPI_INFO_NULL,0,MPI_COMM_SELF) % blocks %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %%% in child octave process %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% MPI_Init % unblocks [info parent]=MPI_Comm_get_parent % next blocks [info NEWORLD] = MPI_Intercomm_merge (parent, 1) %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % always same stuff, could make it a script/function % called mergeParent (function better than script) % and directly spawn "octave -r mergeParent" % we'll do that next time %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% [info NEWORLD] = MPI_Intercomm_merge (children, 0) % unblocks help modes %------------------------------------------------------- % We'll first send a "big" message (greater than tcp_short=64KB) % so that the different modes are better understood later %------------------------------------------------------- A=ones(8,1024); whos A % 64KB info=MPI_Send(A,1,7,NEWORLD) % returns immediately (rpi tcp) % (lamd blocks) [info stat]=MPI_Probe(-1,-1,NEWORLD) B=zeros(8,1024); whos B B(1,1), recvd = all(all(B)) % not recvd yet [info stat]=MPI_Recv(B,0,7,NEWORLD) B(1,1), recvd = all(all(B)) % there it is A=ones(9,1024); whos A % greater than 64KB sa = prod(size(A)) * 8 % 8B/double -> 73728 sa = sizeof(A) info=MPI_Send(A,1,7,NEWORLD) % returns when recvd (rpi tcp) [info stat]=MPI_Probe(-1,-1,NEWORLD) % not yet B=zeros(9,1024); whos B B(1,1), recvd = all(all(B)) % not recvd yet [info stat]=MPI_Recv(B,0,7,NEWORLD) % unblocks A B(1,1), recvd = all(all(B)) % there it is %------------------------------------------------------- % User can provide buffer, instead of system buffer %------------------------------------------------------- help MPI_Bsend help MPI_Buffer_attach help MPI_Buffer_detach help MPI_BSEND_OVERHEAD B='example of string', whos B, sb = sizeof(B) % 17 chars -> 17B B=repmat(B,4,1024); whos B, sb = sizeof(B) % 69632 bytes> 64KB sz=sa+sb+2*MPI_BSEND_OVERHEAD % 143440 bytes>128KB snd=repmat(' ',1,sz); whos snd % room for both info=MPI_Buffer_attach(snd) % user buffer info=MPI_Bsend (A,1,7,NEWORLD) % won't block info=MPI_Bsend (B,1,7,NEWORLD) % won't block, and next % won't block,will fail MPI_Errhandler_set(NEWORLD,MPI_ERRORS_RETURN) % avoid default abort info=MPI_Bsend (A,1,7,NEWORLD) % info==16 [nfo msg ]=MPI_Error_string(info) % unclassified help MPI_ERRCODES % look for it in docs help MPI_ERR_OTHER % undocumented in man info==MPI_ERR_OTHER [info stat]=MPI_Probe(-1,-1,NEWORLD) % stat.len=73728 [info elems]=MPI_Get_elements(stat,[]) % elems=9216 dbls A=zeros(100); whos A, A(1,1) % room for 10000 [info stat]=MPI_Recv(A,0,7,NEWORLD) % blocks !?!? :-( [info flag]=MPI_Iprobe (-1,-1,NEWORLD) % must be awaken A(1,1), recv_A = all(all(A(:,1:92))) % now unblocks A(10:20,90:95) % recall fortran-order [info stat]=MPI_Probe(-1,-1,NEWORLD) % stat.len=69632 B='plenty of room for B' B=repmat(B,4,1024); whos B % looong string room_enough = sizeof(B)>stat.len % 81920 > 69632 [info stat]=MPI_Recv(B,0,7,NEWORLD) % blocked again !!! [info flag]=MPI_Iprobe (-1,-1,NEWORLD) % again same trick B(1,1:40), B(1,end-40:end) % unblocked now B(4,1:40), B(4,end-40:end) % B % if you want to page, but don't press q !!! info = MPI_Buffer_detach % back to sysbuf %------------------------------------------------------- % User can send synchronous, waiting for starting Recv (Send may finish sooner) % LAM may buffer Sends (specially in tcp mode) and make MPI_Send finish sooner % Ssend is a way of not allowing that % you need msgsize<=64KB in tcp rpi to see any difference %------------------------------------------------------- help modes help MPI_Ssend help MPI_Send A=ones(8,1024); whos A % 64KB sa = sizeof(A) % 8B/double -> 65536 info=MPI_Send (A,1,7,NEWORLD) % returns immediately (rpi tcp) info=MPI_Send (A,1,7,NEWORLD) % returns immediately info=MPI_Ssend(A,1,7,NEWORLD) % blocks until Recv [info stat]=MPI_Probe(-1,-1,NEWORLD) B=zeros(8,1024); whos B, B(1,1) [info stat]=MPI_Recv(B,0,7,NEWORLD) % matches Send recv_B = B(1,1), B(1,1)=0; B(1,1) [info stat]=MPI_Probe(-1,-1,NEWORLD) [info stat]=MPI_Recv(B,0,7,NEWORLD) % matches 2nd recv_B = B(1,1), B(1,1)=0; B(1,1) [info stat]=MPI_Probe(-1,-1,NEWORLD) % matches Ssend [info stat]=MPI_Recv(B,0,7,NEWORLD) % & unblocks it recv_B = B(1,1), B(1,1)=0; B(1,1) %------------------------------------------------------- % User can send to an already waiting recv ("ready" send) % DOUBT: no error? defaults to MPI_Send when no recv posted? %------------------------------------------------------- help MPI_Rsend [info stat]=MPI_Probe(-1,-1,NEWORLD) % blocks info=MPI_Rsend(A,1,7,NEWORLD) % returns immediately, unblocks probe % defaults to MPI_Send? info=MPI_Rsend(A,1,7,NEWORLD) % returns immediately ?!? [info stat]=MPI_Recv(B,0,7,NEWORLD) % matches 1st recv_B = B(1,1), B(1,1)=0; B(1,1) [info stat]=MPI_Probe(-1,-1,NEWORLD) % doesn't blk [info stat]=MPI_Recv(B,0,7,NEWORLD) % matches 2nd recv_B = B(1,1), B(1,1)=0; B(1,1) [info stat]=MPI_Recv(B,0,7,NEWORLD) % blocks info=MPI_Rsend(A,1,7,NEWORLD) % unblocks recv_B = B(1,1) info=MPI_Finalize % this time clean exit quit info=MPI_Finalize quit %------------------------------------------------------- % You might want to repeat this past session with rpi lamd instead of tcp... % but it only gets harder to understand %------------------------------------------------------- %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % mergeParent "startup" idea % we want to make a function for this repetitive/boring slave-startup %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% MPI_Init %------------------------------------------------------------------------ % excerpt from "info octave" output % Invoking Octave % =============== % ... % You can also specify the name of a file on the command line, and % Octave will read and execute the commands from the named file and then % exit when it is finished. %------------------------------------------------------------------------ [info children errs] = MPI_Comm_spawn ('/usr/X11R6/bin/xterm',... {'-e','octave','startups/startup_mergeParent.m'},... 1,MPI_INFO_NULL,0,MPI_COMM_SELF) %%%%% this _mergeParent file argument saves us typing %%%%% child gets blocked in _merge [info NEWORLD] = MPI_Intercomm_merge (children, 0) % child unblocks %%%%% yow! way too fast!!! %%%%%%%%%%%%%%%%% %%%%% file args to Octave simply run & quit %%%%% %------------------------------------------------------------------------ % excerpt from "octave --help" output % Usage: octave [options] % Options: % ... % --funcall FUNC Call Octave function FUNC with no arguments. %------------------------------------------------------------------------ % extracted from $OCTAVE/src/octave.cc % // The name of the optional initial function to call at startup. % // (--funcall FUNC) % static std::string fun_to_call; % ... % case FUNCALL_OPTION: % if (args.optarg ()) % fun_to_call = args.optarg (); % break; % ... % int retval = main_loop (fun_to_call); %------------------------------------------------------------------------ % extracted from $OCTAVE/src/toplev.cc % if (! fun_to_call.empty ()) % feval (fun_to_call); %------------------------------------------------------------------------ [info children errs] = MPI_Comm_spawn ('/usr/X11R6/bin/xterm',... {'-e','octave','--funcall','startups/startup_mergeParent.m'},... 1,MPI_INFO_NULL,0,MPI_COMM_SELF) %%%%% ouch! errmsg under octave-2.1.60 %%%%% error: feval: the symbol `startups/startup_mergeParent.m' %%%%% is not valid as a function MPI_Init % to unblock master octave MPI_Finalize % let's retry quit [info children errs] = MPI_Comm_spawn ('/usr/X11R6/bin/xterm',... {'-e','octave','--funcall','date','startups/startup_mergeParent.m'},... 1,MPI_INFO_NULL,0,MPI_COMM_SELF) %%%%% No idea of why prepending "date" makes it work :-) %%%%% this _mergeParent file argument saves us typing %%%%% child gets blocked in _merge [info NEWORLD] = MPI_Intercomm_merge (children, 0) % child unblocks %%%%% yow! way too fast!!! %%%%%%%%%%%%%%%%% %%%%% --funcall args to Octave also simply run & quit %%%%% MPI_Finalize % ok, we'll try with startup file's else branch quit %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % so we can't have the interpreter waiting for our input % if we specify a FILE or FUNCALL argument to Octave... too bad, % we'll have to stick to the "else" branch in .octaverc %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % in the UNIX shell, edit the .octaverc file and remove the % comment in the line before last % % if isempty(getenv('LAMPARENT')) % Parent task - welcome message % disp('Set SSI rpi to tcp with the command:') % disp(' putenv(''LAM_MPI_SSI_rpi'',''tcp''), MPI_Init') % disp('Help on MPI: help mpi') % else % spawned MATLAB process % % startup_mergeParent % this line, remove the % % end %------------------------------------------------------- % do this in the UNIX shell %------------------------------------------------------- pushd $MPITB_HOME; ls # usually ~/octave/mpitb ls -la startups # there is a startups subdir ls -la .oct* # there should be a link .octaverc->startups/ less .octaverc # this is the normal MPITB startup octaverc_MPI cd startups # and it adds this subdir to Octave path ls # more startup fragments less startup_mergeParent.m # this will be our startup for this session popd # or wherever, as long as .octaverc uses MPITB vi .octaverc # remove the comment to the mergeParent script octave # again back to work %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % mergeParent "startup" idea -- else branch of the normal startup %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% MPI_Init [info children errs] = MPI_Comm_spawn ('/usr/X11R6/bin/xterm',... {'-e','octave'},1,MPI_INFO_NULL,0,MPI_COMM_SELF) %%%%% no file argument this time %%%%% _mergeParent gets interpreted from .octaverc %%%%% child gets blocked in _merge [info NEWORLD] = MPI_Intercomm_merge (children, 0) % child unblocks %%%%% yup! there it is !!! %%%%%%%%%%%%%%%%% MPI_Finalize quit %%%%% oops! forgot again about child %%%%%%%%%%%% lamclean %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % Setting rpi on the fly with putenv/MPI_Init %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % do this in the UNIX shell %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% cd $MPITB_HOME # or wherever, as long as .octaverc uses MPITB export -n LAM_MPI_SSI_rpi # un-export any previous LAM_MPI_SSI_rpi env | egrep "LAM|MPI" # got rid of rpi envvar LAM_MPI_SSI_rpi octave # again back to work %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % Setting rpi on the fly with putenv/MPI_Init %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% help getenv help putenv putenv('LAM_MPI_SSI_rpi', 'lamd') % lamd behavior MPI_Init [info children errs] = MPI_Comm_spawn ('/usr/X11R6/bin/xterm',... {'-e','octave'}, 1,MPI_INFO_NULL,0,MPI_COMM_SELF) %%%%% we assume you edited .octaverc to call _merge %%%%% child gets blocked in _merge [info NEWORLD] = MPI_Intercomm_merge (children, 0) % child unblocks %%%%% type this on child Octave %%%%%%%%%%%%%%%%% type octaverc_MPI % found in OctavePath getenv('LAMPARENT') % last 2 lines octaverc type startup_mergeParent % found in OctavePath [info eh]=MPI_Errhandler_get(NEWORLD) % last 2 _merge errors_return = eh==MPI_ERRORS_RETURN % ok, safer [info eh]=MPI_Errhandler_get(NEWORLD) errors_return = eh==MPI_ERRORS_RETURN % ouch, each process MPI_Errhandler_set(NEWORLD,MPI_ERRORS_RETURN) % must set its own %------------------------------------------------------- % Checking for lamd behavior %------------------------------------------------------- A=ones(8,1024); whos A, sa=sizeof(A) % 64KB info=MPI_Send(A,1,7,NEWORLD) % blocks<=64KB B=zeros(8,1024); recv_B = all(all(B)) [info stat]=MPI_Recv(B,0,7,NEWORLD) % unblocks recv_B = all(all(B)) % ok B(1,1)=0; % clean & again recv_B = all(all(B)) [info stat]=MPI_Recv(B,0,7,NEWORLD) % ready info=MPI_Send(A,1,7,NEWORLD) % won't block due to ready recv recv_B = all(all(B)) % ok %------------------------------------------------------- % Starting all over within Octave %------------------------------------------------------- MPI_Finalize [info flag]=MPI_Initialized % it's certainly not amnesiac clear MPI_* % clear all MPITB from mem [info flag]=MPI_Initialized % love that getenv('LAM_MPI_SSI_rpi') % should be 'lamd' putenv('LAM_MPI_SSI_rpi','tcp') % if you followed instructions getenv('LAM_MPI_SSI_rpi') % should be 'tcp' now MPI_Init %%%%% ouch!!! forgot about the other child Octave MPI_Finalize quit [info children errs] = MPI_Comm_spawn ('/usr/X11R6/bin/xterm',... {'-e','octave'}, 1,MPI_INFO_NULL,0,MPI_COMM_SELF) [info NEWORLD] = MPI_Intercomm_merge (children, 0) % child unblocks %%%%% type this on new child %%%%%%%%%%%%%%%%% getenv('LAM_MPI_SSI_rpi') % should be 'tcp' %------------------------------------------------------- % Checking for tcp behavior %------------------------------------------------------- A=ones(8,1024); whos A, sa=sizeof(A) % 64KB info=MPI_Send(A,1,7,NEWORLD) % won't block B=zeros(8,1024); recv_B = all(all(B)) [info stat]=MPI_Recv(B,0,7,NEWORLD) % receives recv_B = all(all(B)) % ok A=ones(9,1024); whos A, sa=sizeof(A) % >64KB info=MPI_Send(A,1,7,NEWORLD) % blocks B=zeros(9,1024); recv_B = all(all(B)) [info stat]=MPI_Recv(B,0,7,NEWORLD) % unblocks recv_B = all(all(B)) % ok B(1,1)=0; recv_B = all(all(B)) % clean&again [info stat]=MPI_Recv(B,0,7,NEWORLD) % ready recv info=MPI_Send(A,1,7,NEWORLD) % won't block, unblocks recv recv_B = all(all(B)) % ok MPI_Finalize quit MPI_Finalize quit %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % Non-blocking (immediate) point-to-point % MPI_Isend, _Ibsend, _Issend, _Irsend, _Irecv, _Iprobe % MPI_Test, _Testall, _Testany, _Testsome % MPI_Wait, _Waitall, _Waitany, _Waitsome %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% (remember we assume $LAMBHOST includes ox1 & ox0, master node last in list) (remember we assume you can ssh to the other computer ox1 w/out passwd/output) (lamprep/lamboot before starting octave if this is a new ssh session) (you should have edited .octaverc so that it calls _mergeParent if child task) putenv('LAM_MPI_SSI_rpi','tcp') % better understood with tcp MPI_Init [info children errs] = MPI_Comm_spawn ('/usr/X11R6/bin/xterm',... {'-e','octave'}, 1,MPI_INFO_NULL,0,MPI_COMM_SELF) [info NEWORLD] = MPI_Intercomm_merge (children, 0) % child unblocks MPI_Errhandler_set(NEWORLD,MPI_ERRORS_RETURN) % safer this way A='Repeat this string until _8KB_! ' % some assorted vars A=repmat(A,1,256); whos A, sa=sizeof(A) % 8KB B=ones(8,1024); whos B, sb=sizeof(B) % 64KB C=ones(9,1024); whos C, sc=sizeof(C) % > 64KB (73728 bytes) help modes help nonblocking %------------------------------------------------------- % _Isend: Immediate (non-blocking) send % must _Test/_Wait for it to progress (at least under tcp rpi) %------------------------------------------------------- help MPI_Isend %%%%%%%%% under child Octave %%%%%%%%%%%%%%%%%%%% [info stat]=MPI_Probe(-1,-1,NEWORLD) % blocks [info rqa]=MPI_Isend(A,1,7,NEWORLD) % returns immediately/unblocks [info rqb]=MPI_Isend(B,1,8,NEWORLD) % that doesn't mean [info rqc]=MPI_Isend(C,1,9,NEWORLD) % xmit has progressed [info stat]=MPI_Probe(0,7,NEWORLD) % tag 7 queued [info stat]=MPI_Probe(0,8,NEWORLD) % tag 8 too [info stat]=MPI_Probe(0,9,NEWORLD) % all them help MPI_Testall % until we test/wait help MPI_Testany help MPI_Testsome [info flag stats] = MPI_Testall (rqa,rqb,rqc) % info==0, flag==0 - not ALL any([stats.len]) % all len==0 stats(1).src==MPI_ANY_SOURCE % or src==MPI_PROC_NULL stats(2).tag==MPI_ANY_TAG % or tag==MPI_UNDEFINED stats(3).err==MPI_ERR_PENDING % err==PENDING % none of them, in fact [info flag stats] = MPI_Testall (rqa,rqb,rqc) % info==0, flag==0, no progress [info flag stats] = MPI_Testall (rqa,rqb,rqc) % no progress [info idx flag stat] = MPI_Testany (rqa,rqb,rqc)% ANY of them? yes, progress stat.src==MPI_PROC_NULL % flag==1, idx=0 (1st msg) stat.tag==MPI_UNDEFINED stat.err==MPI_SUCCESS stat.len==0 % len==4 in LAM-7.0.4 ?!? help MPI_PROC_NULL help MPI_UNDEFINED [info idxs stats] = MPI_Testsome (rqa,rqb,rqc) % SOME of them? yes, progress stats.src==MPI_PROC_NULL % idxs=1 stats.tag==MPI_UNDEFINED % just 1 stat in this case stats.err==MPI_SUCCESS % but might be more stats.len==0 all_done = all([rqa rqb rqc]==0) % not finished yet rqa, rqb, rqc % rqc still pending [info idx stat ] = MPI_Waitany (rqa,rqb,rqc) % blocks due to rqc %%%%%%%%% under child Octave %%%%%%%%%%%%%%%%%%%% [info stat]=MPI_Probe(-1,-1,NEWORLD) % tag 7 1st A=repmat(' ',1,stat.len); % stat.len=8K A(1:60) [info stat]=MPI_Recv(A,0,7,NEWORLD) A(1:60) % there it is [info stat]=MPI_Probe(-1,-1,NEWORLD) % tag 8 then C=zeros(9,1024); s=sizeof(C) % > 64KB room_enough = s>=stat.len C(:,910:912) [info stat]=MPI_Recv(C,0,8,NEWORLD) stat.len % 64KB recvd C(:,910:912) % row-major C(:,end-2:end) % untouched [info stat]=MPI_Probe(-1,-1,NEWORLD) % tag 9 then [info stat]=MPI_Recv(C,0,9,NEWORLD) % unblocks Wait C(:,910:912) % overwritten C(:,end-2:end) % filled all_done = all([rqa rqb rqc]==0) % all done now idx==2 % rqc was the last pending stats.src==MPI_PROC_NULL stats.tag==MPI_UNDEFINED stats.err==MPI_SUCCESS stats.len==0 %%%%%%%%% another turn, clear buffers %%%%%%%%%%% A(1:30)=' '; A(1:60) % clear start B=zeros(8,1024); s=sizeof(B) % 64KB B(:,1:3) C(:,1:2)=0; C(:,1:3) % clear start [info rqa]=MPI_Irecv (A,0,7,NEWORLD) % 3 non-blcking [info rqb]=MPI_Irecv (B,0,8,NEWORLD) % concurrent [info rqc]=MPI_Irecv (C,0,9,NEWORLD) % requests [info flag]=MPI_Iprobe(-1,-1,NEWORLD) % empty queue help MPI_Waitany help MPI_Waitsome help MPI_Waitall [info idx stat ]=MPI_Waitany(rqa,rqb,rqc) % blocks info=MPI_Send(A,1,7,NEWORLD) % won't block, unblocks Waitany idx==0, stat.src==0, stat.tag==7, stat.len==8192 A(1:60) C(:,1:3) [info idxs stats]=MPI_Waitsome(rqa,rqb,rqc) % blocks info=MPI_Send(C,1,9,NEWORLD) % won't block, unblks Waitsome idxs==2, stats.src==0, stats.tag==9, stats.len==73728 C(:,1:3) % notice rqc unblocked... B(:,1:3) % ...before rqb all_done = all([rqa rqb rqc]==0) rqa, rqb, rqc [info stats]=MPI_Waitall(rqa,rqb,rqc) % blocks info=MPI_Send(A,1,7,NEWORLD) % won't block(tcp rpi), neither unblock(tag 7) info=MPI_Send(B,1,8,NEWORLD) % won't block, unblocks Waitall B(:,1:3) % recvd ok all_done = all([rqa rqb rqc]==0) stats(2).src==0, stats(2).tag==8, stats(2).len==65536 stats(2).err==MPI_SUCCESS all([stats(1).src, stats(3).src]==MPI_ANY_SOURCE) all([stats(1).tag, stats(3).tag]==MPI_ANY_TAG) all([stats(1).err, stats(3).err]==MPI_ERR_PENDING) all([stats(1).len, stats(3).len]==0) % that's the stat you get when you wait on finished req [info stats]=MPI_Waitall(rqa,rqb,rqc) % done stats(1), stats(2), stats(3) % can't wait anymore [info flag stats]=MPI_Testall(rqa,rqb,rqc) % flag==1 stats(1), stats(2), stats(3) % all done [info flag stat]=MPI_Iprobe(-1,-1,NEWORLD) % recall A A(1:30)=' '; A(1:60) [info stat]=MPI_Recv(A,0,7,NEWORLD) A(1:60) [info flag stat]=MPI_Iprobe(-1,-1,NEWORLD) % emptyqueue clear flag idx idxs info rqa rqb rqc s stat stats whos -variables clear all_done room_enough clear errs flag idx idxs info rqa rqb rqc s stat stats all_done whos -variables MPI_Finalize % too large for just 1 session quit % jump over the next... MPI_Finalize % ...Init cycle if you want quit % (not tired yet? great! :-) %------------------------------------------------------- % _Ibsend: Immediate (non-blocking) buffered (user provided) send % User provides buffer, instead of system buffer % Init cycle again... skip it if you didn't Finalize just above %------------------------------------------------------- putenv('LAM_MPI_SSI_rpi','tcp'), MPI_Init % better understood with tcp [info children errs] = MPI_Comm_spawn ( '/usr/X11R6/bin/xterm',{'-e','octave'},... 1,MPI_INFO_NULL,0,MPI_COMM_SELF) [info NEWORLD] = MPI_Intercomm_merge (children, 0) % child unblocks MPI_Errhandler_set(NEWORLD,MPI_ERRORS_RETURN) % safer this way A='Repeat this string until _8KB_! ' % some assorted vars A=repmat(A,1,256); whos A, sa=sizeof(A) % 8KB B=ones(8,1024); whos B, sb=sizeof(B) % 64KB C=ones(9,1024); whos C, sc=sizeof(C) % > 64KB (73728 bytes) whos -variables %%%%%%% ON CHILD %%%%%%%%%%%%%%%%%%%%%%%%%%%%%% A=repmat(' ',1,8192); % recv counterparts B=zeros(8,1024); C=zeros(9,1024); whos -variables help nonblocking %------------------------------------------------------- % Init cycle done... jump here if you didn't Finalize just above %------------------------------------------------------- help MPI_Ibsend whos A B C, sa, sb, sc % A,B,C, -- 8,64,72 KB sz=sa+sb+2*MPI_BSEND_OVERHEAD % room for A & B, later C C_fits_where_AB = sc+MPI_BSEND_OVERHEAD < sz snd=repmat(' ',1,sz); s=sizeof(snd) % around 72KB info = MPI_Buffer_attach(snd) % for Ibsend %%%%%%%%% under child Octave %%%%%%%%%%%%%%%%%%%% [info flag stat]=MPI_Iprobe(-1,-1,NEWORLD) % emptyqueue [info rqa]=MPI_Ibsend(A,1,7,NEWORLD) % doesn't block or progress [info rqb]=MPI_Ibsend(B,1,8,NEWORLD) % neither [info rqc]=MPI_Ibsend(C,1,9,NEWORLD) % neither [info stats]=MPI_Waitall(rqa,rqb,rqc) % nothing to wait stats(1), stats(2), stats(3) % len==0 stats(1).src==MPI_PROC_NULL stats(2).tag==MPI_UNDEFINED stats(3).err==MPI_SUCCESS [info stat]=MPI_Probe(0,7,NEWORLD) % tag 7 queued [info flag stat]=MPI_Iprobe(0,8,NEWORLD)% tag 8 too [info flag stat]=MPI_Iprobe(0,9,NEWORLD)% tag 9 as well A(1:30)=' '; A(1:60) % cleared to B(:,1:2)=0; B(:,1:3) % detect when C(:,1:2)=0; C(:,1:3) % it's recvd [info rqa]=MPI_Irecv (A,0,7,NEWORLD) % 3 non-blcking [info rqb]=MPI_Irecv (B,0,8,NEWORLD) % concurrent [info rqc]=MPI_Irecv (C,0,9,NEWORLD) % requests A(1:60), B(:,1:3), C(:,1:3) % C not yet... why? % yup, snd buffer [info idxs stats]=MPI_Testsome(rqa,rqb,rqc) % idxs 0/1 stats(1), stats(2) % tags 7/8 [info flag stats]=MPI_Testall(rqa,rqb,rqc) % flag==0 rqa, rqb, rqc [info stats]=MPI_Waitall(rqa,rqb,rqc) % blocks!!! [info flag stat]=MPI_Test(rqc) % must push it :-) C(:,1:3) rqa, rqb, rqc stats(1), stats(2), stats(3) % ok, unblocked rqc [info flag stat]=MPI_Iprobe(-1,-1,NEWORLD) % done clear flag idxs info rqa rqb rqc stat stats whos -variables info = MPI_Buffer_detach clear errs flag info rqa rqb rqc s snd stat stats sz C_* whos -variables MPI_Finalize % too large for just 1 session quit % either Init cycle or jump MPI_Finalize % over to the next session quit %------------------------------------------------------- % _Issend: Immediate (non-blocking) synchronous (waits for matching Recv) send % Rather self-contradictory :-) basically sending without buffers % paradoxically, far easier to understand ;-) % Init cycle again... skip it if you didn't Finalize just above %------------------------------------------------------- putenv('LAM_MPI_SSI_rpi','tcp'), MPI_Init % better understood with tcp [info children errs] = MPI_Comm_spawn ( '/usr/X11R6/bin/xterm',{'-e','octave'},... 1,MPI_INFO_NULL,0,MPI_COMM_SELF) [info NEWORLD] = MPI_Intercomm_merge (children, 0) % child unblocks MPI_Errhandler_set(NEWORLD,MPI_ERRORS_RETURN) % safer this way A='Repeat this string until _8KB_! ' % some assorted vars A=repmat(A,1,256); whos A, sa=sizeof(A) % 8KB B=ones(8,1024); whos B, sb=sizeof(B) % 64KB C=ones(9,1024); whos C, sc=sizeof(C) % > 64KB (73728 bytes) whos -variables %%%%%%% ON CHILD %%%%%%%%%%%%%%%%%%%%%%%%%%%%%% A=repmat(' ',1,8192); % recv counterparts B=zeros(8,1024); C=zeros(9,1024); whos -variables help nonblocking %------------------------------------------------------- % Init cycle done... jump here if you didn't Finalize just above %------------------------------------------------------- help MPI_Issend %%%%%%%%% under child Octave %%%%%%%%%%%%%%%%%%%% [info flag stat]=MPI_Iprobe(-1,-1,NEWORLD) % emptyqueue [info rqa]=MPI_Issend(A,1,7,NEWORLD) % doesn't block or progress [info rqb]=MPI_Issend(B,1,8,NEWORLD) % neither [info rqc]=MPI_Issend(C,1,9,NEWORLD) % neither [info idx stat ]=MPI_Waitany(rqa,rqb,rqc) % blocks, of course [info flag stat]=MPI_Iprobe(0,7,NEWORLD)% tag 7 queued [info flag stat]=MPI_Iprobe(0,8,NEWORLD)% tag 8 too [info flag stat]=MPI_Iprobe(0,9,NEWORLD)% tag 9 as well A(1:30)=' '; A(1:60) % cleared to B(:,1:2)=0; B(:,1:3) % detect when C(:,1:2)=0; C(:,1:3) % it's recvd [info stat]=MPI_Recv(C,0,9,NEWORLD) % unblks idx=2 C(:,1:3) % any order would do B(:,1:3), A(1:60) rqa, rqb, rqc [info idxs stats]=MPI_Waitsome(rqa,rqb,rqc) % blocks rqa/b [info rqa]=MPI_Irecv (A,0,7,NEWORLD) % unblks idxs=0 B(:,1:3), A(1:60) rqa, rqb, rqc [info stats]=MPI_Waitall(rqa,rqb,rqc) % blocks due to rqb [info stat]=MPI_Recv (B,0,8,NEWORLD) % unblocks B(:,1:3) rqa, rqb, rqc stats(1), stats(2), stats(3) [info flag stat]=MPI_Iprobe(-1,-1,NEWORLD) % done clear flag info rqa stat whos -variables clear errs idx idxs info rqa rqb rqc s snd stat stats whos -variables MPI_Finalize % too large for just 1 session quit % either Init cycle or jump MPI_Finalize % over to the next session quit %------------------------------------------------------- % _Irsend: Immediate (non-blocking) ready ("requires" matching Recv) send % Init cycle again... skip it if you didn't Finalize just above %------------------------------------------------------- putenv('LAM_MPI_SSI_rpi','tcp'), MPI_Init % better understood with tcp [info children errs] = MPI_Comm_spawn ( '/usr/X11R6/bin/xterm',{'-e','octave'},... 1,MPI_INFO_NULL,0,MPI_COMM_SELF) [info NEWORLD] = MPI_Intercomm_merge (children, 0) % child unblocks MPI_Errhandler_set(NEWORLD,MPI_ERRORS_RETURN) % safer this way A='Repeat this string until _8KB_! ' % some assorted vars A=repmat(A,1,256); whos A, sa=sizeof(A) % 8KB B=ones(8,1024); whos B, sb=sizeof(B) % 64KB C=ones(9,1024); whos C, sc=sizeof(C) % > 64KB (73728 bytes) whos -variables %%%%%%% ON CHILD %%%%%%%%%%%%%%%%%%%%%%%%%%%%%% A=repmat(' ',1,8192); % recv counterparts B=zeros(8,1024); C=zeros(9,1024); whos -variables help nonblocking %------------------------------------------------------- % Init cycle done... jump here if you didn't Finalize just above %------------------------------------------------------- help MPI_Irsend %%%%%%%%% under child Octave %%%%%%%%%%%%%%%%%%%% [info flag stat]=MPI_Iprobe(-1,-1,NEWORLD) % emptyqueue [info rqa]=MPI_Irsend(A,1,7,NEWORLD) % doesn't block or progress [info rqb]=MPI_Irsend(B,1,8,NEWORLD) % neither [info rqc]=MPI_Irsend(C,1,9,NEWORLD) % neither [info idx stat ]=MPI_Waitany (rqa,rqb,rqc) % idx ==0 done [info idxs stats]=MPI_Waitsome(rqa,rqb,rqc) % idxs==1 done rqa, rqb, rqc [info stats]=MPI_Waitall (rqa,rqb,rqc) % blocks due to rqc %%%%%%%%% under child Octave %%%%%%%%%%%%%%%%%%%% [info flag stat]=MPI_Iprobe(0,7,NEWORLD)% tag 7 queued [info flag stat]=MPI_Iprobe(0,8,NEWORLD)% tag 8 too [info flag stat]=MPI_Iprobe(0,9,NEWORLD)% tag 9 as well A(1:30)=' '; A(1:60) % cleared to B(:,1:2)=0; B(:,1:3) % detect when C(:,1:2)=0; C(:,1:3) % it's recvd [info stat]=MPI_Recv(C,0,9,NEWORLD) % unblks Wtall C(:,1:3) B(:,1:3), A(1:60) rqa, rqb, rqc stats(1), stats(2), stats(3) [info stats]=MPI_Waitall (rqa,rqb,rqc) % can't wait anymore [info stat]=MPI_Recv(A,0,7,NEWORLD) B(:,1:3), A(1:60) [info stat]=MPI_Recv(B,0,8,NEWORLD) B(:,1:3) [info flag stat]=MPI_Iprobe(-1,-1,NEWORLD) % done MPI_Finalize quit MPI_Finalize quit %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % Concurrent Send/Recv % MPI_Sendrecv, _Sendrecv_replace %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % *** WE ARE USING 3 COMPUTERS THIS TIME :-) *** %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% (this time $LAMBHOST includes ox1 ox2 & ox0, master node last in list) (you can ssh to the nodes (ox1, ox2) from the headed node w/out passwd/output) (Octave is installed for all hosts) (lamprep/lamboot before starting octave if this is a new ssh session) (you should have edited .octaverc so that it calls _mergeParent if child task) putenv('LAM_MPI_SSI_rpi','tcp'), MPI_Init % better understood with tcp [info children errs] = MPI_Comm_spawn ( '/usr/X11R6/bin/xterm',{'-e','octave'},... 2,MPI_INFO_NULL,0,MPI_COMM_SELF) % notice that 2 [info NEWORLD] = MPI_Intercomm_merge (children, 0) % children unblocks MPI_Errhandler_set(NEWORLD,MPI_ERRORS_RETURN) % isn't it cool? % string hostname same size everywhere [info rank]=MPI_Comm_rank(NEWORLD) hnam=system('hostname'), whos hnam hnam(1,60)=' '; whos hnam A(1:60)=' '; whos A %%%%% ON FIRST CHILD %%%%%%% [info rank]=MPI_Comm_rank(NEWORLD) hnam=system('hostname') hnam(1,60)=' '; A(1:60)=' '; whos A hnam %%%%% ON SECOND CHILD %%%%%%% [info rank]=MPI_Comm_rank(NEWORLD) hnam=system('hostname') hnam(1,60)=' '; A(1:60)=' '; whos A hnam help modes %------------------------------------------------------- % Can Sendrecv from same child %------------------------------------------------------- help MPI_Sendrecv [info stat]=MPI_Sendrecv(hnam,1,7, A,1,8, NEWORLD), A % blocks (recv part) %%%%% ON FIRST CHILD %%%%%%% [info stat]=MPI_Probe(-1,-1,NEWORLD) [info stat]=MPI_Recv (A,0,7,NEWORLD), A info =MPI_Send(hnam,0,8,NEWORLD) % now it unblocks %------------------------------------------------------- % It is ...recv what really blocks Sendrecv, send... gets buffered %------------------------------------------------------- %%%%% ON SECOND CHILD %%%%%%% info =MPI_Send(hnam,0,9,NEWORLD) [info stat]=MPI_Recv( A,0,7,NEWORLD), A % blcks A [info stat]=MPI_Sendrecv(hnam,2,7, A,2,9, NEWORLD) % doesn't block, unblks A %------------------------------------------------------- % Can Sendrecv from/to different children (doesnot cascade :-) %------------------------------------------------------- %%%%% ON SECOND CHILD %%%%%%% [info stat]=MPI_Recv(A,0,7,NEWORLD), A % blocks [info stat]=MPI_Sendrecv(hnam,2,7, A,1,8, NEWORLD), A % blocks(recv) % _and_ unblocks 2nd %%%%% ON FIRST CHILD %%%%%%% info=MPI_Send(hnam,0,8,NEWORLD) % unblocks, no cascade %------------------------------------------------------- % Can Sendrecv against Sendrecv %------------------------------------------------------- A(1:10)=' ' [info stat]=MPI_Sendrecv(hnam,1,7, A,1,8, NEWORLD), A % blocks (recv) %%%%% ON FIRST CHILD %%%%%%% A(1:15)=' ' % unblocks [info stat]=MPI_Sendrecv(hnam,0,8, A,0,7, NEWORLD), A %------------------------------------------------------- % Circular Sendrecv %------------------------------------------------------- A(1:10)=' ' [info stat]=MPI_Sendrecv(hnam,1,7, A,2,9, NEWORLD), A % blocks (recv) %%%%% ON FIRST CHILD %%%%%%% A(1:15)=' ' % doesn't block [info stat]=MPI_Sendrecv(hnam,2,8, A,0,7, NEWORLD), A %%%%% ON SECOND CHILD %%%%%%% A(1:15)=' ' % unblocks [info stat]=MPI_Sendrecv(hnam,0,9, A,1,8, NEWORLD), A %------------------------------------------------------- % Can Sendrecv in-place (same buffer), as long as there is room enough for recv %------------------------------------------------------- A(1:10)=' ' % blocks (recv) [info stat]=MPI_Sendrecv_replace(hnam,1,7, 2,9, NEWORLD), hnam %%%%% ON FIRST CHILD %%%%%%% A(1:15)=' ' % doesn't block [info stat]=MPI_Sendrecv_replace(hnam,2,8, 0,7, NEWORLD), hnam %%%%% ON SECOND CHILD %%%%%%% A(1:15)=' ' [info stat]=MPI_Recv( A,1,8,NEWORLD), A info =MPI_Send(hnam,0,9,NEWORLD) % unblocks MPI_Finalize quit MPI_Finalize quit MPI_Finalize quit %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % Recv cancellation (no-op for Send) % MPI_Cancel, _Test_cancelled %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% (remember we assume $LAMBHOST includes ox1 & ox0, master node last in list) (remember we assume you can ssh to the other computer ox1 w/out passwd/output) (lamprep/lamboot before starting octave if this is a new ssh session) (you should have edited .octaverc so that it calls _mergeParent if child task) putenv('LAM_MPI_SSI_rpi','tcp'), MPI_Init % better understood with tcp [info children errs] = MPI_Comm_spawn ( '/usr/X11R6/bin/xterm',{'-e','octave'},... 1,MPI_INFO_NULL,0,MPI_COMM_SELF) [info NEWORLD] = MPI_Intercomm_merge (children, 0) % child unblocks MPI_Errhandler_set(NEWORLD,MPI_ERRORS_RETURN) % safer this way A='Repeat this string until _8KB_! ' % some assorted vars A=repmat(A,1,256); whos A, sa=size(A) % 8KB B=ones(8,1024); whos B, sb=size(B) % 64KB C=ones(9,1024); whos C, sc=size(C) % > 64KB (73728 bytes) whos -variables %%%%%%% ON CHILD %%%%%%%%%%%%%%%%%%%%%%%%%%%%%% A=repmat(' ',1,8192); % recv counterparts B=zeros(8,1024); C=zeros(9,1024); whos -variables %------------------------------------------------------- % Can cancel only receives, not sends %------------------------------------------------------- help cancel help MPI_Cancel help MPI_Test_cancelled %%%%%%% ON CHILD %%%%%%%%%%%%%%%%%%%%%%%%%%%%%% [info rqa]=MPI_Irecv (A,0,7,NEWORLD) % 3 non-blcking [info rqb]=MPI_Irecv (B,0,8,NEWORLD) % concurrent [info rqc]=MPI_Irecv (C,0,9,NEWORLD) % requests A(1:60), B(:,1:3), C(:,1:3) % original data rqa, rqb, rqc % all active [info flag]=MPI_Iprobe(-1,-1,NEWORLD) % empty queue rqa, rqb, rqc % all active info =MPI_Cancel(rqb) % cancel B rqa, rqb, rqc % all active [info idx stat ]=MPI_Waitany(rqa,rqb,rqc) % idx==1 rqa, rqb, rqc % rqb finished stat.src==MPI_CANCEL_SOURCE stat.tag==MPI_UNDEFINED stat.err==MPI_SUCCESS [info flag statb]=MPI_Test_cancelled(stat) %%%%%%%%%%%% notice how cancel info "washes out" stat, statb, rqb [info stat ]=MPI_Wait(rqb) % won't wait stat.src==MPI_ANY_SOURCE stat.tag==MPI_ANY_TAG stat.err==MPI_ERR_PENDING [info flag stat2]=MPI_Test_cancelled(stat ) % flag==0 [info flag stat2]=MPI_Test_cancelled(statb) % flag==1 rqa, rqb, rqc % recall: rqb cancelled A(1:60), B(:,1:3), C(:,1:3) % still original data [info idx stat ]=MPI_Waitany(rqa,rqb,rqc) % blocks %%%%%% back to parent Octave %%%%%% info =MPI_Send (A,1,7,NEWORLD) % won't block, unblks Waitany info =MPI_Send (B,1,8,NEWORLD) % not expected at receiver [info rqc]=MPI_Isend(C,1,9,NEWORLD) % can't be cancelled? sure? info =MPI_Cancel(rqc), rqc % no-op for send ?!? info==0 [info flag statc] = MPI_Test(rqc), rqc % certainly, not completed [info flag stat ] = MPI_Test(rqc), rqc % nope, no progress [info flag stat ] = MPI_Test_cancelled(statc) % nope, not cancelled %[info stat ] = MPI_Wait(rqc) % don't do that, would block %%%%%%%%%%% again at child Octave %%%%%%%%%%%%%%%%%%% A(1:60), B(:,1:3), C(:,1:3) % there A is rqa, rqb, rqc % rqa finished too [info flag stat]=MPI_Iprobe(0,-1,NEWORLD) % tag 8 1st [info flag stat]=MPI_Iprobe(0, 9,NEWORLD) %no 9 waiting [info flag stat]=MPI_Iprobe(0, 7,NEWORLD) %no 7 waiting rqa, rqb, rqc % rqc no progrs info =MPI_Cancel(rqa) % info==13 info ==MPI_ERR_ARG % error arg [nfo msg ] =MPI_Error_string(info) % invalid rqa rqa, rqb, rqc % it's NULL rqa==MPI_REQUEST_NULL info =MPI_Cancel(rqc) % cancelled!!! rqa, rqb, rqc [info flag statc] = MPI_Test(rqc) % not completed rqa, rqb, rqc [info flag stat ] = MPI_Test_cancelled(statc) % nope %%%%% [info stat ] = MPI_Wait(rqc) % would block %%%%%% again at parent Octave %%%%%% info =MPI_Cancel(rqc), rqc % double-cancelled ?!? info==0 %%%%%%%%%%% again at child Octave %%%%%%%%%%%%%%%%%%% rqa, rqb, rqc % rqc is rambo [info flag stats] = MPI_Testall (rqa,rqb,rqc) % not ALL A(1:60) % there A was B(:,1:3), C(:,1:3) % untouched clear stat stats [info flag stat ] = MPI_Test(rqa) % completed [info flag stat ] = MPI_Test_cancelled(stat) % nope [info stat ] = MPI_Wait(rqa) % won't wait [info flag stat ] = MPI_Test(rqb) % cancel info [info flag stat ] = MPI_Test_cancelled(stat) % vanished [info flag stat ] = MPI_Test_cancelled(statb) [info stat ] = MPI_Wait(rqb) % won't wait [info flag stat ] = MPI_Test(rqc) % not completed [info flag stat ] = MPI_Test_cancelled(stat) % not [info flag stat ] = MPI_Test_cancelled(statc) % really %%%%%% [info statc] = MPI_Wait(rqc) % would block B(:,1:3) % untouched [info flag stat ] = MPI_Iprobe(-1,-1,NEWORLD)% tag 8 [info stat ] = MPI_Wait(rqb) % didn't block B(:,1:3) % recall unexpected [info stat ] = MPI_Recv(B,0,8,NEWORLD) B(:,1:3) MPI_Finalize quit MPI_Finalize quit %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % Persistent requests % MPI_[-BSR]send_init, _Recv_init, _Start[all], _Request_free % MPI_Address useful at last! :-) %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% (remember we assume $LAMBHOST includes ox1 & ox0, master node last in list) (remember we assume you can ssh to the other computer ox1 w/out passwd/output) (lamprep/lamboot before starting octave if this is a new ssh session) (you should have edited .octaverc so that it calls _mergeParent if child task) putenv('LAM_MPI_SSI_rpi','tcp'), MPI_Init % better understood with tcp [info children errs] = MPI_Comm_spawn ( '/usr/X11R6/bin/xterm',{'-e','octave'},... 1,MPI_INFO_NULL,0,MPI_COMM_SELF) [info NEWORLD] = MPI_Intercomm_merge (children, 0) % child unblocks MPI_Errhandler_set(NEWORLD,MPI_ERRORS_RETURN) % safer this way A='Repeat this string until _8KB_! ' % some assorted vars A=repmat(A,1,256); whos A, sa=size(A) % 8KB B=ones(8,1024); whos B, sb=size(B) % 64KB C=ones(9,1024); whos C, sc=size(C) % > 64KB (73728 bytes) whos -variables %%%%%%% ON CHILD %%%%%%%%%%%%%%%%%%%%%%%%%%%%%% A=repmat(' ',1,8192); % recv counterparts B=zeros(8,1024); C=zeros(9,1024); whos -variables %------------------------------------------------------- % A small discussion about MPI buffers, Octave variables and shallow copies %------------------------------------------------------- [info addr] = MPI_Address (A) % MPI_Address prints by itself A_rep_data = dec2hex(addr) % the returned addr in hex print_info(A) % which is rep->data address %-----------------------------------------------% % This address is passed to LAM/MPI % % on every Send/Recv/Pack/Unpack routine % % All MPITB commands that require an MPI buffer % % call the same aux function get_MPI_Buff % %-----------------------------------------------% A3 = A2 = A1 = A; % get 3 copies of A whos A* A_rep_data print_info(A) % notice rep->data addresses print_info(A1) % are all equal print_info(A2) % that would be a problem print_info(A3) % if we wanted to recv there [info addr] = MPI_Address (A) % MPI_Address useful at last! A_rep_data = dec2hex(addr) % new address in hex == repdata print_info(A) % rep->data has changed!!! [info addr] = MPI_Address (A) % no need to change again A_rep_data print_info(A) % A address is different print_info(A1) % the others are still equal print_info(A2) % A was deep-copied by print_info(A3) % get_MPI_Buff from MPI_Address %-------------------------------------------------- %%% MPITB uses an auxiliary function get_MPI_Buff() %%% which takes care of guessing the rep->data address for the %%% different variable classes (even scalars). %-------------------------------------------------- %%% If an MPITB command asks for get_MPI_Buff(), the probable reason is that %%% it needs the address to pass it to MPI, who may overwrite there, %%% so it was thought it is a good idea to unshare the rep->data address %%% It is thus an unavoidable side-effect, by design of get_MPI_Buff() %-------------------------------------------------- %%% MPI_Address calls get_MPI_Buff(), so the side effect is that %%% variables whose MPI_Address is asked, are deeply made_unique. %%% MPITB user has not to worry about shallow/deep copies: %%% whenever she uses a variable as MPI buffer, get_MPI_Buff unshares it %%% whenever Octave writes to a variable, Octave unshares it (of course) %-------------------------------------------------- print_info(A1) % notice current address A1(1:10)='1st copy! '; A1(1:30) % Octave unshares var on writes print_info(A1) % A1 address is now different [info addr1] = MPI_Address (A1) % so get_MPI_Buff doesn't need A1_rep_data = dec2hex(addr1) % to change/deep-copy it addr, addr1 print_info(A2) % only A2 and A3 shared now print_info(A3) % shallow copies of same addr %------------------------------------------------------- % Persistent requests: concept (Recv) %------------------------------------------------------- help persistent help MPI_Recv_init help MPI_Start help MPI_Request_free %%%%%%%%% ON CHILD %%%%%%%%%%%%%%%%%%%%%%%%%%% [info req]=MPI_Recv_init(A,0,7,NEWORLD) [info flag stat]=MPI_Test(req) % flag==1, done stat.src==MPI_ANY_SOURCE stat.tag==MPI_ANY_TAG stat.err==MPI_ERR_PENDING [info flag stat]=MPI_Test(req) % nothing happens [info stat]=MPI_Wait(req) % doesn't block %%%%% copy-paste this whole section (for loop) %%%%% at once, as one complete block %%%%% ------------------------------ %%%%% oldPSO = page_screen_output page_screen_output = 0 for n = 1:3 info =MPI_Start(req) % active [info flag stat]=MPI_Test (req) % flag==0 [info stat]=MPI_Wait (req) % blocks A(1:30) % see results end page_screen_output = oldPSO %%%%% ------------------------------ %%%%% %%%%% copy-paste these lines one after another %%%%% see the slave Octave window meanwhile info = MPI_Send(A1,1,7,NEWORLD) % won't block, unblks Wait, A='1st...' info = MPI_Send(A2,1,7,NEWORLD) % won't block, unblks Wait, A='Repeat' info = MPI_Send(A3,1,7,NEWORLD) % won't block, unblks Wait, A='Repeat' %--------------------------------------------------------- % side-effect: A2 was unshared when sent (in the 2nd line) %--------------------------------------------------------- print_info(A2) % now they're deep copies as well print_info(A3) % copies of the same data, of course %%%%%%%%% ON CHILD %%%%%%%%%%%%%%%%%%%%%%%%%%% req % req still ok % we could repeat the for loop again if we wanted info = MPI_Request_free (req), req % cool, isn't? info = MPI_Request_free (req) % info==7 [nfo msg ] =MPI_Error_string(info) % invalid req info== MPI_ERR_REQUEST help MPI_ERR_REQUEST req==MPI_REQUEST_NULL %------------------------------------------------------- % Persistent requests: concept (Send) %------------------------------------------------------- help persistent help MPI_Send_init [info req]=MPI_Send_init(C,1,9,NEWORLD) % C > 64KB (tcp_short) [info flag stat]=MPI_Test(req) % flag==1, done stat.src==MPI_ANY_SOURCE stat.tag==MPI_ANY_TAG stat.err==MPI_ERR_PENDING [info flag stat]=MPI_Test(req) % nothing happens [info stat]=MPI_Wait(req) % doesn't block %%%%%%%%% ON CHILD %%%%%%%%%%%%%%%%%%%%%%%%%%% [info flag stat ] = MPI_Iprobe(-1,-1,NEWORLD) % clean %%%%% copy-paste this whole section (for loop) %%%%% at once, as one complete block %%%%% ------------------------------ %%%%% oldPSO = page_screen_output page_screen_output = 0 for n = 1:3 C(:,1:3)=n; % C > 64KB (tcp_short) info =MPI_Start(req) % activate, will block (tcp rpi) info =MPI_Start(req) % can't start next, wait for this info ==MPI_ERR_REQUEST [info msg]=MPI_Error_string(info) [info flag stat]=MPI_Test (req) % flag==0, not completed [info stat]=MPI_Wait (req) % blocks end page_screen_output = oldPSO %%%%% ------------------------------ %%%%% %%%%%%%%% ON CHILD %%%%%%%%%%%%%%%%%%%%%%%%%%% %%%%% copy-paste these lines one after another %%%%% see the master Octave window meanwhile [info stat]=MPI_Probe(-1,-1,NEWORLD) % envelope here C(:,1:3) % C==0 [info stat]=MPI_Recv (C,0,9,NEWORLD) % unblocks C(:,1:3) % C==1 [info stat]=MPI_Recv (C,0,9,NEWORLD) C(:,1:3) % C==2 [info stat]=MPI_Recv (C,0,9,NEWORLD) C(:,1:3) % C==3 [info flag stat ] = MPI_Iprobe(-1,-1,NEWORLD) % clean info = MPI_Request_free (req) % how about B==64KB ??? [info req]=MPI_Send_init(B,1,8,NEWORLD) % B = 64KB (tcp_short) %%%%% copy-paste this whole section (for loop) %%%%% at once, as one complete block %%%%% ------------------------------ %%%%% page_screen_output = 0 for n = 1:3 B(:,1:3)=n; % B = 64KB (tcp_short) info =MPI_Start(req) % activate, will block (tcp rpi) info =MPI_Start(req) % can't start next, wait for this info ==MPI_ERR_REQUEST % 1st turn / 2nd turn / 3rd turn (blks) [info flag stat]=MPI_Test (req) % flag==1 / flag==0 / flag=0 (blocks) stat.src==MPI_PROC_NULL stat.tag==MPI_UNDEFINED stat.err==MPI_SUCCESS [info stat]=MPI_Wait (req) % won't blk / won't /) stat.src==[MPI_ANY_SOURCE, MPI_PROC_NULL] stat.tag==[MPI_ANY_TAG, MPI_UNDEFINED] stat.err==[MPI_ERR_PENDING, MPI_SUCCESS] end page_screen_output = oldPSO %%%%% ------------------------------ %%%%% %%%%%%%%% ON CHILD %%%%%%%%%%%%%%%%%%%%%%%%%%% %%%%% copy-paste these lines one after another %%%%% see the master Octave window meanwhile [info stat]=MPI_Probe(-1,-1,NEWORLD) % unblocks!!! B(:,1:3) % B==0 [info stat]=MPI_Recv (B,0,8,NEWORLD) B(:,1:3) % B==1 [info stat]=MPI_Recv (B,0,8,NEWORLD) B(:,1:3) % B==2 [info stat]=MPI_Recv (B,0,8,NEWORLD) B(:,1:3) % B==3 [info flag stat ] = MPI_Iprobe(-1,-1,NEWORLD) % clean info = MPI_Request_free (req) MPI_Finalize % too large for just 1 session quit % either Init cycle or jump MPI_Finalize % over to the next session quit %------------------------------------------------------- % Persistent requests: modes ([BSR]send) % Several requests at once: _Startall % Init cycle again... skip it if you didn't Finalize just above %------------------------------------------------------- putenv('LAM_MPI_SSI_rpi','tcp'), MPI_Init % better understood with tcp [info children errs] = MPI_Comm_spawn ( '/usr/X11R6/bin/xterm',{'-e','octave'},... 1,MPI_INFO_NULL,0,MPI_COMM_SELF) [info NEWORLD] = MPI_Intercomm_merge (children, 0) % child unblocks MPI_Errhandler_set(NEWORLD,MPI_ERRORS_RETURN) % safer this way A='Repeat this string until _8KB_! ' % some assorted vars A=repmat(A,1,256); whos A, sa=sizeof(A) % 8KB B=ones(8,1024); whos B, sb=sizeof(B) % 64KB C=ones(9,1024); whos C, sc=sizeof(C) % > 64KB (73728 bytes) whos -variables %%%%%%% ON CHILD %%%%%%%%%%%%%%%%%%%%%%%%%%%%%% A=repmat(' ',1,8192); % recv counterparts B=zeros(8,1024); C=zeros(9,1024); whos -variables %------------------------------------------------------- % Init cycle done... jump here if you didn't Finalize just above %------------------------------------------------------- help persistent help MPI_Bsend_init help MPI_Rsend_init help MPI_Ssend_init help MPI_Startall sz=sa+MPI_BSEND_OVERHEAD % 8232 bytes snd=repmat(' ',1,sz); s=sizeof(snd) % room for Bsend(A) info=MPI_Buffer_attach(snd), whos snd % user buffer [info rqa]=MPI_Bsend_init(A,1,7,NEWORLD) [info rqb]=MPI_Rsend_init(B,1,8,NEWORLD) [info rqc]=MPI_Ssend_init(C,1,9,NEWORLD) A(1:60)=' '; B(:,1:3)=0; C(:,1:3)=0; [info flag stat ] = MPI_Iprobe(-1,-1,NEWORLD) % clean A(1:60), B(:,1:3), C(:,1:3) % none recv yet info=MPI_Startall(rqa,rqb,rqc) % envelopes snt eagerly [info flag stat ] = MPI_Iprobe(-1,-1,NEWORLD) % A,tag 7 [info flag stat ] = MPI_Iprobe(-1, 8,NEWORLD) % B too [info flag stat ] = MPI_Iprobe(-1, 9,NEWORLD) % and C [info idxs stats]=MPI_Testsome(rqa,rqb,rqc) % A-B can progress A_B_done = all (idxs==[0 1]) % A - it's buffered stats(1).src==MPI_PROC_NULL % B - it's <= 64KB stats(1).tag==MPI_UNDEFINED stats(1).err==MPI_SUCCESS [info idxs stats]=MPI_Testsome(rqa,rqb,rqc) % idxs=[], C can't [info flag stats]=MPI_Testall (rqa,rqb,rqc) % not ALL A(1:60), B(:,1:3), C(:,1:3) % none recv yet [info rqa]=MPI_Irecv (A,0,7,NEWORLD) % 3 non-blcking [info rqb]=MPI_Irecv (B,0,8,NEWORLD) % concurrent [info rqc]=MPI_Irecv (C,0,9,NEWORLD) % requests A(1:60), B(:,1:3), C(:,1:3) % A, B, just recvd [info flag stats]=MPI_Testall(rqa,rqb,rqc) % "pushing" [info idxs stats]=MPI_Testsome(rqa,rqb,rqc) % idxs[0 1] stats(1), stats(2) % A & B A(1:60), B(:,1:3), C(:,1:3) % nothing changed (A/B) [info idxs stats]=MPI_Waitsome(rqa,rqb,rqc) % blocks!!! :-) [info stats]=MPI_Waitall(rqa,rqb,rqc) % gosh, hard to push ;-) stats(1), stats(2), stats(3) % rqc finally finished !!! C(:,1:3) [info flag stat ] = MPI_Iprobe(-1,-1,NEWORLD) % clean rqa, rqb, rqc info=MPI_Request_free(rqa) % not usually done info=MPI_Request_free(rqb) % for Irecv/I[brs]send info=MPI_Request_free(rqc) % see help page help MPI_Request_free info ==MPI_ERR_REQUEST % can't free NULL req [inf2 msg]=MPI_Error_string(info) MPI_Finalize quit info = MPI_Buffer_detach rqa, rqb, rqc info=MPI_Request_free(rqa) % usually done for MPI_*_init info=MPI_Request_free(rqb) info=MPI_Request_free(rqc) all([rqa rqb rqc]==MPI_REQUEST_NULL) [info flag stat]=MPI_Test (rqc) % graceful testing on NULL [info stat]=MPI_Wait (rqc) % at least doesn't block [info flag stat]=MPI_Test_cancelled(stat) % no, cancel ~= free MPI_Finalize quit %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % Collective operation % MPI_Barrier, _Bcast, _Scatter, _Scatterv, _Gather, _Gatherv % MPI_Allgather, _Allgatherv, _Alltoall, _Alltoallv % MPI_Reduce, _Allreduce, _Scan, _Reduce_scatter %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % *** WE ARE USING 3 COMPUTERS FROM NOW ON :-) *** %------------------------------------------------------- (this time $LAMBHOST includes ox1 ox2 & ox0, master node last in list) (you can ssh to the nodes (ox1, ox2) from the headed node w/out passwd/output) (Octave is installed for all hosts) (lamprep/lamboot before starting octave if this is a new ssh session) (you should have edited .octaverc so that it calls _mergeParent if child task) putenv('LAM_MPI_SSI_rpi','tcp'), MPI_Init % better understood with tcp [info children errs] = MPI_Comm_spawn ( '/usr/X11R6/bin/xterm',{'-e','octave'},... 2,MPI_INFO_NULL,0,MPI_COMM_SELF) % notice that 2 [info NEWORLD] = MPI_Intercomm_merge (children, 0) % children unblock MPI_Errhandler_set(NEWORLD,MPI_ERRORS_RETURN) % isn't it cool? [info rank]=MPI_Comm_rank(NEWORLD) hnam=system('hostname') %%%%% ON FIRST CHILD %%%%%%% [info rank]=MPI_Comm_rank(NEWORLD) hnam=system('hostname') %%%%% ON SECOND CHILD %%%%%%% [info rank]=MPI_Comm_rank(NEWORLD) hnam=system('hostname') help collective %------------------------------------------------------- % Can synchronize all ranks in communicator group %------------------------------------------------------- help MPI_Barrier %%%%% ON SECOND CHILD %%%%%%% MPI_Barrier(NEWORLD) % 2nd child blocks %%%%% ON FIRST CHILD %%%%%%% info=MPI_Barrier(NEWORLD) % 1st child blocks %%%%% ON PARENT Octave %%%%%%% info=MPI_Barrier(NEWORLD) % unblocks both %------------------------------------------------------- % Can broadcast a msg to all ranks in communicator group %------------------------------------------------------- help MPI_Bcast %%%%% ON SECOND CHILD %%%%%%% C=[1 3 5 7] MPI_Bcast(C,0,NEWORLD),C% blocks %%%%% ON PARENT Octave %%%%%%% A=[2 4;8 16] info=MPI_Bcast(A,0,NEWORLD), A % unblocks %%%%% ON FIRST CHILD %%%%%%% B=[2;4;6;8] info=MPI_Bcast(B,0,NEWORLD), B % doesn't block %------------------------------------------------------- % Parent had [2 4 gets same value % 8 16] % child1 had [2;4;6;8] gets [2;8;4;16] % child2 had [1 3 5 7] gets [2 8 4 16] notice row-major order %------------------------------------------------------- %------------------------------------------------------- % Can scatter a msg among all ranks in communicator group % variant with displacements, too %------------------------------------------------------- help MPI_Scatter %%%%% ON SECOND CHILD %%%%%%% R=[7 8] MPI_Scatter([],R,0,NEWORLD), R % blocks %%%%% ON PARENT Octave %%%%%%% A=[1 2;3 4;5 6], R=[7;8] info=MPI_Scatter(A,R,0,NEWORLD), R % unblocks %%%%% ON FIRST CHILD %%%%%%% R=[7;8] info=MPI_Scatter([],R,0,NEWORLD), R % doesn't blk %------------------------------------------------------- % Parent had [7;8] gets [1;3] from [1 2; % child1 had [7;8] gets [5;2] 3 4; % child2 had [7 8] gets [4 6] 5 6] notice row-major order %------------------------------------------------------- % variant with displacements, too %------------------------------------------------------- help MPI_Scatterv %%%%% ON FIRST CHILD %%%%%%% R=[8 9;8 9] info=MPI_Scatterv([],[],[],R,0,NEWORLD), R % blocks %%%%% ON PARENT Octave %%%%%%% A=[0 1 2 3 4 5 6 7] R=[8 8 9 9] info=MPI_Scatterv(A,[1 2 3],[0 3 5],R,0,NEWORLD) % unblocks R % 1,2,3 elems@pos 0,3,5 %%%%% ON SECOND CHILD %%%%%%% R=[8;8;9;9] % doesn't blk MPI_Scatterv([],[],[],R,0,NEWORLD), R %------------------------------------------------------- % Parent had [8 8 9 9] gets [0 1 2 3] from [0 1 2 3 4 5 6 7] % child1 had [8 9 gets [3 9 which is 2 elems @ pos 3 (3 4) % 8 9] 4 9] % child2 had [8;8;9;9] gets [5;6;7;9] which is 3 elems @ pos 5 (5 6 7) %------------------------------------------------------- % seems as if Parent shouldn't have got this R=[0 1 2 3], should be [0 8 9 9] % read the MPI Std carefully!!! %%% root always receives Rcnt %%% % since the R@root has room for 4, it gets 4 % let's prove it, interchanging the root role among ranks 0-1-2 %------------------------------------------------------- R=[8 8 9 9], A=[0 1 2 3 4 5 6 7] info=MPI_Scatterv(A,[1 2 3],[0 3 5], R,0,NEWORLD), R R=[8 8 9 9] info=MPI_Scatterv([],[],[], R,0,NEWORLD), R R=[8 8 9 9] info=MPI_Scatterv([],[],[], R,0,NEWORLD), R R=[8 8 9 9] info=MPI_Scatterv([],[],[], R,1,NEWORLD), R R=[8 8 9 9], B=[0 1 2 3 4 5 6 7] info=MPI_Scatterv(B,[1 2 3],[0 3 5], R,1,NEWORLD), R R=[8 8 9 9] info=MPI_Scatterv([],[],[], R,1,NEWORLD), R R=[8 8 9 9] info=MPI_Scatterv([],[],[], R,2,NEWORLD), R R=[8 8 9 9] info=MPI_Scatterv([],[],[], R,2,NEWORLD), R R=[8 8 9 9], C=[0 1 2 3 4 5 6 7] MPI_Scatterv(C,[1 2 3],[0 3 5], R,2,NEWORLD), R %------------------------------------------------------- % Can gather a msg from among all ranks in communicator group % variant with displacements, too %------------------------------------------------------- help MPI_Gather S=[4 5 6] % doesn't info=MPI_Gather(S,[],0,NEWORLD) A=ones(3), S=[1;2;3] info=MPI_Gather(S,A,0,NEWORLD), A % blocks S=[7 8 9] info=MPI_Gather(S,[],0,NEWORLD) % unbl %------------------------------------------------------- % Parent sends [1;2;3] gets [1 4 7 % child1 sends [4 5 6] 2 5 8 % child2 sends [7 8 9] 3 6 9] notice row-major order %------------------------------------------------------- % variant with displacements, too %------------------------------------------------------- help MPI_Gatherv A=ones(3), S=[1;2;3] info=MPI_Gatherv(S,A,[3 3 6],[0 2 3],0,NEWORLD), A % blocks S=[4 5 6] % doesn't info=MPI_Gatherv(S,[],[],[],0,NEWORLD) S=[7 9 7;8 8 6] info=MPI_Gatherv(S,[],[],[],0,NEWORLD) % unbl %------------------------------------------------------- % Parent sends [1;2;3] gets [1 7 8 notice overlapping parnt/chld1 % child1 sends [4 5 6] 2 8 7 (elm#2) and chld1/2 (elms#3,4) % child2 sends [7 9 7 4 9 6] higher rank overwrites lower's % 8 8 6] notice row-major order %------------------------------------------------------- MPI_Finalize % ok, too large quit % cycle-Init or MPI_Finalize % skip to next session quit MPI_Finalize quit %------------------------------------------------------- % Init cycle ... skip it if you didn't Finalize just above %------------------------------------------------------- putenv('LAM_MPI_SSI_rpi','tcp'), MPI_Init [info children errs] = MPI_Comm_spawn ( '/usr/X11R6/bin/xterm',{'-e','octave'},... 2,MPI_INFO_NULL,0,MPI_COMM_SELF) [info NEWORLD] = MPI_Intercomm_merge (children, 0) MPI_Errhandler_set(NEWORLD,MPI_ERRORS_RETURN) [info rank]=MPI_Comm_rank(NEWORLD) hnam=system('hostname') %%%%% ON FIRST CHILD %%%%%%% [info rank]=MPI_Comm_rank(NEWORLD) hnam=system('hostname') %%%%% ON SECOND CHILD %%%%%%% [info rank]=MPI_Comm_rank(NEWORLD) hnam=system('hostname') %------------------------------------------------------- % Init cycle done... jump here if you didn't Finalize just above %------------------------------------------------------- help collective %------------------------------------------------------- % Can gather in all ranks, not only at root % variant with displacements, too %------------------------------------------------------- help MPI_Allgather R=ones(3,2), S=[3 4] info=MPI_Allgather(S,R,NEWORLD),R % blocks A=zeros(3,2), S=[1;2] info=MPI_Allgather(S,A,NEWORLD), A % blocks R=ones(2,3), S=[5 6] info=MPI_Allgather(S,R,NEWORLD),R % unbl %------------------------------------------------------- % Parent had [0 0 sends [1;2] gets [1 4 % 0 0 2 5 % 0 0] 3 6] % child1 had [1 1 sends [3 4] gets [1 4 % 1 1 2 5 % 1 1] 3 6] % child2 had [1 1 1 sends [5 6] gets [1 3 5 notice row-major order % 1 1 1] 2 4 6] %------------------------------------------------------- % variant with displacements, too %------------------------------------------------------- help MPI_Allgatherv A=ones(3), S=[1;2;3] info=MPI_Allgatherv(S,A,[3 3 6],[0 2 3],NEWORLD), A % blocks B=linspace(0,0,9), S=[4 5 6] % blocks info=MPI_Allgatherv(S,B,[3 3 6],[0 2 3],NEWORLD), B C=repmat(0,2,5), S=[7 9 7;8 8 6] % unblocks info=MPI_Allgatherv(S,C,[3 3 6],[0 2 3],NEWORLD), C %------------------------------------------------------- % Parent had [1 1 1 sends [1;2;3] gets [1 7 8 % overlapping % 1 1 1 2 8 7 % as seen in % 1 1 1] 4 9 6] % MPI_Gatherv % child1 had [0 0... sends [4 5 6] gets [1 2 4 7 8 9 8 7 6] % child2 had [0 0... sends [7 9 7 gets [1 4 8 8 6 % 0 0... 8 8 6] 2 7 9 7 0] % greater room %------------------------------------------------------- % sizes (cnts in help file) must be equal in all ranks (or somebody is lying) % in particular, varsize must correspond to declared size in cnts % positions (disps in help file) need not be equal, can be altered at will %------------------------------------------------------- A=zeros(3), S=[1 2 3] info=MPI_Allgatherv(S,A,[3 2 3],[0 3 6],NEWORLD), A % blocks B=zeros(3), S=[4 5] % blocks info=MPI_Allgatherv(S,B,[3 2 3],[0 6 3],NEWORLD), B C=zeros(3), S=[7 8 9] % unblocks info=MPI_Allgatherv(S,C,[3 2 3],[3 0 6],NEWORLD), C %------------------------------------------------------- % All them had [0 0 0 parent sends 1 2 3 % 0 0 0 child1 sends 4 5 % 0 0 0] child2 sends 7 8 9 % % Each one parent [1 4 7 child1 [1 7 4 child2 [4 1 7 % reorders 2 5 8 2 8 5 5 2 8 % at will 3 0 9] 3 9 0] 0 3 9] %------------------------------------------------------- %------------------------------------------------------- % Can scatter-gather from all to all % variant with displacements, too %------------------------------------------------------- help MPI_Alltoall R=zeros(3,2), S=[1 2 3;1 2 4] info=MPI_Alltoall(S,R,2,NEWORLD),R % blocks R=zeros(3,2), S=[1 1 2 2 1 2] info=MPI_Alltoall(S,R,2,NEWORLD), R % blocks R=zeros(2,3), S=[1;1;2;2;5;6] MPI_Alltoall(S,R,2,NEWORLD),R % unblocks %------------------------------------------------------- % Parent had [0 0 sends [1 1 2 2 1 2] gets [1 1 % 0 0 1 1 % 0 0] 1 1] % child1 had [0 0 sends [1 2 3 gets [2 2 % 0 0 1 2 4] 2 2 % 0 0] 2 2] % child2 had [0 0 0 sends [1 1 2 2 5 6]' gets [1 3 5 % row-major % 0 0 0] 2 4 6] %------------------------------------------------------- % variant with displacements, too %------------------------------------------------------- help MPI_Alltoallv R=[0 0 0], S=[1 2 2 3 3 3]' MPI_Alltoallv ( S,[1 2 3],[0 1 3],... R,[1 1 1],[2 1 0],NEWORLD), R % blocks R=zeros(2), S=[1 4 3;4 3 3] % blocks MPI_Alltoallv ( S,[1 2 3],[0 1 3],... R,[2 2 2],[0 2 1],NEWORLD), R R=zeros(4,3), S=[5 3;2 3;2 3] % unblocks MPI_Alltoallv ( S,[1 2 3],[0 1 3],... R,[3 3 3],[1 5 9],NEWORLD),R %------------------------------------------------------- % Parent had [0 0 0] sends [1 2 2 3 3 3]' gets [5 1 1] % child1 had [0 0 sends [1 4 3 gets [2 2 % 0 0] 4 3 3] 2 4] % child2 had [0 0 0 sends [5 3 gets [0 0 0 % 0 0 0 2 3 3 3 3 % 0 0 0 2 3] 3 3 3 % 0 0 0] 3 3 3] %------------------------------------------------------- % as with Allgatherv, % sizes sent (scnts in help file) must be coherent with sizes recvd (rcnts) % (or somebody is lying) % positions (disps in help file) need not be equal, can be altered at will % again, there can be overlapping, and higher ranks overwrite lower ones %------------------------------------------------------- MPI_Finalize % ok, too large quit % cycle-Init MPI_Finalize % this time, don't quit % skip directly MPI_Finalize % to next session quit % do cycle-init %------------------------------------------------------- % Init cycle ... don't skip. We need ErrorHandler Abort %------------------------------------------------------- putenv('LAM_MPI_SSI_rpi','tcp'), MPI_Init [info children errs] = MPI_Comm_spawn ( '/usr/X11R6/bin/xterm',{'-e','octave'},... 2,MPI_INFO_NULL,0,MPI_COMM_SELF) [info NEWORLD] = MPI_Intercomm_merge (children, 0) % % -------------------------- % MPI_Errhandler_set(NEWORLD,MPI_ERRORS_RETURN) % be sure to read help below % % -------------------------- [info rank]=MPI_Comm_rank(NEWORLD) hnam=system('hostname') %%%%% ON FIRST CHILD %%%%%%% [info rank]=MPI_Comm_rank(NEWORLD) hnam=system('hostname') %%%%% ON SECOND CHILD %%%%%%% [info rank]=MPI_Comm_rank(NEWORLD) hnam=system('hostname') help collective %------------------------------------------------------- % Can reduce distributed values to its single (sum,prod,max,min...) %------------------------------------------------------- % -------------------------- help MPI_Reduce % read remark about errors % -------------------------- help MPI_MAX help MPI_MIN %%%%% ON PARENT Octave %%%%%%% A=[00 11 22], R=[10 11 12] info=MPI_Reduce(A,R,MPI_MIN,0,NEWORLD), R % blocks %%%%% ON FIRST CHILD %%%%%%% B=[20 01 12] info=MPI_Reduce(B,[],MPI_MIN,0,NEWORLD) % doesn't %%%%% ON SECOND CHILD %%%%%%% C=[10 21 02] MPI_Reduce(C,[],MPI_MIN,0,NEWORLD) % unblocks %------------------------------------------------------- % Parent had [10 11 12] min [00 -- --] gets [00 01 02] % child1 min [-- 01 --] % child2 min [-- -- 02] %------------------------------------------------------- % Can perform the reduction in all, not just root %------------------------------------------------------- help MPI_Allreduce R=[10 11 12] info=MPI_Allreduce(A,R,MPI_MAX,NEWORLD), R % blocks R=[10 11 12]' info=MPI_Allreduce(B,R,MPI_MAX,NEWORLD), R % DOES block R=[10 11;12 13] MPI_Allreduce(C,R,MPI_MAX,NEWORLD),R % unblocks %------------------------------------------------------- % Parent had [10 11 12] max [-- -- 22] gets [20 21 22] % child1 had [10 11 12]' max [20 -- --] gets [20 21 22]' % child2 had [10 11 max [-- 21 --] gets [20 22 % 12 13] 21 13] % NOTICE that rank 1 DOES get blocked... all them like root blocked before %------------------------------------------------------- % Can perform partial reductions in each rank (with its lesser ranks) %------------------------------------------------------- help MPI_Scan help MPI_SUM help MPI_PROD R=[10 11 12] info=MPI_Scan(A,R,MPI_SUM,NEWORLD), R % doesn't hang R=[10 11;12 13] MPI_Scan(C,R,MPI_SUM,NEWORLD),R % depends on... R=[10 11 12]' info=MPI_Scan(B,R,MPI_SUM,NEWORLD), R % unblocks %------------------------------------------------------- % Parent had [10 11 12] sends [00 11 22] gets [00 11 22] % child1 had [10 11 12]' sends [20 01 12] gets [20 12 34]' % --------PARTIAL SUM IS [20 12 34]------ % child2 had [10 11 sends [10 21 02] gets [30 36 % 12 13] --------SUM IS [30 33 36]------ 33 13] % NOTICE that each rank depends on its lower ranks, blocking if not available %------------------------------------------------------- % And scatter the reduction after computing it %------------------------------------------------------- help MPI_Reduce_scatter A=[0 1 3;1 8 1], R=[1 1 1] info=MPI_Reduce_scatter(A,R,[1 1 4],MPI_PROD,NEWORLD), R % blocks B=[1 4 1;0 3 2], R=[2;2;2] MPI_Reduce_scatter(B,R,[1 1 4],MPI_PROD,NEWORLD)% blocks too R C=[1 6 6;1 1 9], R=[3 3;3 3] % unblocks MPI_Reduce_scatter(C,R,[1 1 4],MPI_PROD,NEWORLD) R %------------------------------------------------------- % Parent had [1 1 1] sends [0 1 3 gets [0 1 1] % 1 8 1] % child1 had [2 2 2]'sends [1 4 1 gets [0 2 2]' % 0 3 2] % child2 had [3 3 sends [1 6 6 gets [24 18 % 3 3] 1 1 9] 24 18] % --------PROD IS [0 24 18-------- % ---------------- 0 24 18]------- % NOTICE that counts must be coherent, as with Allgatherv / Alltoallv %------------------------------------------------------- R=[1 1 1] MPI_Reduce_scatter(A,R,[2 2 2],MPI_PROD,NEWORLD), R % blocks R=[2;2;2] % blocks!!! info=MPI_Reduce_scatter(B,R,[1 2 3],MPI_PROD,NEWORLD) R R=[3 3;3 3] % unblocks MPI_Reduce_scatter(C,R,[1 1 4],MPI_PROD,NEWORLD) R info==MPI_SUCCESS % info==0 !!! %------------------------------------------------------- % Parent had [1 1 1] sends [0 1 3 gets [0 0 1] the two reqstd % 1 8 1] from self % child1 had [2 2 2]'sends [1 4 1 gets [24 24 2]' % 0 3 2] % child2 had [3 3 sends [1 6 6 gets [18 3 % 3 3] 1 1 9] 18 3] % --------PROD IS [0 24 18-------- % ---------------- 0 24 18]------- % try [1 1 3] as counts in 1st child: info~=MPI_SUCCESS %------------------------------------------------------- % The easiest reduction operation %------------------------------------------------------- help MPI_REPLACE %%%%% ON PARENT Octave %%%%%%% A=[00 01 02], R=[99 99 99] info=MPI_Reduce(A,R,MPI_REPLACE,0,NEWORLD), R % blocks %%%%% ON FIRST CHILD %%%%%%% B=[10 11 12] info=MPI_Reduce(B,[],MPI_REPLACE,0,NEWORLD) % doesn't %%%%% ON SECOND CHILD %%%%%%% C=[20 21 22] MPI_Reduce(C,[],MPI_REPLACE,0,NEWORLD) % unbl %------------------------------------------------------- % Parent had [99 99 99] sends [00 01 02] ---> gets [00 01 02] % child1 sends [10 11 12] % child2 sends [20 21 22] %------------------------------------------------------- % init-cycle or jump below, as you want %------------------------------------------------------- MPI_Finalize quit MPI_Finalize quit MPI_Finalize quit %------------------------------------------------------- % Other operations (logical, replace) % (on bool, string, complex, range) %------------------------------------------------------- putenv('LAM_MPI_SSI_rpi','tcp'), MPI_Init [info children errs] = MPI_Comm_spawn ( '/usr/X11R6/bin/xterm',{'-e','octave'},... 2,MPI_INFO_NULL,0,MPI_COMM_SELF) [info NEWORLD] = MPI_Intercomm_merge (children, 0) % % -------------------------- % MPI_Errhandler_set(NEWORLD,MPI_ERRORS_RETURN) % recall remark about errors % % -------------------------- [info rank]=MPI_Comm_rank(NEWORLD) hnam=system('hostname') %%%%% ON FIRST CHILD %%%%%%% [info rank]=MPI_Comm_rank(NEWORLD) hnam=system('hostname') %%%%% ON SECOND CHILD %%%%%%% [info rank]=MPI_Comm_rank(NEWORLD) hnam=system('hostname') %------------------------------------------------------- help MPI_Reduce help MPI_LAND help MPI_LOR help MPI_LXOR %%%%% ON PARENT Octave %%%%%%% A=logical([2 1 0]), R=logical([0 0 0]) info=MPI_Reduce(A,R,MPI_LAND,0,NEWORLD), R % blocks %%%%% ON FIRST CHILD %%%%%%% B=logical([3 0 0]) info=MPI_Reduce(B,[],MPI_LAND,0,NEWORLD) % doesn't %%%%% ON SECOND CHILD %%%%%%% C=logical([4 5 6]) MPI_Reduce(C,[],MPI_LAND,0,NEWORLD) % unbl %------------------------------------------------------- % Parent had [0 0 0] sends [1 1 0] gets [1 0 0] % child1 sends [1 0 0] % child2 sends [1 1 1] % --LOGICAL AND-- [1 0 0] %------------------------------------------------------- %%%%% ON PARENT Octave %%%%%%% R=logical([0 0 0]) info=MPI_Reduce(A,R,MPI_LOR,0,NEWORLD), R % blocks %%%%% ON FIRST CHILD %%%%%%% info=MPI_Reduce(B,[],MPI_LOR,0,NEWORLD) % doesn't %%%%% ON SECOND CHILD %%%%%%% MPI_Reduce(C,[],MPI_LOR,0,NEWORLD) % unblocks %------------------------------------------------------- % Parent had [0 0 0] sends [1 1 0] gets [1 1 1] % child1 sends [1 0 0] % child2 sends [1 1 1] % --LOGICAL OR -- [1 1 1] %------------------------------------------------------- %%%%% ON PARENT Octave %%%%%%% R=logical([0 0 0]) info=MPI_Reduce(A,R,MPI_LXOR,0,NEWORLD), R % blocks %%%%% ON FIRST CHILD %%%%%%% info=MPI_Reduce(B,[],MPI_LXOR,0,NEWORLD) % doesn't %%%%% ON SECOND CHILD %%%%%%% MPI_Reduce(C,[],MPI_LXOR,0,NEWORLD) % unblocks %------------------------------------------------------- % Parent had [0 0 0] sends [1 1 0] gets [1 0 1] % child1 sends [1 0 0] % child2 sends [1 1 1] % --LOGICAL XOR-- [1 0 1] %------------------------------------------------------- %%%%% ON PARENT Octave %%%%%%% R=logical([0 0 0]) info=MPI_Reduce(A,R,MPI_REPLACE,0,NEWORLD), R % blocks %%%%% ON FIRST CHILD %%%%%%% info=MPI_Reduce(B,[],MPI_REPLACE,0,NEWORLD) % doesn't %%%%% ON SECOND CHILD %%%%%%% MPI_Reduce(C,[],MPI_REPLACE,0,NEWORLD) % unbl %------------------------------------------------------- % Parent had [0 0 0] sends [1 1 0] ------> gets [1 1 0] % child1 sends [1 0 0] % child2 sends [1 1 1] %------------------------------------------------------- %%%%% ON PARENT Octave %%%%%%% A="ABC", R="DEF" info=MPI_Reduce(A,R,MPI_REPLACE,0,NEWORLD), R % blocks %%%%% ON FIRST CHILD %%%%%%% B="GHI" info=MPI_Reduce(B,"",MPI_REPLACE,0,NEWORLD) % doesn't %%%%% ON SECOND CHILD %%%%%%% C="JKL" MPI_Reduce(C,"",MPI_REPLACE,0,NEWORLD) % unbl %------------------------------------------------------- % Parent had "DEF" sends "ABC" ------> gets "ABC" % child1 sends "GHI" % child2 sends "JKL" %------------------------------------------------------- %%%%% ON PARENT Octave %%%%%%% A=[01 5i 02], R=[0 i 0] info=MPI_Reduce(A,R,MPI_MAX,0,NEWORLD), R % blocks %%%%% ON FIRST CHILD %%%%%%% B=[1i 02 6i] info=MPI_Reduce(B,[],MPI_MAX,0,NEWORLD) % doesn't %%%%% ON SECOND CHILD %%%%%%% C=[4i 01 03] MPI_Reduce(C,[],MPI_MAX,0,NEWORLD) % unblocks %------------------------------------------------------- % Parent had [00 1i 00] max r/i [01 5i --] gets [01 02 03] % child1 max r/i [-- 02 6i] +4i +5i +6i % child2 max r/i [4i -- 03] %------------------------------------------------------- % TODO: this will probably change when MPI_CXX_DOUBLE_COMPLEX is supported % _MIN and _MAX operations for complex are not supported by LAM % _SUM and _PROD are supported however %------------------------------------------------------- %%%%% ON PARENT Octave %%%%%%% R=[0 i 0] info=MPI_Reduce(A,R,MPI_REPLACE,0,NEWORLD), R % blocks %%%%% ON FIRST CHILD %%%%%%% info=MPI_Reduce(B,[],MPI_REPLACE,0,NEWORLD) % doesn't %%%%% ON SECOND CHILD %%%%%%% MPI_Reduce(C,[],MPI_REPLACE,0,NEWORLD) % unbl %------------------------------------------------------- % Parent had [00 1i 00] sends [01 5i 02] ---> gets [01 5i 02] % child1 sends [ i 2 6i] % child2 sends [4i 1 3 ] %------------------------------------------------------- %%%%% ON PARENT Octave %%%%%%% A=1:2:11, R=1:1:3 info=MPI_Reduce(A,R,MPI_MAX,0,NEWORLD), R % blocks %%%%% ON FIRST CHILD %%%%%%% B=5:1:7 info=MPI_Reduce(B,[],MPI_MAX,0,NEWORLD) % doesn't %%%%% ON SECOND CHILD %%%%%%% C=7:-4:-10 MPI_Reduce(C,[],MPI_MAX,0,NEWORLD) % unblocks %------------------------------------------------------- % Parent had 01:01:03 sends 01:02: 11 gets 7:2:11 % child1 sends 05:01: 07 == [7 9 11] % child2 sends 07:-4:-10 % --MAX-- 07:02: 11 %------------------------------------------------------- %%%%% ON PARENT Octave %%%%%%% R=1:1:3 info=MPI_Reduce(A,R,MPI_MIN,0,NEWORLD), R % blocks %%%%% ON FIRST CHILD %%%%%%% info=MPI_Reduce(B,[],MPI_MIN,0,NEWORLD) % doesn't %%%%% ON SECOND CHILD %%%%%%% MPI_Reduce(C,[],MPI_MIN,0,NEWORLD) % unblocks %------------------------------------------------------- % Parent had 01:01:03 sends 01:02: 11 gets 1:-4:-10 % child1 sends 05:01: 07 == [1 -3 -7] % child2 sends 07:-4:-10 % --MIN-- 01:-4:-10 %------------------------------------------------------- %%%%% ON PARENT Octave %%%%%%% R=1:1:3 info=MPI_Reduce(A,R,MPI_SUM,0,NEWORLD), R % blocks %%%%% ON FIRST CHILD %%%%%%% info=MPI_Reduce(B,[],MPI_SUM,0,NEWORLD) % doesn't %%%%% ON SECOND CHILD %%%%%%% MPI_Reduce(C,[],MPI_SUM,0,NEWORLD) % unblocks %------------------------------------------------------- % Parent had 01:01:03 sends 01:02: 11 gets 13:-1:08 % child1 sends 05:01: 07 == [13 12 11] % child2 sends 07:-4:-10 3elems due to initial R % --SUM-- 13:-1: 08 repeat with R=1:7 %------------------------------------------------------- %%%%% ON PARENT Octave %%%%%%% R=1:7 info=MPI_Reduce(A,R,MPI_SUM,0,NEWORLD), R % blocks %%%%% ON FIRST CHILD %%%%%%% info=MPI_Reduce(B,[],MPI_SUM,0,NEWORLD) % doesn't %%%%% ON SECOND CHILD %%%%%%% MPI_Reduce(C,[],MPI_SUM,0,NEWORLD) % unblocks %------------------------------------------------------- % Parent had 01:01:03 sends 01:02: 11 gets 13:-1:08 % child1 sends 05:01: 07 == [13 ... 7] % child2 sends 07:-4:-10 initial R had 7 elems % --SUM-- 13:-1: 08 ranges are double[3] %------------------------------------------------------- %%%%% ON PARENT Octave %%%%%%% R=1:6 info=MPI_Reduce(A,R,MPI_PROD,0,NEWORLD), R % blocks %%%%% ON FIRST CHILD %%%%%%% info=MPI_Reduce(B,[],MPI_PROD,0,NEWORLD) % doesn't %%%%% ON SECOND CHILD %%%%%%% MPI_Reduce(C,[],MPI_PROD,0,NEWORLD) % unblocks %------------------------------------------------------- % Parent had 01:01:03 sends 01:02: 11 gets 35:-8:-770 % child1 sends 05:01: 07 ==[35 27 19 11 3 -5] % child2 sends 07:-4:-10 fortunately R had 6cols % --PROD- 35:-8:-770 %------------------------------------------------------- %------------------------------------------------------- % I'm not sure wether that was useful, (reducing ranges as doubles[3]) % but it's certainly funny %------------------------------------------------------- % init-cycle or jump below, as you want %------------------------------------------------------- MPI_Finalize quit MPI_Finalize quit MPI_Finalize quit %------------------------------------------------------- % Reduce operations over integer types ([u]int16/32/64 - not [u]int8) % (all operations: max/min, sum/prod, [l/b]and/or/xor, replace) % Most MPI reduce-ops on LAM do not support [u]int8 types % see $LAM_SOURCE/share/mpi/lamreduce.c % int8 - MPI_BYTE: lam_replace, lam_band/or/xor % uint8 - MPI_UNSIGNED_CHAR: lam_replace %------------------------------------------------------- putenv('LAM_MPI_SSI_rpi','tcp'), MPI_Init [info children errs] = MPI_Comm_spawn ( '/usr/X11R6/bin/xterm',{'-e','octave'},... 2,MPI_INFO_NULL,0,MPI_COMM_SELF) [info NEWORLD] = MPI_Intercomm_merge (children, 0) % % -------------------------- % MPI_Errhandler_set(NEWORLD,MPI_ERRORS_RETURN) % recall remark about errors % % -------------------------- [info rank]=MPI_Comm_rank(NEWORLD) hnam=system('hostname') %%%%% ON FIRST CHILD %%%%%%% [info rank]=MPI_Comm_rank(NEWORLD) hnam=system('hostname') %%%%% ON SECOND CHILD %%%%%%% [info rank]=MPI_Comm_rank(NEWORLD) hnam=system('hostname') %------------------------------------------------------- % same indented format: % left = parent octave % middle = first child % right = second child %------------------------------------------------------- A=int16([00 11 22]), R=int16([10 11 12]) info=MPI_Reduce(A,R,MPI_MIN,0,NEWORLD), R % blocks B=int16([20 01 12]) info=MPI_Reduce(B,[],MPI_MIN,0,NEWORLD) % doesn't C=int16([10 21 02]) MPI_Reduce(C,[],MPI_MIN,0,NEWORLD) % unblocks %------------------------------------------------------- % Parent had [10 11 12] sends [00 11 22] gets [0 1 2] % child1 sends [20 01 12] % child2 sends [10 21 02] % --MIN-- 00 01 02 %------------------------------------------------------- A=uint32([00 11 22]), R=uint32([10 11 12]) info=MPI_Reduce(A,R,MPI_MAX,0,NEWORLD), R % blocks B=uint32([20 01 12]) info=MPI_Reduce(B,[],MPI_MAX,0,NEWORLD) % doesn't C=uint32([10 21 02]) MPI_Reduce(C,[],MPI_MAX,0,NEWORLD) % unblocks %------------------------------------------------------- % Parent had [10 11 12] sends [00 11 22] gets [20 21 22] % child1 sends [20 01 12] % child2 sends [10 21 02] % --MAX-- 20 21 22 %------------------------------------------------------- A=int64([00 11 22]), R=int64([10 11 12]) info=MPI_Reduce(A,R,MPI_SUM,0,NEWORLD), R % blocks B=int64([20 01 12]) info=MPI_Reduce(B,[],MPI_SUM,0,NEWORLD) % doesn't C=int64([10 21 02]) MPI_Reduce(C,[],MPI_SUM,0,NEWORLD) % unblocks %------------------------------------------------------- % Parent had [10 11 12] sends [00 11 22] gets [30 33 36] % child1 sends [20 01 12] % child2 sends [10 21 02] % --SUM-- 30 33 36 %------------------------------------------------------- A=uint64([0 1 2]), R=uint64([0 0 0]) info=MPI_Reduce(A,R,MPI_PROD,0,NEWORLD), R % blocks B=uint64([2 4 6]) info=MPI_Reduce(B,[],MPI_PROD,0,NEWORLD) % doesn't C=uint64([3 6 9]) MPI_Reduce(C,[],MPI_PROD,0,NEWORLD) % unblocks %------------------------------------------------------- % Parent had [0 0 0] sends [ 0 1 2] gets [0 24 108] % child1 sends [ 2 4 6] % child2 sends [ 3 6 9] % -PROD- 00 24 108 %------------------------------------------------------- A=uint16([2 1 0]), R=uint16([0 0 0]) info=MPI_Reduce(A,R,MPI_LAND,0,NEWORLD), R % blocks B=uint16([3 0 0]) info=MPI_Reduce(B,[],MPI_LAND,0,NEWORLD) % doesn't C=uint16([4 5 6]) MPI_Reduce(C,[],MPI_LAND,0,NEWORLD) % unbl %------------------------------------------------------- % Parent had [0 0 0] sends [1 1 0] gets [1 0 0] % child1 sends [1 0 0] % child2 sends [1 1 1] % --LOGICAL AND-- [1 0 0] %------------------------------------------------------- A=int32([2 1 0]), R=int32([0 0 0]) info=MPI_Reduce(A,R,MPI_LOR,0,NEWORLD), R % blocks B=int32([3 0 0]) info=MPI_Reduce(B,[],MPI_LOR,0,NEWORLD) % doesn't C=uint32([0 -1 -2]) % notice uint MPI_Reduce(C,[],MPI_LOR,0,NEWORLD) % unblocks %------------------------------------------------------- % Parent had [0 0 0] sends [1 1 0] gets [1 1 0] % child1 sends [1 0 0] % child2 sends [0 0 0] (uint saturates <0 to 0) % --LOGICAL OR -- [1 1 0] %------------------------------------------------------- A=uint64([2 1 0]), R=uint64([0 0 0]) info=MPI_Reduce(A,R,MPI_LXOR,0,NEWORLD), R % blocks B=uint64([3 0 0]) info=MPI_Reduce(B,[],MPI_LXOR,0,NEWORLD) % doesn't C=uint64([0 -1 -2]) MPI_Reduce(C,[],MPI_LXOR,0,NEWORLD) % unblocks %------------------------------------------------------- % Parent had [0 0 0] sends [1 1 0] gets [0 1 0] % child1 sends [1 0 0] % child2 sends [0 0 0] % --LOGICAL XOR-- [0 1 0] %------------------------------------------------------- A=uint8([2 1 0]), R=uint8([0 0 0]) info=MPI_Reduce(A,R,MPI_REPLACE,0,NEWORLD), R % blocks B=uint8([3 0 0]) info=MPI_Reduce(B,[],MPI_REPLACE,0,NEWORLD) % doesn't C=uint8([0 -1 -2]) MPI_Reduce(C,[],MPI_REPLACE,0,NEWORLD) % unbl %------------------------------------------------------- % Parent had [0 0 0] sends [2 1 0] ------> gets [2 1 0] % child1 sends [3 0 0] % child2 sends [0 0 0] % --REPLACE-- only op for UINT8 (INT8 can BAND/OR/XOR too) %------------------------------------------------------- A=intmax('int8'), A=[A-0 A-1 A-2], R=int8([0 0 0]) info=MPI_Reduce(A,R,MPI_BAND,0,NEWORLD), R % blocks B=intmax('int8')/2, B=[B-0 B-4 B-1] info=MPI_Reduce(B,[],MPI_BAND,0,NEWORLD) % doesn't C=intmax('int8')/4, C=[C-0 C-2 C-4] MPI_Reduce(C,[],MPI_BAND,0,NEWORLD) % unbl %------------------------------------------------------- % Parent had [0 0 0] sends [127 126 125] gets [0 28 28] % child1 sends [ 64 60 63] % child2 sends [ 32 30 28] % --BITWISE AND-- [ 0 28 28] -- INT8 can BAND/OR/XOR %------------------------------------------------------- A=uint16([2 1 0]), R=uint16([0 0 0]) info=MPI_Reduce(A,R,MPI_BOR,0,NEWORLD), R % blocks B=uint16([4 5 6]) info=MPI_Reduce(B,[],MPI_BOR,0,NEWORLD) % doesn't C=uint16([8 4 4]) MPI_Reduce(C,[],MPI_BOR,0,NEWORLD) % unblocks %------------------------------------------------------- % Parent had [0 0 0] sends [2 1 0] gets [14 5 6] % child1 sends [4 5 6] % child2 sends [8 4 4] % --BITWISE OR --[14 5 6] %------------------------------------------------------- A=int32([2 1 0]), R=int32([0 0 0]) info=MPI_Reduce(A,R,MPI_BXOR,0,NEWORLD), R % blocks B=int32([4 5 6]) info=MPI_Reduce(B,[],MPI_BXOR,0,NEWORLD) % doesn't C=int32([8 4 4]) MPI_Reduce(C,[],MPI_BXOR,0,NEWORLD) % unblocks %------------------------------------------------------- % Parent had [0 0 0] sends [2 1 0] gets [14 0 2] % child1 sends [4 5 6] % child2 sends [8 4 4] % --BITWISE XOR--[14 0 2] %------------------------------------------------------- %------------------------------------------------------- % done with integer reductions %------------------------------------------------------- MPI_Finalize quit MPI_Finalize quit MPI_Finalize quit %%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % Groups, Communicators, etc % %%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % Communicators: % MPI_Comm_spawn, _Comm_get_parent, _Intercomm_merge %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % *** WE KEEP ON USING 3 COMPUTERS :-) *** %------------------------------------------------------- (this time $LAMBHOST includes ox1 ox2 & ox0, master node last in list) (you can ssh to the nodes (ox1, ox2) from the headed node w/out passwd/output) (lamprep/lamboot before starting octave if this is a new ssh session) %------------------------------------------------------- (edit again .octaverc so that it _DOES_NOT_ call _mergeParent if child task) %------------------------------------------------------- % We are not using the "mergeParent" Octave startup idea this time % you recall, we wanted to make a function for same boring slave startup % it saved us a lot of typing in these tutorial sessions % but now we want to study it in more detail %------------------------------------------------------- putenv('LAM_MPI_SSI_rpi','tcp') % this is for rpi tcp behavior MPI_Init % most easily understood help comms help groups help process %------------------------------------------------------- % This is our normal Spawn command %------------------------------------------------------- [info children errs] = MPI_Comm_spawn ( % blocked waiting for children _Init '/usr/X11R6/bin/xterm',{'-e','octave'},... 2,MPI_INFO_NULL,0,MPI_COMM_SELF) %------------------------------------------------------- % If you followed instructions, edited .octaverc and _mergeParent is not called % then parent gets blocked, waiting for children to MPI_Init %------------------------------------------------------- %%%%% ON FIRST CHILD %%%%%%% type startup_mergeParent % read from child1 %%%%% ON SECOND CHILD %%%%%%% getenv('LAMPARENT') % type on child2 MPI_Init % ouch! blocked!!! %%%%% ON FIRST CHILD %%%%%%% % DON'T USE "q" key % simply press MPI_Init % unblocks all type startup_mergeParent % go on reading %%%%% ON SECOND CHILD %%% go on typing WORLD = MPI_COMM_WORLD MPI_Errhandler_set( WORLD,MPI_ERRORS_RETURN) help MPI_Comm_get_parent [info parent] =MPI_Comm_get_parent help MPI_Intercomm_merge [info NEWORLD]=MPI_Intercomm_merge(parent,1) %%%%% ON FIRST CHILD %%%%%%% key % get out of "type" WORLD = MPI_COMM_WORLD MPI_Errhandler_set( WORLD,MPI_ERRORS_RETURN) [info parent] =MPI_Comm_get_parent [info NEWORLD]=MPI_Intercomm_merge(parent,1) % blocks too %%%%% ON PARENT Octave %%%%%%% [info NEWORLD] = MPI_Intercomm_merge (children, 0) % children unblock MPI_Errhandler_set(NEWORLD,MPI_ERRORS_RETURN) %%%%% ON FIRST CHILD %%%%%%% type startup_mergeParent % go on reading %%%%% ON SECOND CHILD %%% go on typing help MPI_Errhandler_set help MPI_Errhandler_get [info eh] = MPI_Errhandler_get (NEWORLD) eh==MPI_ERRORS_ARE_FATAL MPI_Errhandler_set(NEWORLD,MPI_ERRORS_RETURN) %%%%% ON FIRST CHILD %%%%%%% key % get out of "type" [info eh] = MPI_Errhandler_get (NEWORLD) eh==MPI_ERRORS_ARE_FATAL % each one its own errh MPI_Errhandler_set(NEWORLD,MPI_ERRORS_RETURN) %%%%% ON PARENT Octave %%%%%%% [info rank]=MPI_Comm_rank(NEWORLD) hnam=system('hostname') %%%%% ON FIRST CHILD %%%%%%% [info rank]=MPI_Comm_rank(NEWORLD) hnam=system('hostname') %%%%% ON SECOND CHILD %%%%%%% [info rank]=MPI_Comm_rank(NEWORLD) hnam=system('hostname') MPI_Finalize quit MPI_Finalize quit MPI_Finalize quit %------------------------------------------------------- % edit again .octaverc so that it _DOES_ call _mergeParent if child task % we will never again avoid the _mergeParent startup ;-) %------------------------------------------------------- %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % Communicators: % MPI_Comm_rank, _size, _remote_size, _test_inter % MPI_Comm_dup, _compare, _free % MPI_COMM_WORLD, _SELF, _NULL % MPI_IDENT, _CONGRUENT, _SIMILAR, _UNEQUAL %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% putenv('LAM_MPI_SSI_rpi','tcp'), MPI_Init [info children errs] = MPI_Comm_spawn ( '/usr/X11R6/bin/xterm',{'-e','octave'},... 2,MPI_INFO_NULL,0,MPI_COMM_SELF) [info NEWORLD] = MPI_Intercomm_merge (children, 0) MPI_Errhandler_set(NEWORLD,MPI_ERRORS_RETURN) hnam=system('hostname') %%%%% ON FIRST CHILD %%%%%%% hnam=system('hostname') [info size]=MPI_Comm_size(NEWORLD) % size==3 [info rank]=MPI_Comm_rank(NEWORLD) % rank==1 %%%%% ON SECOND CHILD %%%%%%% hnam=system('hostname') [info size]=MPI_Comm_size(NEWORLD) [info rank]=MPI_Comm_rank(NEWORLD) help comms help MPI_Comm_size help MPI_Comm_rank help MPI_COMM_NULL %------------------------------------------------------- % Do not play with MPI_COMM_NULL %------------------------------------------------------- % MPI_Errhandler_set(MPI_COMM_NULL,MPI_ERRORS_RETURN) % can't change errhndlr % [info size]=MPI_Comm_size(MPI_COMM_NULL) % and asking for size in NULL % [info size]=MPI_Comm_rank(MPI_COMM_NULL) % is error, so don't try that [info size]=MPI_Comm_size(MPI_COMM_SELF) % SELF is just this process [info size]=MPI_Comm_size(MPI_COMM_WORLD) % WORLD are all spawned at once %---------------------------------------------------------------% % since we don't start Octave with mpirun, but exec MPI_Init, % % we will never see an MPI_COMM_WORLD sized > 1 % %---------------------------------------------------------------% % want to see WORLD > 1 ? Hmpf... % % Starting octave from mpirun is a little bit tricky, % % since it does not wait for console input % % BTW, that's why we have to edit .octaverc instead of % % spawning "xterm" {'-e' 'octave' 'mergeParent'}, % % because octave would source mergeParent and _END_ % % without letting the user to type in commands % % we tried that around line 1666 in this tutorial, where it says: %---------------------------------------------------------------% % %%%%% yow! way too fast!!! %%%%%%%%%%%%%%%%% % %%%%% file args to Octave simply run & quit %%%%% %---------------------------------------------------------------% [info size]=MPI_Comm_size(NEWORLD) % size==3 [info rank]=MPI_Comm_rank(NEWORLD) % rank==0 %------------------------------------------------------- % Can check for intercommunicators and their remote size %------------------------------------------------------- help MPI_Comm_remote_size help MPI_Comm_test_inter [info flag] = MPI_Comm_test_inter (MPI_COMM_WORLD) % flag==0 [info flag] = MPI_Comm_test_inter (MPI_COMM_SELF) % flag==0 [info flag] = MPI_Comm_test_inter (NEWORLD) % flag==0 [info flag] = MPI_Comm_test_inter (children) % flag==1 [info rsiz] = MPI_Comm_remote_size(children) % rem.size==2 children [info lsiz] = MPI_Comm_size (children) % loc.size==1 parent [info size] = MPI_Comm_remote_size (NEWORLD) % info==5 [inf2 msg] = MPI_Error_string(info) % invalid communicator info==MPI_ERR_COMM % it's intra, not inter %%%%% ON FIRST CHILD %%%%%%% WORLD == MPI_COMM_WORLD % _mergeParent [info flag] = MPI_Comm_test_inter (NEWORLD) % 0 [info flag] = MPI_Comm_test_inter ( WORLD) % 0 [info size] = MPI_Comm_size ( WORLD) % size==2 [info flag] = MPI_Comm_test_inter (parent) % flag==1 [info rsiz] = MPI_Comm_remote_size(parent) % rsiz==1 parnt [info lsiz] = MPI_Comm_size (parent) % lsiz==2 chldr %------------------------------------------------------- % Can duplicate, compare and free communicators %------------------------------------------------------- help MPI_Comm_dup help MPI_Comm_compare help MPI_Comm_free [info WORLDNEW] = MPI_Comm_dup (NEWORLD) % blocks %%%%% ON FIRST CHILD %%%%%%% [info WORLDNEW] = MPI_Comm_dup (NEWORLD) % blocks %%%%% ON SECOND CHILD %%%%%%% [info WORLDNEW] = MPI_Comm_dup (NEWORLD)% unblocks [info flag]=MPI_Comm_test_inter(WORLDNEW) % 0 info = MPI_Comm_free (WORLDNEW) [info flag]=MPI_Comm_test_inter(WORLDNEW) % flag==0 info = MPI_Comm_free (WORLDNEW) [info flag]=MPI_Comm_test_inter(WORLDNEW) % flag==0 % keep it [info child2] = MPI_Comm_dup (children) % blocks [info par2] = MPI_Comm_dup (parent) % blocks [info par2] = MPI_Comm_dup (parent) % unblocks [info flag]=MPI_Comm_test_inter(par2) % flag==1 info = MPI_Comm_free (par2) [info flag]=MPI_Comm_test_inter(par2) % flag==1 info = MPI_Comm_free (par2) [info flag]=MPI_Comm_test_inter(child2) % flag==1 info = MPI_Comm_free (child2) print_info(WORLDNEW) % remember Octave's WORLD3 = WORLDNEW % lazy copy mechanism print_info(WORLDNEW) % one copy more print_info(WORLD3 ) % 3 copies are: WORLDNEW, WORLD3 % and args(0) for print_info itself [info addr]=MPI_Address(WORLD3) % side-effect MPI_Addr [info addr]=MPI_Address(WORLDNEW) % now they're unique print_info(WORLDNEW) % just one copy print_info(WORLD3 ) % count:2 for args(0) WORLD3==WORLDNEW % Octave can compare [info result] = MPI_Comm_compare (WORLD3, WORLDNEW) % for ident copies result==MPI_IDENT % (same pointers) WORLD3==NEWORLD % but not for [info result] = MPI_Comm_compare (WORLD3, NEWORLD) % congruent result==MPI_CONGRUENT % obtained with _dup WORLD3==MPI_COMM_WORLD [info result] = MPI_Comm_compare (WORLD3, MPI_COMM_WORLD) result==MPI_UNEQUAL help MPI_Comm_compare help MPI_SIMILAR % can't try it yet %------------------------------------------------------- % We will check for similar when we build similar groups (changing order) %------------------------------------------------------- % Can't Init-cycle right now, we want to keep both communicators for later use % remember they were obtained using % [info WORLDNEW]=MPI_Comm_dup(NEWORLD) % WORLD3=WORLDNEW %------------------------------------------------------- %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % Groups: % MPI_Comm_group, MPI_Group_size, _rank, % MPI_Group_compare, _free % MPI_GROUP_NULL, _EMPTY %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% help groups %------------------------------------------------------- % Groups: can extract them from communicators, check for rank & size, %------------------------------------------------------- help MPI_Comm_group help MPI_Group_size help MPI_Group_rank [info size]= MPI_Comm_size (NEWORLD) % 3 [info rank]= MPI_Comm_rank (NEWORLD) % 0 [info grp ]= MPI_Comm_group(NEWORLD) [info size]= MPI_Group_size(grp) % 3 [info rank]= MPI_Group_rank(grp) % 0 [info size]= MPI_Comm_size (NEWORLD) % 3 [info rank]= MPI_Comm_rank (NEWORLD) % 1 [info grp ]= MPI_Comm_group(NEWORLD) [info size]= MPI_Group_size(grp) % 3 [info rank]= MPI_Group_rank(grp) % 1 [info size]= MPI_Comm_size (NEWORLD) [info rank]= MPI_Comm_rank (NEWORLD) [info grp ]= MPI_Comm_group(NEWORLD) [info size]= MPI_Group_size(grp) % 3 [info rank]= MPI_Group_rank(grp) % 2 %------------------------------------------------------- % Groups: can compare and free them %------------------------------------------------------- help MPI_Group_compare help MPI_Group_free [info gr2 ]=MPI_Comm_group(WORLDNEW) % congruent to NEWORLD [info size]=MPI_Group_size(gr2) % size==3 [info rslt]=MPI_Group_compare(grp,gr2) % result==1 rslt==MPI_IDENT % ident. groups, same order & members [info rslt]=MPI_Comm_compare (WORLDNEW, NEWORLD) rslt==MPI_CONGRUENT %------------------------------------------------------- % remember they were obtained using % [info WORLDNEW]=MPI_Comm_dup(NEWORLD) %------------------------------------------------------- print_info(gr2) gr3 = gr2 % new identical group print_info(gr2) % one copy more print_info(gr3) % this is the new copy % notice we didn't ask for MPI_Address % no side-effect this time [info rsl1]=MPI_Group_compare(gr2, gr3) % result1==ident [info rsl2]=MPI_Group_compare(grp, gr3) % result2==ident all ([rsl1,rsl2]==MPI_IDENT) info= MPI_Group_free (gr2) % info==MPI_SUCCESS, side-effect redef help MPI_Group_free % on output, MPI_GROUP_NULL gr2 == MPI_GROUP_NULL % indeed it is gr3, grp==gr3 % this is still non-null, same old val print_info(gr2) % _Group_free re-defined grp2 print_info(gr3) % they are different variables now %------------------------------------------------------- % remember they were obtained using % [info grp ]= MPI_Comm_group(NEWORLD) % [info gr2 ]= MPI_Comm_group(WORLDNEW) % congruent to NEWORLD % gr3 = gr2 % new identical group %------------------------------------------------------- MPI_Errhandler_set(MPI_COMM_WORLD,MPI_ERRORS_RETURN) info= MPI_Group_free (gr3) % recall gr3=gr2, can free it again info= MPI_Group_free (gr3) % but not twice, since it's null gr3== MPI_GROUP_NULL % indeed it is [inf2 msg]=MPI_Error_string(info) % invalid group (for _Group_free) info ==MPI_ERR_GROUP help MPI_GROUP_NULL help MPI_GROUP_EMPTY info = MPI_Comm_free (WORLD3) info = MPI_Comm_free (WORLDNEW) % same for Comms all (MPI_COMM_NULL==[WORLDNEW,WORLD3]) info = MPI_Comm_free (WORLDNEW) % can't free twice [nfo msg]=MPI_Error_string(info) % invalid communicator info ==MPI_ERR_COMM %------------------------------------------------------- % Too long for a single session? finishing %------------------------------------------------------- %MPI_Errhandler_set(MPI_COMM_WORLD,MPI_ERRORS_ARE_FATAL) % original Errhandler % % we'll do that later :-) info =MPI_Group_free(MPI_GROUP_EMPTY) % any of these would crash :-) info =MPI_Group_free(MPI_GROUP_NULL) info =MPI_Group_free(gr3) % copy already freed!!! [nfo msg]=MPI_Error_string(info) % invalid group info ==MPI_ERR_GROUP info =MPI_Comm_free (WORLDNEW) % copy already freed info =MPI_Comm_free (MPI_COMM_NULL) [nfo msg]=MPI_Error_string(info) % invalid comm info ==MPI_ERR_COMM MPI_Errhandler_set(NEWORLD,MPI_ERRORS_ARE_FATAL) % original Errhandler info =MPI_Group_free(MPI_GROUP_EMPTY) % any of these will do :-) info =MPI_Group_free(MPI_GROUP_NULL) info =MPI_Comm_free (MPI_COMM_NULL) MPI_Finalize quit %------------------------------------------------------- % Ouch! forgot again about children %------------------------------------------------------- lamclean %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % Groups: % MPI_Group_incl, _excl, _range_incl, _range_excl % MPI_Comm_remote_group, _Comm_create % MPI_Group_translate_ranks % MPI_Group_intersection, _union, _difference %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% putenv('LAM_MPI_SSI_rpi','tcp'), MPI_Init [info children errs] = MPI_Comm_spawn ( '/usr/X11R6/bin/xterm',{'-e','octave'},... 2,MPI_INFO_NULL,0,MPI_COMM_SELF) [info NEWORLD] = MPI_Intercomm_merge (children, 0) MPI_Errhandler_set(NEWORLD,MPI_ERRORS_RETURN) hnam=system('hostname') %%%%% ON FIRST CHILD %%%%%%% hnam=system('hostname') [info rank]=MPI_Comm_rank(NEWORLD) % rank==1 %%%%% ON SECOND CHILD %%%%%%% hnam=system('hostname') [info rank]=MPI_Comm_rank(NEWORLD) help groups %------------------------------------------------------- % Groups: can operate with them, including & excluding ranks %------------------------------------------------------- help MPI_Group_incl help MPI_Group_excl [info grp ]=MPI_Comm_group (NEWORLD) % grp ==[par ch1 ch2] [info size]=MPI_Group_size (grp ) % size==3 [info grp1]=MPI_Group_excl (grp, [0]) % grp1== [ch1 ch2] [info size]=MPI_Group_size (grp1) % size==2 [info grp2]=MPI_Group_incl (grp, [2 1]) % grp2== [ch2 ch1] [info size]=MPI_Group_size (grp2) [info grp3]=MPI_Group_incl (grp, [1 2]) % grp3== [ch1 ch2] [info size]=MPI_Group_size (grp3) [info grp0]=MPI_Comm_group (MPI_COMM_WORLD) % grp0==[par] [info size]=MPI_Group_size (grp0) % size==1 %------------------------------------------------------- % Groups: can extract groups from remote part of intercomm %------------------------------------------------------- help MPI_Comm_remote_group [info grpc]=MPI_Comm_group (children) % children: local [parent] [info size]=MPI_Group_size (grpc) % : remote [ch1 ch2] [info size]=MPI_Comm_size (children) % returns local group size 1 [info rslt]=MPI_Group_compare(grp0, grpc) % grp0 we got it from WORLD rslt==MPI_IDENT % children's local side is par info =MPI_Group_free (grpc) % done with grpc grpc==MPI_GROUP_NULL % grpc NULL %info rslt]=MPI_Group_compare(grpc, MPI_GROUP_NULL) % don't, don't [info grpc]=MPI_Comm_remote_group(children) % remote side [ch1 ch2] [info size]=MPI_Group_size (grpc) % size==2 [info size]=MPI_Comm_remote_size (children) % ok, understood [info rslt]=MPI_Group_compare (grp3, grpc) % grp3 we built it from [1 2] rslt==MPI_IDENT [info rslt]=MPI_Group_compare (grp2, grpc) % grp3 we built it from [2 1] rslt==MPI_SIMILAR [info rslt]=MPI_Group_compare (grp1, grpc) % grp1 from [0 1 2] removing 0 rslt==MPI_IDENT [info rslt]=MPI_Group_compare (grp0, grpc) % grp1 from WORLD, it's [0] rslt==MPI_UNEQUAL info =MPI_Group_free (grpc) % done again with this grpc grpc==MPI_GROUP_NULL % grpc NULL info =MPI_Comm_free (children) % done with intercomm as well children==MPI_COMM_NULL % children NULL %------------------------------------------------------- % Groups: can translate ranks in different groups %------------------------------------------------------- help MPI_Group_translate_ranks MPI_IDENT, MPI_SIMILAR, MPI_UNEQUAL, MPI_UNDEFINED [info result] = MPI_Group_compare (grp, grp1) % unequal [info ranks ] = MPI_Group_translate_ranks (grp, [0 1 2], grp1) % missing 0 all ( ranks == [MPI_UNDEFINED, 0, 1] ) [info result] = MPI_Group_compare (grp1, grp2) % similar [info ranks ] = MPI_Group_translate_ranks (grp1, [0 1], grp2) % reverse order all ( ranks == [1, 0] ) [info result] = MPI_Group_compare (grp1, grp3) % ident [info ranks ] = MPI_Group_translate_ranks (grp1, [0 1], grp3) % same order all ( ranks == [0, 1] ) [info result] = MPI_Group_compare (grp2, grp3) % similar [info ranks ] = MPI_Group_translate_ranks (grp2, [0 1], grp3) % reversed all ( ranks == [1, 0] ) %------------------------------------------------------- % Groups: can compute intersection/union/difference among groups %------------------------------------------------------- help MPI_Group_intersection help MPI_Group_union help MPI_Group_difference [info grpi]=MPI_Group_intersection(grp, grp0) % grp =[par ch1 ch2] [info size]=MPI_Group_size(grpi) % grp0=[par] = grpi [info rnks]=MPI_Group_translate_ranks(grp,[0 1 2],grpi) % only rank 0 all ( rnks==[0 MPI_UNDEFINED MPI_UNDEFINED] ) [info rslt]=MPI_Group_compare(grp0, grpi) % ident rslt==MPI_IDENT info =MPI_Group_free(grpi) % done w/intersection grpi==MPI_GROUP_NULL [info grpd]=MPI_Group_difference(grp, grp0) % grpd=[ch1 ch2] [info size]=MPI_Group_size(grpd) % size==2 [info rnks]=MPI_Group_translate_ranks(grp,[0 1 2],grpd) % missing 0 all ( rnks==[MPI_UNDEFINED 0 1] ) [info rslt]=MPI_Group_compare(grp1,grpd) % ident info =MPI_Group_free (grpd) info = MPI_Group_free (grp2) info = MPI_Group_free (grp3) %info = MPI_Group_free (grp3) % don't, didn't set errhandler/WORLD [info grps]=MPI_Group_union (grp1, grp0) % grp0=[par] [info size]=MPI_Group_size (grps) % grp1=[ch1 ch2] [info rslt]=MPI_Group_compare (grp, grps) % similar rslt==MPI_SIMILAR [info rnks]=MPI_Group_translate_ranks(grps,[0 1 2],grp) % 0 is last all ( rnks==[1 2 0] ) %------------------------------------------------------- % Groups: can operate with them, including & excluding rank ranges %------------------------------------------------------- help MPI_Group_range_incl help MPI_Group_range_excl info =MPI_Group_free(grp0) % we're creating a new grp0/1 info =MPI_Group_free(grp1) % free former ones/no mem leak MPI_Errhandler_set(MPI_COMM_WORLD,MPI_ERRORS_RETURN) info =MPI_Group_free(grp1) % can't do it twice [info2 msg]=MPI_Error_string(info) % invalid group info ==MPI_ERR_GROUP all ( MPI_GROUP_NULL==[grp0,grp1] ) [info grp0]=MPI_Group_range_excl (grp,[1 2 1] ) % grp=[par ch1 ch2] [info size]=MPI_Group_size (grp0) % exclude 1 to 2 step 1 [info rnks]=MPI_Group_translate_ranks(grp,[0 1 2],grp0) % only parent left all ( rnks==[0 MPI_UNDEFINED MPI_UNDEFINED] ) [info grp1]=MPI_Group_range_incl (grp,[1 2 1] ) % include 1 to 2 step 1 [info size]=MPI_Group_size (grp1) % size==2 [info rnks]=MPI_Group_translate_ranks(grp,[0 1 2],grp1) % [ch1 ch2] left all ( rnks==[MPI_UNDEFINED 0 1] ) [info rnks]=MPI_Group_translate_ranks(grp1,[0 1 2],grp) % [ch1 ch2] left [info2 msg]=MPI_Error_string(info) % there is no rank 2 info ==MPI_ERR_RANK % invalid rank in grp1 [info rnks]=MPI_Group_translate_ranks(grp1,[0 1 ],grp) % avoid it all ( rnks==[1 2] ) [info grps]=MPI_Group_union(grp1, grp0) % notice order: [ch1 ch2 par] [info size]=MPI_Group_size (grps) % size==3 [info rnks]=MPI_Group_translate_ranks(grps,[0 1 2],grp) % ranks 0 1 2 in grps all ( rnks==[1 2 0] ) % are the ranks 1 2 0 in grp info = MPI_Group_free (grp0) % done info = MPI_Group_free (grp1) % keep grps to create comm %------------------------------------------------------- % Communicators: can create them. Recall we said % "We will check for similar when we build similar groups (changing order)" %------------------------------------------------------- help MPI_Comm_create [info comms] = MPI_Comm_create (NEWORLD, grps) % blocks % ouch! no grps in children!!! %%%%% ON FIRST CHILD %%%%%%% [info grp0]=MPI_Comm_group (parent) % grp0=[par] [info size]=MPI_Group_size (grp0) % local size 2 info =MPI_Group_free (grp0) % not interested info =MPI_Comm_free (parent) % anymore on that side MPI_COMM_NULL==parent info =MPI_Comm_free (parent) % can't do it twice [info2 msg]=MPI_Error_string(info) % invalid communicator info ==MPI_ERR_COMM [info grp ]=MPI_Comm_group (NEWORLD) % grp=[par ch1 ch2] [info grps]=MPI_Group_range_incl(grp,... [1 2 1 ;... % include 1 to 2 step 1 0 0 1]) % then 0 to 0 step 1 [info rnks]=MPI_Group_translate_ranks(grps,[0 1 2],grp) all ( rnks== [1 2 0] ) % grps==[ch1 ch2 par] [info comms] = MPI_Comm_create (NEWORLD, grps) %%%%% ON SECOND CHILD %%%%%%% [info grp ]=MPI_Comm_group(NEWORLD) [info grps]=MPI_Group_range_incl(grp,... [1 2 1;0 0 1]) [info comms]=MPI_Comm_create(NEWORLD, grps) %%%%%%%%%%%%%% % unblocks all %%%%%%%%%%%%%% [info rslt]=MPI_Comm_compare(NEWORLD, comms) % similar rslt==MPI_SIMILAR %------------------------------------------------------- % cleaning %------------------------------------------------------- info = MPI_Group_free (grps) info = MPI_Comm_free (comms) MPI_Finalize quit %%%%% ON FIRST CHILD %%%%%%% info = MPI_Group_free (grp ) info = MPI_Group_free (grps) info = MPI_Comm_free (comms) MPI_Finalize quit %%%%% ON SECOND CHILD %%%%%%% info = MPI_Group_free (grp ) info = MPI_Group_free (grps) info = MPI_Comm_free (comms) MPI_Finalize quit %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % Communicators: % MPI_Intercomm_create, MPI_Comm_split %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% putenv('LAM_MPI_SSI_rpi','tcp'), MPI_Init [info children errs] = MPI_Comm_spawn ( '/usr/X11R6/bin/xterm',{'-e','octave'},... 2,MPI_INFO_NULL,0,MPI_COMM_SELF) [info NEWORLD] = MPI_Intercomm_merge (children, 0) MPI_Errhandler_set(NEWORLD,MPI_ERRORS_RETURN) hnam=system('hostname') %%%%% ON FIRST CHILD %%%%%%% hnam=system('hostname') [info size]=MPI_Comm_size(NEWORLD) % size==3 [info rank]=MPI_Comm_rank(NEWORLD) % rank==1 %%%%% ON SECOND CHILD %%%%%%% hnam=system('hostname') [info size]=MPI_Comm_size(NEWORLD) [info rank]=MPI_Comm_rank(NEWORLD) help comms % we're almost done help MPI_Intercomm_create help MPI_Comm_split %------------------------------------------------------- % Groups: one more experiment on creating Comms %------------------------------------------------------- help MPI_Comm_create help MPI_Comm_group help MPI_Group_incl [info grp ]=MPI_Comm_group(NEWORLD) % [par ch1 ch2] [info grp1]=MPI_Group_incl(grp,[1]) % [ch1] [info com1]=MPI_Comm_create(NEWORLD, grp1) % blocks [info grp ]=MPI_Comm_group(NEWORLD) [info grp1]=MPI_Group_incl(grp,[1]) % [ch1] [info com1]=MPI_Comm_create(NEWORLD, grp1) % blcks [info grp ]=MPI_Comm_group(NEWORLD) [info grp1]=MPI_Group_incl(grp,[1]) % unbl [info com1]=MPI_Comm_create(NEWORLD, grp1) % naturally, blocks in NEWORLD, but only meaningful in rank 1 MPI_Errhandler_set(MPI_COMM_WORLD,MPI_ERRORS_RETURN) % avoid abort [info size]=MPI_Comm_size(com1) % we're not in com1 [inf2 msg ]=MPI_Error_string(info) % invalid comm info ==MPI_ERR_COMM [info size]=MPI_Comm_size(com1) % no problem, size==1 [info size]=MPI_Comm_size(com1) info==MPI_ERR_COMM % didn't need Errhndlr %nfo = MPI_Group_free(grp1) % keep grp1/later use info = MPI_Comm_free (com1) % invalid info ==MPI_ERR_COMM %%%%% %nfo = MPI_Group_free(grp1) % keep it info = MPI_Comm_free (com1) info ==MPI_SUCCESS info = MPI_Group_free(grp1) info ==MPI_SUCCESS info = MPI_Comm_free (com1) info ==MPI_ERR_COMM % not in it info = MPI_Group_free(grp1) info ==MPI_ERR_GROUP % already freed info = MPI_Group_size(grp1) info ==MPI_ERR_GROUP [info size] = MPI_Group_size(grp1) % still valid here info = MPI_Group_free(grp1) % done [info size] = MPI_Group_size(grp1) % funny, size==1 info = MPI_Group_free(grp1) % done as well [info size] = MPI_Comm_size (com1) % not in it info ==MPI_ERR_COMM %------------------------------------------------------- % Can create intercomms with local/remote parts %------------------------------------------------------- help MPI_Intercomm_create [info icom01]=MPI_Intercomm_create(MPI_COMM_SELF,0, ... % local part: parent children,0, 666) % remote: children % remote peer: ch1 [info icom01]=MPI_Intercomm_create( MPI_COMM_WORLD,0, ... % local: children parent,0, 666) % remote: parent %%%%% ON SECOND CHILD %%%%%%% [info icom01]=MPI_Intercomm_create( MPI_COMM_WORLD,0, ... parent,0, 666) [info flag] = MPI_Comm_test_inter (icom01) % flag==1 [info size] = MPI_Comm_remote_size (icom01) % remote size==2 chld [info grpr] = MPI_Comm_remote_group(icom01) % remote group=children [info size] = MPI_Group_size(grpr) % size==2 [info size] = MPI_Group_size(grp ) % size==3 from NEWORLD [info grpc] = MPI_Group_excl(grp,0) % grpc=[ch1 ch2] [info size] = MPI_Group_size(grpc) % size==2 now [info rslt] = MPI_Group_compare(grpr, grpc) % identical rslt == MPI_IDENT info = MPI_Group_free (grpc) % done info = MPI_Group_free (grpr) [info size] = MPI_Comm_size (icom01) % local size==1 parent [info grpl] = MPI_Comm_group(icom01) % local group=myself [info size] = MPI_Group_size(grpl) % size==1 [info grps] = MPI_Comm_group(MPI_COMM_SELF) % myself [info size] = MPI_Group_size(grps) % size==1 [info rslt] = MPI_Group_compare(grpl, grps) % identical rslt == MPI_IDENT info = MPI_Group_free (grps) % done info = MPI_Group_free (grpl) [info flag]=MPI_Comm_test_inter(icom01) % flag==1 [info size]=MPI_Comm_size (icom01) % local group 2 chldrn [info grpl]=MPI_Comm_group(icom01) [info size]=MPI_Group_size(grpl) [info grpw]=MPI_Comm_group(MPI_COMM_WORLD) [info rslt]=MPI_Group_compare(grpl,grpw)% identical rslt == MPI_IDENT info = MPI_Group_free(grpw) % done info = MPI_Group_free(grpl) info = MPI_Comm_free(icom01) % done %%%%% ON SECOND CHILD %%%%%%% info = MPI_Comm_free(icom01) info = MPI_Comm_free (icom01) MPI_COMM_NULL==icom01 %------------------------------------------------------- % Can split communicators in parts %------------------------------------------------------- help MPI_Comm_split [info black]=MPI_Comm_split(NEWORLD, 0, 69) % color 0 ("black") order 69 % blocks [info white]=MPI_Comm_split(NEWORLD, 1, 69) % blocks %%%%% ON SECOND CHILD %%%%%%% % unblocks [info black]=MPI_Comm_split(NEWORLD, 0, 66) % color 0 order 66 before 69 [info size]=MPI_Comm_size (black) % size==2 [info grpb]=MPI_Comm_group(black) % [child2 parent] [info rnks]=MPI_Group_translate_ranks(grp,[0 1 2],grpb) % new ranks are: all ( rnks==[1 MPI_UNDEFINED 0] ) info = MPI_Group_free(grpb) info = MPI_Comm_free (black) [info size]=MPI_Comm_size (white) % size==1 [info grpw]=MPI_Comm_group(white) % group=[ch1] [info rnks]=MPI_Group_translate_ranks(grp,[0 1 2],grpw) all ( rnks==[MPI_UNDEFINED 0 MPI_UNDEFINED] ) info = MPI_Group_free(grpw) info = MPI_Comm_free (white) %%%%% ON SECOND CHILD %%%%%%% info = MPI_Comm_free (black) %------------------------------------------------------- % cleaning %------------------------------------------------------- info = MPI_Group_free(grp) info = MPI_Comm_free (NEWORLD) MPI_Finalize quit info = MPI_Group_free (grp) info = MPI_Comm_free (NEWORLD) MPI_Finalize quit info = MPI_Group_free (grp) info = MPI_Comm_free (NEWORLD) MPI_Finalize quit %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % The MPI_Info object (MPI-2.0) % MPI_Info_create, _dup, _free % MPI_Info_set, _get, _get_valuelen, _get_nkeys, _get_nthkey % MPI_INFO_NULL %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % no children this time %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% MPI_Init help info %------------------------------------------------------- % Info: create an object, add entries, lookup entries, query length %------------------------------------------------------- help MPI_Info_create help MPI_Info_set help MPI_Info_get help MPI_Info_get_valuelen [info inf] = MPI_Info_create info = MPI_Info_set (inf, 'Key1', 'first element''s value') info = MPI_Info_set (inf, 'Key2', 'second element''s value') [info flag val] = MPI_Info_get (inf, 'Key3') % flag==0 [info flag val] = MPI_Info_get (inf, 'Key2') % flag==1 whos val % val=='second...' [info len flag] = MPI_Info_get_valuelen (inf, 'Key2') % len==22 [info flag val] = MPI_Info_get (inf, 'Key1') % flag==1 whos val % val=='first...' [info len flag] = MPI_Info_get_valuelen (inf, 'Key1') % len==21 [info len flag] = MPI_Info_get_valuelen (inf, 'Key0') % flag==0 %------------------------------------------------------- % Info: query no. entries, get indexed entry %------------------------------------------------------- help MPI_Info_get_nkeys help MPI_Info_get_nthkey [info nkeys] = MPI_Info_get_nkeys (inf) % nkeys==2 [info key ] = MPI_Info_get_nthkey (inf, 1) % key=='Key2' [info len flag] = MPI_Info_get_valuelen (inf, key) % len==22 [info flag val] = MPI_Info_get (inf, key) % val=='second...' MPI_Errhandler_set(MPI_COMM_WORLD,MPI_ERRORS_RETURN) % avoid abort [info key ] = MPI_Info_get_nthkey (inf, 2) % no such key [inf2 msg ] = MPI_Error_string(info) % invalid arg info == MPI_ERR_ARG %------------------------------------------------------- % Info: duplicate / free info objects %------------------------------------------------------- help MPI_Info_dup help MPI_Info_free help MPI_INFO_NULL [info inf2 ] = MPI_Info_dup (inf) [info nkeys] = MPI_Info_get_nkeys (inf2) % same 2 keys [info key ] =MPI_Info_get_nthkey(inf2, 1) % key=='Key2' [info flag val]=MPI_Info_get (inf2, key) % val=='second...' [info key ] =MPI_Info_get_nthkey(inf2, 0) % Key1 [info flag val]=MPI_Info_get (inf2, key) % first... info =MPI_Info_free (inf2) MPI_INFO_NULL==inf2 help MPI_Info_free %info =MPI_Info_free(inf2) % don't try, SIGSEGV, Errhndlr can't help here %------------------------------------------------------- % Info: delete entries %------------------------------------------------------- help info % we're done help MPI_Info_delete % this is the last one info = MPI_Info_delete (inf, 'Key3') [inf2 msg ] = MPI_Error_string(info) % no such info key help MPI_ERRCODES info == MPI_ERR_INFO_NOKEY [info nkeys] = MPI_Info_get_nkeys (inf) % 2 keys [info key ] = MPI_Info_get_nthkey(inf, 0) % 1st one keyed 'Key1' [info flag val]=MPI_Info_get (inf, key) % BTW, value 'first...' info = MPI_Info_delete (inf, key) % delete it [info nkeys] = MPI_Info_get_nkeys (inf) % 1 key left info = MPI_Info_free (inf) % done, clean MPI_Finalize quit %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % Process management % MPI_Comm_spawn[_multiple], MPI_Comm[_accept,_connect,_disconnect], % MPI_[Open_,Close_]port, MPI_[ [Un] publish_, Lookup_] name %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % funny mixing of xterms, octaves, etc % we require 3 hosts in $LAMBHOST, your host computer last in list %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% help process %------------------------------------------------------- % can spawn program copies, as long as they MPI_Init %------------------------------------------------------- help MPI_Comm_spawn MPI_Init [info children errs] = MPI_Comm_spawn ( '/usr/X11R6/bin/xterm',... {},2,MPI_INFO_NULL,0,MPI_COMM_SELF) % blocks %%%%% ON CHILDREN XTERMs %%%%%%% hostname # already in prompt, anyways cd $MPITB_HOME; ls ls stdlones cd stdlones cat MPI_Init.c cat MPI_Finalize.c mpicc -o MPI_Init MPI_Init.c mpicc -o MPI_Finalize MPI_Finalize.c ./MPI_Init # grandchild, not child. Blocks ./MPI_Finalize # doesn't inherit Init from grandchild exit %%%%% DONE WITH XTERMs %%%%%%% MPI_Comm_free(children) tbxpath=getenv('MPITB_HOME') [info children errs] = MPI_Comm_spawn ( '/usr/X11R6/bin/xterm',... {'-e',[tbxpath '/stdlones/MPI_Init']},... 2,MPI_INFO_NULL,0,MPI_COMM_SELF) % quick, isn't it? MPI_Comm_free(children) %------------------------------------------------------- % can spawn other octaves... we've been there %------------------------------------------------------- [info children errs] = MPI_Comm_spawn ( '/usr/X11R6/bin/xterm',{'-e','octave'},... 2,MPI_INFO_NULL,0,MPI_COMM_SELF) [info NEWORLD] = MPI_Intercomm_merge (children, 0) MPI_Comm_free(children) %%%%% ON CHILDREN Octaves %%%%%% MPI_Finalize quit %------------------------------------------------------- % can spawn arbitrary processes, as long as they MPI_Init % our previous example worked because _mergeParent does MPI_Init % edit the .octaverc file so that children Octaves don't call _mergeParent % and try this example: %------------------------------------------------------- help MPI_Comm_spawn_multiple [info children errs] = MPI_Comm_spawn_multiple (... {'/usr/X11R6/bin/xterm' ;... % two xterms '/usr/X11R6/bin/xterm'},... % (column cells) {{'-e','octave'};... % args 1st xterm {} },... % args 2nd xterm {2;2},... % 2 copies of each {MPI_INFO_NULL;MPI_INFO_NULL},... % no additional info 0,MPI_COMM_SELF) % and i'm the parent %%%%% ON xterms %%%%%%% $MPITB_HOME/stdlones/MPI_Init # unblock spawn exit %%%%% ON octaves %%%%%%% MPI_Init % unblock spawn quit % dirty quitting MPI_Comm_free(children) %------------------------------------------------------- % edit the .octaverc file again (let children call _mergeParent) % you can repeat the previous example with _mergeParent, if you want % no need to type MPI_Init in Octaves, since _mergeParent already does it %------------------------------------------------------- % How could we communicate 2 MPI processes not related by spawn? % ie, not mpirun together, not spawned one from the other %------------------------------------------------------- #---- open a new xterm and log into cluster node ------- ssh ox1 # ox1 is the name of my slave cluster node hostname # should be ox1 by now cd $MPITB_HOME # typically $HOME/octave/mpitb octave %---- Octave mode from now on --------------------------- MPI_Init [info size]=MPI_Comm_size(MPI_COMM_WORLD) % size==1 [info rank]=MPI_Comm_rank(MPI_COMM_WORLD) % rank==0 %------------------------------------------------------- % we have no inter-communicator to merge... see the point? %------------------------------------------------------- % can open ports, accept connections on them, connect to them %------------------------------------------------------- help MPI_Open_port help MPI_Comm_accept help MPI_Comm_connect [info port] = MPI_Open_port (MPI_INFO_NULL) % write down port name whos port % it's a simple char string % something like 'n2:i11:323' %%%%% ON cluster node Octave %%%%%%% hnam=system('hostname') % don't get the wrong window:-) port = 'n2:i11:323' % or whatever you got above whos port % requires knowing port name [info svcomm]=MPI_Comm_connect( % blocks, of course port,MPI_INFO_NULL,0,MPI_COMM_WORLD) [info clcomm]=MPI_Comm_accept (port, MPI_INFO_NULL, 0, MPI_COMM_SELF) % unblocks [info size] = MPI_Comm_size (clcomm) % 1 local [info size] = MPI_Comm_remote_size (clcomm) % 1 remote [info size] = MPI_Comm_size (svcomm) % 1 local [info size] = MPI_Comm_remote_size (svcomm) % 1 remote %------------------------------------------------------- % having inter-communicator, it's the same old stuff now %------------------------------------------------------- [info NEWORLD] = MPI_Intercomm_merge(clcomm,0) % blocks [info NEWORLD] = MPI_Intercomm_merge(svcomm,1) % unblocks [info size] = MPI_Comm_size (NEWORLD) % 2 ranks [info rank] = MPI_Comm_rank(NEWORLD) % server rnk 0 info = MPI_Send (port, 1, 0, NEWORLD) % send sthing whos port port(1:10)=' ' % room for recv [info stat] = MPI_Recv (port, 0, 0, NEWORLD) % recv sthing port % if only bfore MPI_Comm_free(NEWORLD) % done with it MPI_Comm_free(NEWORLD) %------------------------------------------------------- % done, disconnecting. Can disconnect from ports, and clos'em %------------------------------------------------------- help MPI_Comm_disconnect help MPI_Close_port %%%%% ON cluster node Octave %%%%%%% info = MPI_Comm_disconnect (svcomm) MPI_COMM_NULL == svcomm info = MPI_Comm_disconnect (clcomm) MPI_COMM_NULL == clcomm % info = MPI_Close_port (port) % not yet, keep it for next try % port %------------------------------------------------------- % nice, but required knowing port name at client side % can publish a port name and look it up from client side % this only requires according previously on a "well-known" service name %------------------------------------------------------- help process % we're almost done help MPI_Publish_name help MPI_Lookup_name help MPI_Unpublish_name info = MPI_Publish_name ('ServName', MPI_INFO_NULL, port) % same port port %%%%% ON cluster node Octave %%%%%%% port % tricky, we knew it beforehand port(1:10)=' ' % clean it [info port] = MPI_Lookup_name('ServName',... % there it is MPI_INFO_NULL) % we just needed SvcNam %--------------------- % same old stuff now %--------------------- [info svcomm]=MPI_Comm_connect(port,MPI_INFO_NULL,... 0,MPI_COMM_WORLD) % blocks [info clcomm] =MPI_Comm_accept (port,MPI_INFO_NULL, 0,MPI_COMM_SELF) % unblocks [info NEWORLD]=MPI_Intercomm_merge(clcomm,0) % blocks [info NEWORLD] = MPI_Intercomm_merge(svcomm,1) % unblocks [info rank] = MPI_Comm_rank (NEWORLD) % rank 1 port(1:10)=' ' % room for recv [info stat] = MPI_Recv (port, 0, 0, NEWORLD) % from rank 0 info = MPI_Send (port,1,0,NEWORLD) % unblocks info = MPI_Comm_free (NEWORLD) % done port % it worked MPI_Comm_free (NEWORLD) % done with it MPI_Comm_disconnect (svcomm) MPI_COMM_NULL == svcomm MPI_COMM_NULL == NEWORLD info = MPI_Comm_disconnect (clcomm) MPI_COMM_NULL == clcomm MPI_COMM_NULL == NEWORLD info = MPI_Unpublish_name ('ServName', MPI_INFO_NULL, port) % done with it info = MPI_Close_port (port) %------------------------------------------------------- % ok, finishing session %------------------------------------------------------- MPI_Errhandler_set(MPI_COMM_SELF, MPI_ERRORS_RETURN) % trying hard MPI_Errhandler_set(MPI_COMM_WORLD,MPI_ERRORS_RETURN) % to survive info = MPI_Unpublish_name ('ServName', % can't twice MPI_INFO_NULL, port) % info==25 [nfo msg]=MPI_Error_string(info) % publish svc help MPI_ERRCODES info == MPI_ERR_SERVICE info = MPI_Close_port (port) % info==16 [nfo msg]=MPI_Error_string(info) % unclassified help MPI_ERR_OTHER info == MPI_ERR_OTHER %------------------------------------------------------- % it's funny to be naughty sometimes %------------------------------------------------------- MPI_COMM_NULL == clcomm % don't play with NULL info = MPI_Comm_disconnect (clcomm) % can't disconnect twice % no matter how hard we tried %------------------------------------------------------- % MPI process rank 0 (n2, p11315) caught a SIGSEGV in MPI_Comm_disconnect. % Rank (0, MPI_COMM_WORLD): Call stack within LAM: % Rank (0, MPI_COMM_WORLD): - MPI_Comm_disconnect() % Rank (0, MPI_COMM_WORLD): - main() %------------------------------------------------------- %%%%% ON cluster node Octave %%%%%%% [info flag]=MPI_Initialized %%%%% ON master node Octave %%%%%%% reset # in xterm lamclean # the other dies %%%%% ON cluster node xterm %%%%%%% reset lamclean %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % Errhandlers & Miscellaneous % MPI_Errhandler_ [create,free,set,get] % MPI_Error_ [class|string] % MPI_ERRORS_ARE_FATAL / RETURN % MPI_ERRHANDLER_NULL / OCTAVE_ERRORS_RETURN %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % Just 1 computer this time %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% MPI_Init help errors %------------------------------------------------------- % can get/set error handler %------------------------------------------------------- help MPI_Errhandler_get help MPI_Errhandler_set [info eh]=MPI_Errhandler_get (MPI_COMM_WORLD) eh==MPI_ERRORS_ARE_FATAL info = MPI_Errhandler_set(MPI_COMM_WORLD,MPI_ERRORS_RETURN) [info eh]=MPI_Errhandler_get (MPI_COMM_WORLD) eh==MPI_ERRORS_RETURN help MPI_ERRORS_ARE_FATAL help MPI_ERRORS_RETURN %------------------------------------------------------- % can obtain informational error string %------------------------------------------------------- help MPI_Error_string help MPI_ERRCODES help MPI_ERR_BASE help MPI_ERR_LASTCODE [info msg] = MPI_Error_string(MPI_ERR_BASE) % notice info==0 [errc msg] = MPI_Error_string(MPI_ERR_LASTCODE) % notice errc==13 [info msg] = MPI_Error_string(errc) % LASTCODE was invalid errc == MPI_ERR_ARG % invalid arg for Error_string [info cls] = MPI_Error_class (errc) % cls==errc [info msg] = MPI_Error_string(cls) cls == MPI_ERR_ARG %------------------------------------------------------- % error codes can be more elaborated, and we should then resume it to classes %------------------------------------------------------- A = [1 2; 3 4] info = MPI_Send(A,0,7,MPI_COMM_WORLD) % send 4 doubles B = zeros(1,3) % (not enough) room for them [info stat] = MPI_Recv(B,0,7,MPI_COMM_WORLD) % info=15, len=24, err=677135 [inf2 msg] = MPI_Error_string(info) % message truncated info== MPI_ERR_TRUNCATE [inf2 cls] = MPI_Error_class (info) % cls==info cls == info [inf2 msg] = MPI_Error_string(stat.err) % message truncated, I/O error [inf2 cls] = MPI_Error_class (stat.err) % cls==info !!! [inf2 msg] = MPI_Error_string(cls) % message truncated cls == MPI_ERR_TRUNCATE help MPI_Abort % code:15; funct:9; class:8bits hexerr = dec2hex(stat.err) % splice errcode into parts % A550F = 5 | 055 | 0F hexerr(end-1:end) % lower 8bits cls =hex2dec(hexerr(end-1:end)) [inf2 msg] = MPI_Error_string(cls) % message truncated func=floor(stat.err/0x100) % remove lower 8bits dec2hex(func) % yup, A55 code=floor(func /0x200) % remove lower 9bits dec2hex(code) % yup, A>>1 == 5 func-=code*0x200 dec2hex(func) % yup, 55 hexreconstr=dec2hex( ( (code*0x200 + func) * 0x100) + cls) all(hexerr==hexreconstr) % yup, reconstructed ok cls % truncate == err #15 func % MPI_Recv == func#85 code % I/O error== code# 5 %------------------------------------------------------- % can play with Octave error handlers %------------------------------------------------------- help MPI_Errhandler_create which OCTAVE_ERRORS_RETURN % suitable Octave cmd type OCTAVE_ERRORS_RETURN % for Errhandler_create [info EHND]=MPI_Errhandler_create('OCTAVE_ERRORS_RETURN') info =MPI_Errhandler_set(MPI_COMM_WORLD,EHND) [info eh ]=MPI_Errhandler_get(MPI_COMM_WORLD) eh ==EHND %------------------------------------------------------- % causing a 'dummy' error %------------------------------------------------------- [errc msg] = MPI_Error_string(MPI_ERR_LASTCODE) %-----------------------------------------------% Whoa! lots of output % returned error: % errclass==13, error==22, errmsg== % in MPI_COMM_WORLD % won't take any action on it, except for writing its MPI_Error_string: % MPI_Error_string: invalid argument %------------------------------------------------------- [info cls] = MPI_Error_class (errc) % same previous error errc == MPI_ERR_ARG % invalid arg for Error_string cls == MPI_ERR_ARG %------------------------------------------------------- % causing the other 'test' error %------------------------------------------------------- info = MPI_Send(A,0,7,MPI_COMM_WORLD) % send 4dbls, no room for recv [info stat] = MPI_Recv(B,0,7,MPI_COMM_WORLD) % info=15, len=24, err=677135 %-----------------------------------------------% Whoa! lots of output % returned error: % errclass==15, error==5, errmsg== % in MPI_COMM_WORLD % won't take any action on it, except for writing its MPI_Error_string: % MPI_Error_string: message truncated %------------------------------------------------------- [info cls] = MPI_Error_class (stat.err) % same previous error cls == MPI_ERR_TRUNCATE % message truncated %------------------------------------------------------- % undoing Octave errhandler %------------------------------------------------------- help MPI_Errhandler_free %info= MPI_Errhandler_set(MPI_COMM_WORLD,MPI_ERRORS_RETURN) % should do that info= MPI_Errhandler_free (EHND) % don't!!! still set in WORLD! MPI_ERRHANDLER_NULL==EHND % free -> set to NULL %------------------------------------------------------- % weird: if we get an error right now, % and since we have not changed ErrHndlr back to ERRS_RET, % % LAM/MPI still will call EHND, the internal MPITB wrapper for % OCTAVE_ERRORS_RETURN (we set that wrapper when MPI_Errh_create) % % but the internal wrapper has deleted the string OCTAVE_ERRORS_RETURN % that should be feval()-uated when the wrapper is called by LAM/MPI %------------------------------------------------------- info= MPI_Errhandler_free (EHND) % can't do it twice % I told you... :-) %-----------------------------------------------% MPI keeps calling wrapper % error: feval: the symbol `' is not valid as a function %----------------------------------------------- [info eh2]=MPI_Errhandler_get (MPI_COMM_WORLD) eh ==eh2 % it's still old EHND info= MPI_Errhandler_set(MPI_COMM_WORLD, % back to ERR_RET MPI_ERRORS_RETURN) info= MPI_Errhandler_free (EHND) % can't do it twice, info=13 [inf2 msg] = MPI_Error_string(info) % invalid argument help errors % well, we have seen it all MPI_Finalize quit %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % Attribute Caching % MPI_Keyval_ [create|free|], MPI_Attr_ [put|get|delete] %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % Again, just 1 computer, although the interesting part % of attributes is that they're stored in the communicator % and thus immediately accessible to all ranks in comm % % That's true for predefined attributes (ints) % MPITB could also do the same if attrs restricted to ints % MPITB tries to be able to store any octave variable as attr % In doing so, other ranks can see only the stored pointer, not the value %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% help environ help caching MPI_Init MPI_Errhandler_set(MPI_COMM_WORLD,MPI_ERRORS_RETURN) % try to survive A={1,2;3,4} print_info(A) % only 1 copy (2nd is for print_info) %------------------------------------------------------- % Creating, setting and accessing attributes %------------------------------------------------------- help MPI_Keyval_create help MPI_Attr_put help MPI_Attr_get [info kv] = MPI_Keyval_create ('NULL', 'NULL', {}) % kv==19 or so help MPI_ATTRKVALS % around 18 predef.attr info = MPI_Attr_put (MPI_COMM_WORLD, kv, A) % store attr. in WORLD print_info(A) % 2 copies now(+p_info) [info B flag] = MPI_Attr_get (MPI_COMM_WORLD, kv) % flag 1, B==A print_info(A) % 3 copies now(+p_info) print_info(B) % A, B & stored@MPITB [info C flag] = MPI_Attr_get (MPI_COMM_WORLD, 666) % info==28 [inf2 msg] = MPI_Error_string(info) % invalid key value info ==MPI_ERR_KEYVAL %------------------------------------------------------- % Overwriting, deleting attributes %------------------------------------------------------- B = [1 2;3 4], WORLD=MPI_COMM_WORLD % tired of typing % overwriting attr A info = MPI_Attr_put (WORLD, kv, [1 2;3 4]) % this temporary will % be saved/stored [info C flag] = MPI_Attr_get (WORLD, kv) % flag==1, C==B all(all(B==C)) % but 1 difference: print_info(C) % 2 copies: C & stored (+1) print_info(B) % 1 copy : B (+1 @ print_info) info = MPI_Attr_put (WORLD, kv, B) % overwrite temporary [info C flag] = MPI_Attr_get (WORLD, kv) % flag==1, C==B print_info (C) % 3 copies B,C,stored print_info (A) % 1 copy (+print_info) % old stored copy ovrwr B = 'hello' print_info (C) % 2 copies: C & attr info = MPI_Attr_put (WORLD, kv, B) % attr overwritten print_info (C) % last copy print_info (B) % 2 copies, here & attr [info C flag] = MPI_Attr_get (WORLD, kv) % flag==1, C==B print_info (C) % attr,B,C are 'hello' all(B==C), C %------------------------------------------------------- % Deleting attributes % The delete-function stuff - problem %------------------------------------------------------- help MPI_Attr_delete info = MPI_Attr_delete(WORLD, kv) % should delete attr print_info (B) % still 3 copies ?!? %--------------------------------------------- % MPI_Attr_delete just detaches kv from comm % In MPITB, attr is deleted only by overwriting % or by freeing the keyval with MPI_Keyval_free % In LAM/MPI, if you want actual deletion, you must % define a delete function when creating the keyval % recall we used "NULL" as delete function %--------------------------------------------- % no longer accessible [info C flag] = MPI_Attr_get (WORLD, kv) % flag==0, C==[] %--------------------------------------------- % That was a silly way of killing C copy % We could just have done C=[] altogether, and same result %--------------------------------------------- print_info (B) % 2 copies, C died % HOW COULD we erase attr? %------------------------------------------------------- % The copy-function stuff - problem %------------------------------------------------------- print_info(A), A % recall A was alone C = {A,'hi';'hello',A} print_info(A) % now 3 copies (used twice) info = MPI_Attr_put (WORLD, kv, C) % attr copy destroyed/reattachd print_info (B) % THIS WAY we have erased attr [info B flag] = MPI_Attr_get (WORLD, kv) % flag==1, B==C print_info (B) % 3 copies:B,C,attr(+pr_info) [info WORLD2] = MPI_Comm_dup (WORLD) % copy_fn was NULL [info B flag] = MPI_Attr_get (WORLD2,kv) % attr not dup'ed, flag==0 [info B flag] = MPI_Attr_get (WORLD, kv) % exists in WORLD, flag==1 %--------------------------------------------- % one intuitively thinks that attributes should % also be copied when communicator is duplicated % should specify a copy func when creating the keyval % recall we used "NULL" as copy function %--------------------------------------------- info = MPI_Comm_free (WORLD2) %------------------------------------------------------- % The 'static' copy 'feature' (bug?) - problem %------------------------------------------------------- % This "copy-attribute-internally-to-MPITB" could have been % implemented in other ways, such as requiring that the % attr is a variable whose symtable definition address is % stored (it works, and is funny when you change/clear the variable) %------------------------------------------------------- % Try to guess how attr could get lost in a "weird" way... % yup, you guessed, let's prove it %------------------------------------------------------- print_info (C) % 3 copies, print_info (B) % same rep whos MPI_* % where the wrapper table clear MPI_Keyval_create % is defined print_info (C) % sure, 3 copies [info B flag] = MPI_Attr_get (WORLD, kv) % ouch! indeed print_info (B) % 3 copies %-------------------------------------------- % What's wrong? how could it survive the clear? % yup, it's 3 DEFUN_DLDs in the same source % must clear them all to destroy the wrapper table % we have also used Attr_put(), but not Keyval_free() %-------------------------------------------- clear MPI_Attr_put % might be defined whos MPI_* % no wrapper tables left print_info (C) % gosh! still 3 copies! :-) MPIOCTLIST = whos MPI_* % 10 MPITB cmds left clear MPI_Attr_get MPI_Comm_dup MPI_Error_string print_info (C) % finally 2 copies!!! print_info (B) % don't have explanation handy MPIOCTLIST = whos MPI_* %------------------------------------------------------- % The 'static' copy problem - accessing attr when table cleared %------------------------------------------------------- clear B C % no copies left help MPI_Keyval_create % pathetically trying to load which MPI_Keyval_create % the table, even if it's clear [info B flag] = MPI_Attr_get (WORLD, kv) % B=oct_base_val() -> 'unknown' %-----------------------------------------------% Octave tried to display B % error: octave_base_value::print (): wrong type argument `' %------------------------------------------------ whos B, B % B seems undefined [info B flag] = MPI_Attr_get (WORLD, kv); % using ; Octave won't display whos B, B % but that doesn't fix anything %-------------------------------------------- % Ok, loading an empty table was not pathetic at all % any of "which" or "help" would do it % let's prove that, and also try with no loaded table % before taking a break %-------------------------------------------- MPIOCTLIST = whos MPI_* % 9 MPITB cmds left clear MPI_Keyval_create MPI_Attr_get % again no explanation for get help MPI_Keyval_create [info B flag] = MPI_Attr_get (WORLD, kv); % help does load it clear MPI_Keyval_create MPI_Attr_get which MPI_Keyval_create [info B flag] = MPI_Attr_get (WORLD, kv); % which does load it clear MPI_Keyval_create MPI_Attr_get [info B flag] = MPI_Attr_get (WORLD, kv); % stubborn as only we can %------------------------------------------------------- % MPI process rank 0 (n2, p2581) caught a SIGSEGV. %------------------------------------------------------- % Taking a break - _Finalize not really needed after SIGSEGV :-) %------------------------------------------------------- MPI_Finalize quit %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % Attribute Caching % MPI_DUP_FN, MPI_NULL_COPY_FN, MPI_NULL_DELETE_FN %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % Ok, again from the beginning, copy-function problem revisited %------------------------------------------------------- MPI_Init, WORLD = MPI_COMM_WORLD % really lazy :-) % eager to save typing MPI_Errhandler_set(WORLD, MPI_ERRORS_RETURN) % try to survive [info kv] = MPI_Keyval_create ('NULL', 'NULL', {}) % kv==19 or so A = [1 2;3 4] B = 'hello' C = {A,B;A,B} info = MPI_Attr_put (WORLD, kv, C) % notice we put C [info D flag] = MPI_Attr_get (WORLD, kv) % flag==1, D==C(==attr, 3 copy) print_info(D) % C, D, attr, print_info print_info(C) % C, D, attr, print_info print_info(B) % B, 2 uses to build C, p_info print_info(A) % A, 2 uses to build C, p_info [info WORLD2] = MPI_Comm_dup (WORLD) % copy_fn was NULL [info D flag] = MPI_Attr_get (WORLD2,kv) % attr not dup, flag==0 print_info(C) % C & attr [info D flag] = MPI_Attr_get (WORLD, kv) % exists in WORLD, flag==1 print_info(C) % C,D,attr %------------------------------------------------------- % starting over, this time with some copy-function %------------------------------------------------------- help MPI_Keyval_free help MPI_KEYVAL_INVALID info = MPI_Comm_free (WORLD2), WORLD2 % first free Comm, then Key WORLD2 ==MPI_COMM_NULL % WORLD2 finished info = MPI_Keyval_free (kv), kv % kv==-1, info==0, ok kv ==MPI_KEYVAL_INVALID % (we didn't Attr_delete) % kv 19 unusable forever [info D flag] = MPI_Attr_get (WORLD, kv) % invalid key (also in WORLD) [inf2 msg] = MPI_Error_string(info) % invalid key value info ==MPI_ERR_KEYVAL %------------------------------------------------------- % standard LAM/MPI copy function %------------------------------------------------------- help MPI_Keyval_create help MPI_NULL_DELETE_FN % we can't really use these ints help MPI_NULL_COPY_FN % since the args to MPI_Keyval_create were help MPI_DUP_FN % better thought to be strings - later see why MPI_NULL_DELETE_FN % MPI_Keyval_create translates the strings MPI_NULL_COPY_FN % "NULL" & "DUP" to the corresponding MPI_DUP_FN % LAM/MPI copy/del functions A = [1 2;3 4] % now we use the std [info kv] = MPI_Keyval_create ('DUP','NULL',{}) % MPI dup callback info = MPI_Attr_put (WORLD, kv, A) % kv==20 or so [info WORLD2] = MPI_Comm_dup (WORLD) % kv 19 locked above [info B flag] = MPI_Attr_get (WORLD2,kv) % & get B==A @ 2nd comm print_info(B) % 3 copies now [info C flag] = MPI_Attr_get (WORLD, kv) % have both, WORLD & W2 print_info(C) % 4 copies now clear A % the "static" attr [info B flag] = MPI_Attr_get (WORLD2, kv) % is stored with kv [info C flag] = MPI_Attr_get (WORLD , kv) % accessible in both info = MPI_Attr_delete(WORLD2, kv) % deleted from WORLD2 [info B flag] = MPI_Attr_get (WORLD2, kv) % from here (flag==0) [info C flag] = MPI_Attr_get (WORLD , kv) % not here, still can % this time, first... info = MPI_Keyval_free (kv), kv % free Key, then Comm kv ==MPI_KEYVAL_INVALID % same result [info B flag] = MPI_Attr_get ( WORLD2, kv) % here it was deleted [info C flag] = MPI_Attr_get ( WORLD , kv) % here simply freed info ==MPI_ERR_KEYVAL % both cases, INVALID info = MPI_Comm_free (WORLD2) % same result % you can _Attr_free after / before _Comm_free info = MPI_Comm_free (WORLD ) % Yep!, can't do that info ==MPI_ERR_COMM % not with WORLD! where are you living? %------------------------------------------------------- % Taking a break %------------------------------------------------------- MPI_Finalize quit %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % Attribute Caching % OCTAVE_COPY_FN, OCTAVE_DEL_FN %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % Ok, again from the beginning, 'static' copy 'feature' (bug?) revisited %------------------------------------------------------- MPI_Init, WORLD = MPI_COMM_WORLD % really lazy :-) MPI_Errhandler_set(WORLD,MPI_ERRORS_RETURN) % try to survive help OCTAVE_COPY_FN type OCTAVE_COPY_FN help OCTAVE_DEL_FN type OCTAVE_DEL_FN [info kv] = MPI_Keyval_create('OCTAVE_COPY_FN',... % using Octave skeletns 'OCTAVE_DEL_FN',{'hi'}) % kv==19 or so A = [1 2;3 4] info = MPI_Attr_put (WORLD, kv, A) [info WORLD2] = MPI_Comm_dup (WORLD) % OctaveCallback called %------------------------------------------------------- % For information purposes, this wrapper will not use but wants to print: % 0.-Current communicator MPI_COMM_WORLD: 1089287552 % 1.- The OLDcomm communicator argument: 1089287552 % 2.- The keyval integer argument: 19 % 3.- The extra_st octave_value argument: extra_st = % { % [1,1] = hi % ------------------- % } % we get this message % --------------------------- % from OCTAVE_COPY_FN % Copying attribute keyed 19: % ------------------- % 1 2 % 3 4 %------------------------------------------------------- WORLD % notice it's the same value as printed in 1.- type OCTAVE_COPY_FN [info B flag] = MPI_Attr_get (WORLD , kv) [info C flag] = MPI_Attr_get (WORLD2, kv) % get 2 copies clear A % clear original copy [info B flag] = MPI_Attr_get ( WORLD , kv) % attr stored with kv [info C flag] = MPI_Attr_get ( WORLD2, kv) % accessible from all info = MPI_Attr_delete( WORLD2, kv) % OctaveCallback called %----------------------------------------------------------------------- % For information purposes, this wrapper will not use but wants to print: % 0.-Current communicator MPI_COMM_WORLD: 1089287552 % 1.- Passed comm communicator argument: 145786840 % 2.- Passed keyval integer argument: 19 % 3.- Passed attr. octave_value argument: attr = % % ------------------- % 1 2 % we get this message % 3 4 % from OCTAVE_DEL_FN % % ------------------- % 4.- The extra_st octave_value argument: extra_st = % { % [1,1] = hi % } % ---------------------------------------------------------------------- % With the current MPITB design, only MPI_Keyval_free.oct should destroy attr % No work left for an Octave-M-file callback (see Keyval.cc source code) %----------------------------------------------------------------------- WORLD2 % notice it's the same value as printed in 1.- info = MPI_Comm_free (WORLD2) % attr de-attached, % now comm freed [info B flag] = MPI_Attr_get (WORLD, kv) % attr still there info = MPI_Attr_delete(WORLD, kv) % Callback called again % ---------------------------------------------------------------------- % With the current MPITB design, only MPI_Keyval_free.oct should destroy attr % No work left for an Octave-M-file callback (see Keyval.cc source code) %----------------------------------------------------------------------- %------------------------------ % if the 1st Attr_delete would have % destroyed the "static" copy, then % _this_ delete would have SIGSEGV %------------------------------ clear C print_info (B) % 2 copies: attr, B info = MPI_Keyval_free (kv), kv % attr finally destryd kv == MPI_KEYVAL_INVALID print_info (B) % B alone now %------------------------------------------------------- % done, quitting %------------------------------------------------------- MPI_Finalize quit %------------------------------------------------------- % you might want to repeat the previous session with the % OCTAVE_COPY_FN "user" mechanism, but we warned: you'll SIGSEGV %------------------------------------------------------- %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % More attribute Caching % predefined attributes % MPI_ TAG_UB, _HOST, _IO, _WTIME_IS_GLOBAL, _UNIVERSE_SIZE % MPI_APPNUM, _WIN_BASE, _WIN_SIZE, _WIN_DISP_UNIT % LAM_UNIVERSE_NCPUS, _NNODES %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % Try with a lamboot of say 9 computers, master ox0 and slaves ox1-ox8 % if some of them are multiprocessor, better (difference nnodes/ncpus) % just LAM/MPI required on remaining 8 computers ox1-ox8 % and Octave available on 1st slave ox1 (in addition to local Octave @ ox0) %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% (recall: we assume that:) (master node last in list $LAMBHOST) (you can ssh to the nodes (ox1-ox8) from the headed node w/out passwd/output) (lamprep/lamboot before starting octave if this is a new ssh session) (standard MPITB .octaverc w/out edited else branch, children call mergeParent) %------------------------------------------------------- putenv('LAM_MPI_SSI_rpi','tcp'), MPI_Init WORLD = MPI_COMM_WORLD %------------------------------------------------------- % Put an attribute just to check that they're not copied w/Spawn %------------------------------------------------------- A = [1 2;3 4] % MPI dup callback [info kv] = MPI_Keyval_create ('DUP','NULL',{}) % kv==19 or so info = MPI_Attr_put (WORLD, kv, A) % attr not copied... [info children errs] = MPI_Comm_spawn ( % ...to spawned comms '/usr/X11R6/bin/xterm',{'-e','octave'},... 1,MPI_INFO_NULL,0,MPI_COMM_SELF) [info NEWORLD] = MPI_Intercomm_merge (children, 0) %%%%% ON CHILD %%%%%%%%%% MPI_Errhandler_set(WORLD,MPI_ERRORS_RETURN) [info rank]=MPI_Comm_rank(NEWORLD) % rank==1 MPI_Errhandler_set( WORLD,MPI_ERRORS_RETURN) % avoid future aborts MPI_Errhandler_set(NEWORLD,MPI_ERRORS_RETURN) [info rank]=MPI_Comm_rank(NEWORLD) % rank==0 %------------------------------------------------------- % Check attr not copied %------------------------------------------------------- [info B flag] = MPI_Attr_get ( WORLD,kv) % A is in WORLD [info B flag] = MPI_Attr_get (NEWORLD,kv) % A is not in NEWORLD kv % flag==0, kv==19 %%%%% ON CHILD %%%%%%%%%% kv=19 % or whatever you got [info B flag] = MPI_Attr_get( WORLD,kv)% certainly not in WRLD info == MPI_ERR_KEYVAL % kv not even created [info B flag] = MPI_Attr_get(NEWORLD,kv)% neither in NEWRLD info == MPI_ERR_KEYVAL % doesn't work that way help environ %------------------------------------------------------- % We already used processor_name %------------------------------------------------------- help MPI_Get_processor_name [info name] = MPI_Get_processor_name help MPI_ATTRKVALS %------------------------------------------------------- % Those 18 predefined attributes are the reason that % newly created attributes are keyed from 19 on %------------------------------------------------------- % Accessing predefined attributes %------------------------------------------------------- help MPI_TAG_UB [info tub flag]=MPI_Attr_get(WORLD,MPI_TAG_UB) % tub==2147483647 %%%%% ON CHILD %%%%%%%%%% [info tub flag]=MPI_Attr_get(WORLD,MPI_TAG_UB) MPI_Send(tub,0,tub,NEWORLD) % same 2147483647 [info stat]=MPI_Recv(tub ,1,tub ,NEWORLD) tub1 =tub+1 info =MPI_Send(tub1,1,tub1,NEWORLD) % should be info==4 %nfo msg] =MPI_Error_string(info) % invalid tag %nfo ==MPI_ERR_TAG % but octave traps it [info tub flag]=MPI_Attr_get(NEWORLD,MPI_TAG_UB) % tub=={}, flag==0 % not copied @ spawn %------------------------------------------------------- help MPI_HOST [info hst flag]=MPI_Attr_get(WORLD,MPI_HOST) % hst==0, rank0 here %%%%% ON CHILD %%%%%%%%%% [info hst flag]=MPI_Attr_get(WORLD,MPI_HOST) % hst==-2 hst==MPI_PROC_NULL % not in local procssor [info hst flag]=MPI_Attr_get(NEWORLD,MPI_HOST) % hst=={} not found % intracomms - have no predef. attrs %------------------------------------------------------- help MPI_IO [info mio flag]=MPI_Attr_get(WORLD,MPI_IO) % mio==-1 mio==MPI_ANY_SOURCE %%%%% ON CHILD %%%%%%%%%% [info mio flag]=MPI_Attr_get(WORLD,MPI_IO) % mio==-2 mio==MPI_PROC_NULL [info mio flag]=MPI_Attr_get(NEWORLD,MPI_IO) % mio not found %------------------------------------------------------- help MPI_WTIME_IS_GLOBAL [info wtg flag]=MPI_Attr_get(WORLD,MPI_WTIME_IS_GLOBAL) % wtg==0, time not glob %%%%% ON CHILD %%%%%%%%%% [info wtg flag]=MPI_Attr_get(WORLD,MPI_WTIME_IS_GLOBAL) %==0 [info wtg flag]=MPI_Attr_get(NEWORLD,MPI_WTIME_IS_GLOBAL) % wtg not found %------------------------------------------------------- help MPI_UNIVERSE_SIZE [info mus flag]=MPI_Attr_get(WORLD,MPI_UNIVERSE_SIZE) % mus==9 lambooted [info mus flag]=MPI_Attr_get( WORLD,MPI_UNIVERSE_SIZE) % mus==9 [info mus flag]=MPI_Attr_get(NEWORLD,MPI_UNIVERSE_SIZE) % not found [info mus flag]=MPI_Attr_get(NEWORLD,MPI_UNIVERSE_SIZE) % mus not found %------------------------------------------------------- help MPI_APPNUM [info apn flag]=MPI_Attr_get( WORLD,MPI_APPNUM) % apn not found [info apn flag]=MPI_Attr_get(NEWORLD,MPI_APPNUM) % anywhere [info apn flag]=MPI_Attr_get( WORLD,MPI_APPNUM) % apn==0!!! found! [info apn flag]=MPI_Attr_get(NEWORLD,MPI_APPNUM) % not here % slave octave was spawn with MPI_Comm_spawn %------------------------------------------------------- help MPI_WIN_BASE [info mwb flag]=MPI_Attr_get( WORLD,MPI_WIN_BASE) % info==28 [info mwb flag]=MPI_Attr_get(NEWORLD,MPI_WIN_BASE) % info==28 [info mwb flag]=MPI_Attr_get( WORLD,MPI_WIN_BASE) % info==28 [info mwb flag]=MPI_Attr_get(NEWORLD,MPI_WIN_BASE) % info==28 [nfo msg] =MPI_Error_string(info) % invalid key value info ==MPI_ERR_KEYVAL % !?! it's documented %------------------------------------------------------- help MPI_WIN_SIZE [info mws flag]=MPI_Attr_get( WORLD,MPI_WIN_SIZE) % info==28 [info mws flag]=MPI_Attr_get(NEWORLD,MPI_WIN_SIZE) % info==28 [info mws flag]=MPI_Attr_get( WORLD,MPI_WIN_SIZE) % info==28 [info mws flag]=MPI_Attr_get(NEWORLD,MPI_WIN_SIZE) % info==28 [nfo msg] =MPI_Error_string(info) % invalid key value info ==MPI_ERR_KEYVAL % !?! it's documented %------------------------------------------------------- help MPI_WIN_DISP_UNIT [info mwu flag]=MPI_Attr_get( WORLD,MPI_WIN_DISP_UNIT) % info==28 [info mwu flag]=MPI_Attr_get(NEWORLD,MPI_WIN_DISP_UNIT) % info==28 [info mwu flag]=MPI_Attr_get( WORLD,MPI_WIN_DISP_UNIT) % info==28 [info mwu flag]=MPI_Attr_get(NEWORLD,MPI_WIN_DISP_UNIT) % info==28 [nfo msg] =MPI_Error_string(info) % invalid key value info ==MPI_ERR_KEYVAL % !?! it's documented %------------------------------------------------------- help LAM_UNIVERSE_NCPUS [info lnc flag]=MPI_Attr_get( WORLD,LAM_UNIVERSE_NCPUS) % lnc==9 [info lnc flag]=MPI_Attr_get(NEWORLD,LAM_UNIVERSE_NCPUS) % not found, flag==0 %%%%% ON CHILD %%%%%%%%%% [info lnc flag]=MPI_Attr_get( WORLD,LAM_UNIVERSE_NCPUS) % lnc==9 [info lnc flag]=MPI_Attr_get(NEWORLD,LAM_UNIVERSE_NCPUS) % not found %------------------------------------------------------- help LAM_UNIVERSE_NNODES [info lnn flag]=MPI_Attr_get( WORLD,LAM_UNIVERSE_NNODES) % lnn==9 [info lnn flag]=MPI_Attr_get(NEWORLD,LAM_UNIVERSE_NNODES) % flag==0 %%%%% ON CHILD %%%%%%%%%% [info lnn flag]=MPI_Attr_get( WORLD,LAM_UNIVERSE_NNODES) % lnn==9 [info lnn flag]=MPI_Attr_get(NEWORLD,LAM_UNIVERSE_NNODES) % flag==0 %------------------------------------------------------- % done %------------------------------------------------------- MPI_Finalize quit %%%%% ON CHILD %%%%%%%%%% ouch! forgot about child! MPI_Finalize quit %------------------------------------------------------- % alternatively, on xterm %------------------------------------------------------- lamclean # child dies %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % Topologies % MPI_[Dims/Cart/Graph]_create, Topo_test, [Graphdims/Cartdim]_get % MPI_[Graph/Cart]_get, Cart_[rank/coords], Graph_neighbors[_count] % MPI_Cart_[shift/sub], MPI_[Cart/Graph]_map %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % requires 9-machines-wide LAMBOOT :-) your computer last in $LAMBHOST % ok, it may work with less hosts, if you don't have Octave everywhere % anyways, for real work, it's not a big deal 2 Octaves on same host % (even on a biprocessor ;-) %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% (recall: we assume that:) (master node last in list $LAMBHOST) (you can ssh to the nodes (ox1-ox8) from the headed node w/out passwd/output) (lamprep/lamboot before starting octave if this is a new ssh session) (standard MPITB .octaverc w/out edited else branch, children call mergeParent) %------------------------------------------------------- putenv('LAM_MPI_SSI_rpi','tcp'), MPI_Init [info children errs] = MPI_Comm_spawn ( '/usr/X11R6/bin/xterm',{'-e','octave'},... 8,MPI_INFO_NULL,0,MPI_COMM_SELF) [info NEWORLD] = MPI_Intercomm_merge (children, 0) WORLD=MPI_COMM_WORLD MPI_Errhandler_set( WORLD,MPI_ERRORS_RETURN) % avoid future aborts MPI_Errhandler_set(NEWORLD,MPI_ERRORS_RETURN) [info Nsiz]=MPI_Comm_size(NEWORLD) % Nsiz==9 [info Nrnk]=MPI_Comm_rank(NEWORLD) % Nrnk==0 hnam=system('hostname') %%%%% ON CHILDREN %%%%%%% [info eh]=MPI_Errhandler_get(NEWORLD) eh==MPI_ERRORS_RETURN % already set @ master MPI_Errhandler_set( WORLD,MPI_ERRORS_RETURN) [info Nrnk]=MPI_Comm_rank(NEWORLD) % Nrnk==i hnam=system('hostname') % ranks $LAMBHOST order %------------------------------------------------------- % CAVEAT: take your time to order your xterm/Octave windows based on rank % (that's why we asked for it right now) % or you'll end up in a mess and you won't understand anything at all % Yet better if you "cascade" them such that you can copy-paste easily % In this session, you will paste the same text to every child Octave % You have been warned :-) %------------------------------------------------------- help topo %------------------------------------------------------- % Topologies: computing dims (not so easy if you have 1,354,297 computers) %------------------------------------------------------- help MPI_Dims_create ndims = 2 % for grid? [info dims] = MPI_Dims_create (Nsiz, ndims) % (3x3) ndims = 3 % for hypercube? [info dims] = MPI_Dims_create (Nsiz, ndims) % (3x3x1), because 9=3x3 [info dims] = MPI_Dims_create (8 , ndims) % (2x2x2), for 8 in 3-D grid [info dims] = MPI_Dims_create (8 , 2) % (4x2), for 8 in 2-D %------------------------------------------------------- % Topologies: creating cartesian dims (low level call, DO NOT USE) %------------------------------------------------------- % <-> 0 <-> 1 <-> 2 <-> (3x3) grid % ^ ^ ^ periodic (toroidal) % | | | on 2nd dim (columns) % v v v (less significant dim in C-style) % <-> 3 <-> 4 <-> 5 <-> % ^ ^ ^ % | | | % v v v % <-> 6 <-> 7 <-> 8 <-> %------------------------------------------------------- help MPI_Cart_map dims = [3 3] % (3x3) grid periods = [0 1] % periodic in 2nd dim [info rank]=MPI_Cart_map(NEWORLD,dims,periods) % this would be rank 0 Nrnk==rank % not too difficult :-) %%%%% ON CHILDREN %%%%%%% [info rank]=MPI_Cart_map(NEWORLD,[3 3],[0 1]) Nrnk==rank % not too difficult :-) %------------------------------------------------------- % Topologies: creating graph dims (low level call, DO NOT USE) %------------------------------------------------------- % ---- 3 <-> 4 - This time we want a ring % / \ like this one with 8 ranks % 0 <-> 1 <-> 2 5 and rank 8 out again % \ / We indicate neighbors and % ---- 7 <-> 6 - cummulative neighbor count %------------------------------------------------------- % nodes 0 1 2 3 4 5 6 7 8 % neighbs 1 0-2 1-3-7 2-4 3-5 4-6 5-7 6-2 - % num-ngb 1 2 3 2 2 2 2 2 0 % cummul 1 3 6 8 10 12 14 16 16 % index = [1 3 6 8 10 12 14 16 ] % edges = [1 0 2 1 3 7 2 4 3 5 4 6 5 7 6 2 ] %------------------------------------------------------- help MPI_Graph_map index = [1 3 6 8 10 12 14 16 ] % see above edges = [1 0 2 1 3 7 2 4 3 5 4 6 5 7 6 2] % rnk8 makes graph unconnected [info grnk]=MPI_Graph_map(NEWORLD,index,edges) % grnk==0 Nrnk==grnk % again, not too complex %%%%% ON CHILDREN %%%%%%% index = [1 3 6 8 10 12 14 16 0] edges = [1 0 2 1 3 7 2 4 3 5 4 6 5 7 6 2] [info grnk]=MPI_Graph_map(NEWORLD,index,edges) Nrnk==grnk % not too complex either :-) % Subtle difference between undefined and unconnected %%%%% ON CHILD #8 %%%%%%% index = [1 3 6 8 10 12 14 16 ] % missing 0 end [info grnk]=MPI_Graph_map(NEWORLD,index,edges) grnk==[Nrnk MPI_UNDEFINED] % [0 1] @ rank8 %------------------------------------------------------- % Topologies: creating cartesian communicator %------------------------------------------------------- % <-> 0 <-> 1 <-> 2 <-> (3x3) grid % ^ ^ ^ periodic (toroidal) % | | | on 2nd dim (less significant C-style) % v v v % <-> 3 <-> 4 <-> 5 <-> % ^ ^ ^ % | | | % v v v % <-> 6 <-> 7 <-> 8 <-> %------------------------------------------------------- help MPI_Cart_create help MPI_Topo_test % reorder flag ignored [info GRID]=MPI_Cart_create(NEWORLD,[3 3],[0 1],1) % collective call, [info topo]=MPI_Topo_test (GRID) % got blocked topo==MPI_CART % Topo_test not collect %%%%% ON CHILDREN %%%%%%% [info GRID]=MPI_Cart_create(NEWORLD,[3 3],[0 1],1) [info Gsiz]=MPI_Comm_size (GRID) % Gsiz==9 [info Grnk]=MPI_Comm_rank (GRID) Nrnk==Grnk % easy to understand [info topo] = MPI_Topo_test(NEWORLD) topo == MPI_UNDEFINED [info Gsiz] = MPI_Comm_size(GRID) [info Grnk] = MPI_Comm_rank(GRID) Nrnk==Grnk %%%%% ON ANY CHILD %%%%%%% Just 1 of them, just to check [info eh]=MPI_Errhandler_get(GRID) % inherited from eh==MPI_ERRORS_RETURN % NEWORLD? [info eh]=MPI_Errhandler_get(GRID) eh==MPI_ERRORS_RETURN %info =MPI_Errhandler_set(GRID,MPI_ERRORS_RETURN) % not required then %------------------------------------------------------- % % |/ |/ % --4 ----- 5 -- Yup, I know you can't see anything % |/| |/| Well, imagine it's a 3-D (not-so-hyper-)cube % -- 0 +---- 1 +- 4 -- 5 % /|-6 ---/+ 7 -- 0 -- 1 | % |/| |/| | 6 -|-7 % -- 2 ----- 3 -- 2 -- 3 % / / and it's periodic (cyclic, toroidal) on each dim % %------------------------------------------------------- [info CUBE]=MPI_Cart_create(NEWORLD,[2 2 2],[1 0 1],1) % rank 8 is out! [info topo]=MPI_Topo_test (CUBE) topo==MPI_CART %%%%% ON CHILDREN %%%%%%% [info CUBE]=MPI_Cart_create(NEWORLD,[2 2 2],[1 0 1],1) [info Csiz]=MPI_Comm_size (CUBE) [info Crnk]=MPI_Comm_rank (CUBE) % easy to understand Nrnk==Crnk % false @ rank8 % no problem for pasting previous text also to child 8, % you only get invalid comm & errcodes % in fact, you must _Cart_create to unblock NEWORLD %%%%% ON CHILD #8 %%%%%%% CUBE==MPI_COMM_NULL % rank 8 out, CUBE=NULL info ==MPI_ERR_COMM % from MPI_Rank [info topo]=MPI_Topo_test (CUBE) % info==5 info ==MPI_ERR_COMM % can't ask topo(NULL) [info Csiz] = MPI_Comm_size(CUBE) [info Crnk] = MPI_Comm_rank(CUBE) Nrnk==Crnk % obvious %%%%% ON ANY CHILD %%%%%%% Just 1 of them, just to check [info eh]=MPI_Errhandler_get(CUBE) % inherited from eh==MPI_ERRORS_RETURN % NEWORLD? [info eh]=MPI_Errhandler_get(CUBE) eh==MPI_ERRORS_RETURN %info =MPI_Errhandler_set(CUBE,MPI_ERRORS_RETURN) % not required then %------------------------------------------------------- % Topologies: creating graph communicator %------------------------------------------------------- % ---- 3 <-> 4 - The same previous ring % / \ % 0 <-> 1 <-> 2 5 % \ / % ---- 7 <-> 6 - %------------------------------------------------------- % nodes 0 1 2 3 4 5 6 7 8 % neighbs 1 0-2 1-3-7 2-4 3-5 4-6 5-7 6-2 - % num-ngb 1 2 3 2 2 2 2 2 0 % cummul 1 3 6 8 10 12 14 16 16 %------------------------------------------------------- help MPI_Graph_create index = [1 3 6 8 10 12 14 16 ] % see above edges = [1 0 2 1 3 7 2 4 3 5 4 6 5 7 6 2] % rnk8 makes graph unconnected [info RING] = MPI_Graph_create (NEWORLD,index,edges,1) % reorder flag ignored [info topo] = MPI_Topo_test (RING) topo == MPI_GRAPH %%%%% ON CHILD #8 %%%%%%% index = [1 3 6 8 10 12 14 16 ] edges = [1 0 2 1 3 7 2 4 3 5 4 6 5 7 6 2] [info RING]=MPI_Graph_create(NEWORLD,index,edges,1) RING==MPI_COMM_NULL % true @ rank8 [info topo]=MPI_Topo_test (RING) % rank8 out info ==MPI_ERR_COMM %%%%% ON CHILDREN %%%%%%% index = [1 3 6 8 10 12 14 16 ] edges = [1 0 2 1 3 7 2 4 3 5 4 6 5 7 6 2] [info RING]=MPI_Graph_create (NEWORLD,index,edges,1) [info Rsiz]=MPI_Comm_size(RING) % it says 8 [info Rrnk]=MPI_Comm_rank(RING) Nrnk==Rrnk % speechless [info Rsiz] = MPI_Comm_size(RING) % 8 didn't enter graph [info Rrnk] = MPI_Comm_rank(RING) Nrnk==Rrnk % no comment %%%%% ON ANY CHILD %%%%%%% Just 1 of them, just to check [info eh]=MPI_Errhandler_get(RING) eh==MPI_ERRORS_RETURN [info eh]=MPI_Errhandler_get(RING) % protected against eh==MPI_ERRORS_RETURN % aborts %------------------------------------------------------- % Topologies: retrieving graph connectivity (topology) and neighbors %------------------------------------------------------- help MPI_Graphdims_get help MPI_Graph_get [info nnodes nedges] = MPI_Graphdims_get (CUBE) % nope! not a graph! [info2 string] = MPI_Error_string (info) % invalid topology info == MPI_ERR_TOPOLOGY [info nnodes nedges] = MPI_Graphdims_get (RING) % 8 nodes, 16 edges %%%%% ON ANY CHILD %%%%%%% Just 1 of them, just to check [info nnodes nedges] = MPI_Graphdims_get (RING) clear index edges [info index edges] = MPI_Graph_get (RING) nnodes==length(index) nedges==length(edges) %%%%% ON ANY CHILD %%%%%%% Just 1 of them, just to check index, edges [info index2 edges2] = MPI_Graph_get (RING) all(index==index2) all(edges==edges2) [info nneighs]= MPI_Graph_neighbors_count (RING, Rrnk) % rank in RING (same) [info neighs] = MPI_Graph_neighbors (RING, Rrnk) % can ask for other rnk %------------------------------------------------------- % ---- 3 <-> 4 - Recall rank 0 is neighbor to 1 % / \ % 0 <-> 1 <-> 2 5 % \ / % ---- 7 <-> 6 - rank 2 neighbor to 1,3,7 %------------------------------------------------------- %%%%% ON ANY CHILD %%%%%%% Just 1 of them, just to check [info nneighs]= MPI_Graph_neighbors_count (RING, 2) [info neighs] = MPI_Graph_neighbors (RING, 2) %------------------------------------------------------- % Topologies: retrieving cartesian connectivity %------------------------------------------------------- help MPI_Cartdim_get help MPI_Cart_get [info ndims] = MPI_Cartdim_get (RING) % nope! it's not cartesian [inf2 strng] = MPI_Error_string(info) % invalid topology info == MPI_ERR_TOPOLOGY [info ndims] = MPI_Cartdim_get (CUBE) clear dims periods [info dims periods coords]=MPI_Cart_get(CUBE) % each one asks for theirs %%%%% ON EVERY CHILD %%%%%%% Trying to understand dim signific [info dims periods coords]=MPI_Cart_get(CUBE) %------------------------------------------------------- % Particularly, for this simple case (3-D cube) % the 3 dims are of length 2 and coords==rank in binary % ie: rank 0 is at (0 0 0), rank 1 at (0 0 1), 2 @ (0 1 0) %------------------------------------------------------- %------------------------------------------------------- % % |/ |/ % --4 ----- 5 -- In these drawings, less significant dimension is % |/| |/| horizontal (X), next is vertical (Y), Z is depth % -- 0 +---- 1 +- 4 -- 5 % /|-6 ---/+ 7 -- 0 -- 1 | % |/| |/| | 6 -|-7 % -- 2 ----- 3 -- 2 -- 3 % / / % %------------------------------------------------------- % Topologies: rank <-> coords translations %------------------------------------------------------- help MPI_Cart_rank help MPI_Cart_coords [info rank ]=MPI_Cart_rank (CUBE, coords) all (rank == [Crnk, Nrnk, Grnk] ) [info coords]=MPI_Cart_coords(CUBE, 0) % can ask for other's %%%%% ON ANY CHILD %%%%%%% Just 1 of them, just to check [info coords]=MPI_Cart_coords(CUBE, 0 ) [info rank] = MPI_Cart_rank (CUBE, coords) [info coords]=MPI_Cart_coords(CUBE, 1 ) [info rank] = MPI_Cart_rank (CUBE, coords) [info coords]=MPI_Cart_coords(CUBE, 2 ) [info rank] = MPI_Cart_rank (CUBE, coords) [info coords]=MPI_Cart_coords(CUBE, 4 ) [info rank] = MPI_Cart_rank (CUBE, coords) %------------------------------------------------------- % Topologies: retrieving cartesian routing %------------------------------------------------------- help MPI_Cart_shift % routing operations disp=1, horz=2, vert=1, dpth=0 % 1 hop along each dim Horz/Vert/Depth [info Hsrc Hdst] = MPI_Cart_shift (CUBE, horz, disp) % LSB routing [info Vsrc Vdst] = MPI_Cart_shift (CUBE, vert, disp) % middle-bit [info Dsrc Ddst] = MPI_Cart_shift (CUBE, dpth, disp) % MSB routing %------------------------------------------------------- % 4 -- 5 Who is routed from/to node 0? % 0 -- 1 | Horizontal: from left (none) / to right (1) % | 6 -|-7 Vertical: from up (none) / to down (2) % 2 -- 3 Depth: from fore (none) / to back (4) % but recall X-Z are periodic, so 1,4 again %------------------------------------------------------- periods % 2nd dim (vert) not periodic all ( [Hsrc Vsrc Dsrc] == [Hdst MPI_PROC_NULL Ddst] ) %%%%% ON ANY CHILD %%%%%%% 7 is a nice dual of 0, all bits set disp=1, horz=2, vert=1, dpth=0 [info Hsrc Hdst]=MPI_Cart_shift(CUBE,horz,disp) % LSB routing [info Vsrc Vdst]=MPI_Cart_shift(CUBE,vert,disp) % middle-bit [info Dsrc Ddst]=MPI_Cart_shift(CUBE,dpth,disp) % MSB routing all ( [Hsrc MPI_PROC_NULL Dsrc] == [Hdst Vdst Ddst] ) MPI_PROC_NULL % when no neighbor, PROC_NULL help MPI_PROC_NULL help MPI_Cart_shift % see the MPI_PROC_NULL remark %------------------------------------------------------- % Topologies: sub-dividing cartesian grid in cartesian sub-grids %------------------------------------------------------- help MPI_Cart_sub [info NEW] = MPI_Cart_sub (RING, [0 1]) % nope! not cartesian info == MPI_ERR_TOPOLOGY % we'll drop depth (most signf) [info LMS] = MPI_Cart_sub (CUBE, [0 1 1]) % two halves: MSHalf, LSHalf %%%%% ON CHILD 8 %%%%%%%%%%%%%%%%% CUBE == MPI_COMM_NULL %%%%% ON EVERY CHILD but 8 %%%%%%% % Blocks only in CUBE, rank 8 not needed / not valid %------------------------------------------------------- [info LMS] = MPI_Cart_sub (CUBE, [0 1 1]) %------------------------------------------------------- % 4 -- 5 From depth point of view, there is a % 0 -- 1 | less significant half 0,1,2,3, and a % | 6 -|-7 most significant half 4,5,6,7 % 2 -- 3 (the MSHalf has the least depth coordinate possible) % In more general cases, several communicators ("slices") are produced % with ranks sharing the same coordinate in the dropped dimension(s) %------------------------------------------------------- [info Lsiz]= MPI_Comm_size(LMS) % 2 halves sized (2x2) [info Lrnk]= MPI_Comm_rank(LMS) % of course we're on LSHalf %%%%% ON EVERY CHILD but 8 %%%%%%% [info Lsiz]= MPI_Comm_size(LMS) % Lsiz==4 [info Lrnk]= MPI_Comm_rank(LMS) % first 4 are 0,1,2,3 and again [info gLMS] = MPI_Comm_group (LMS) [info gsiz] = MPI_Group_size (gLMS) [info gCUB] = MPI_Comm_group (CUBE) [info gsiz] = MPI_Group_size (gCUB) [info ranks]= MPI_Group_translate_ranks(gCUB, [0:gsiz-1], gLMS) all (ranks==[0 1 2 3 MPI_UNDEFINED MPI_UNDEFINED MPI_UNDEFINED MPI_UNDEFINED]) %%%%% ON SOME CHILD of the other half %%%%%%% say, 7 [info gLMS] = MPI_Comm_group (LMS) [info gsiz] = MPI_Group_size (gLMS) [info gCUB] = MPI_Comm_group (CUBE) [info gsiz] = MPI_Group_size (gCUB) [info ranks]= MPI_Group_translate_ranks(gCUB,[0:gsiz-1],gLMS) all (ranks==[MPI_UNDEFINED*ones(1,4) 0 1 2 3]) %------------------------------------------------------- % Recall the 2-D GRID we built at the beginning % % 0 - 1 - 2 We can cut it in 3 slices % | | | along the... say... most significant direction % 3 - 4 - 5 (vertical), so that it goes 0,1,2, then 3,4,5, % | | | and finally 6,7,8 (in three different comms) % 6 - 7 - 8 % %------------------------------------------------------- [info ROW] = MPI_Cart_sub (GRID, [0 1]) % row-sliced, throwing row dim [info size]= MPI_Comm_size(ROW) % 3 rows sized (1x3) %%%%% ON EVERY CHILD %%%%%%% rank8 included, see the graph [info ROW] = MPI_Cart_sub (GRID, [0 1]) [info size]= MPI_Comm_size(ROW) [info gROW] = MPI_Comm_group (ROW) [info gsiz] = MPI_Group_size (gROW) [info gRID] = MPI_Comm_group (GRID) [info gsiz] = MPI_Group_size (gRID) [info ranks]= MPI_Group_translate_ranks(gRID, [0:gsiz-1], gROW) all (ranks==[0 1 2 MPI_UNDEFINED*ones(1,6)]) %%%%% ON EVERY CHILD %%%%%%% [info gROW] = MPI_Comm_group (ROW) [info gsiz] = MPI_Group_size (gROW) [info gRID] = MPI_Comm_group (GRID) [info gsiz] = MPI_Group_size (gRID) [info ranks]= MPI_Group_translate_ranks(gRID,[0:gsiz-1],gROW) R1=all(ranks==[0 1 2 MPI_UNDEFINED*ones(1,6)]); % true @ 0 1 2 R2=all(ranks==[MPI_UNDEFINED*ones(1,3) 0 1 2,... MPI_UNDEFINED*ones(1,3)]); % true @ 3 4 5 R3=all(ranks==[MPI_UNDEFINED*ones(1,6) 0 1 2]); % true @ 6 7 8 [R1 R2 R3]' %------------------------------------------------------- % way too tired to try to clean %------------------------------------------------------- %%% MPI_Finalize %%% quit %%% %%%%% ON EVERY CHILD %%%%%%% %%% MPI_Finalize %%% quit quit lamhalt # in the shell. This was faster ;-) ------------------------------------------------------------------------------- ******************************************************************************* ------------------------------------------------------------------------------- help mpi % You're a power user now !!! 6Klines of tutorial !!! help missing % well, MPI std goes on and on and... %%%%%%%%% % TO-DO % %%%%%%%%% Add MPE_Buffer_reattach so that MPI_Buffer_detach info is not spoiled Add MPE_Attr_put/get to store/retrieve integer attributes ?!? Change MPI_DUP/NULL_COPY/DELETE_FN to char[4] so that they can be args to MPI_Keyval_create() ?!? Re-design Makefile system %%%%%%%%%%%%%%%%%%% % WON'T IMPLEMENT % datatypes %%%%%%%%%%%%%%%%%%% REASON: Octave "end"-users have neither 1) info on how does Octave store variables, nor 2) way of controlling data layout (they're objects) 1) should study ov.h etc (all "end"-users understand MPI_Pack.cc/hBuff.h code?) 2) no constructions in Octave languaje to impose data layout in a way compatible with MPI_Type_commit (that I can think of) For MPI layout-compatible types (double/char/bool - scalars/arrays, basically) some MPI type can be used (MPI_DOUBLE, MPI_CHAR, MPI_CXX_BOOL). For the remaining (cells and structs, basically), MPI_Pack/Unpack must be used. These routines use MPI_PACKED datatype and Octave API calls (object.rows(), .type_id(), .type_name(), etc) to analyze (at source) and reconstruct (at destination) the Octave var. Need or usefulness of datatypes is thus obviated