LWP : Bull MPI Lightweight Profiler

Bull MPI Lightweight Profiler, anciennement MPIprof, est un outil de profilage léger, simple à utiliser, et qui ne nécessite aucune modification de code : pas de recompilation ou d’édition de lien.

La documentation sur Olympe :

/opt/tools/profilers/bull-lwp/1.1.0/share/doc/bull-lwp/Lightweight_Profiler_1.1_User_Guide_86A224GC01.pdf

Pour utiliser cet outil, il faut charger en début du script SLURM les modules correspondant à la bibliothèque MPI (Openmpi vs. intelmpi) ainsi qu’au compilateur (gcc vs. icc) que vous utilisez. Il faut ensuite utiliser la commande lwp devant votre lanceur MPI (srun, mpirun, ...):

module purge
module load intel/18.2
module load intelmpi/18.2
module load profilers/bull-lwp/1.1.0
module load profilers/bull-lwp-mpi-intelmpi2017-icc17.0.5/1.6.4
...
lwp srun .....

ou

module purge
module load intel/18.2
module load openmpi/icc/4.0.3.3
module load profilers/bull-lwp/1.1.0
module load profilers/bull-lwp-mpi-openmpi4.0.2-icc17.0.5/1.7.0
...
lwp srun ....

Le profileur est activé et écrira des informations sur la sortie standard:

===============================================================================================================================================================================
                                                             [[ BULL MPI Lightweight profiler - LWP-MPI - 1.6.4 ]]
===============================================================================================================================================================================

* Communication Summary:

Number of MPI ranks                              : 16
Number of MPI call (without accumulation)        : 400
Number of non-blocking point-to-point sync event : 0
Communication by computation ratio               : 2.69%

** General summary:
Description:                 Elapsed time:                                   
                             Min:         Max:         Sum:         Avg:        
Total MPI environment        21.8945      21.9118      350.491      21.9057     
MPI functions                7.9e-08      0.589946     12.1291      0.0303228   
MPI component overhead       0.000260872  0.00032443   0.0047332    0.000295825

** Call sites summary:
Name:                                   Calls:    Elapsed time:                                   
                                                  Min:          Max:          Sum:          Avg:          
 \_ 0x4011d9                            
  \_ __libc_start_main                  
   \_ 0x40332e                          
    \_ 0x4012f9                         
     \_ MPI_INIT                        16        0.030501      0.0329592     0.523207      0.0327004     

 \_ 0x4011d9                            
  \_ __libc_start_main                  
   \_ 0x40332e                          
    \_ 0x401316                         
     \_ mpi_comm_rank_                  
      \_ MPI_COMM_RANK                  16        1.12e-07      2.71e-07      2.835e-06     1.77188e-07   

 \_ 0x4011d9                            
  \_ __libc_start_main                  
   \_ 0x40332e                          
    \_ 0x401316                         16        1.7548e-05    3.0984e-05    0.000344901   2.15563e-05   

 \_ 0x4011d9                            
  \_ __libc_start_main                  
   \_ 0x40332e                          
    \_ 0x401333                         
     \_ mpi_comm_size_                  
      \_ PMPI_COMM_SIZE                 16        7.9e-08       1.66e-07      1.813e-06     1.13313e-07   

 \_ 0x4011d9                            
  \_ __libc_start_main                  
   \_ 0x40332e                          
    \_ 0x401333                         16        7.599e-06     1.0091e-05    0.00013322    8.32625e-06   

 \_ 0x4011d9                            
  \_ __libc_start_main                  
   \_ 0x40332e                          
    \_ 0x401456                         
     \_ mpi_bcast_                      
      \_ pmpi_bcast                     16        1.2113e-05    0.000478927   0.00575821    0.000359888   

 \_ 0x4011d9                            
  \_ __libc_start_main                  
   \_ 0x40332e                          
    \_ 0x401456                         16        3.0907e-05    0.000500045   0.00630974    0.000394359   

 \_ 0x4011d9                            
  \_ __libc_start_main                  
   \_ 0x40332e                          
    \_ 0x4020ef                         
     \_ mpi_cart_create_                
      \_ MPI_CART_CREATE                16        0.000109763   0.589917      5.08111       0.317569      

 \_ 0x4011d9                            
  \_ __libc_start_main                  
   \_ 0x40332e                          
    \_ 0x4020ef                         16        0.000137395   0.589946      5.08154       0.317597      

 \_ 0x4011d9                            
  \_ __libc_start_main                  
   \_ 0x40332e                          
    \_ 0x40210a                         
     \_ 0x404a22                        
      \_ mpi_cart_shift_                
       \_ mpi_cart_shift__              16        6.831e-06     2.1736e-05    0.00018886    1.18038e-05   

 \_ 0x4011d9                            
  \_ __libc_start_main                  
   \_ 0x40332e                          
    \_ 0x40210a                         
     \_ 0x404a22                        16        2.6568e-05    6.4004e-05    0.000584707   3.65442e-05   

 \_ 0x4011d9                            
  \_ __libc_start_main                  
   \_ 0x40332e                          
    \_ 0x4023ff                         
     \_ 0x4038d5                        
      \_ mpi_sendrecv_                  
       \_ pmpi_sendrecv                 16        0.000164156   0.00560177    0.034381      0.00214881    

 \_ 0x4011d9                            
  \_ __libc_start_main                  
   \_ 0x40332e                          
    \_ 0x4023ff                         
     \_ 0x4038d5                        16        0.000190158   0.00563428    0.0348442     0.00217776    

 \_ 0x4011d9                            
  \_ __libc_start_main                  
   \_ 0x40332e                          
    \_ 0x4023ff                         
     \_ 0x403996                        
      \_ mpi_sendrecv_                  
       \_ pmpi_sendrecv                 16        0.000138677   0.00431328    0.0112854     0.000705337   

 \_ 0x4011d9                            
  \_ __libc_start_main                  
   \_ 0x40332e                          
    \_ 0x4023ff                         
     \_ 0x403996                        16        0.000144962   0.00431894    0.0113892     0.000711823   

 \_ 0x4011d9                            
  \_ __libc_start_main                  
   \_ 0x40332e                          
    \_ 0x4024f5                         
     \_ mpi_allreduce_                  
      \_ mpi_allreduce_                 16        2.2683e-05    0.0536125     0.665409      0.0415881     

 \_ 0x4011d9                            
  \_ __libc_start_main                  
   \_ 0x40332e                          
    \_ 0x4024f5                         16        7.8103e-05    0.0536745     0.666324      0.0416452     

 \_ 0x4011d9                            
  \_ __libc_start_main                  
   \_ 0x40332e                          
    \_ 0x402614                         
     \_ mpi_barrier_                    
      \_ pmpi_barrier                   16        2.0421e-05    9.6716e-05    0.000973246   6.08279e-05   

 \_ 0x4011d9                            
  \_ __libc_start_main                  
   \_ 0x40332e                          
    \_ 0x402614                         16        2.675e-05     0.000103755   0.00108211    6.76316e-05   

 \_ 0x4011d9                            
  \_ __libc_start_main                  
   \_ 0x40332e                          
    \_ 0x40264e                         
     \_ mpi_allreduce_                  
      \_ mpi_allreduce_                 16        1.1876e-05    5.8053e-05    0.000625332   3.90833e-05   

 \_ 0x4011d9                            
  \_ __libc_start_main                  
   \_ 0x40332e                          
    \_ 0x40264e                         16        1.9272e-05    6.5853e-05    0.000743544   4.64715e-05   

 \_ 0x4011d9                            
  \_ __libc_start_main                  
   \_ 0x40332e                          
    \_ 0x403105                         
     \_ mpi_barrier_                    
      \_ pmpi_barrier                   16        3.409e-06     6.846e-05     0.00100789    6.29932e-05   

 \_ 0x4011d9                            
  \_ __libc_start_main                  
   \_ 0x40332e                          
    \_ 0x403105                         16        7.987e-06     7.3146e-05    0.00107914    6.7446e-05    

 \_ 0x4011d9                            
  \_ __libc_start_main                  
   \_ 0x40332e                          
    \_ 0x403219                         
     \_ mpi_comm_free_                  
      \_ MPI_COMM_FREE                  16        1.445e-05     2.1661e-05    0.000264967   1.65604e-05   

 \_ 0x4011d9                            
  \_ __libc_start_main                  
   \_ 0x40332e                          
    \_ 0x403219                         16        2.8123e-05    4.4795e-05    0.000537943   3.36214e-05   

** MPI events summary:Elapsed time      
Name:                                   Calls:              Min:          Max:          Sum:          Avg:          
MPI_Init                                16                  0.030501      0.0329592     0.523207      0.0327004     
MPI_Comm_rank                           16                  1.12e-07      2.71e-07      2.835e-06     1.77188e-07   
mpi_comm_rank_                          16                  1.7548e-05    3.0984e-05    0.000344901   2.15563e-05   
MPI_Comm_size                           16                  7.9e-08       1.66e-07      1.813e-06     1.13313e-07   
mpi_comm_size_                          16                  7.599e-06     1.0091e-05    0.00013322    8.32625e-06   
MPI_Bcast                               16                  1.2113e-05    0.000478927   0.00575821    0.000359888   
mpi_bcast_                              16                  3.0907e-05    0.000500045   0.00630974    0.000394359   
MPI_Cart_create                         16                  0.000109763   0.589917      5.08111       0.317569      
mpi_cart_create_                        16                  0.000137395   0.589946      5.08154       0.317597      
MPI_Cart_shift                          16                  6.831e-06     2.1736e-05    0.00018886    1.18038e-05   
mpi_cart_shift_                         16                  2.6568e-05    6.4004e-05    0.000584707   3.65442e-05   
MPI_Sendrecv                            32                  0.000138677   0.00560177    0.0456664     0.00142707    
mpi_sendrecv_                           32                  0.000144962   0.00563428    0.0462333     0.00144479    
MPI_Allreduce                           32                  1.1876e-05    0.0536125     0.666034      0.0208136     
mpi_allreduce_                          32                  1.9272e-05    0.0536745     0.667067      0.0208458     
MPI_Barrier                             32                  3.409e-06     9.6716e-05    0.00198114    6.19105e-05   
mpi_barrier_                            32                  7.987e-06     0.000103755   0.00216124    6.75388e-05   
MPI_Comm_free                           16                  1.445e-05     2.1661e-05    0.000264967   1.65604e-05   
mpi_comm_free_                          16                  2.8123e-05    4.4795e-05    0.000537943   3.36214e-05   

* Communication Details:

Call site:
 \_ 0x4011d9                            
  \_ __libc_start_main                  
   \_ 0x40332e                          
    \_ 0x4012f9                         
     \_ MPI_INIT                        
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Type: [GENERIC]                             Elapsed time inside MPI function                
id:     name                hits:   rank:   min:        max:        sum:        
1       MPI_Init            1       0       0.0329592   0.0329592   0.0329592    
26      MPI_Init            1       1       0.0329414   0.0329414   0.0329414    
51      MPI_Init            1       2       0.032956    0.032956    0.032956     
76      MPI_Init            1       3       0.0329445   0.0329445   0.0329445    
101     MPI_Init            1       4       0.0329478   0.0329478   0.0329478    
126     MPI_Init            1       5       0.030501    0.030501    0.030501     
151     MPI_Init            1       6       0.0329547   0.0329547   0.0329547    
176     MPI_Init            1       7       0.032955    0.032955    0.032955     
201     MPI_Init            1       8       0.0327536   0.0327536   0.0327536    
226     MPI_Init            1       9       0.0327487   0.0327487   0.0327487    
251     MPI_Init            1       10      0.0327567   0.0327567   0.0327567    
276     MPI_Init            1       11      0.032761    0.032761    0.032761     
301     MPI_Init            1       12      0.0327581   0.0327581   0.0327581    
326     MPI_Init            1       13      0.0327605   0.0327605   0.0327605    
351     MPI_Init            1       14      0.0327477   0.0327477   0.0327477    
376     MPI_Init            1       15      0.0327608   0.0327608   0.0327608    

Call site:
 \_ 0x4011d9                            
  \_ __libc_start_main                  
   \_ 0x40332e                          
    \_ 0x401316                         
     \_ mpi_comm_rank_                  
      \_ MPI_COMM_RANK                  
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Type: [GENERIC]                             Elapsed time inside MPI function                
id:     name                hits:   rank:   min:        max:        sum:        
2       MPI_Comm_rank       1       0       1.76e-07    1.76e-07    1.76e-07     
27      MPI_Comm_rank       1       1       1.46e-07    1.46e-07    1.46e-07     
52      MPI_Comm_rank       1       2       1.69e-07    1.69e-07    1.69e-07     
77      MPI_Comm_rank       1       3       1.62e-07    1.62e-07    1.62e-07     
102     MPI_Comm_rank       1       4       1.56e-07    1.56e-07    1.56e-07     
127     MPI_Comm_rank       1       5       1.12e-07    1.12e-07    1.12e-07     
152     MPI_Comm_rank       1       6       1.28e-07    1.28e-07    1.28e-07     
177     MPI_Comm_rank       1       7       1.94e-07    1.94e-07    1.94e-07     
202     MPI_Comm_rank       1       8       2.7e-07     2.7e-07     2.7e-07      
227     MPI_Comm_rank       1       9       1.69e-07    1.69e-07    1.69e-07     
252     MPI_Comm_rank       1       10      2.71e-07    2.71e-07    2.71e-07     
277     MPI_Comm_rank       1       11      1.84e-07    1.84e-07    1.84e-07     
302     MPI_Comm_rank       1       12      1.34e-07    1.34e-07    1.34e-07     
327     MPI_Comm_rank       1       13      2.1e-07     2.1e-07     2.1e-07      
352     MPI_Comm_rank       1       14      1.54e-07    1.54e-07    1.54e-07     
377     MPI_Comm_rank       1       15      2e-07       2e-07       2e-07        

Call site:
 \_ 0x4011d9                            
  \_ __libc_start_main                  
   \_ 0x40332e                          
    \_ 0x401316                         
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Type: [GENERIC]                             Elapsed time inside MPI function                
id:     name                hits:   rank:   min:        max:        sum:        
3       mpi_comm_rank_      1       0       2.6276e-05  2.6276e-05  2.6276e-05   
28      mpi_comm_rank_      1       1       1.8692e-05  1.8692e-05  1.8692e-05   
53      mpi_comm_rank_      1       2       2.4563e-05  2.4563e-05  2.4563e-05   
78      mpi_comm_rank_      1       3       3.0091e-05  3.0091e-05  3.0091e-05   
103     mpi_comm_rank_      1       4       2.3906e-05  2.3906e-05  2.3906e-05   
128     mpi_comm_rank_      1       5       3.0984e-05  3.0984e-05  3.0984e-05   
153     mpi_comm_rank_      1       6       2.0323e-05  2.0323e-05  2.0323e-05   
178     mpi_comm_rank_      1       7       2.1421e-05  2.1421e-05  2.1421e-05   
203     mpi_comm_rank_      1       8       1.7833e-05  1.7833e-05  1.7833e-05   
228     mpi_comm_rank_      1       9       1.8801e-05  1.8801e-05  1.8801e-05   
253     mpi_comm_rank_      1       10      1.7548e-05  1.7548e-05  1.7548e-05   
278     mpi_comm_rank_      1       11      1.8309e-05  1.8309e-05  1.8309e-05   
303     mpi_comm_rank_      1       12      1.8563e-05  1.8563e-05  1.8563e-05   
328     mpi_comm_rank_      1       13      1.9309e-05  1.9309e-05  1.9309e-05   
353     mpi_comm_rank_      1       14      2.0179e-05  2.0179e-05  2.0179e-05   
378     mpi_comm_rank_      1       15      1.8103e-05  1.8103e-05  1.8103e-05   

Call site:
 \_ 0x4011d9                            
  \_ __libc_start_main                  
   \_ 0x40332e                          
    \_ 0x401333                         
     \_ mpi_comm_size_                  
      \_ PMPI_COMM_SIZE                 
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Type: [GENERIC]                             Elapsed time inside MPI function                
id:     name                hits:   rank:   min:        max:        sum:        
4       MPI_Comm_size       1       0       8e-08       8e-08       8e-08        
29      MPI_Comm_size       1       1       1.19e-07    1.19e-07    1.19e-07     
54      MPI_Comm_size       1       2       1.31e-07    1.31e-07    1.31e-07     
79      MPI_Comm_size       1       3       9.5e-08     9.5e-08     9.5e-08      
104     MPI_Comm_size       1       4       1.44e-07    1.44e-07    1.44e-07     
129     MPI_Comm_size       1       5       1.17e-07    1.17e-07    1.17e-07     
154     MPI_Comm_size       1       6       1.05e-07    1.05e-07    1.05e-07     
179     MPI_Comm_size       1       7       1.21e-07    1.21e-07    1.21e-07     
204     MPI_Comm_size       1       8       1.06e-07    1.06e-07    1.06e-07     
229     MPI_Comm_size       1       9       1.66e-07    1.66e-07    1.66e-07     
254     MPI_Comm_size       1       10      1.12e-07    1.12e-07    1.12e-07     
279     MPI_Comm_size       1       11      1.46e-07    1.46e-07    1.46e-07     
304     MPI_Comm_size       1       12      1.03e-07    1.03e-07    1.03e-07     
329     MPI_Comm_size       1       13      1.1e-07     1.1e-07     1.1e-07      
354     MPI_Comm_size       1       14      7.9e-08     7.9e-08     7.9e-08      
379     MPI_Comm_size       1       15      7.9e-08     7.9e-08     7.9e-08      

Call site:
 \_ 0x4011d9                            
  \_ __libc_start_main                  
   \_ 0x40332e                          
    \_ 0x401333                         
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Type: [GENERIC]                             Elapsed time inside MPI function                
id:     name                hits:   rank:   min:        max:        sum:        
5       mpi_comm_size_      1       0       9.863e-06   9.863e-06   9.863e-06    
30      mpi_comm_size_      1       1       8.314e-06   8.314e-06   8.314e-06    
55      mpi_comm_size_      1       2       7.696e-06   7.696e-06   7.696e-06    
80      mpi_comm_size_      1       3       7.599e-06   7.599e-06   7.599e-06    
105     mpi_comm_size_      1       4       8.522e-06   8.522e-06   8.522e-06    
130     mpi_comm_size_      1       5       7.789e-06   7.789e-06   7.789e-06    
155     mpi_comm_size_      1       6       7.813e-06   7.813e-06   7.813e-06    
180     mpi_comm_size_      1       7       9.353e-06   9.353e-06   9.353e-06    
205     mpi_comm_size_      1       8       7.704e-06   7.704e-06   7.704e-06    
230     mpi_comm_size_      1       9       7.742e-06   7.742e-06   7.742e-06    
255     mpi_comm_size_      1       10      7.74e-06    7.74e-06    7.74e-06     
280     mpi_comm_size_      1       11      7.73e-06    7.73e-06    7.73e-06     
305     mpi_comm_size_      1       12      1.0091e-05  1.0091e-05  1.0091e-05   
330     mpi_comm_size_      1       13      7.72e-06    7.72e-06    7.72e-06     
355     mpi_comm_size_      1       14      9.892e-06   9.892e-06   9.892e-06    
380     mpi_comm_size_      1       15      7.652e-06   7.652e-06   7.652e-06    

Call site:
 \_ 0x4011d9                            
  \_ __libc_start_main                  
   \_ 0x40332e                          
    \_ 0x401456                         
     \_ mpi_bcast_                      
      \_ pmpi_bcast                     
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Type: [GENERIC]                             Elapsed time inside MPI function                
id:     name                hits:   rank:   min:        max:        sum:        
6       MPI_Bcast           1       0       1.2113e-05  1.2113e-05  1.2113e-05   
31      MPI_Bcast           1       1       1.5518e-05  1.5518e-05  1.5518e-05   
56      MPI_Bcast           1       2       0.000444989 0.000444989 0.000444989  
81      MPI_Bcast           1       3       0.000424654 0.000424654 0.000424654  
106     MPI_Bcast           1       4       0.000425851 0.000425851 0.000425851  
131     MPI_Bcast           1       5       0.000415098 0.000415098 0.000415098  
156     MPI_Bcast           1       6       2.7488e-05  2.7488e-05  2.7488e-05   
181     MPI_Bcast           1       7       0.000463485 0.000463485 0.000463485  
206     MPI_Bcast           1       8       0.000429593 0.000429593 0.000429593  
231     MPI_Bcast           1       9       0.000430527 0.000430527 0.000430527  
256     MPI_Bcast           1       10      0.000435314 0.000435314 0.000435314  
281     MPI_Bcast           1       11      0.000433138 0.000433138 0.000433138  
306     MPI_Bcast           1       12      0.000438136 0.000438136 0.000438136  
331     MPI_Bcast           1       13      0.000439513 0.000439513 0.000439513  
356     MPI_Bcast           1       14      0.000478927 0.000478927 0.000478927  
381     MPI_Bcast           1       15      0.000443861 0.000443861 0.000443861  

Call site:
 \_ 0x4011d9                            
  \_ __libc_start_main                  
   \_ 0x40332e                          
    \_ 0x401456                         
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Type: [GENERIC]                             Elapsed time inside MPI function                
id:     name                hits:   rank:   min:        max:        sum:        
7       mpi_bcast_          1       0       5.2283e-05  5.2283e-05  5.2283e-05   
32      mpi_bcast_          1       1       3.0907e-05  3.0907e-05  3.0907e-05   
57      mpi_bcast_          1       2       0.000468035 0.000468035 0.000468035  
82      mpi_bcast_          1       3       0.000459183 0.000459183 0.000459183  
107     mpi_bcast_          1       4       0.000473722 0.000473722 0.000473722  
132     mpi_bcast_          1       5       0.000454397 0.000454397 0.000454397  
157     mpi_bcast_          1       6       4.7708e-05  4.7708e-05  4.7708e-05   
182     mpi_bcast_          1       7       0.000480312 0.000480312 0.000480312  
207     mpi_bcast_          1       8       0.000463969 0.000463969 0.000463969  
232     mpi_bcast_          1       9       0.000473772 0.000473772 0.000473772  
257     mpi_bcast_          1       10      0.000476253 0.000476253 0.000476253  
282     mpi_bcast_          1       11      0.000473896 0.000473896 0.000473896  
307     mpi_bcast_          1       12      0.000491481 0.000491481 0.000491481  
332     mpi_bcast_          1       13      0.00048197  0.00048197  0.00048197   
357     mpi_bcast_          1       14      0.000500045 0.000500045 0.000500045  
382     mpi_bcast_          1       15      0.000481806 0.000481806 0.000481806  

Call site:
 \_ 0x4011d9                            
  \_ __libc_start_main                  
   \_ 0x40332e                          
    \_ 0x4020ef                         
     \_ mpi_cart_create_                
      \_ MPI_CART_CREATE                
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Type: [GENERIC]                             Elapsed time inside MPI function                
id:     name                hits:   rank:   min:        max:        sum:        
8       MPI_Cart_create     1       0       0.000109763 0.000109763 0.000109763  
33      MPI_Cart_create     1       1       0.317422    0.317422    0.317422     
58      MPI_Cart_create     1       2       0.438817    0.438817    0.438817     
83      MPI_Cart_create     1       3       0.305307    0.305307    0.305307     
108     MPI_Cart_create     1       4       0.151558    0.151558    0.151558     
133     MPI_Cart_create     1       5       0.217676    0.217676    0.217676     
158     MPI_Cart_create     1       6       0.589917    0.589917    0.589917     
183     MPI_Cart_create     1       7       0.507344    0.507344    0.507344     
208     MPI_Cart_create     1       8       0.085735    0.085735    0.085735     
233     MPI_Cart_create     1       9       0.326352    0.326352    0.326352     
258     MPI_Cart_create     1       10      0.39722     0.39722     0.39722      
283     MPI_Cart_create     1       11      0.277789    0.277789    0.277789     
308     MPI_Cart_create     1       12      0.164632    0.164632    0.164632     
333     MPI_Cart_create     1       13      0.256542    0.256542    0.256542     
358     MPI_Cart_create     1       14      0.464221    0.464221    0.464221     
383     MPI_Cart_create     1       15      0.580465    0.580465    0.580465     

Call site:
 \_ 0x4011d9                            
  \_ __libc_start_main                  
   \_ 0x40332e                          
    \_ 0x4020ef                         
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Type: [GENERIC]                             Elapsed time inside MPI function                
id:     name                hits:   rank:   min:        max:        sum:        
9       mpi_cart_create_    1       0       0.000137395 0.000137395 0.000137395  
34      mpi_cart_create_    1       1       0.317448    0.317448    0.317448     
59      mpi_cart_create_    1       2       0.438849    0.438849    0.438849     
84      mpi_cart_create_    1       3       0.30534     0.30534     0.30534      
109     mpi_cart_create_    1       4       0.151589    0.151589    0.151589     
134     mpi_cart_create_    1       5       0.217702    0.217702    0.217702     
159     mpi_cart_create_    1       6       0.589946    0.589946    0.589946     
184     mpi_cart_create_    1       7       0.507368    0.507368    0.507368     
209     mpi_cart_create_    1       8       0.0857599   0.0857599   0.0857599    
234     mpi_cart_create_    1       9       0.326377    0.326377    0.326377     
259     mpi_cart_create_    1       10      0.397246    0.397246    0.397246     
284     mpi_cart_create_    1       11      0.277815    0.277815    0.277815     
309     mpi_cart_create_    1       12      0.164659    0.164659    0.164659     
334     mpi_cart_create_    1       13      0.256567    0.256567    0.256567     
359     mpi_cart_create_    1       14      0.464252    0.464252    0.464252     
384     mpi_cart_create_    1       15      0.58049     0.58049     0.58049      

Call site:
 \_ 0x4011d9                            
  \_ __libc_start_main                  
   \_ 0x40332e                          
    \_ 0x40210a                         
     \_ 0x404a22                        
      \_ mpi_cart_shift_                
       \_ mpi_cart_shift__              
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Type: [GENERIC]                             Elapsed time inside MPI function                
id:     name                hits:   rank:   min:        max:        sum:        
10      MPI_Cart_shift      1       0       7.044e-06   7.044e-06   7.044e-06    
35      MPI_Cart_shift      1       1       9.493e-06   9.493e-06   9.493e-06    
60      MPI_Cart_shift      1       2       1.8757e-05  1.8757e-05  1.8757e-05   
85      MPI_Cart_shift      1       3       1.4346e-05  1.4346e-05  1.4346e-05   
110     MPI_Cart_shift      1       4       1.8539e-05  1.8539e-05  1.8539e-05   
135     MPI_Cart_shift      1       5       1.8892e-05  1.8892e-05  1.8892e-05   
160     MPI_Cart_shift      1       6       2.1736e-05  2.1736e-05  2.1736e-05   
185     MPI_Cart_shift      1       7       1.3397e-05  1.3397e-05  1.3397e-05   
210     MPI_Cart_shift      1       8       7.14e-06    7.14e-06    7.14e-06     
235     MPI_Cart_shift      1       9       7.177e-06   7.177e-06   7.177e-06    
260     MPI_Cart_shift      1       10      6.831e-06   6.831e-06   6.831e-06    
285     MPI_Cart_shift      1       11      8.976e-06   8.976e-06   8.976e-06    
310     MPI_Cart_shift      1       12      1.0492e-05  1.0492e-05  1.0492e-05   
335     MPI_Cart_shift      1       13      9.068e-06   9.068e-06   9.068e-06    
360     MPI_Cart_shift      1       14      9.472e-06   9.472e-06   9.472e-06    
385     MPI_Cart_shift      1       15      7.5e-06     7.5e-06     7.5e-06      

Call site:
 \_ 0x4011d9                            
  \_ __libc_start_main                  
   \_ 0x40332e                          
    \_ 0x40210a                         
     \_ 0x404a22                        
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Type: [GENERIC]                             Elapsed time inside MPI function                
id:     name                hits:   rank:   min:        max:        sum:        
11      mpi_cart_shift_     1       0       2.6568e-05  2.6568e-05  2.6568e-05   
36      mpi_cart_shift_     1       1       2.7091e-05  2.7091e-05  2.7091e-05   
61      mpi_cart_shift_     1       2       4.3973e-05  4.3973e-05  4.3973e-05   
86      mpi_cart_shift_     1       3       3.8554e-05  3.8554e-05  3.8554e-05   
111     mpi_cart_shift_     1       4       3.8537e-05  3.8537e-05  3.8537e-05   
136     mpi_cart_shift_     1       5       6.4004e-05  6.4004e-05  6.4004e-05   
161     mpi_cart_shift_     1       6       3.8246e-05  3.8246e-05  3.8246e-05   
186     mpi_cart_shift_     1       7       3.0314e-05  3.0314e-05  3.0314e-05   
211     mpi_cart_shift_     1       8       2.7505e-05  2.7505e-05  2.7505e-05   
236     mpi_cart_shift_     1       9       3.6812e-05  3.6812e-05  3.6812e-05   
261     mpi_cart_shift_     1       10      2.757e-05   2.757e-05   2.757e-05    
286     mpi_cart_shift_     1       11      3.7858e-05  3.7858e-05  3.7858e-05   
311     mpi_cart_shift_     1       12      3.855e-05   3.855e-05   3.855e-05    
336     mpi_cart_shift_     1       13      4.0869e-05  4.0869e-05  4.0869e-05   
361     mpi_cart_shift_     1       14      3.1741e-05  3.1741e-05  3.1741e-05   
386     mpi_cart_shift_     1       15      3.6515e-05  3.6515e-05  3.6515e-05   

Call site:
 \_ 0x4011d9                            
  \_ __libc_start_main                  
   \_ 0x40332e                          
    \_ 0x4023ff                         
     \_ 0x4038d5                        
      \_ mpi_sendrecv_                  
       \_ pmpi_sendrecv                 
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Type: [GENERIC]                             Elapsed time inside MPI function                
id:     name                hits:   rank:   min:        max:        sum:        
12      MPI_Sendrecv        1       0       0.000164156 0.000164156 0.000164156  
37      MPI_Sendrecv        1       1       0.00560177  0.00560177  0.00560177   
62      MPI_Sendrecv        1       2       0.00249461  0.00249461  0.00249461   
87      MPI_Sendrecv        1       3       0.00173429  0.00173429  0.00173429   
112     MPI_Sendrecv        1       4       0.000170191 0.000170191 0.000170191  
137     MPI_Sendrecv        1       5       0.00114674  0.00114674  0.00114674   
162     MPI_Sendrecv        1       6       0.0039134   0.0039134   0.0039134    
187     MPI_Sendrecv        1       7       0.00260901  0.00260901  0.00260901   
212     MPI_Sendrecv        1       8       0.000191262 0.000191262 0.000191262  
237     MPI_Sendrecv        1       9       0.00381841  0.00381841  0.00381841   
262     MPI_Sendrecv        1       10      0.0034411   0.0034411   0.0034411    
287     MPI_Sendrecv        1       11      0.000171837 0.000171837 0.000171837  
312     MPI_Sendrecv        1       12      0.00213222  0.00213222  0.00213222   
337     MPI_Sendrecv        1       13      0.000169342 0.000169342 0.000169342  
362     MPI_Sendrecv        1       14      0.00436879  0.00436879  0.00436879   
387     MPI_Sendrecv        1       15      0.00225386  0.00225386  0.00225386   

Call site:
 \_ 0x4011d9                            
  \_ __libc_start_main                  
   \_ 0x40332e                          
    \_ 0x4023ff                         
     \_ 0x4038d5                        
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Type: [GENERIC]                             Elapsed time inside MPI function                
id:     name                hits:   rank:   min:        max:        sum:        
13      mpi_sendrecv_       1       0       0.000190158 0.000190158 0.000190158  
38      mpi_sendrecv_       1       1       0.00563428  0.00563428  0.00563428   
63      mpi_sendrecv_       1       2       0.00252576  0.00252576  0.00252576   
88      mpi_sendrecv_       1       3       0.00176223  0.00176223  0.00176223   
113     mpi_sendrecv_       1       4       0.000196658 0.000196658 0.000196658  
138     mpi_sendrecv_       1       5       0.00117232  0.00117232  0.00117232   
163     mpi_sendrecv_       1       6       0.00394447  0.00394447  0.00394447   
188     mpi_sendrecv_       1       7       0.00263678  0.00263678  0.00263678   
213     mpi_sendrecv_       1       8       0.00022106  0.00022106  0.00022106   
238     mpi_sendrecv_       1       9       0.00384912  0.00384912  0.00384912   
263     mpi_sendrecv_       1       10      0.00346813  0.00346813  0.00346813   
288     mpi_sendrecv_       1       11      0.000198258 0.000198258 0.000198258  
313     mpi_sendrecv_       1       12      0.00215833  0.00215833  0.00215833   
338     mpi_sendrecv_       1       13      0.000196806 0.000196806 0.000196806  
363     mpi_sendrecv_       1       14      0.00439647  0.00439647  0.00439647   
388     mpi_sendrecv_       1       15      0.00229335  0.00229335  0.00229335   

Call site:
 \_ 0x4011d9                            
  \_ __libc_start_main                  
   \_ 0x40332e                          
    \_ 0x4023ff                         
     \_ 0x403996                        
      \_ mpi_sendrecv_                  
       \_ pmpi_sendrecv                 
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Type: [GENERIC]                             Elapsed time inside MPI function                
id:     name                hits:   rank:   min:        max:        sum:        
14      MPI_Sendrecv        1       0       0.000228811 0.000228811 0.000228811  
39      MPI_Sendrecv        1       1       0.000242938 0.000242938 0.000242938  
64      MPI_Sendrecv        1       2       0.00228629  0.00228629  0.00228629   
89      MPI_Sendrecv        1       3       0.000138677 0.000138677 0.000138677  
114     MPI_Sendrecv        1       4       0.000251184 0.000251184 0.000251184  
139     MPI_Sendrecv        1       5       0.000263078 0.000263078 0.000263078  
164     MPI_Sendrecv        1       6       0.0011169   0.0011169   0.0011169    
189     MPI_Sendrecv        1       7       0.000161437 0.000161437 0.000161437  
214     MPI_Sendrecv        1       8       0.000160928 0.000160928 0.000160928  
239     MPI_Sendrecv        1       9       0.000402319 0.000402319 0.000402319  
264     MPI_Sendrecv        1       10      0.000141113 0.000141113 0.000141113  
289     MPI_Sendrecv        1       11      0.000927715 0.000927715 0.000927715  
314     MPI_Sendrecv        1       12      0.00015597  0.00015597  0.00015597   
339     MPI_Sendrecv        1       13      0.00024213  0.00024213  0.00024213   
364     MPI_Sendrecv        1       14      0.000252619 0.000252619 0.000252619  
389     MPI_Sendrecv        1       15      0.00431328  0.00431328  0.00431328   

Call site:
 \_ 0x4011d9                            
  \_ __libc_start_main                  
   \_ 0x40332e                          
    \_ 0x4023ff                         
     \_ 0x403996                        
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Type: [GENERIC]                             Elapsed time inside MPI function                
id:     name                hits:   rank:   min:        max:        sum:        
15      mpi_sendrecv_       1       0       0.0002352   0.0002352   0.0002352    
40      mpi_sendrecv_       1       1       0.000249515 0.000249515 0.000249515  
65      mpi_sendrecv_       1       2       0.0022934   0.0022934   0.0022934    
90      mpi_sendrecv_       1       3       0.000144962 0.000144962 0.000144962  
115     mpi_sendrecv_       1       4       0.000257583 0.000257583 0.000257583  
140     mpi_sendrecv_       1       5       0.000270074 0.000270074 0.000270074  
165     mpi_sendrecv_       1       6       0.00112372  0.00112372  0.00112372   
190     mpi_sendrecv_       1       7       0.000167609 0.000167609 0.000167609  
215     mpi_sendrecv_       1       8       0.000167822 0.000167822 0.000167822  
240     mpi_sendrecv_       1       9       0.000408335 0.000408335 0.000408335  
265     mpi_sendrecv_       1       10      0.000147363 0.000147363 0.000147363  
290     mpi_sendrecv_       1       11      0.000934695 0.000934695 0.000934695  
315     mpi_sendrecv_       1       12      0.000162416 0.000162416 0.000162416  
340     mpi_sendrecv_       1       13      0.000248442 0.000248442 0.000248442  
365     mpi_sendrecv_       1       14      0.000259092 0.000259092 0.000259092  
390     mpi_sendrecv_       1       15      0.00431894  0.00431894  0.00431894   

Call site:
 \_ 0x4011d9                            
  \_ __libc_start_main                  
   \_ 0x40332e                          
    \_ 0x4024f5                         
     \_ mpi_allreduce_                  
      \_ mpi_allreduce_                 
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Type: [GENERIC]                             Elapsed time inside MPI function                
id:     name                hits:   rank:   min:        max:        sum:        
16      MPI_Allreduce       1       0       0.039597    0.039597    0.039597     
41      MPI_Allreduce       1       1       0.0480108   0.0480108   0.0480108    
66      MPI_Allreduce       1       2       0.0514626   0.0514626   0.0514626    
91      MPI_Allreduce       1       3       0.0280434   0.0280434   0.0280434    
116     MPI_Allreduce       1       4       0.0489742   0.0489742   0.0489742    
141     MPI_Allreduce       1       5       0.0488824   0.0488824   0.0488824    
166     MPI_Allreduce       1       6       0.00210632  0.00210632  0.00210632   
191     MPI_Allreduce       1       7       0.0485342   0.0485342   0.0485342    
216     MPI_Allreduce       1       8       0.0511526   0.0511526   0.0511526    
241     MPI_Allreduce       1       9       0.0468175   0.0468175   0.0468175    
266     MPI_Allreduce       1       10      0.0536125   0.0536125   0.0536125    
291     MPI_Allreduce       1       11      0.0510486   0.0510486   0.0510486    
316     MPI_Allreduce       1       12      2.2683e-05  2.2683e-05  2.2683e-05   
341     MPI_Allreduce       1       13      0.0456448   0.0456448   0.0456448    
366     MPI_Allreduce       1       14      0.0521313   0.0521313   0.0521313    
391     MPI_Allreduce       1       15      0.0493683   0.0493683   0.0493683    

Call site:
 \_ 0x4011d9                            
  \_ __libc_start_main                  
   \_ 0x40332e                          
    \_ 0x4024f5                         
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Type: [GENERIC]                             Elapsed time inside MPI function                
id:     name                hits:   rank:   min:        max:        sum:        
17      mpi_allreduce_      1       0       0.0396458   0.0396458   0.0396458    
42      mpi_allreduce_      1       1       0.0480887   0.0480887   0.0480887    
67      mpi_allreduce_      1       2       0.0515196   0.0515196   0.0515196    
92      mpi_allreduce_      1       3       0.0280881   0.0280881   0.0280881    
117     mpi_allreduce_      1       4       0.0490359   0.0490359   0.0490359    
142     mpi_allreduce_      1       5       0.048926    0.048926    0.048926     
167     mpi_allreduce_      1       6       0.00215907  0.00215907  0.00215907   
192     mpi_allreduce_      1       7       0.0485982   0.0485982   0.0485982    
217     mpi_allreduce_      1       8       0.051209    0.051209    0.051209     
242     mpi_allreduce_      1       9       0.0468709   0.0468709   0.0468709    
267     mpi_allreduce_      1       10      0.0536745   0.0536745   0.0536745    
292     mpi_allreduce_      1       11      0.0511208   0.0511208   0.0511208    
317     mpi_allreduce_      1       12      7.8103e-05  7.8103e-05  7.8103e-05   
342     mpi_allreduce_      1       13      0.0456987   0.0456987   0.0456987    
367     mpi_allreduce_      1       14      0.0521875   0.0521875   0.0521875    
392     mpi_allreduce_      1       15      0.0494228   0.0494228   0.0494228    

Call site:
 \_ 0x4011d9                            
  \_ __libc_start_main                  
   \_ 0x40332e                          
    \_ 0x402614                         
     \_ mpi_barrier_                    
      \_ pmpi_barrier                   
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Type: [GENERIC]                             Elapsed time inside MPI function                
id:     name                hits:   rank:   min:        max:        sum:        
18      MPI_Barrier         1       0       4.8889e-05  4.8889e-05  4.8889e-05   
43      MPI_Barrier         1       1       2.0421e-05  2.0421e-05  2.0421e-05   
68      MPI_Barrier         1       2       7.7619e-05  7.7619e-05  7.7619e-05   
93      MPI_Barrier         1       3       9.6716e-05  9.6716e-05  9.6716e-05   
118     MPI_Barrier         1       4       3.9534e-05  3.9534e-05  3.9534e-05   
143     MPI_Barrier         1       5       7.6087e-05  7.6087e-05  7.6087e-05   
168     MPI_Barrier         1       6       5.6836e-05  5.6836e-05  5.6836e-05   
193     MPI_Barrier         1       7       5.9879e-05  5.9879e-05  5.9879e-05   
218     MPI_Barrier         1       8       5.2269e-05  5.2269e-05  5.2269e-05   
243     MPI_Barrier         1       9       5.4645e-05  5.4645e-05  5.4645e-05   
268     MPI_Barrier         1       10      6.8476e-05  6.8476e-05  6.8476e-05   
293     MPI_Barrier         1       11      5.8129e-05  5.8129e-05  5.8129e-05   
318     MPI_Barrier         1       12      6.075e-05   6.075e-05   6.075e-05    
343     MPI_Barrier         1       13      7.4832e-05  7.4832e-05  7.4832e-05   
368     MPI_Barrier         1       14      6.2951e-05  6.2951e-05  6.2951e-05   
393     MPI_Barrier         1       15      6.5213e-05  6.5213e-05  6.5213e-05   

Call site:
 \_ 0x4011d9                            
  \_ __libc_start_main                  
   \_ 0x40332e                          
    \_ 0x402614                         
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Type: [GENERIC]                             Elapsed time inside MPI function                
id:     name                hits:   rank:   min:        max:        sum:        
19      mpi_barrier_        1       0       5.5233e-05  5.5233e-05  5.5233e-05   
44      mpi_barrier_        1       1       2.675e-05   2.675e-05   2.675e-05    
69      mpi_barrier_        1       2       8.3991e-05  8.3991e-05  8.3991e-05   
94      mpi_barrier_        1       3       0.000103755 0.000103755 0.000103755  
119     mpi_barrier_        1       4       4.5788e-05  4.5788e-05  4.5788e-05   
144     mpi_barrier_        1       5       8.3534e-05  8.3534e-05  8.3534e-05   
169     mpi_barrier_        1       6       6.3167e-05  6.3167e-05  6.3167e-05   
194     mpi_barrier_        1       7       6.6819e-05  6.6819e-05  6.6819e-05   
219     mpi_barrier_        1       8       6.0421e-05  6.0421e-05  6.0421e-05   
244     mpi_barrier_        1       9       6.2249e-05  6.2249e-05  6.2249e-05   
269     mpi_barrier_        1       10      7.4796e-05  7.4796e-05  7.4796e-05   
294     mpi_barrier_        1       11      6.4736e-05  6.4736e-05  6.4736e-05   
319     mpi_barrier_        1       12      6.7141e-05  6.7141e-05  6.7141e-05   
344     mpi_barrier_        1       13      8.2952e-05  8.2952e-05  8.2952e-05   
369     mpi_barrier_        1       14      6.9185e-05  6.9185e-05  6.9185e-05   
394     mpi_barrier_        1       15      7.1588e-05  7.1588e-05  7.1588e-05   

Call site:
 \_ 0x4011d9                            
  \_ __libc_start_main                  
   \_ 0x40332e                          
    \_ 0x40264e                         
     \_ mpi_allreduce_                  
      \_ mpi_allreduce_                 
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Type: [GENERIC]                             Elapsed time inside MPI function                
id:     name                hits:   rank:   min:        max:        sum:        
20      MPI_Allreduce       1       0       5.5321e-05  5.5321e-05  5.5321e-05   
45      MPI_Allreduce       1       1       5.8053e-05  5.8053e-05  5.8053e-05   
70      MPI_Allreduce       1       2       2.4967e-05  2.4967e-05  2.4967e-05   
95      MPI_Allreduce       1       3       1.1876e-05  1.1876e-05  1.1876e-05   
120     MPI_Allreduce       1       4       5.7177e-05  5.7177e-05  5.7177e-05   
145     MPI_Allreduce       1       5       3.6751e-05  3.6751e-05  3.6751e-05   
170     MPI_Allreduce       1       6       4.7992e-05  4.7992e-05  4.7992e-05   
195     MPI_Allreduce       1       7       2.9998e-05  2.9998e-05  2.9998e-05   
220     MPI_Allreduce       1       8       4.9349e-05  4.9349e-05  4.9349e-05   
245     MPI_Allreduce       1       9       4.8636e-05  4.8636e-05  4.8636e-05   
270     MPI_Allreduce       1       10      3.2265e-05  3.2265e-05  3.2265e-05   
295     MPI_Allreduce       1       11      2.423e-05   2.423e-05   2.423e-05    
320     MPI_Allreduce       1       12      4.2118e-05  4.2118e-05  4.2118e-05   
345     MPI_Allreduce       1       13      2.8875e-05  2.8875e-05  2.8875e-05   
370     MPI_Allreduce       1       14      4.0928e-05  4.0928e-05  4.0928e-05   
395     MPI_Allreduce       1       15      3.6796e-05  3.6796e-05  3.6796e-05   

Call site:
 \_ 0x4011d9                            
  \_ __libc_start_main                  
   \_ 0x40332e                          
    \_ 0x40264e                         
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Type: [GENERIC]                             Elapsed time inside MPI function                
id:     name                hits:   rank:   min:        max:        sum:        
21      mpi_allreduce_      1       0       6.039e-05   6.039e-05   6.039e-05    
46      mpi_allreduce_      1       1       6.5853e-05  6.5853e-05  6.5853e-05   
71      mpi_allreduce_      1       2       3.2158e-05  3.2158e-05  3.2158e-05   
96      mpi_allreduce_      1       3       1.9272e-05  1.9272e-05  1.9272e-05   
121     mpi_allreduce_      1       4       6.5023e-05  6.5023e-05  6.5023e-05   
146     mpi_allreduce_      1       5       4.4175e-05  4.4175e-05  4.4175e-05   
171     mpi_allreduce_      1       6       5.5342e-05  5.5342e-05  5.5342e-05   
196     mpi_allreduce_      1       7       3.7368e-05  3.7368e-05  3.7368e-05   
221     mpi_allreduce_      1       8       5.8095e-05  5.8095e-05  5.8095e-05   
246     mpi_allreduce_      1       9       5.5998e-05  5.5998e-05  5.5998e-05   
271     mpi_allreduce_      1       10      3.9601e-05  3.9601e-05  3.9601e-05   
296     mpi_allreduce_      1       11      3.1908e-05  3.1908e-05  3.1908e-05   
321     mpi_allreduce_      1       12      4.9608e-05  4.9608e-05  4.9608e-05   
346     mpi_allreduce_      1       13      3.6268e-05  3.6268e-05  3.6268e-05   
371     mpi_allreduce_      1       14      4.84e-05    4.84e-05    4.84e-05     
396     mpi_allreduce_      1       15      4.4085e-05  4.4085e-05  4.4085e-05   

Call site:
 \_ 0x4011d9                            
  \_ __libc_start_main                  
   \_ 0x40332e                          
    \_ 0x403105                         
     \_ mpi_barrier_                    
      \_ pmpi_barrier                   
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Type: [GENERIC]                             Elapsed time inside MPI function                
id:     name                hits:   rank:   min:        max:        sum:        
22      MPI_Barrier         1       0       3.409e-06   3.409e-06   3.409e-06    
47      MPI_Barrier         1       1       6.5573e-05  6.5573e-05  6.5573e-05   
72      MPI_Barrier         1       2       6.7853e-05  6.7853e-05  6.7853e-05   
97      MPI_Barrier         1       3       6.8143e-05  6.8143e-05  6.8143e-05   
122     MPI_Barrier         1       4       6.6437e-05  6.6437e-05  6.6437e-05   
147     MPI_Barrier         1       5       6.6439e-05  6.6439e-05  6.6439e-05   
172     MPI_Barrier         1       6       6.6139e-05  6.6139e-05  6.6139e-05   
197     MPI_Barrier         1       7       6.8273e-05  6.8273e-05  6.8273e-05   
222     MPI_Barrier         1       8       6.5279e-05  6.5279e-05  6.5279e-05   
247     MPI_Barrier         1       9       6.634e-05   6.634e-05   6.634e-05    
272     MPI_Barrier         1       10      6.6928e-05  6.6928e-05  6.6928e-05   
297     MPI_Barrier         1       11      6.8204e-05  6.8204e-05  6.8204e-05   
322     MPI_Barrier         1       12      6.7359e-05  6.7359e-05  6.7359e-05   
347     MPI_Barrier         1       13      6.6553e-05  6.6553e-05  6.6553e-05   
372     MPI_Barrier         1       14      6.6502e-05  6.6502e-05  6.6502e-05   
397     MPI_Barrier         1       15      6.846e-05   6.846e-05   6.846e-05    

Call site:
 \_ 0x4011d9                            
  \_ __libc_start_main                  
   \_ 0x40332e                          
    \_ 0x403105                         
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Type: [GENERIC]                             Elapsed time inside MPI function                
id:     name                hits:   rank:   min:        max:        sum:        
23      mpi_barrier_        1       0       7.987e-06   7.987e-06   7.987e-06    
48      mpi_barrier_        1       1       6.9838e-05  6.9838e-05  6.9838e-05   
73      mpi_barrier_        1       2       7.2171e-05  7.2171e-05  7.2171e-05   
98      mpi_barrier_        1       3       7.2485e-05  7.2485e-05  7.2485e-05   
123     mpi_barrier_        1       4       7.086e-05   7.086e-05   7.086e-05    
148     mpi_barrier_        1       5       7.0886e-05  7.0886e-05  7.0886e-05   
173     mpi_barrier_        1       6       7.1009e-05  7.1009e-05  7.1009e-05   
198     mpi_barrier_        1       7       7.2702e-05  7.2702e-05  7.2702e-05   
223     mpi_barrier_        1       8       6.9784e-05  6.9784e-05  6.9784e-05   
248     mpi_barrier_        1       9       7.0829e-05  7.0829e-05  7.0829e-05   
273     mpi_barrier_        1       10      7.1254e-05  7.1254e-05  7.1254e-05   
298     mpi_barrier_        1       11      7.2782e-05  7.2782e-05  7.2782e-05   
323     mpi_barrier_        1       12      7.1749e-05  7.1749e-05  7.1749e-05   
348     mpi_barrier_        1       13      7.0872e-05  7.0872e-05  7.0872e-05   
373     mpi_barrier_        1       14      7.0782e-05  7.0782e-05  7.0782e-05   
398     mpi_barrier_        1       15      7.3146e-05  7.3146e-05  7.3146e-05   

Call site:
 \_ 0x4011d9                            
  \_ __libc_start_main                  
   \_ 0x40332e                          
    \_ 0x403219                         
     \_ mpi_comm_free_                  
      \_ MPI_COMM_FREE                  
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Type: [GENERIC]                             Elapsed time inside MPI function                
id:     name                hits:   rank:   min:        max:        sum:        
24      MPI_Comm_free       1       0       1.5026e-05  1.5026e-05  1.5026e-05   
49      MPI_Comm_free       1       1       1.6489e-05  1.6489e-05  1.6489e-05   
74      MPI_Comm_free       1       2       1.5224e-05  1.5224e-05  1.5224e-05   
99      MPI_Comm_free       1       3       1.5518e-05  1.5518e-05  1.5518e-05   
124     MPI_Comm_free       1       4       1.8565e-05  1.8565e-05  1.8565e-05   
149     MPI_Comm_free       1       5       2.1299e-05  2.1299e-05  2.1299e-05   
174     MPI_Comm_free       1       6       2.1661e-05  2.1661e-05  2.1661e-05   
199     MPI_Comm_free       1       7       1.459e-05   1.459e-05   1.459e-05    
224     MPI_Comm_free       1       8       1.5466e-05  1.5466e-05  1.5466e-05   
249     MPI_Comm_free       1       9       1.6826e-05  1.6826e-05  1.6826e-05   
274     MPI_Comm_free       1       10      1.955e-05   1.955e-05   1.955e-05    
299     MPI_Comm_free       1       11      1.4598e-05  1.4598e-05  1.4598e-05   
324     MPI_Comm_free       1       12      1.445e-05   1.445e-05   1.445e-05    
349     MPI_Comm_free       1       13      1.5365e-05  1.5365e-05  1.5365e-05   
374     MPI_Comm_free       1       14      1.473e-05   1.473e-05   1.473e-05    
399     MPI_Comm_free       1       15      1.561e-05   1.561e-05   1.561e-05    

Call site:
 \_ 0x4011d9                            
  \_ __libc_start_main                  
   \_ 0x40332e                          
    \_ 0x403219                         
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Type: [GENERIC]                             Elapsed time inside MPI function                
id:     name                hits:   rank:   min:        max:        sum:        
25      mpi_comm_free_      1       0       2.9623e-05  2.9623e-05  2.9623e-05   
50      mpi_comm_free_      1       1       3.0247e-05  3.0247e-05  3.0247e-05   
75      mpi_comm_free_      1       2       3.5899e-05  3.5899e-05  3.5899e-05   
100     mpi_comm_free_      1       3       3.2201e-05  3.2201e-05  3.2201e-05   
125     mpi_comm_free_      1       4       3.983e-05   3.983e-05   3.983e-05    
150     mpi_comm_free_      1       5       4.4795e-05  4.4795e-05  4.4795e-05   
175     mpi_comm_free_      1       6       3.773e-05   3.773e-05   3.773e-05    
200     mpi_comm_free_      1       7       2.8123e-05  2.8123e-05  2.8123e-05   
225     mpi_comm_free_      1       8       3.2366e-05  3.2366e-05  3.2366e-05   
250     mpi_comm_free_      1       9       3.7998e-05  3.7998e-05  3.7998e-05   
275     mpi_comm_free_      1       10      4.3271e-05  4.3271e-05  4.3271e-05   
300     mpi_comm_free_      1       11      2.8538e-05  2.8538e-05  2.8538e-05   
325     mpi_comm_free_      1       12      2.9037e-05  2.9037e-05  2.9037e-05   
350     mpi_comm_free_      1       13      2.8875e-05  2.8875e-05  2.8875e-05   
375     mpi_comm_free_      1       14      2.9405e-05  2.9405e-05  2.9405e-05   
400     mpi_comm_free_      1       15      3.0005e-05  3.0005e-05  3.0005e-05   


===============================================================================================================================================================================
                                                                              [[ end of report ]]
===============================================================================================================================================================================

#################################################################################
#                                                                               #
#                          Bull Lightweight Profiler                            #
#                                                                               #
#################################################################################

General info:
job_start=Wed 2021-03-10 14:19:11
job_finish=Wed 2021-03-10 14:19:33
job_duration=22 s
command='srun --cpu_bind=mask_cpu:0x1,0x2,0x4,0x8,0x10,0x20,0x40,0x80,0x40000,0x80000,0x100000,0x200000,0x400000,0x800000,0x1000000,0x2000000 ./poisson.65536'
user='renon'

#################################################################################

Voir aussi

FAQ

Questions fréquentes et messages d'erreurs

ITAC (Intel Trace Analyzer Collector)

ITAC s’utilise en deux temps : Étape 1 : on lance un job pour la collecte des informations Étape 2 : on lance l’outil de post-traitement graphique afin d’analyser les résultats.