MPI send/recv latency jump at 32 MiB message size
up vote
0
down vote
favorite
For a university project I'm currently working on a slurm supercomputer cluster and have written a number of C programs using MPI.
While profiling one of them I have observed that the time elapsed between an MPI_Send and an accompanying MPI_Recv operation is a mostly linear function of the message length. However, at around 32 MiB there is a sudden jump in latency from around 10ms to around 20ms. This happens both for two processes on the same node and two processes on separate nodes.
Now I would like to find out why this happens. I'm aware that this is not an MPI intrinsic phenomenon but must be related to the underlying hardware setup, but I'm not sure where to begin looking for an explanation. What are some possible explanations for this and how could I check whether they apply in my case?
mpi hpc
asked yesterday
Peter
31819
add a comment |
up vote
0
down vote
favorite
For a university project I'm currently working on a slurm supercomputer cluster and have written a number of C programs using MPI.
While profiling one of them I have observed that the time elapsed between an MPI_Send and an accompanying MPI_Recv operation is a mostly linear function of the message length. However, at around 32 MiB there is a sudden jump in latency from around 10ms to around 20ms. This happens both for two processes on the same node and two processes on separate nodes.
Now I would like to find out why this happens. I'm aware that this is not an MPI intrinsic phenomenon but must be related to the underlying hardware setup, but I'm not sure where to begin looking for an explanation. What are some possible explanations for this and how could I check whether they apply in my case?
mpi hpc
asked yesterday
Peter
31819
Is there a spike or a jump ? I mean, does the latency for messages larger than 32MiB go down again, or stay high ? For anything better than guesses and waffle you will have to tell us what hardware your cluster comprises, especially what kind of interconnect it has.
– High Performance Mark
18 hours ago
It's a jump, you're right, spike is the wrong word. I'm not completely sure which hardware specs are relevant here but I know that the nodes I'm working on are comprised of around two dozen somewhat current Intel Xeon processors and Infiniband (FDR) is used for interconnect.
– Peter
16 hours ago
You should first measure the latency/bandwidth with a trusted benchmark such as IMB (from Intel) or the OSU benchmark.
– Gilles Gouaillardet
16 hours ago
add a comment |
up vote
0
down vote
favorite
up vote
0
down vote
favorite
For a university project I'm currently working on a slurm supercomputer cluster and have written a number of C programs using MPI.
While profiling one of them I have observed that the time elapsed between an MPI_Send and an accompanying MPI_Recv operation is a mostly linear function of the message length. However, at around 32 MiB there is a sudden jump in latency from around 10ms to around 20ms. This happens both for two processes on the same node and two processes on separate nodes.
Now I would like to find out why this happens. I'm aware that this is not an MPI intrinsic phenomenon but must be related to the underlying hardware setup, but I'm not sure where to begin looking for an explanation. What are some possible explanations for this and how could I check whether they apply in my case?
mpi hpc
asked yesterday
Peter
31819
For a university project I'm currently working on a slurm supercomputer cluster and have written a number of C programs using MPI.
While profiling one of them I have observed that the time elapsed between an MPI_Send and an accompanying MPI_Recv operation is a mostly linear function of the message length. However, at around 32 MiB there is a sudden jump in latency from around 10ms to around 20ms. This happens both for two processes on the same node and two processes on separate nodes.
Now I would like to find out why this happens. I'm aware that this is not an MPI intrinsic phenomenon but must be related to the underlying hardware setup, but I'm not sure where to begin looking for an explanation. What are some possible explanations for this and how could I check whether they apply in my case?
mpi hpc
mpi hpc
asked yesterday
Peter
31819
asked yesterday
Peter
31819
edited 16 hours ago
asked yesterday
Peter
31819
asked yesterday
Peter
31819
asked yesterday
Peter
31819
31819
Is there a spike or a jump ? I mean, does the latency for messages larger than 32MiB go down again, or stay high ? For anything better than guesses and waffle you will have to tell us what hardware your cluster comprises, especially what kind of interconnect it has.
– High Performance Mark
18 hours ago
It's a jump, you're right, spike is the wrong word. I'm not completely sure which hardware specs are relevant here but I know that the nodes I'm working on are comprised of around two dozen somewhat current Intel Xeon processors and Infiniband (FDR) is used for interconnect.
– Peter
16 hours ago
You should first measure the latency/bandwidth with a trusted benchmark such as IMB (from Intel) or the OSU benchmark.
– Gilles Gouaillardet
16 hours ago
add a comment |
Is there a spike or a jump ? I mean, does the latency for messages larger than 32MiB go down again, or stay high ? For anything better than guesses and waffle you will have to tell us what hardware your cluster comprises, especially what kind of interconnect it has.
– High Performance Mark
18 hours ago
It's a jump, you're right, spike is the wrong word. I'm not completely sure which hardware specs are relevant here but I know that the nodes I'm working on are comprised of around two dozen somewhat current Intel Xeon processors and Infiniband (FDR) is used for interconnect.
– Peter
16 hours ago
You should first measure the latency/bandwidth with a trusted benchmark such as IMB (from Intel) or the OSU benchmark.
– Gilles Gouaillardet
16 hours ago
Is there a spike or a jump ? I mean, does the latency for messages larger than 32MiB go down again, or stay high ? For anything better than guesses and waffle you will have to tell us what hardware your cluster comprises, especially what kind of interconnect it has.
– High Performance Mark
18 hours ago
Is there a spike or a jump ? I mean, does the latency for messages larger than 32MiB go down again, or stay high ? For anything better than guesses and waffle you will have to tell us what hardware your cluster comprises, especially what kind of interconnect it has.
– High Performance Mark
18 hours ago
It's a jump, you're right, spike is the wrong word. I'm not completely sure which hardware specs are relevant here but I know that the nodes I'm working on are comprised of around two dozen somewhat current Intel Xeon processors and Infiniband (FDR) is used for interconnect.
– Peter
16 hours ago
It's a jump, you're right, spike is the wrong word. I'm not completely sure which hardware specs are relevant here but I know that the nodes I'm working on are comprised of around two dozen somewhat current Intel Xeon processors and Infiniband (FDR) is used for interconnect.
– Peter
16 hours ago
You should first measure the latency/bandwidth with a trusted benchmark such as IMB (from Intel) or the OSU benchmark.
– Gilles Gouaillardet
16 hours ago
You should first measure the latency/bandwidth with a trusted benchmark such as IMB (from Intel) or the OSU benchmark.
– Gilles Gouaillardet
16 hours ago
add a comment |
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53401577%2fmpi-send-recv-latency-jump-at-32-mib-message-size%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Is there a spike or a jump ? I mean, does the latency for messages larger than 32MiB go down again, or stay high ? For anything better than guesses and waffle you will have to tell us what hardware your cluster comprises, especially what kind of interconnect it has.
– High Performance Mark
18 hours ago
It's a jump, you're right, spike is the wrong word. I'm not completely sure which hardware specs are relevant here but I know that the nodes I'm working on are comprised of around two dozen somewhat current Intel Xeon processors and Infiniband (FDR) is used for interconnect.
– Peter
16 hours ago
You should first measure the latency/bandwidth with a trusted benchmark such as IMB (from Intel) or the OSU benchmark.
– Gilles Gouaillardet
16 hours ago