Strange bottleneck copying files on EFS shares
up vote
0
down vote
favorite
A bit of a weird problem that has popped up out of nowhere and will little to no information. I work in support of a PHP (Laravel) application that stores asset files for our users. The application runs on across a few EC2 instances, and the assets are stored in EFS mounts.
The other day, one of our core routes which is responsible for triggering the server to copy a directory from A to B began to take a significant amount of time (from 5s to 90s) with no changes to the contents or application code.
I was able to isolate the delay to an xcopy call that moves 1 folder on the EFS to another folder in the same EFS. When I reached out to Amazon for assistance looking at the EFS metrics in CloudWatch, they indicated that everything looked strong, but they noted the fall off. Throughput is provisioned at 10Mb/s, and I've only spiked at 4-5Mb/s in the past few days.
Using NetData, I'm able to observe performance of RAM, CPU, NFS Client calls, and much more. From what I can tell, the NFS client is taking its sweet time. When testing locally without the EFS, the call is very speedy. I tried to modify the app code so that rather than a PHP copy, it would perform a system call to cp. This didn't seem to work.
Has anyone else experienced random latency issues like this before when using EFS? Does anyone have any recommendations on debugging techniques to see why this might be happening?
Thank you for reading and information you can offer up.
php laravel filesystems latency amazon-efs
asked yesterday
Neurax
6301515
add a comment |
up vote
0
down vote
favorite
A bit of a weird problem that has popped up out of nowhere and will little to no information. I work in support of a PHP (Laravel) application that stores asset files for our users. The application runs on across a few EC2 instances, and the assets are stored in EFS mounts.
The other day, one of our core routes which is responsible for triggering the server to copy a directory from A to B began to take a significant amount of time (from 5s to 90s) with no changes to the contents or application code.
I was able to isolate the delay to an xcopy call that moves 1 folder on the EFS to another folder in the same EFS. When I reached out to Amazon for assistance looking at the EFS metrics in CloudWatch, they indicated that everything looked strong, but they noted the fall off. Throughput is provisioned at 10Mb/s, and I've only spiked at 4-5Mb/s in the past few days.
Using NetData, I'm able to observe performance of RAM, CPU, NFS Client calls, and much more. From what I can tell, the NFS client is taking its sweet time. When testing locally without the EFS, the call is very speedy. I tried to modify the app code so that rather than a PHP copy, it would perform a system call to cp. This didn't seem to work.
Has anyone else experienced random latency issues like this before when using EFS? Does anyone have any recommendations on debugging techniques to see why this might be happening?
Thank you for reading and information you can offer up.
php laravel filesystems latency amazon-efs
asked yesterday
Neurax
6301515
add a comment |
up vote
0
down vote
favorite
up vote
0
down vote
favorite
A bit of a weird problem that has popped up out of nowhere and will little to no information. I work in support of a PHP (Laravel) application that stores asset files for our users. The application runs on across a few EC2 instances, and the assets are stored in EFS mounts.
The other day, one of our core routes which is responsible for triggering the server to copy a directory from A to B began to take a significant amount of time (from 5s to 90s) with no changes to the contents or application code.
I was able to isolate the delay to an xcopy call that moves 1 folder on the EFS to another folder in the same EFS. When I reached out to Amazon for assistance looking at the EFS metrics in CloudWatch, they indicated that everything looked strong, but they noted the fall off. Throughput is provisioned at 10Mb/s, and I've only spiked at 4-5Mb/s in the past few days.
Using NetData, I'm able to observe performance of RAM, CPU, NFS Client calls, and much more. From what I can tell, the NFS client is taking its sweet time. When testing locally without the EFS, the call is very speedy. I tried to modify the app code so that rather than a PHP copy, it would perform a system call to cp. This didn't seem to work.
Has anyone else experienced random latency issues like this before when using EFS? Does anyone have any recommendations on debugging techniques to see why this might be happening?
Thank you for reading and information you can offer up.
php laravel filesystems latency amazon-efs
asked yesterday
Neurax
6301515
A bit of a weird problem that has popped up out of nowhere and will little to no information. I work in support of a PHP (Laravel) application that stores asset files for our users. The application runs on across a few EC2 instances, and the assets are stored in EFS mounts.
The other day, one of our core routes which is responsible for triggering the server to copy a directory from A to B began to take a significant amount of time (from 5s to 90s) with no changes to the contents or application code.
I was able to isolate the delay to an xcopy call that moves 1 folder on the EFS to another folder in the same EFS. When I reached out to Amazon for assistance looking at the EFS metrics in CloudWatch, they indicated that everything looked strong, but they noted the fall off. Throughput is provisioned at 10Mb/s, and I've only spiked at 4-5Mb/s in the past few days.
Using NetData, I'm able to observe performance of RAM, CPU, NFS Client calls, and much more. From what I can tell, the NFS client is taking its sweet time. When testing locally without the EFS, the call is very speedy. I tried to modify the app code so that rather than a PHP copy, it would perform a system call to cp. This didn't seem to work.
Has anyone else experienced random latency issues like this before when using EFS? Does anyone have any recommendations on debugging techniques to see why this might be happening?
Thank you for reading and information you can offer up.
php laravel filesystems latency amazon-efs
php laravel filesystems latency amazon-efs
asked yesterday
Neurax
6301515
asked yesterday
Neurax
6301515
asked yesterday
Neurax
6301515
asked yesterday
Neurax
6301515
asked yesterday
Neurax
6301515
6301515
add a comment |
add a comment |
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53401750%2fstrange-bottleneck-copying-files-on-efs-shares%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
