How do I asynchronously calculate the checksum of a file on the hard disk in Rust?











up vote
-1
down vote

favorite












I have a TCP file server in Rust / Tokio stack.



When a client is uploading a file, the data is being read from a tokio::net::TcpStream and written to a futures_fs::FsWriteSink, which has been started on a separate futures_fs::FsPool.



When the file is completely uploaded, I need to check its consistency by checking its checksum against the one sent by the client.



What is the easiest way to asynchronously calculate the checksum, especially if the file does not fit into RAM?










share|improve this question
























  • Using a BufReader?
    – hellow
    Nov 21 at 13:19






  • 2




    You might also want to explore calculating the checksum for the actual data uploaded as it's being uploaded. You already have the data passing through your processing at that point, so calculating the checksum then is almost a free operation. This is especially true if you have performance considerations, because by calculating the checksum off the data after it's been saved to disk you're effectively doubling the IO operations you need to do to support file uploading.
    – Andrew Henle
    Nov 21 at 13:49










  • Yeah, I totally agree. But it seems that I need to implement such thing myself, doesn't it? I'd like to have something like another Sink, where I could feed the chunks so the Sink could calculate the checksum on the fly.
    – hedgar2017
    Nov 21 at 14:47










  • What checksum? Is the implementation of the checksum naturally asynchronous? If so, just use it. If it's not, then this is a duplicate of What is the best approach to encapsulate blocking I/O in future-rs?.
    – Shepmaster
    Nov 21 at 17:59















up vote
-1
down vote

favorite












I have a TCP file server in Rust / Tokio stack.



When a client is uploading a file, the data is being read from a tokio::net::TcpStream and written to a futures_fs::FsWriteSink, which has been started on a separate futures_fs::FsPool.



When the file is completely uploaded, I need to check its consistency by checking its checksum against the one sent by the client.



What is the easiest way to asynchronously calculate the checksum, especially if the file does not fit into RAM?










share|improve this question
























  • Using a BufReader?
    – hellow
    Nov 21 at 13:19






  • 2




    You might also want to explore calculating the checksum for the actual data uploaded as it's being uploaded. You already have the data passing through your processing at that point, so calculating the checksum then is almost a free operation. This is especially true if you have performance considerations, because by calculating the checksum off the data after it's been saved to disk you're effectively doubling the IO operations you need to do to support file uploading.
    – Andrew Henle
    Nov 21 at 13:49










  • Yeah, I totally agree. But it seems that I need to implement such thing myself, doesn't it? I'd like to have something like another Sink, where I could feed the chunks so the Sink could calculate the checksum on the fly.
    – hedgar2017
    Nov 21 at 14:47










  • What checksum? Is the implementation of the checksum naturally asynchronous? If so, just use it. If it's not, then this is a duplicate of What is the best approach to encapsulate blocking I/O in future-rs?.
    – Shepmaster
    Nov 21 at 17:59













up vote
-1
down vote

favorite









up vote
-1
down vote

favorite











I have a TCP file server in Rust / Tokio stack.



When a client is uploading a file, the data is being read from a tokio::net::TcpStream and written to a futures_fs::FsWriteSink, which has been started on a separate futures_fs::FsPool.



When the file is completely uploaded, I need to check its consistency by checking its checksum against the one sent by the client.



What is the easiest way to asynchronously calculate the checksum, especially if the file does not fit into RAM?










share|improve this question















I have a TCP file server in Rust / Tokio stack.



When a client is uploading a file, the data is being read from a tokio::net::TcpStream and written to a futures_fs::FsWriteSink, which has been started on a separate futures_fs::FsPool.



When the file is completely uploaded, I need to check its consistency by checking its checksum against the one sent by the client.



What is the easiest way to asynchronously calculate the checksum, especially if the file does not fit into RAM?







file asynchronous rust checksum rust-tokio






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited Nov 21 at 14:53









Shepmaster

144k11271402




144k11271402










asked Nov 21 at 13:17









hedgar2017

3471723




3471723












  • Using a BufReader?
    – hellow
    Nov 21 at 13:19






  • 2




    You might also want to explore calculating the checksum for the actual data uploaded as it's being uploaded. You already have the data passing through your processing at that point, so calculating the checksum then is almost a free operation. This is especially true if you have performance considerations, because by calculating the checksum off the data after it's been saved to disk you're effectively doubling the IO operations you need to do to support file uploading.
    – Andrew Henle
    Nov 21 at 13:49










  • Yeah, I totally agree. But it seems that I need to implement such thing myself, doesn't it? I'd like to have something like another Sink, where I could feed the chunks so the Sink could calculate the checksum on the fly.
    – hedgar2017
    Nov 21 at 14:47










  • What checksum? Is the implementation of the checksum naturally asynchronous? If so, just use it. If it's not, then this is a duplicate of What is the best approach to encapsulate blocking I/O in future-rs?.
    – Shepmaster
    Nov 21 at 17:59


















  • Using a BufReader?
    – hellow
    Nov 21 at 13:19






  • 2




    You might also want to explore calculating the checksum for the actual data uploaded as it's being uploaded. You already have the data passing through your processing at that point, so calculating the checksum then is almost a free operation. This is especially true if you have performance considerations, because by calculating the checksum off the data after it's been saved to disk you're effectively doubling the IO operations you need to do to support file uploading.
    – Andrew Henle
    Nov 21 at 13:49










  • Yeah, I totally agree. But it seems that I need to implement such thing myself, doesn't it? I'd like to have something like another Sink, where I could feed the chunks so the Sink could calculate the checksum on the fly.
    – hedgar2017
    Nov 21 at 14:47










  • What checksum? Is the implementation of the checksum naturally asynchronous? If so, just use it. If it's not, then this is a duplicate of What is the best approach to encapsulate blocking I/O in future-rs?.
    – Shepmaster
    Nov 21 at 17:59
















Using a BufReader?
– hellow
Nov 21 at 13:19




Using a BufReader?
– hellow
Nov 21 at 13:19




2




2




You might also want to explore calculating the checksum for the actual data uploaded as it's being uploaded. You already have the data passing through your processing at that point, so calculating the checksum then is almost a free operation. This is especially true if you have performance considerations, because by calculating the checksum off the data after it's been saved to disk you're effectively doubling the IO operations you need to do to support file uploading.
– Andrew Henle
Nov 21 at 13:49




You might also want to explore calculating the checksum for the actual data uploaded as it's being uploaded. You already have the data passing through your processing at that point, so calculating the checksum then is almost a free operation. This is especially true if you have performance considerations, because by calculating the checksum off the data after it's been saved to disk you're effectively doubling the IO operations you need to do to support file uploading.
– Andrew Henle
Nov 21 at 13:49












Yeah, I totally agree. But it seems that I need to implement such thing myself, doesn't it? I'd like to have something like another Sink, where I could feed the chunks so the Sink could calculate the checksum on the fly.
– hedgar2017
Nov 21 at 14:47




Yeah, I totally agree. But it seems that I need to implement such thing myself, doesn't it? I'd like to have something like another Sink, where I could feed the chunks so the Sink could calculate the checksum on the fly.
– hedgar2017
Nov 21 at 14:47












What checksum? Is the implementation of the checksum naturally asynchronous? If so, just use it. If it's not, then this is a duplicate of What is the best approach to encapsulate blocking I/O in future-rs?.
– Shepmaster
Nov 21 at 17:59




What checksum? Is the implementation of the checksum naturally asynchronous? If so, just use it. If it's not, then this is a duplicate of What is the best approach to encapsulate blocking I/O in future-rs?.
– Shepmaster
Nov 21 at 17:59












2 Answers
2






active

oldest

votes

















up vote
0
down vote













It depends on what checksum algorithm you want to use, but using the md5 crate as an example, you can compute the checksum on the fly. Something like this should do it:



// When starting the file transfer
let mut md5_context = md5::Context::new();

// ...

// as part of your existing processing for each block of data
md5_context.consume (&block);

// ...

// once the last block has been processed
return md5_context.compute();





share|improve this answer



















  • 1




    This appears to be completely synchronous. The OP has explicitly requested an asynchronous solution.
    – Shepmaster
    Nov 21 at 15:11






  • 1




    @Shepmaster I don't see how you can tell whether this is synchronous or asynchronous: I only said that for each block of data he should call md5_context.consume, I never said that he should do it in a synchronous loop.
    – Jmb
    Nov 21 at 15:46








  • 1




    @Shepmaster edited the answer to make it clearer
    – Jmb
    Nov 21 at 15:50


















up vote
0
down vote



accepted










Actually, making simple hashing algorithms asynchronous is in such cases somewhat redundant, as long as one MD5 calculation takes less then 1 us (about 500 ns).






share|improve this answer























    Your Answer






    StackExchange.ifUsing("editor", function () {
    StackExchange.using("externalEditor", function () {
    StackExchange.using("snippets", function () {
    StackExchange.snippets.init();
    });
    });
    }, "code-snippets");

    StackExchange.ready(function() {
    var channelOptions = {
    tags: "".split(" "),
    id: "1"
    };
    initTagRenderer("".split(" "), "".split(" "), channelOptions);

    StackExchange.using("externalEditor", function() {
    // Have to fire editor after snippets, if snippets enabled
    if (StackExchange.settings.snippets.snippetsEnabled) {
    StackExchange.using("snippets", function() {
    createEditor();
    });
    }
    else {
    createEditor();
    }
    });

    function createEditor() {
    StackExchange.prepareEditor({
    heartbeatType: 'answer',
    convertImagesToLinks: true,
    noModals: true,
    showLowRepImageUploadWarning: true,
    reputationToPostImages: 10,
    bindNavPrevention: true,
    postfix: "",
    imageUploader: {
    brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
    contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
    allowUrls: true
    },
    onDemand: true,
    discardSelector: ".discard-answer"
    ,immediatelyShowMarkdownHelp:true
    });


    }
    });














    draft saved

    draft discarded


















    StackExchange.ready(
    function () {
    StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53412922%2fhow-do-i-asynchronously-calculate-the-checksum-of-a-file-on-the-hard-disk-in-rus%23new-answer', 'question_page');
    }
    );

    Post as a guest















    Required, but never shown

























    2 Answers
    2






    active

    oldest

    votes








    2 Answers
    2






    active

    oldest

    votes









    active

    oldest

    votes






    active

    oldest

    votes








    up vote
    0
    down vote













    It depends on what checksum algorithm you want to use, but using the md5 crate as an example, you can compute the checksum on the fly. Something like this should do it:



    // When starting the file transfer
    let mut md5_context = md5::Context::new();

    // ...

    // as part of your existing processing for each block of data
    md5_context.consume (&block);

    // ...

    // once the last block has been processed
    return md5_context.compute();





    share|improve this answer



















    • 1




      This appears to be completely synchronous. The OP has explicitly requested an asynchronous solution.
      – Shepmaster
      Nov 21 at 15:11






    • 1




      @Shepmaster I don't see how you can tell whether this is synchronous or asynchronous: I only said that for each block of data he should call md5_context.consume, I never said that he should do it in a synchronous loop.
      – Jmb
      Nov 21 at 15:46








    • 1




      @Shepmaster edited the answer to make it clearer
      – Jmb
      Nov 21 at 15:50















    up vote
    0
    down vote













    It depends on what checksum algorithm you want to use, but using the md5 crate as an example, you can compute the checksum on the fly. Something like this should do it:



    // When starting the file transfer
    let mut md5_context = md5::Context::new();

    // ...

    // as part of your existing processing for each block of data
    md5_context.consume (&block);

    // ...

    // once the last block has been processed
    return md5_context.compute();





    share|improve this answer



















    • 1




      This appears to be completely synchronous. The OP has explicitly requested an asynchronous solution.
      – Shepmaster
      Nov 21 at 15:11






    • 1




      @Shepmaster I don't see how you can tell whether this is synchronous or asynchronous: I only said that for each block of data he should call md5_context.consume, I never said that he should do it in a synchronous loop.
      – Jmb
      Nov 21 at 15:46








    • 1




      @Shepmaster edited the answer to make it clearer
      – Jmb
      Nov 21 at 15:50













    up vote
    0
    down vote










    up vote
    0
    down vote









    It depends on what checksum algorithm you want to use, but using the md5 crate as an example, you can compute the checksum on the fly. Something like this should do it:



    // When starting the file transfer
    let mut md5_context = md5::Context::new();

    // ...

    // as part of your existing processing for each block of data
    md5_context.consume (&block);

    // ...

    // once the last block has been processed
    return md5_context.compute();





    share|improve this answer














    It depends on what checksum algorithm you want to use, but using the md5 crate as an example, you can compute the checksum on the fly. Something like this should do it:



    // When starting the file transfer
    let mut md5_context = md5::Context::new();

    // ...

    // as part of your existing processing for each block of data
    md5_context.consume (&block);

    // ...

    // once the last block has been processed
    return md5_context.compute();






    share|improve this answer














    share|improve this answer



    share|improve this answer








    edited Nov 21 at 15:49

























    answered Nov 21 at 15:04









    Jmb

    2,313521




    2,313521








    • 1




      This appears to be completely synchronous. The OP has explicitly requested an asynchronous solution.
      – Shepmaster
      Nov 21 at 15:11






    • 1




      @Shepmaster I don't see how you can tell whether this is synchronous or asynchronous: I only said that for each block of data he should call md5_context.consume, I never said that he should do it in a synchronous loop.
      – Jmb
      Nov 21 at 15:46








    • 1




      @Shepmaster edited the answer to make it clearer
      – Jmb
      Nov 21 at 15:50














    • 1




      This appears to be completely synchronous. The OP has explicitly requested an asynchronous solution.
      – Shepmaster
      Nov 21 at 15:11






    • 1




      @Shepmaster I don't see how you can tell whether this is synchronous or asynchronous: I only said that for each block of data he should call md5_context.consume, I never said that he should do it in a synchronous loop.
      – Jmb
      Nov 21 at 15:46








    • 1




      @Shepmaster edited the answer to make it clearer
      – Jmb
      Nov 21 at 15:50








    1




    1




    This appears to be completely synchronous. The OP has explicitly requested an asynchronous solution.
    – Shepmaster
    Nov 21 at 15:11




    This appears to be completely synchronous. The OP has explicitly requested an asynchronous solution.
    – Shepmaster
    Nov 21 at 15:11




    1




    1




    @Shepmaster I don't see how you can tell whether this is synchronous or asynchronous: I only said that for each block of data he should call md5_context.consume, I never said that he should do it in a synchronous loop.
    – Jmb
    Nov 21 at 15:46






    @Shepmaster I don't see how you can tell whether this is synchronous or asynchronous: I only said that for each block of data he should call md5_context.consume, I never said that he should do it in a synchronous loop.
    – Jmb
    Nov 21 at 15:46






    1




    1




    @Shepmaster edited the answer to make it clearer
    – Jmb
    Nov 21 at 15:50




    @Shepmaster edited the answer to make it clearer
    – Jmb
    Nov 21 at 15:50












    up vote
    0
    down vote



    accepted










    Actually, making simple hashing algorithms asynchronous is in such cases somewhat redundant, as long as one MD5 calculation takes less then 1 us (about 500 ns).






    share|improve this answer



























      up vote
      0
      down vote



      accepted










      Actually, making simple hashing algorithms asynchronous is in such cases somewhat redundant, as long as one MD5 calculation takes less then 1 us (about 500 ns).






      share|improve this answer

























        up vote
        0
        down vote



        accepted







        up vote
        0
        down vote



        accepted






        Actually, making simple hashing algorithms asynchronous is in such cases somewhat redundant, as long as one MD5 calculation takes less then 1 us (about 500 ns).






        share|improve this answer














        Actually, making simple hashing algorithms asynchronous is in such cases somewhat redundant, as long as one MD5 calculation takes less then 1 us (about 500 ns).







        share|improve this answer














        share|improve this answer



        share|improve this answer








        edited Nov 22 at 21:04

























        answered Nov 22 at 20:56









        hedgar2017

        3471723




        3471723






























            draft saved

            draft discarded




















































            Thanks for contributing an answer to Stack Overflow!


            • Please be sure to answer the question. Provide details and share your research!

            But avoid



            • Asking for help, clarification, or responding to other answers.

            • Making statements based on opinion; back them up with references or personal experience.


            To learn more, see our tips on writing great answers.





            Some of your past answers have not been well-received, and you're in danger of being blocked from answering.


            Please pay close attention to the following guidance:


            • Please be sure to answer the question. Provide details and share your research!

            But avoid



            • Asking for help, clarification, or responding to other answers.

            • Making statements based on opinion; back them up with references or personal experience.


            To learn more, see our tips on writing great answers.




            draft saved


            draft discarded














            StackExchange.ready(
            function () {
            StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53412922%2fhow-do-i-asynchronously-calculate-the-checksum-of-a-file-on-the-hard-disk-in-rus%23new-answer', 'question_page');
            }
            );

            Post as a guest















            Required, but never shown





















































            Required, but never shown














            Required, but never shown












            Required, but never shown







            Required, but never shown

































            Required, but never shown














            Required, but never shown












            Required, but never shown







            Required, but never shown







            Popular posts from this blog

            Berounka

            Sphinx de Gizeh

            Different font size/position of beamer's navigation symbols template's content depending on regular/plain...