Requesting size of Stream from consumer, before loading all the data
up vote
0
down vote
favorite
Hy all,
I have a problem with the camel component I am developing, where I'm not sure how to implement it in a way, that goes in line with the concepts of camel.
The producer I'm developing talks to the http api for our server, which is used to send messages with attachments.
Those attachments can potentially be very big, which is why the server expects the total filesize before any upload is done.
Currently the producer only accepts io.Files, nio.Paths and GenericFile, because there I can read the file size, before I upload the file.
Of course this is not a good way to do things, because it requires the (big) file to be available locally.
Connecting, for a example, a ftp server as the consumer would mean, that I have to download each file locally so I can upload it afterwards.
The obvious solution is using streams to access and upload the data, but with this I do not know how big the file is, before I'm done uploading, which is not an option, I need the size in advance.
My question now is, what are best practices to stream files through camel and also make the consumer give me the filesize in advance.
Greets
Chris
apache-camel
asked Nov 21 at 9:30
Chris
363
add a comment |
up vote
0
down vote
favorite
Hy all,
I have a problem with the camel component I am developing, where I'm not sure how to implement it in a way, that goes in line with the concepts of camel.
The producer I'm developing talks to the http api for our server, which is used to send messages with attachments.
Those attachments can potentially be very big, which is why the server expects the total filesize before any upload is done.
Currently the producer only accepts io.Files, nio.Paths and GenericFile, because there I can read the file size, before I upload the file.
Of course this is not a good way to do things, because it requires the (big) file to be available locally.
Connecting, for a example, a ftp server as the consumer would mean, that I have to download each file locally so I can upload it afterwards.
The obvious solution is using streams to access and upload the data, but with this I do not know how big the file is, before I'm done uploading, which is not an option, I need the size in advance.
My question now is, what are best practices to stream files through camel and also make the consumer give me the filesize in advance.
Greets
Chris
apache-camel
asked Nov 21 at 9:30
Chris
363
add a comment |
up vote
0
down vote
favorite
up vote
0
down vote
favorite
Hy all,
I have a problem with the camel component I am developing, where I'm not sure how to implement it in a way, that goes in line with the concepts of camel.
The producer I'm developing talks to the http api for our server, which is used to send messages with attachments.
Those attachments can potentially be very big, which is why the server expects the total filesize before any upload is done.
Currently the producer only accepts io.Files, nio.Paths and GenericFile, because there I can read the file size, before I upload the file.
Of course this is not a good way to do things, because it requires the (big) file to be available locally.
Connecting, for a example, a ftp server as the consumer would mean, that I have to download each file locally so I can upload it afterwards.
The obvious solution is using streams to access and upload the data, but with this I do not know how big the file is, before I'm done uploading, which is not an option, I need the size in advance.
My question now is, what are best practices to stream files through camel and also make the consumer give me the filesize in advance.
Greets
Chris
apache-camel
asked Nov 21 at 9:30
Chris
363
Hy all,
I have a problem with the camel component I am developing, where I'm not sure how to implement it in a way, that goes in line with the concepts of camel.
The producer I'm developing talks to the http api for our server, which is used to send messages with attachments.
Those attachments can potentially be very big, which is why the server expects the total filesize before any upload is done.
Currently the producer only accepts io.Files, nio.Paths and GenericFile, because there I can read the file size, before I upload the file.
Of course this is not a good way to do things, because it requires the (big) file to be available locally.
Connecting, for a example, a ftp server as the consumer would mean, that I have to download each file locally so I can upload it afterwards.
The obvious solution is using streams to access and upload the data, but with this I do not know how big the file is, before I'm done uploading, which is not an option, I need the size in advance.
My question now is, what are best practices to stream files through camel and also make the consumer give me the filesize in advance.
Greets
Chris
apache-camel
apache-camel
asked Nov 21 at 9:30
Chris
363
asked Nov 21 at 9:30
Chris
363
asked Nov 21 at 9:30
Chris
363
asked Nov 21 at 9:30
Chris
363
asked Nov 21 at 9:30
Chris
363
363
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
up vote
0
down vote
For File/FTP consumer, the exchange in header has a key CamelFileLength (Exchange.FILE_LENGTH) which return the file size in remote ftp server from consumer's scan result.
Unlike the file size obtain from local, the file size in key CamelFileLength might differ from actual file size your application received
- The ASCII mode will potentially change the linefeed when there is OS differ
- The file size might change between consumer scan action and consumer pick actionn
answered Nov 22 at 2:48
hk6279
1,00611024
So if I understand you correctly, this would result in two issues: 1. I can not rely on the CamelFileSize to be there at all 2. The file size in my application (which is the only one relevant to me) is basically wrong If so, this unfortunatelly does not help me with my problem =(
– Chris
Nov 26 at 7:48
1
@Chris The CamelFileSize is reliable if you accept delay receive like using readlock or receive single file (exist in folder at least minutes) in each batch.
– hk6279
Nov 27 at 2:24
Well as long as the CamelFileSize is the correct value, when I have actually read the whole file, this is fine. But all this is only the case for the file component, isn't it? I can't rely on this header, when I receive a file from a different component, or am I wrong here?
– Chris
yesterday
@Chris The answer's point 2 also apply to file component. As state in above comment, the headerCamelFileSizeis reliable under restricted condition.
– hk6279
23 hours ago
add a comment |
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
0
down vote
For File/FTP consumer, the exchange in header has a key CamelFileLength (Exchange.FILE_LENGTH) which return the file size in remote ftp server from consumer's scan result.
Unlike the file size obtain from local, the file size in key CamelFileLength might differ from actual file size your application received
- The ASCII mode will potentially change the linefeed when there is OS differ
- The file size might change between consumer scan action and consumer pick actionn
answered Nov 22 at 2:48
hk6279
1,00611024
So if I understand you correctly, this would result in two issues: 1. I can not rely on the CamelFileSize to be there at all 2. The file size in my application (which is the only one relevant to me) is basically wrong If so, this unfortunatelly does not help me with my problem =(
– Chris
Nov 26 at 7:48
1
@Chris The CamelFileSize is reliable if you accept delay receive like using readlock or receive single file (exist in folder at least minutes) in each batch.
– hk6279
Nov 27 at 2:24
Well as long as the CamelFileSize is the correct value, when I have actually read the whole file, this is fine. But all this is only the case for the file component, isn't it? I can't rely on this header, when I receive a file from a different component, or am I wrong here?
– Chris
yesterday
@Chris The answer's point 2 also apply to file component. As state in above comment, the headerCamelFileSizeis reliable under restricted condition.
– hk6279
23 hours ago
add a comment |
up vote
0
down vote
For File/FTP consumer, the exchange in header has a key CamelFileLength (Exchange.FILE_LENGTH) which return the file size in remote ftp server from consumer's scan result.
Unlike the file size obtain from local, the file size in key CamelFileLength might differ from actual file size your application received
- The ASCII mode will potentially change the linefeed when there is OS differ
- The file size might change between consumer scan action and consumer pick actionn
answered Nov 22 at 2:48
hk6279
1,00611024
So if I understand you correctly, this would result in two issues: 1. I can not rely on the CamelFileSize to be there at all 2. The file size in my application (which is the only one relevant to me) is basically wrong If so, this unfortunatelly does not help me with my problem =(
– Chris
Nov 26 at 7:48
1
@Chris The CamelFileSize is reliable if you accept delay receive like using readlock or receive single file (exist in folder at least minutes) in each batch.
– hk6279
Nov 27 at 2:24
Well as long as the CamelFileSize is the correct value, when I have actually read the whole file, this is fine. But all this is only the case for the file component, isn't it? I can't rely on this header, when I receive a file from a different component, or am I wrong here?
– Chris
yesterday
@Chris The answer's point 2 also apply to file component. As state in above comment, the headerCamelFileSizeis reliable under restricted condition.
– hk6279
23 hours ago
add a comment |
up vote
0
down vote
up vote
0
down vote
For File/FTP consumer, the exchange in header has a key CamelFileLength (Exchange.FILE_LENGTH) which return the file size in remote ftp server from consumer's scan result.
Unlike the file size obtain from local, the file size in key CamelFileLength might differ from actual file size your application received
- The ASCII mode will potentially change the linefeed when there is OS differ
- The file size might change between consumer scan action and consumer pick actionn
answered Nov 22 at 2:48
hk6279
1,00611024
For File/FTP consumer, the exchange in header has a key CamelFileLength (Exchange.FILE_LENGTH) which return the file size in remote ftp server from consumer's scan result.
Unlike the file size obtain from local, the file size in key CamelFileLength might differ from actual file size your application received
- The ASCII mode will potentially change the linefeed when there is OS differ
- The file size might change between consumer scan action and consumer pick actionn
answered Nov 22 at 2:48
hk6279
1,00611024
answered Nov 22 at 2:48
hk6279
1,00611024
answered Nov 22 at 2:48
hk6279
1,00611024
answered Nov 22 at 2:48
hk6279
1,00611024
1,00611024
So if I understand you correctly, this would result in two issues: 1. I can not rely on the CamelFileSize to be there at all 2. The file size in my application (which is the only one relevant to me) is basically wrong If so, this unfortunatelly does not help me with my problem =(
– Chris
Nov 26 at 7:48
1
@Chris The CamelFileSize is reliable if you accept delay receive like using readlock or receive single file (exist in folder at least minutes) in each batch.
– hk6279
Nov 27 at 2:24
Well as long as the CamelFileSize is the correct value, when I have actually read the whole file, this is fine. But all this is only the case for the file component, isn't it? I can't rely on this header, when I receive a file from a different component, or am I wrong here?
– Chris
yesterday
@Chris The answer's point 2 also apply to file component. As state in above comment, the headerCamelFileSizeis reliable under restricted condition.
– hk6279
23 hours ago
add a comment |
So if I understand you correctly, this would result in two issues: 1. I can not rely on the CamelFileSize to be there at all 2. The file size in my application (which is the only one relevant to me) is basically wrong If so, this unfortunatelly does not help me with my problem =(
– Chris
Nov 26 at 7:48
1
@Chris The CamelFileSize is reliable if you accept delay receive like using readlock or receive single file (exist in folder at least minutes) in each batch.
– hk6279
Nov 27 at 2:24
Well as long as the CamelFileSize is the correct value, when I have actually read the whole file, this is fine. But all this is only the case for the file component, isn't it? I can't rely on this header, when I receive a file from a different component, or am I wrong here?
– Chris
yesterday
@Chris The answer's point 2 also apply to file component. As state in above comment, the headerCamelFileSizeis reliable under restricted condition.
– hk6279
23 hours ago
So if I understand you correctly, this would result in two issues: 1. I can not rely on the CamelFileSize to be there at all 2. The file size in my application (which is the only one relevant to me) is basically wrong If so, this unfortunatelly does not help me with my problem =(
– Chris
Nov 26 at 7:48
So if I understand you correctly, this would result in two issues: 1. I can not rely on the CamelFileSize to be there at all 2. The file size in my application (which is the only one relevant to me) is basically wrong If so, this unfortunatelly does not help me with my problem =(
– Chris
Nov 26 at 7:48
1
1
@Chris The CamelFileSize is reliable if you accept delay receive like using readlock or receive single file (exist in folder at least minutes) in each batch.
– hk6279
Nov 27 at 2:24
@Chris The CamelFileSize is reliable if you accept delay receive like using readlock or receive single file (exist in folder at least minutes) in each batch.
– hk6279
Nov 27 at 2:24
Well as long as the CamelFileSize is the correct value, when I have actually read the whole file, this is fine. But all this is only the case for the file component, isn't it? I can't rely on this header, when I receive a file from a different component, or am I wrong here?
– Chris
yesterday
Well as long as the CamelFileSize is the correct value, when I have actually read the whole file, this is fine. But all this is only the case for the file component, isn't it? I can't rely on this header, when I receive a file from a different component, or am I wrong here?
– Chris
yesterday
@Chris The answer's point 2 also apply to file component. As state in above comment, the header
CamelFileSize is reliable under restricted condition.– hk6279
23 hours ago
@Chris The answer's point 2 also apply to file component. As state in above comment, the header
CamelFileSize is reliable under restricted condition.– hk6279
23 hours ago
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53408933%2frequesting-size-of-stream-from-consumer-before-loading-all-the-data%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
