RxJs Observable make requests in parallel until it fails
I want to reach an API that returns data in pages of 50 items, but I dont know how many items (and therefore pages) there are.
My idea is to send 20 requests in parallel, each one will request the ith page and then the ith+20 page, and so on, until a page returns blank, in which case I end.
With this approach I would do at most 20 unnecessary requests.
The thing is, I don't know how to structure this loop using observables.
I imagined something like this:
return Observable.from(_.range(0, 20))
.map((pageNo) => fetchPage(pageNo))
.while((page) => isValid(page));
but this while method or similars dont exist/work
I found this similar question but he uses interval, which seems inefficient RxJs Observable interval until reached desired value
From my understanding, I can't use takeWhilebecause it validates the condition already met, and not a response of the request still to be made.
javascript node.js typescript rxjs observer-pattern
asked Nov 22 at 16:21
luisforque
508
add a comment |
I want to reach an API that returns data in pages of 50 items, but I dont know how many items (and therefore pages) there are.
My idea is to send 20 requests in parallel, each one will request the ith page and then the ith+20 page, and so on, until a page returns blank, in which case I end.
With this approach I would do at most 20 unnecessary requests.
The thing is, I don't know how to structure this loop using observables.
I imagined something like this:
return Observable.from(_.range(0, 20))
.map((pageNo) => fetchPage(pageNo))
.while((page) => isValid(page));
but this while method or similars dont exist/work
I found this similar question but he uses interval, which seems inefficient RxJs Observable interval until reached desired value
From my understanding, I can't use takeWhilebecause it validates the condition already met, and not a response of the request still to be made.
javascript node.js typescript rxjs observer-pattern
asked Nov 22 at 16:21
luisforque
508
add a comment |
I want to reach an API that returns data in pages of 50 items, but I dont know how many items (and therefore pages) there are.
My idea is to send 20 requests in parallel, each one will request the ith page and then the ith+20 page, and so on, until a page returns blank, in which case I end.
With this approach I would do at most 20 unnecessary requests.
The thing is, I don't know how to structure this loop using observables.
I imagined something like this:
return Observable.from(_.range(0, 20))
.map((pageNo) => fetchPage(pageNo))
.while((page) => isValid(page));
but this while method or similars dont exist/work
I found this similar question but he uses interval, which seems inefficient RxJs Observable interval until reached desired value
From my understanding, I can't use takeWhilebecause it validates the condition already met, and not a response of the request still to be made.
javascript node.js typescript rxjs observer-pattern
asked Nov 22 at 16:21
luisforque
508
I want to reach an API that returns data in pages of 50 items, but I dont know how many items (and therefore pages) there are.
My idea is to send 20 requests in parallel, each one will request the ith page and then the ith+20 page, and so on, until a page returns blank, in which case I end.
With this approach I would do at most 20 unnecessary requests.
The thing is, I don't know how to structure this loop using observables.
I imagined something like this:
return Observable.from(_.range(0, 20))
.map((pageNo) => fetchPage(pageNo))
.while((page) => isValid(page));
but this while method or similars dont exist/work
I found this similar question but he uses interval, which seems inefficient RxJs Observable interval until reached desired value
From my understanding, I can't use takeWhilebecause it validates the condition already met, and not a response of the request still to be made.
javascript node.js typescript rxjs observer-pattern
javascript node.js typescript rxjs observer-pattern
asked Nov 22 at 16:21
luisforque
508
asked Nov 22 at 16:21
luisforque
508
edited Nov 22 at 16:37
asked Nov 22 at 16:21
luisforque
508
asked Nov 22 at 16:21
luisforque
508
asked Nov 22 at 16:21
luisforque
508
508
add a comment |
add a comment |
2 Answers
2
active
oldest
votes
This might help
return Observable.from(_.range(0, 20)).pipe(
mergeMap(pageNo => ajax.getJSON(`/api/fetchPage/${pageNo}`).pipe(
mergeMap(result =>
of(addPersonFulfilled(result), secondFunc(foo)),
retryWhen(error => tap(console.log('error on page', error)))
)
))
)
answered Nov 22 at 17:03
Steven McConnon
930717
can you explain what this does?
– luisforque
Nov 22 at 18:34
add a comment |
You can create 20 requests and wait for all of them to complete with forkJoin and then use takeWhile to complete the chain when the array of results is empty:
const fetchPage = page => {
...
return forkJoin(...);
};
range(0, 20).pipe(
concatMap(page => fetchPage(page)),
takeWhile(arr => arr.length > 0),
)
.subscribe(console.log);
Complete demo: https://stackblitz.com/edit/rxjs-zw1sr2?devtoolsheight=60
answered Nov 22 at 17:07
martin
41.6k1183124
Nice, thank you. I wasn't familiar with this pipe function.
– luisforque
Nov 22 at 17:19
after further examination of your answer, I think you misunderstood my question. Your code fires 20 fetchPage, and each fetches (currently logs) 20 times. The stopping clause should be on the second loop, not the first.
– luisforque
Nov 22 at 18:33
You mean inside fetchPage()?
– martin
Nov 22 at 18:42
the 20 in the range is correct, the 20 inside the for loop needs to be something that takes in consideration the response from the request, i.e. if there is data returning in that request. This is where the stopping condition should be (from my way of thinking how to solve the problem, unless you are suggesting something different)
– luisforque
Nov 22 at 18:46
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53434933%2frxjs-observable-make-requests-in-parallel-until-it-fails%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
This might help
return Observable.from(_.range(0, 20)).pipe(
mergeMap(pageNo => ajax.getJSON(`/api/fetchPage/${pageNo}`).pipe(
mergeMap(result =>
of(addPersonFulfilled(result), secondFunc(foo)),
retryWhen(error => tap(console.log('error on page', error)))
)
))
)
answered Nov 22 at 17:03
Steven McConnon
930717
can you explain what this does?
– luisforque
Nov 22 at 18:34
add a comment |
This might help
return Observable.from(_.range(0, 20)).pipe(
mergeMap(pageNo => ajax.getJSON(`/api/fetchPage/${pageNo}`).pipe(
mergeMap(result =>
of(addPersonFulfilled(result), secondFunc(foo)),
retryWhen(error => tap(console.log('error on page', error)))
)
))
)
answered Nov 22 at 17:03
Steven McConnon
930717
can you explain what this does?
– luisforque
Nov 22 at 18:34
add a comment |
This might help
return Observable.from(_.range(0, 20)).pipe(
mergeMap(pageNo => ajax.getJSON(`/api/fetchPage/${pageNo}`).pipe(
mergeMap(result =>
of(addPersonFulfilled(result), secondFunc(foo)),
retryWhen(error => tap(console.log('error on page', error)))
)
))
)
answered Nov 22 at 17:03
Steven McConnon
930717
This might help
return Observable.from(_.range(0, 20)).pipe(
mergeMap(pageNo => ajax.getJSON(`/api/fetchPage/${pageNo}`).pipe(
mergeMap(result =>
of(addPersonFulfilled(result), secondFunc(foo)),
retryWhen(error => tap(console.log('error on page', error)))
)
))
)
answered Nov 22 at 17:03
Steven McConnon
930717
answered Nov 22 at 17:03
Steven McConnon
930717
answered Nov 22 at 17:03
Steven McConnon
930717
answered Nov 22 at 17:03
Steven McConnon
930717
930717
can you explain what this does?
– luisforque
Nov 22 at 18:34
add a comment |
can you explain what this does?
– luisforque
Nov 22 at 18:34
can you explain what this does?
– luisforque
Nov 22 at 18:34
can you explain what this does?
– luisforque
Nov 22 at 18:34
add a comment |
You can create 20 requests and wait for all of them to complete with forkJoin and then use takeWhile to complete the chain when the array of results is empty:
const fetchPage = page => {
...
return forkJoin(...);
};
range(0, 20).pipe(
concatMap(page => fetchPage(page)),
takeWhile(arr => arr.length > 0),
)
.subscribe(console.log);
Complete demo: https://stackblitz.com/edit/rxjs-zw1sr2?devtoolsheight=60
answered Nov 22 at 17:07
martin
41.6k1183124
Nice, thank you. I wasn't familiar with this pipe function.
– luisforque
Nov 22 at 17:19
after further examination of your answer, I think you misunderstood my question. Your code fires 20 fetchPage, and each fetches (currently logs) 20 times. The stopping clause should be on the second loop, not the first.
– luisforque
Nov 22 at 18:33
You mean inside fetchPage()?
– martin
Nov 22 at 18:42
the 20 in the range is correct, the 20 inside the for loop needs to be something that takes in consideration the response from the request, i.e. if there is data returning in that request. This is where the stopping condition should be (from my way of thinking how to solve the problem, unless you are suggesting something different)
– luisforque
Nov 22 at 18:46
add a comment |
You can create 20 requests and wait for all of them to complete with forkJoin and then use takeWhile to complete the chain when the array of results is empty:
const fetchPage = page => {
...
return forkJoin(...);
};
range(0, 20).pipe(
concatMap(page => fetchPage(page)),
takeWhile(arr => arr.length > 0),
)
.subscribe(console.log);
Complete demo: https://stackblitz.com/edit/rxjs-zw1sr2?devtoolsheight=60
answered Nov 22 at 17:07
martin
41.6k1183124
Nice, thank you. I wasn't familiar with this pipe function.
– luisforque
Nov 22 at 17:19
after further examination of your answer, I think you misunderstood my question. Your code fires 20 fetchPage, and each fetches (currently logs) 20 times. The stopping clause should be on the second loop, not the first.
– luisforque
Nov 22 at 18:33
You mean inside fetchPage()?
– martin
Nov 22 at 18:42
the 20 in the range is correct, the 20 inside the for loop needs to be something that takes in consideration the response from the request, i.e. if there is data returning in that request. This is where the stopping condition should be (from my way of thinking how to solve the problem, unless you are suggesting something different)
– luisforque
Nov 22 at 18:46
add a comment |
You can create 20 requests and wait for all of them to complete with forkJoin and then use takeWhile to complete the chain when the array of results is empty:
const fetchPage = page => {
...
return forkJoin(...);
};
range(0, 20).pipe(
concatMap(page => fetchPage(page)),
takeWhile(arr => arr.length > 0),
)
.subscribe(console.log);
Complete demo: https://stackblitz.com/edit/rxjs-zw1sr2?devtoolsheight=60
answered Nov 22 at 17:07
martin
41.6k1183124
You can create 20 requests and wait for all of them to complete with forkJoin and then use takeWhile to complete the chain when the array of results is empty:
const fetchPage = page => {
...
return forkJoin(...);
};
range(0, 20).pipe(
concatMap(page => fetchPage(page)),
takeWhile(arr => arr.length > 0),
)
.subscribe(console.log);
Complete demo: https://stackblitz.com/edit/rxjs-zw1sr2?devtoolsheight=60
answered Nov 22 at 17:07
martin
41.6k1183124
answered Nov 22 at 17:07
martin
41.6k1183124
answered Nov 22 at 17:07
martin
41.6k1183124
answered Nov 22 at 17:07
martin
41.6k1183124
41.6k1183124
Nice, thank you. I wasn't familiar with this pipe function.
– luisforque
Nov 22 at 17:19
after further examination of your answer, I think you misunderstood my question. Your code fires 20 fetchPage, and each fetches (currently logs) 20 times. The stopping clause should be on the second loop, not the first.
– luisforque
Nov 22 at 18:33
You mean inside fetchPage()?
– martin
Nov 22 at 18:42
the 20 in the range is correct, the 20 inside the for loop needs to be something that takes in consideration the response from the request, i.e. if there is data returning in that request. This is where the stopping condition should be (from my way of thinking how to solve the problem, unless you are suggesting something different)
– luisforque
Nov 22 at 18:46
add a comment |
Nice, thank you. I wasn't familiar with this pipe function.
– luisforque
Nov 22 at 17:19
after further examination of your answer, I think you misunderstood my question. Your code fires 20 fetchPage, and each fetches (currently logs) 20 times. The stopping clause should be on the second loop, not the first.
– luisforque
Nov 22 at 18:33
You mean inside fetchPage()?
– martin
Nov 22 at 18:42
the 20 in the range is correct, the 20 inside the for loop needs to be something that takes in consideration the response from the request, i.e. if there is data returning in that request. This is where the stopping condition should be (from my way of thinking how to solve the problem, unless you are suggesting something different)
– luisforque
Nov 22 at 18:46
Nice, thank you. I wasn't familiar with this pipe function.
– luisforque
Nov 22 at 17:19
Nice, thank you. I wasn't familiar with this pipe function.
– luisforque
Nov 22 at 17:19
after further examination of your answer, I think you misunderstood my question. Your code fires 20 fetchPage, and each fetches (currently logs) 20 times. The stopping clause should be on the second loop, not the first.
– luisforque
Nov 22 at 18:33
after further examination of your answer, I think you misunderstood my question. Your code fires 20 fetchPage, and each fetches (currently logs) 20 times. The stopping clause should be on the second loop, not the first.
– luisforque
Nov 22 at 18:33
You mean inside fetchPage()?
– martin
Nov 22 at 18:42
You mean inside fetchPage()?
– martin
Nov 22 at 18:42
the 20 in the range is correct, the 20 inside the for loop needs to be something that takes in consideration the response from the request, i.e. if there is data returning in that request. This is where the stopping condition should be (from my way of thinking how to solve the problem, unless you are suggesting something different)
– luisforque
Nov 22 at 18:46
the 20 in the range is correct, the 20 inside the for loop needs to be something that takes in consideration the response from the request, i.e. if there is data returning in that request. This is where the stopping condition should be (from my way of thinking how to solve the problem, unless you are suggesting something different)
– luisforque
Nov 22 at 18:46
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53434933%2frxjs-observable-make-requests-in-parallel-until-it-fails%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
