How to use Bash to create arrays with values from the same line of many files?
I have a number of files (in the same folder) all with the same number of lines:
a.txt
20
3
10
15
15
b.txt
19
4
5
8
8
c.txt
2
4
9
21
5
Using Bash, I'd like to create an array of arrays that contain the value of each line in every file. So, line 1 from a.txt
, b.txt
, and c.txt
. The same for lines 2 to 5, so that in the end it looks like:
[
[20, 19, 2],
[3, 4, 4],
...
[15, 8, 5]
]
Note: I messed up the formatting and wording. I've changed this now.
I'm actually using jq
to get these lists in the first place, as they're originally specific values within a JSON file I download every X minutes. I used jq
to get the values I needed into different files as I thought that would get me further, but now I'm not sure that was the way to go. If it helps, here is the original JSON file I download and start with.
I've looked at various questions that somewhat deal with this:
- Creating an array from a text file in Bash
- Bash Script to create a JSON file
- JQ create json array using bash
Among others. But none of these deal with taking the value of the same line from various files. I don't know Bash well enough to do this and any help is greatly appreciated.
json bash jq multiple-files
add a comment |
I have a number of files (in the same folder) all with the same number of lines:
a.txt
20
3
10
15
15
b.txt
19
4
5
8
8
c.txt
2
4
9
21
5
Using Bash, I'd like to create an array of arrays that contain the value of each line in every file. So, line 1 from a.txt
, b.txt
, and c.txt
. The same for lines 2 to 5, so that in the end it looks like:
[
[20, 19, 2],
[3, 4, 4],
...
[15, 8, 5]
]
Note: I messed up the formatting and wording. I've changed this now.
I'm actually using jq
to get these lists in the first place, as they're originally specific values within a JSON file I download every X minutes. I used jq
to get the values I needed into different files as I thought that would get me further, but now I'm not sure that was the way to go. If it helps, here is the original JSON file I download and start with.
I've looked at various questions that somewhat deal with this:
- Creating an array from a text file in Bash
- Bash Script to create a JSON file
- JQ create json array using bash
Among others. But none of these deal with taking the value of the same line from various files. I don't know Bash well enough to do this and any help is greatly appreciated.
json bash jq multiple-files
add a comment |
I have a number of files (in the same folder) all with the same number of lines:
a.txt
20
3
10
15
15
b.txt
19
4
5
8
8
c.txt
2
4
9
21
5
Using Bash, I'd like to create an array of arrays that contain the value of each line in every file. So, line 1 from a.txt
, b.txt
, and c.txt
. The same for lines 2 to 5, so that in the end it looks like:
[
[20, 19, 2],
[3, 4, 4],
...
[15, 8, 5]
]
Note: I messed up the formatting and wording. I've changed this now.
I'm actually using jq
to get these lists in the first place, as they're originally specific values within a JSON file I download every X minutes. I used jq
to get the values I needed into different files as I thought that would get me further, but now I'm not sure that was the way to go. If it helps, here is the original JSON file I download and start with.
I've looked at various questions that somewhat deal with this:
- Creating an array from a text file in Bash
- Bash Script to create a JSON file
- JQ create json array using bash
Among others. But none of these deal with taking the value of the same line from various files. I don't know Bash well enough to do this and any help is greatly appreciated.
json bash jq multiple-files
I have a number of files (in the same folder) all with the same number of lines:
a.txt
20
3
10
15
15
b.txt
19
4
5
8
8
c.txt
2
4
9
21
5
Using Bash, I'd like to create an array of arrays that contain the value of each line in every file. So, line 1 from a.txt
, b.txt
, and c.txt
. The same for lines 2 to 5, so that in the end it looks like:
[
[20, 19, 2],
[3, 4, 4],
...
[15, 8, 5]
]
Note: I messed up the formatting and wording. I've changed this now.
I'm actually using jq
to get these lists in the first place, as they're originally specific values within a JSON file I download every X minutes. I used jq
to get the values I needed into different files as I thought that would get me further, but now I'm not sure that was the way to go. If it helps, here is the original JSON file I download and start with.
I've looked at various questions that somewhat deal with this:
- Creating an array from a text file in Bash
- Bash Script to create a JSON file
- JQ create json array using bash
Among others. But none of these deal with taking the value of the same line from various files. I don't know Bash well enough to do this and any help is greatly appreciated.
json bash jq multiple-files
json bash jq multiple-files
edited Nov 23 '18 at 9:46
asked Nov 23 '18 at 1:10
duxk.gh
12916
12916
add a comment |
add a comment |
3 Answers
3
active
oldest
votes
Use paste
to join the files, then read the input as raw text, splitting on the tabs inserted by paste
:
$ paste a.txt b.txt c.txt | jq -Rc 'split("t") | map(tonumber)'
[20,19,2]
[3,4,4]
[10,5,9]
[15,8,21]
[15,8,5]
If you want to gather the entire result into a single array, pipe it into another instance of jq
in slurp mode. (There's probably a way to do it with a single invocation of jq
, but this seems simpler.)
$ paste a.txt b.txt c.txt | jq -R 'split("t") | map(tonumber)' | jq -sc
[[20,19,2],[3,4,4],[10,5,9],[15,8,21],[15,8,5]]
Thanks a lot, bothpaste
and thejq
command were extremely useful
– duxk.gh
Nov 23 '18 at 13:03
add a comment |
Here’s one approach:
$ jq -c -n '[$a,$b,$c] | transpose' --slurpfile a a.txt --slurpfile b b.txt --slurpfile c c.txt
Generalization to an arbitrary number of files
In the following, we'll assume that the files to be processed can be specified by *.txt in the current directory:
jq -n -c '
[reduce inputs as $i ({}; .[input_filename] += [$i]) | .]
| transpose' *.txt
add a comment |
I could not come up with a simple way, but here's one I got to do this.
1. Join files and create CSV-like file
If your machine have join
, you can create joined records from two files (like join
command in SQL).
To do this, make sure your file is sorted.
The easiest way I think is just numbering each lines. This works as Primary ID in SQL.
$ cat a.txt | nl > a.txt.nl
$ cat b.txt | nl > b.txt.nl
$ cat c.txt | nl > c.txt.nl
Now you can join
sorted files into one. Note that join
can join only two files at once. This is why I piped output to next join
.
$ join a.txt.nl b.txt.nl | join - c.txt.nl > conc.txt
now conc.txt
is:
1 20 19 2
2 3 4 4
3 10 5 9
4 15 8 21
5 15 8 5
2. Create json from the CSV-like file
It seems little complicated.
jq -Rsn '
[inputs
| . / "n"
| (. | select((. | length) > 0) | . / " ") as $input
| [$input[1], $input[2], $input[3] ] ]
' <conc.txt
Actually I do not know detailed syntex or usage of jq
, it seems like doing:
- split input file by
n
- split a given line by space, then select valid data
- put splitted records in appropriate location by their index
I used this question as a reference:
https://stackoverflow.com/a/44781106/10675437
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53439594%2fhow-to-use-bash-to-create-arrays-with-values-from-the-same-line-of-many-files%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
3 Answers
3
active
oldest
votes
3 Answers
3
active
oldest
votes
active
oldest
votes
active
oldest
votes
Use paste
to join the files, then read the input as raw text, splitting on the tabs inserted by paste
:
$ paste a.txt b.txt c.txt | jq -Rc 'split("t") | map(tonumber)'
[20,19,2]
[3,4,4]
[10,5,9]
[15,8,21]
[15,8,5]
If you want to gather the entire result into a single array, pipe it into another instance of jq
in slurp mode. (There's probably a way to do it with a single invocation of jq
, but this seems simpler.)
$ paste a.txt b.txt c.txt | jq -R 'split("t") | map(tonumber)' | jq -sc
[[20,19,2],[3,4,4],[10,5,9],[15,8,21],[15,8,5]]
Thanks a lot, bothpaste
and thejq
command were extremely useful
– duxk.gh
Nov 23 '18 at 13:03
add a comment |
Use paste
to join the files, then read the input as raw text, splitting on the tabs inserted by paste
:
$ paste a.txt b.txt c.txt | jq -Rc 'split("t") | map(tonumber)'
[20,19,2]
[3,4,4]
[10,5,9]
[15,8,21]
[15,8,5]
If you want to gather the entire result into a single array, pipe it into another instance of jq
in slurp mode. (There's probably a way to do it with a single invocation of jq
, but this seems simpler.)
$ paste a.txt b.txt c.txt | jq -R 'split("t") | map(tonumber)' | jq -sc
[[20,19,2],[3,4,4],[10,5,9],[15,8,21],[15,8,5]]
Thanks a lot, bothpaste
and thejq
command were extremely useful
– duxk.gh
Nov 23 '18 at 13:03
add a comment |
Use paste
to join the files, then read the input as raw text, splitting on the tabs inserted by paste
:
$ paste a.txt b.txt c.txt | jq -Rc 'split("t") | map(tonumber)'
[20,19,2]
[3,4,4]
[10,5,9]
[15,8,21]
[15,8,5]
If you want to gather the entire result into a single array, pipe it into another instance of jq
in slurp mode. (There's probably a way to do it with a single invocation of jq
, but this seems simpler.)
$ paste a.txt b.txt c.txt | jq -R 'split("t") | map(tonumber)' | jq -sc
[[20,19,2],[3,4,4],[10,5,9],[15,8,21],[15,8,5]]
Use paste
to join the files, then read the input as raw text, splitting on the tabs inserted by paste
:
$ paste a.txt b.txt c.txt | jq -Rc 'split("t") | map(tonumber)'
[20,19,2]
[3,4,4]
[10,5,9]
[15,8,21]
[15,8,5]
If you want to gather the entire result into a single array, pipe it into another instance of jq
in slurp mode. (There's probably a way to do it with a single invocation of jq
, but this seems simpler.)
$ paste a.txt b.txt c.txt | jq -R 'split("t") | map(tonumber)' | jq -sc
[[20,19,2],[3,4,4],[10,5,9],[15,8,21],[15,8,5]]
answered Nov 23 '18 at 3:21
chepner
245k32232323
245k32232323
Thanks a lot, bothpaste
and thejq
command were extremely useful
– duxk.gh
Nov 23 '18 at 13:03
add a comment |
Thanks a lot, bothpaste
and thejq
command were extremely useful
– duxk.gh
Nov 23 '18 at 13:03
Thanks a lot, both
paste
and the jq
command were extremely useful– duxk.gh
Nov 23 '18 at 13:03
Thanks a lot, both
paste
and the jq
command were extremely useful– duxk.gh
Nov 23 '18 at 13:03
add a comment |
Here’s one approach:
$ jq -c -n '[$a,$b,$c] | transpose' --slurpfile a a.txt --slurpfile b b.txt --slurpfile c c.txt
Generalization to an arbitrary number of files
In the following, we'll assume that the files to be processed can be specified by *.txt in the current directory:
jq -n -c '
[reduce inputs as $i ({}; .[input_filename] += [$i]) | .]
| transpose' *.txt
add a comment |
Here’s one approach:
$ jq -c -n '[$a,$b,$c] | transpose' --slurpfile a a.txt --slurpfile b b.txt --slurpfile c c.txt
Generalization to an arbitrary number of files
In the following, we'll assume that the files to be processed can be specified by *.txt in the current directory:
jq -n -c '
[reduce inputs as $i ({}; .[input_filename] += [$i]) | .]
| transpose' *.txt
add a comment |
Here’s one approach:
$ jq -c -n '[$a,$b,$c] | transpose' --slurpfile a a.txt --slurpfile b b.txt --slurpfile c c.txt
Generalization to an arbitrary number of files
In the following, we'll assume that the files to be processed can be specified by *.txt in the current directory:
jq -n -c '
[reduce inputs as $i ({}; .[input_filename] += [$i]) | .]
| transpose' *.txt
Here’s one approach:
$ jq -c -n '[$a,$b,$c] | transpose' --slurpfile a a.txt --slurpfile b b.txt --slurpfile c c.txt
Generalization to an arbitrary number of files
In the following, we'll assume that the files to be processed can be specified by *.txt in the current directory:
jq -n -c '
[reduce inputs as $i ({}; .[input_filename] += [$i]) | .]
| transpose' *.txt
edited Nov 23 '18 at 10:35
answered Nov 23 '18 at 3:06
peak
30.4k83956
30.4k83956
add a comment |
add a comment |
I could not come up with a simple way, but here's one I got to do this.
1. Join files and create CSV-like file
If your machine have join
, you can create joined records from two files (like join
command in SQL).
To do this, make sure your file is sorted.
The easiest way I think is just numbering each lines. This works as Primary ID in SQL.
$ cat a.txt | nl > a.txt.nl
$ cat b.txt | nl > b.txt.nl
$ cat c.txt | nl > c.txt.nl
Now you can join
sorted files into one. Note that join
can join only two files at once. This is why I piped output to next join
.
$ join a.txt.nl b.txt.nl | join - c.txt.nl > conc.txt
now conc.txt
is:
1 20 19 2
2 3 4 4
3 10 5 9
4 15 8 21
5 15 8 5
2. Create json from the CSV-like file
It seems little complicated.
jq -Rsn '
[inputs
| . / "n"
| (. | select((. | length) > 0) | . / " ") as $input
| [$input[1], $input[2], $input[3] ] ]
' <conc.txt
Actually I do not know detailed syntex or usage of jq
, it seems like doing:
- split input file by
n
- split a given line by space, then select valid data
- put splitted records in appropriate location by their index
I used this question as a reference:
https://stackoverflow.com/a/44781106/10675437
add a comment |
I could not come up with a simple way, but here's one I got to do this.
1. Join files and create CSV-like file
If your machine have join
, you can create joined records from two files (like join
command in SQL).
To do this, make sure your file is sorted.
The easiest way I think is just numbering each lines. This works as Primary ID in SQL.
$ cat a.txt | nl > a.txt.nl
$ cat b.txt | nl > b.txt.nl
$ cat c.txt | nl > c.txt.nl
Now you can join
sorted files into one. Note that join
can join only two files at once. This is why I piped output to next join
.
$ join a.txt.nl b.txt.nl | join - c.txt.nl > conc.txt
now conc.txt
is:
1 20 19 2
2 3 4 4
3 10 5 9
4 15 8 21
5 15 8 5
2. Create json from the CSV-like file
It seems little complicated.
jq -Rsn '
[inputs
| . / "n"
| (. | select((. | length) > 0) | . / " ") as $input
| [$input[1], $input[2], $input[3] ] ]
' <conc.txt
Actually I do not know detailed syntex or usage of jq
, it seems like doing:
- split input file by
n
- split a given line by space, then select valid data
- put splitted records in appropriate location by their index
I used this question as a reference:
https://stackoverflow.com/a/44781106/10675437
add a comment |
I could not come up with a simple way, but here's one I got to do this.
1. Join files and create CSV-like file
If your machine have join
, you can create joined records from two files (like join
command in SQL).
To do this, make sure your file is sorted.
The easiest way I think is just numbering each lines. This works as Primary ID in SQL.
$ cat a.txt | nl > a.txt.nl
$ cat b.txt | nl > b.txt.nl
$ cat c.txt | nl > c.txt.nl
Now you can join
sorted files into one. Note that join
can join only two files at once. This is why I piped output to next join
.
$ join a.txt.nl b.txt.nl | join - c.txt.nl > conc.txt
now conc.txt
is:
1 20 19 2
2 3 4 4
3 10 5 9
4 15 8 21
5 15 8 5
2. Create json from the CSV-like file
It seems little complicated.
jq -Rsn '
[inputs
| . / "n"
| (. | select((. | length) > 0) | . / " ") as $input
| [$input[1], $input[2], $input[3] ] ]
' <conc.txt
Actually I do not know detailed syntex or usage of jq
, it seems like doing:
- split input file by
n
- split a given line by space, then select valid data
- put splitted records in appropriate location by their index
I used this question as a reference:
https://stackoverflow.com/a/44781106/10675437
I could not come up with a simple way, but here's one I got to do this.
1. Join files and create CSV-like file
If your machine have join
, you can create joined records from two files (like join
command in SQL).
To do this, make sure your file is sorted.
The easiest way I think is just numbering each lines. This works as Primary ID in SQL.
$ cat a.txt | nl > a.txt.nl
$ cat b.txt | nl > b.txt.nl
$ cat c.txt | nl > c.txt.nl
Now you can join
sorted files into one. Note that join
can join only two files at once. This is why I piped output to next join
.
$ join a.txt.nl b.txt.nl | join - c.txt.nl > conc.txt
now conc.txt
is:
1 20 19 2
2 3 4 4
3 10 5 9
4 15 8 21
5 15 8 5
2. Create json from the CSV-like file
It seems little complicated.
jq -Rsn '
[inputs
| . / "n"
| (. | select((. | length) > 0) | . / " ") as $input
| [$input[1], $input[2], $input[3] ] ]
' <conc.txt
Actually I do not know detailed syntex or usage of jq
, it seems like doing:
- split input file by
n
- split a given line by space, then select valid data
- put splitted records in appropriate location by their index
I used this question as a reference:
https://stackoverflow.com/a/44781106/10675437
answered Nov 23 '18 at 2:36
toshim
12
12
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53439594%2fhow-to-use-bash-to-create-arrays-with-values-from-the-same-line-of-many-files%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown