Caching results from a web API using with open












1














I am using a lot of data returned from a WEB-api.The function below calls the API 22 times, decodes and loads json into python format. Then I store the results in a big list of 22 pages with each 100 art objects as data.



fourteen_list = return_14th_century_works_list() 


To limit the necessary API-calls, I want to build a function that stores this list as a file if it is not present, and when it is present I want to load the file from my computer. I came up with the following:



with open('fourteenth_century_list.txt', 'w') as fourteenth_century_file:
print(fourteen_list, file=fourteenth_century_file)

try:
with open('fourteenth_century_list.txt', 'r') as fourteenth_century_file:
fourteenth_list_cache = fourteenth_century_file.read()
count_objects(fourteenth_list_cache)
except FileNotFoundError:
fourteenth_list = return_14th_century_works_list() Calls API again
count_objects(fourteen_list)


I use the count_objects function to check if everything still works, but the file that is opened in the try block doesn't seem to return in the same way I saved it; When I run this code, the function call in the try-block returns a type-error. For me this is an indication that the file opened from disk is in a somewhat different format then if I load it directly from the API.



When i call the function count_objects() with the non-cached version of my list, so fourteen_list in this case,works fine.



Does with_open(filename, 'w') and then with_open(filename, 'r') mutate your data, and if not what am I doing wrong here?










share|improve this question
























  • correct the indentation first please :-)
    – Nimish Bansal
    Nov 22 '18 at 20:42










  • @NimishBansal mb
    – Psychotechnopath
    Nov 22 '18 at 20:43










  • using with to open the file ensures that file will be closed as soon as with block ends
    – Nimish Bansal
    Nov 22 '18 at 20:44










  • What are the type of objects held by the list - are they dicts?
    – Will Keeling
    Nov 22 '18 at 21:43










  • @WillKeeling Yes, they are dicts. The API returns pages of JSON data, when I check it in my browser this data looks like a dictionary, and after I deserialize (With data.decode('utf-8) and then json.loads(data.decoded), and do a type-check on the pages it returns 'dict'....
    – Psychotechnopath
    Nov 23 '18 at 19:02


















1














I am using a lot of data returned from a WEB-api.The function below calls the API 22 times, decodes and loads json into python format. Then I store the results in a big list of 22 pages with each 100 art objects as data.



fourteen_list = return_14th_century_works_list() 


To limit the necessary API-calls, I want to build a function that stores this list as a file if it is not present, and when it is present I want to load the file from my computer. I came up with the following:



with open('fourteenth_century_list.txt', 'w') as fourteenth_century_file:
print(fourteen_list, file=fourteenth_century_file)

try:
with open('fourteenth_century_list.txt', 'r') as fourteenth_century_file:
fourteenth_list_cache = fourteenth_century_file.read()
count_objects(fourteenth_list_cache)
except FileNotFoundError:
fourteenth_list = return_14th_century_works_list() Calls API again
count_objects(fourteen_list)


I use the count_objects function to check if everything still works, but the file that is opened in the try block doesn't seem to return in the same way I saved it; When I run this code, the function call in the try-block returns a type-error. For me this is an indication that the file opened from disk is in a somewhat different format then if I load it directly from the API.



When i call the function count_objects() with the non-cached version of my list, so fourteen_list in this case,works fine.



Does with_open(filename, 'w') and then with_open(filename, 'r') mutate your data, and if not what am I doing wrong here?










share|improve this question
























  • correct the indentation first please :-)
    – Nimish Bansal
    Nov 22 '18 at 20:42










  • @NimishBansal mb
    – Psychotechnopath
    Nov 22 '18 at 20:43










  • using with to open the file ensures that file will be closed as soon as with block ends
    – Nimish Bansal
    Nov 22 '18 at 20:44










  • What are the type of objects held by the list - are they dicts?
    – Will Keeling
    Nov 22 '18 at 21:43










  • @WillKeeling Yes, they are dicts. The API returns pages of JSON data, when I check it in my browser this data looks like a dictionary, and after I deserialize (With data.decode('utf-8) and then json.loads(data.decoded), and do a type-check on the pages it returns 'dict'....
    – Psychotechnopath
    Nov 23 '18 at 19:02
















1












1








1







I am using a lot of data returned from a WEB-api.The function below calls the API 22 times, decodes and loads json into python format. Then I store the results in a big list of 22 pages with each 100 art objects as data.



fourteen_list = return_14th_century_works_list() 


To limit the necessary API-calls, I want to build a function that stores this list as a file if it is not present, and when it is present I want to load the file from my computer. I came up with the following:



with open('fourteenth_century_list.txt', 'w') as fourteenth_century_file:
print(fourteen_list, file=fourteenth_century_file)

try:
with open('fourteenth_century_list.txt', 'r') as fourteenth_century_file:
fourteenth_list_cache = fourteenth_century_file.read()
count_objects(fourteenth_list_cache)
except FileNotFoundError:
fourteenth_list = return_14th_century_works_list() Calls API again
count_objects(fourteen_list)


I use the count_objects function to check if everything still works, but the file that is opened in the try block doesn't seem to return in the same way I saved it; When I run this code, the function call in the try-block returns a type-error. For me this is an indication that the file opened from disk is in a somewhat different format then if I load it directly from the API.



When i call the function count_objects() with the non-cached version of my list, so fourteen_list in this case,works fine.



Does with_open(filename, 'w') and then with_open(filename, 'r') mutate your data, and if not what am I doing wrong here?










share|improve this question















I am using a lot of data returned from a WEB-api.The function below calls the API 22 times, decodes and loads json into python format. Then I store the results in a big list of 22 pages with each 100 art objects as data.



fourteen_list = return_14th_century_works_list() 


To limit the necessary API-calls, I want to build a function that stores this list as a file if it is not present, and when it is present I want to load the file from my computer. I came up with the following:



with open('fourteenth_century_list.txt', 'w') as fourteenth_century_file:
print(fourteen_list, file=fourteenth_century_file)

try:
with open('fourteenth_century_list.txt', 'r') as fourteenth_century_file:
fourteenth_list_cache = fourteenth_century_file.read()
count_objects(fourteenth_list_cache)
except FileNotFoundError:
fourteenth_list = return_14th_century_works_list() Calls API again
count_objects(fourteen_list)


I use the count_objects function to check if everything still works, but the file that is opened in the try block doesn't seem to return in the same way I saved it; When I run this code, the function call in the try-block returns a type-error. For me this is an indication that the file opened from disk is in a somewhat different format then if I load it directly from the API.



When i call the function count_objects() with the non-cached version of my list, so fourteen_list in this case,works fine.



Does with_open(filename, 'w') and then with_open(filename, 'r') mutate your data, and if not what am I doing wrong here?







python api caching






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited Nov 22 '18 at 20:43

























asked Nov 22 '18 at 20:40









Psychotechnopath

413212




413212












  • correct the indentation first please :-)
    – Nimish Bansal
    Nov 22 '18 at 20:42










  • @NimishBansal mb
    – Psychotechnopath
    Nov 22 '18 at 20:43










  • using with to open the file ensures that file will be closed as soon as with block ends
    – Nimish Bansal
    Nov 22 '18 at 20:44










  • What are the type of objects held by the list - are they dicts?
    – Will Keeling
    Nov 22 '18 at 21:43










  • @WillKeeling Yes, they are dicts. The API returns pages of JSON data, when I check it in my browser this data looks like a dictionary, and after I deserialize (With data.decode('utf-8) and then json.loads(data.decoded), and do a type-check on the pages it returns 'dict'....
    – Psychotechnopath
    Nov 23 '18 at 19:02




















  • correct the indentation first please :-)
    – Nimish Bansal
    Nov 22 '18 at 20:42










  • @NimishBansal mb
    – Psychotechnopath
    Nov 22 '18 at 20:43










  • using with to open the file ensures that file will be closed as soon as with block ends
    – Nimish Bansal
    Nov 22 '18 at 20:44










  • What are the type of objects held by the list - are they dicts?
    – Will Keeling
    Nov 22 '18 at 21:43










  • @WillKeeling Yes, they are dicts. The API returns pages of JSON data, when I check it in my browser this data looks like a dictionary, and after I deserialize (With data.decode('utf-8) and then json.loads(data.decoded), and do a type-check on the pages it returns 'dict'....
    – Psychotechnopath
    Nov 23 '18 at 19:02


















correct the indentation first please :-)
– Nimish Bansal
Nov 22 '18 at 20:42




correct the indentation first please :-)
– Nimish Bansal
Nov 22 '18 at 20:42












@NimishBansal mb
– Psychotechnopath
Nov 22 '18 at 20:43




@NimishBansal mb
– Psychotechnopath
Nov 22 '18 at 20:43












using with to open the file ensures that file will be closed as soon as with block ends
– Nimish Bansal
Nov 22 '18 at 20:44




using with to open the file ensures that file will be closed as soon as with block ends
– Nimish Bansal
Nov 22 '18 at 20:44












What are the type of objects held by the list - are they dicts?
– Will Keeling
Nov 22 '18 at 21:43




What are the type of objects held by the list - are they dicts?
– Will Keeling
Nov 22 '18 at 21:43












@WillKeeling Yes, they are dicts. The API returns pages of JSON data, when I check it in my browser this data looks like a dictionary, and after I deserialize (With data.decode('utf-8) and then json.loads(data.decoded), and do a type-check on the pages it returns 'dict'....
– Psychotechnopath
Nov 23 '18 at 19:02






@WillKeeling Yes, they are dicts. The API returns pages of JSON data, when I check it in my browser this data looks like a dictionary, and after I deserialize (With data.decode('utf-8) and then json.loads(data.decoded), and do a type-check on the pages it returns 'dict'....
– Psychotechnopath
Nov 23 '18 at 19:02














1 Answer
1






active

oldest

votes


















1














The issue here is that when you print the list of dictionaries to a file, you create a string representation of the list. You then read that string back and pass it to count_objects(), but that falls over because it expects a list of dictionaries, not a big string.



Rather than printing, a better approach would be to serialize the list to back JSON - which would preserve its structure. You also want to write the list to the cache in the except block after you've retrieved the data from the API.



import json

try:
with open('fourteenth_century_list.json', 'r') as fourteenth_century_file:
fourteenth_list_cache = json.load(fourteenth_century_file)
count_objects(fourteenth_list_cache)
except FileNotFoundError:
# Calls API again
fourteenth_list = return_14th_century_works_list()
count_objects(fourteen_list)

# Cache the API data
with open('fourteenth_century_list.json', 'w') as fourteenth_century_file:
json.dump(fourteen_list, fourteenth_century_file)





share|improve this answer





















  • So when printing python data-structures to a file by using with open, are they always converted to a string representation? Would it have been better to use something like shelve here? It is indeed much more logical to only cache after trying to open, thinking mistake from my part (am still learning). Thanks for this very comprehensive answer sir =)
    – Psychotechnopath
    Nov 23 '18 at 21:22






  • 1




    Yes print will just output the string representation of whatever is passed to it. Using shelve (or pickle) would also work, although keeping the format as JSON means the file is human readable should you need to ever edit it with a text editor.
    – Will Keeling
    Nov 23 '18 at 21:28













Your Answer






StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});


}
});














draft saved

draft discarded


















StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53437757%2fcaching-results-from-a-web-api-using-with-open%23new-answer', 'question_page');
}
);

Post as a guest















Required, but never shown

























1 Answer
1






active

oldest

votes








1 Answer
1






active

oldest

votes









active

oldest

votes






active

oldest

votes









1














The issue here is that when you print the list of dictionaries to a file, you create a string representation of the list. You then read that string back and pass it to count_objects(), but that falls over because it expects a list of dictionaries, not a big string.



Rather than printing, a better approach would be to serialize the list to back JSON - which would preserve its structure. You also want to write the list to the cache in the except block after you've retrieved the data from the API.



import json

try:
with open('fourteenth_century_list.json', 'r') as fourteenth_century_file:
fourteenth_list_cache = json.load(fourteenth_century_file)
count_objects(fourteenth_list_cache)
except FileNotFoundError:
# Calls API again
fourteenth_list = return_14th_century_works_list()
count_objects(fourteen_list)

# Cache the API data
with open('fourteenth_century_list.json', 'w') as fourteenth_century_file:
json.dump(fourteen_list, fourteenth_century_file)





share|improve this answer





















  • So when printing python data-structures to a file by using with open, are they always converted to a string representation? Would it have been better to use something like shelve here? It is indeed much more logical to only cache after trying to open, thinking mistake from my part (am still learning). Thanks for this very comprehensive answer sir =)
    – Psychotechnopath
    Nov 23 '18 at 21:22






  • 1




    Yes print will just output the string representation of whatever is passed to it. Using shelve (or pickle) would also work, although keeping the format as JSON means the file is human readable should you need to ever edit it with a text editor.
    – Will Keeling
    Nov 23 '18 at 21:28


















1














The issue here is that when you print the list of dictionaries to a file, you create a string representation of the list. You then read that string back and pass it to count_objects(), but that falls over because it expects a list of dictionaries, not a big string.



Rather than printing, a better approach would be to serialize the list to back JSON - which would preserve its structure. You also want to write the list to the cache in the except block after you've retrieved the data from the API.



import json

try:
with open('fourteenth_century_list.json', 'r') as fourteenth_century_file:
fourteenth_list_cache = json.load(fourteenth_century_file)
count_objects(fourteenth_list_cache)
except FileNotFoundError:
# Calls API again
fourteenth_list = return_14th_century_works_list()
count_objects(fourteen_list)

# Cache the API data
with open('fourteenth_century_list.json', 'w') as fourteenth_century_file:
json.dump(fourteen_list, fourteenth_century_file)





share|improve this answer





















  • So when printing python data-structures to a file by using with open, are they always converted to a string representation? Would it have been better to use something like shelve here? It is indeed much more logical to only cache after trying to open, thinking mistake from my part (am still learning). Thanks for this very comprehensive answer sir =)
    – Psychotechnopath
    Nov 23 '18 at 21:22






  • 1




    Yes print will just output the string representation of whatever is passed to it. Using shelve (or pickle) would also work, although keeping the format as JSON means the file is human readable should you need to ever edit it with a text editor.
    – Will Keeling
    Nov 23 '18 at 21:28
















1












1








1






The issue here is that when you print the list of dictionaries to a file, you create a string representation of the list. You then read that string back and pass it to count_objects(), but that falls over because it expects a list of dictionaries, not a big string.



Rather than printing, a better approach would be to serialize the list to back JSON - which would preserve its structure. You also want to write the list to the cache in the except block after you've retrieved the data from the API.



import json

try:
with open('fourteenth_century_list.json', 'r') as fourteenth_century_file:
fourteenth_list_cache = json.load(fourteenth_century_file)
count_objects(fourteenth_list_cache)
except FileNotFoundError:
# Calls API again
fourteenth_list = return_14th_century_works_list()
count_objects(fourteen_list)

# Cache the API data
with open('fourteenth_century_list.json', 'w') as fourteenth_century_file:
json.dump(fourteen_list, fourteenth_century_file)





share|improve this answer












The issue here is that when you print the list of dictionaries to a file, you create a string representation of the list. You then read that string back and pass it to count_objects(), but that falls over because it expects a list of dictionaries, not a big string.



Rather than printing, a better approach would be to serialize the list to back JSON - which would preserve its structure. You also want to write the list to the cache in the except block after you've retrieved the data from the API.



import json

try:
with open('fourteenth_century_list.json', 'r') as fourteenth_century_file:
fourteenth_list_cache = json.load(fourteenth_century_file)
count_objects(fourteenth_list_cache)
except FileNotFoundError:
# Calls API again
fourteenth_list = return_14th_century_works_list()
count_objects(fourteen_list)

# Cache the API data
with open('fourteenth_century_list.json', 'w') as fourteenth_century_file:
json.dump(fourteen_list, fourteenth_century_file)






share|improve this answer












share|improve this answer



share|improve this answer










answered Nov 23 '18 at 20:24









Will Keeling

10.7k22329




10.7k22329












  • So when printing python data-structures to a file by using with open, are they always converted to a string representation? Would it have been better to use something like shelve here? It is indeed much more logical to only cache after trying to open, thinking mistake from my part (am still learning). Thanks for this very comprehensive answer sir =)
    – Psychotechnopath
    Nov 23 '18 at 21:22






  • 1




    Yes print will just output the string representation of whatever is passed to it. Using shelve (or pickle) would also work, although keeping the format as JSON means the file is human readable should you need to ever edit it with a text editor.
    – Will Keeling
    Nov 23 '18 at 21:28




















  • So when printing python data-structures to a file by using with open, are they always converted to a string representation? Would it have been better to use something like shelve here? It is indeed much more logical to only cache after trying to open, thinking mistake from my part (am still learning). Thanks for this very comprehensive answer sir =)
    – Psychotechnopath
    Nov 23 '18 at 21:22






  • 1




    Yes print will just output the string representation of whatever is passed to it. Using shelve (or pickle) would also work, although keeping the format as JSON means the file is human readable should you need to ever edit it with a text editor.
    – Will Keeling
    Nov 23 '18 at 21:28


















So when printing python data-structures to a file by using with open, are they always converted to a string representation? Would it have been better to use something like shelve here? It is indeed much more logical to only cache after trying to open, thinking mistake from my part (am still learning). Thanks for this very comprehensive answer sir =)
– Psychotechnopath
Nov 23 '18 at 21:22




So when printing python data-structures to a file by using with open, are they always converted to a string representation? Would it have been better to use something like shelve here? It is indeed much more logical to only cache after trying to open, thinking mistake from my part (am still learning). Thanks for this very comprehensive answer sir =)
– Psychotechnopath
Nov 23 '18 at 21:22




1




1




Yes print will just output the string representation of whatever is passed to it. Using shelve (or pickle) would also work, although keeping the format as JSON means the file is human readable should you need to ever edit it with a text editor.
– Will Keeling
Nov 23 '18 at 21:28






Yes print will just output the string representation of whatever is passed to it. Using shelve (or pickle) would also work, although keeping the format as JSON means the file is human readable should you need to ever edit it with a text editor.
– Will Keeling
Nov 23 '18 at 21:28




















draft saved

draft discarded




















































Thanks for contributing an answer to Stack Overflow!


  • Please be sure to answer the question. Provide details and share your research!

But avoid



  • Asking for help, clarification, or responding to other answers.

  • Making statements based on opinion; back them up with references or personal experience.


To learn more, see our tips on writing great answers.





Some of your past answers have not been well-received, and you're in danger of being blocked from answering.


Please pay close attention to the following guidance:


  • Please be sure to answer the question. Provide details and share your research!

But avoid



  • Asking for help, clarification, or responding to other answers.

  • Making statements based on opinion; back them up with references or personal experience.


To learn more, see our tips on writing great answers.




draft saved


draft discarded














StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53437757%2fcaching-results-from-a-web-api-using-with-open%23new-answer', 'question_page');
}
);

Post as a guest















Required, but never shown





















































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown

































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown







Popular posts from this blog

Berounka

Sphinx de Gizeh

Different font size/position of beamer's navigation symbols template's content depending on regular/plain...