Htykuut

Goal: I am trying to scrape some data off a website.

Problem: I have to use the requests library but it gives me error.

import requests

url = "https://www.random.com"

requests.get(url)

Attempt at solution: I have another program that uses urllib3 to get website data and it works fine:

url = "https://www.random.com"

http = urllib3.PoolManager()

http.request("GET", url)

I discovered that urllib3 and requests uses a common library to send the requests, so I thought I could see what the difference between the two was and maybe change it accordingly. Whilst reading the stack trace below and following it I noticed that requests for some reason can't read the response(the code section is from connectionpool.py of one of the library):

    def begin(self):

    if self.headers is not None:

        # we've already started reading the response

        return



    # read until we get a non-100 response

    while True:

        version, status, reason = self._read_status()

The last line in the above code is where they both differ. requests gets an error and does not get any response. urllib3 gets a response and continues on. My suspicion is that it has to do with the security protocols of requests, but I am lost since there are so many variables it sets before sending the response.

Full stack trace of the first piece of code:

Traceback (most recent call last):

  File "/path/lib/python3.7/site-packages/urllib3/connectionpool.py", line 600, in urlopen

    chunked=chunked)

  File "/path/lib/python3.7/site-packages/urllib3/connectionpool.py", line 384, in _make_request

    six.raise_from(e, None)

  File "<string>", line 2, in raise_from

  File "/path/lib/python3.7/site-packages/urllib3/connectionpool.py", line 380, in _make_request

    httplib_response = conn.getresponse()

  File "/path/lib/python3.7/http/client.py", line 1321, in getresponse

    response.begin()

  File "/path/lib/python3.7/http/client.py", line 296, in begin

    version, status, reason = self._read_status()

  File "/path/lib/python3.7/http/client.py", line 257, in _read_status

    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")

  File "/path/lib/python3.7/socket.py", line 589, in readinto

    return self._sock.recv_into(b)

  File "/path/lib/python3.7/site-packages/urllib3/contrib/pyopenssl.py", line 290, in recv_into

    raise SocketError(str(e))

OSError: (104, 'ECONNRESET')



During handling of the above exception, another exception occurred:



Traceback (most recent call last):

  File "/path/lib/python3.7/site-packages/requests/adapters.py", line 445, in send

    timeout=timeout

  File "/path/lib/python3.7/site-packages/urllib3/connectionpool.py", line 638, in urlopen

    _stacktrace=sys.exc_info()[2])

  File "/path/lib/python3.7/site-packages/urllib3/util/retry.py", line 367, in increment

    raise six.reraise(type(error), error, _stacktrace)

  File "/path/lib/python3.7/site-packages/urllib3/packages/six.py", line 685, in reraise

    raise value.with_traceback(tb)

  File "/path/lib/python3.7/site-packages/urllib3/connectionpool.py", line 600, in urlopen

    chunked=chunked)

  File "/path/lib/python3.7/site-packages/urllib3/connectionpool.py", line 384, in _make_request

    six.raise_from(e, None)

  File "<string>", line 2, in raise_from

  File "/homes/ubalgans/miniconda3/lib/python3.7/site-packages/urllib3/connectionpool.py", line 380, in _make_request

    httplib_response = conn.getresponse()

  File "/path/lib/python3.7/http/client.py", line 1321, in getresponse

    response.begin()

  File "/path/lib/python3.7/http/client.py", line 296, in begin

    version, status, reason = self._read_status()

  File "/path/lib/python3.7/http/client.py", line 257, in _read_status

    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")

  File "/path/lib/python3.7/socket.py", line 589, in readinto

    return self._sock.recv_into(b)

  File "/path/lib/python3.7/site-packages/urllib3/contrib/pyopenssl.py", line 290, in recv_into

    raise SocketError(str(e))

urllib3.exceptions.ProtocolError: ('Connection aborted.', OSError("(104, 'ECONNRESET')"))



During handling of the above exception, another exception occurred:



Traceback (most recent call last):

  File "/path/PycharmProjects/test/reqtest.py", line 8, in <module>

    requests.get(url)

  File "/path/lib/python3.7/site-packages/requests/api.py", line 72, in get

    return request('get', url, params=params, **kwargs)

  File "/path/lib/python3.7/site-packages/requests/api.py", line 58, in request

    return session.request(method=method, url=url, **kwargs)

  File "/path/lib/python3.7/site-packages/requests/sessions.py", line 512, in request

    resp = self.send(prep, **send_kwargs)

  File "/path/lib/python3.7/site-packages/requests/sessions.py", line 622, in send

    r = adapter.send(request, **kwargs)

  File "/path/lib/python3.7/site-packages/requests/adapters.py", line 495, in send

    raise ConnectionError(err, request=request)

requests.exceptions.ConnectionError: ('Connection aborted.', OSError("(104, 'ECONNRESET')"))

EDIT: I enabled logging to see the difference between the two.

requests:

DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): random.com:443

urllib3:

    DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): random.com:443

/path/urllib3/connectionpool.py:857: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings

  InsecureRequestWarning)

DEBUG:urllib3.connectionpool:https://random.com:443 "GET /request HTTP/1.1" 200 15189