I am writing a Python application that queries social media APIs via cURL. Most of the different servers I query (Google+, Reddit, Twitter, Facebook, others) have cURL complaining:
additional stuff not fine transfer.c:1037: 0 0
The unusual thing is that when the application first starts, each service's response will throw this line once or twice. After a few minutes, the line will appear several several times. Obviously cURL is identifying something that it doesn't like. After about half an hour, the servers begin to time out and this line is repeated many tens of times, so it is showing a real problem.
How might I diagnose this? I tried using Wireshark to capture the request and response headers to search for anomalies that might cause cURL to complain, but for all Wireshark's complexity there does not seem to be a way to isolate and display only the headers.
Here is the relevant part of the code:
output = cStringIO.StringIO() c = pycurl.Curl() c.setopt(c.URL, url) c.setopt(c.USERAGENT, 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:17.0) Gecko/20100101 Firefox/17.0') c.setopt(c.WRITEFUNCTION, output.write) c.setopt(c.CONNECTTIMEOUT, 10) c.setopt(c.TIMEOUT, 15) c.setopt(c.FAILONERROR, True) c.setopt(c.NOSIGNAL, 1) try: c.perform() toReturn = output.getvalue() output.close() return toReturn except pycurl.error, error: errno, errstr = error print 'The following cURL error occurred: ', errstr