Is Yahoo data getting flaky again ?

[mike]

A few months back, Yahoo changed the availabilty of its data which needed a re-coding of various macros. Since that time, the Yahoo data source had been pretty reliable.

I have noticed over the past couple of weeks that I am getting a "No data returned from Yahoo" far more frequently. It does correct itself during the day, but there do seem to be more problems at present than has been the case.

Has anyone else noticed this ?
Thanks
Mike


Windows 8.1, Open Office 4.1.5
Simple Yahoo Price Scrape
HYPTUSS (Just downloaded to check it wasn't just my program)

[kiloran]

I've been using HYPTUSS a lot recently while developing the new enhancements and have not seen any Yahoo problems at all. I normally use LibreOffice.

I just checked HYPTUSS version b-i and the Simple Yahoo Price Scrape v1.3a using OpenOffice 4.1.3 Portable and Yahoo refuses to download prices. I check these tools on Windows and Linux, with various versions of LibreOffice and Openoffice before I publish so it sounds like something has changed on the Yahoo side.

I'll have a look at it. Thanks for reporting the problem.

--kiloran

[kyu66]

I've been using HYPTUSS a lot recently while developing the new enhancements and have not seen any Yahoo problems at all. I normally use LibreOffice.

I just checked HYPTUSS version b-i and the Simple Yahoo Price Scrape v1.3a using OpenOffice 4.1.3 Portable and Yahoo refuses to download prices. I check these tools on Windows and Linux, with various versions of LibreOffice and Openoffice before I publish so it sounds like something has changed on the Yahoo side.

I'm not sure if it is related but I had a problem with my homegrown data scrape code (Python 3.5 and LO 6 under MacOS) not retrieving data from Yahoo a couple of weeks ago.

It turned out that the http request was being redirected to https, resulting in no data retrieved. I changed my target url to https without any joy. I finally pinned it down to the SSL lib in Python 3.5, SSL certificate checking was failing (Yahoo not implementing properly?) so the simple fix was to set the default SSL context to ignore cert checks. It seems this behaviour differs from Python 2.

Code: Select all

# Yahoo SSL cert fails in 3.5 so force no check for this version
import ssl

ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE
# end SSL cert no check

HTH
kyu66

[kiloran]

I've been using HYPTUSS a lot recently while developing the new enhancements and have not seen any Yahoo problems at all. I normally use LibreOffice.

I just checked HYPTUSS version b-i and the Simple Yahoo Price Scrape v1.3a using OpenOffice 4.1.3 Portable and Yahoo refuses to download prices. I check these tools on Windows and Linux, with various versions of LibreOffice and Openoffice before I publish so it sounds like something has changed on the Yahoo side.

I'm not sure if it is related but I had a problem with my homegrown data scrape code (Python 3.5 and LO 6 under MacOS) not retrieving data from Yahoo a couple of weeks ago.

It turned out that the http request was being redirected to https, resulting in no data retrieved. I changed my target url to https without any joy. I finally pinned it down to the SSL lib in Python 3.5, SSL certificate checking was failing (Yahoo not implementing properly?) so the simple fix was to set the default SSL context to ignore cert checks. It seems this behaviour differs from Python 2.

Code: Select all

# Yahoo SSL cert fails in 3.5 so force no check for this version
import ssl

ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE
# end SSL cert no check

HTH
kyu66

It's probably something along those lines. My normal Python work uses v3.6 and LibreOffice is v3.5 and these work fine.
OpenOffice uses Python 2.7 and the coding for web scraping differs significantly, so I'm just installing Python 2.7 on my laptop to investigate more thoroughly. I have a vague recollection of SSL differences between 2.7 and 3.x

--kiloran

[kiloran]

Hmm... the code runs fine using my standalone version of Python 2.7 so it looks like the problem lies within OpenOffice.

Investigation continues

--kiloran

[Itsallaguess]

I've just tried the latest Excel version with a fairly large portfolio, and the Yahoo prices downloaded without any issues.

Cheers,

Itsallaguess

[kiloran]

I've just tried the latest Excel version with a fairly large portfolio, and the Yahoo prices downloaded without any issues.

Cheers,

Itsallaguess

It's definitely an OpenOffice issue.

Standalone Python 3.6 works fine
Libreoffice (all versions) with Python 3.x work fine
Standalone Python 2.7.15 works fine
OpenOffice (all versions) with Python 2.7 all fail
But.... if I use the OpenOffice Python installation as a standalone Python, it works fine!!!

So something very mysterious within OpenOffice itself.

I see Python 2.7 will be deprecated from 01Jan2020 so I guess OpenOffice will have to be updated to Python 3.x. Hooray!!!

--kiloran

[mike]

Thanks guys. This is an intermittent fault I'm seeing, but only over the last fortnight or so. Before there was the major disruption of Yahoo changing their address for scraping the data a few months back, the download did fail from time to time, but one or two repeats normally obtained the data.

I am now noticing the download failing for periods of hours, rather than just re-initiating the price scrape to solve the problem.

It has been OK for me since late morning. If others are noticing this, then I will assume it is general, but if I am the only one then it is me. If the feeling is that it is only OpenOffice users, time possibly for a re-think by me.

Thanks to all

[kiloran]

I'm getting confused!
I've been trying OpenOffice Portable versions 3.2.0, 4.1.1 and 4.1.3 and they have consistently failed today when using the Simple Yahoo Price Scrape v1.3a

I installed OpenOffice Portable 4.1.5 (the latest), fired it up and set the macro security level to Medium (my usual preference), opened the price scrape and it worked. Every time. Closed OO, opened it again, and the price scrape now fails every time! Rebooted the Win10 laptop and still have the problem. Disabled the Comodo Firewall and it still fails.

Meanwhile, all my LibreOffice versions are working fine.

What on earth is going on?

--kiloran

[kyu66]

I'm getting confused!
...
What on earth is going on?

--kiloran

In my case, to pin down the error I was getting I had to let Python exceptions filter up and be reported by LO. If your code is anything like this it may be worth a shot.

I was accessing the url in a try/except block and not propagating the specific error back to LO as below, the LO code just relied on a non 0 return value.

Code: Select all

   
   try:
     if sys.version[0:1] == '3':
        response = urllib.urlopen(url, context=ctx)
     else:
        response = urllib.urlopen(url)
     # updated for Yahoo JSON API
     raw = response.read()
     decoded_raw = raw.decode('utf-8')  # Or any other ISO format
     result = ConvertedJSON(decoded_raw)
     return result
   except IOError:
     return 0

HTH

[mike]

I'm getting confused!
I've been trying OpenOffice Portable versions 3.2.0, 4.1.1 and 4.1.3 and they have consistently failed today when using the Simple Yahoo Price Scrape v1.3a

I installed OpenOffice Portable 4.1.5 (the latest), fired it up and set the macro security level to Medium (my usual preference), opened the price scrape and it worked. Every time. Closed OO, opened it again, and the price scrape now fails every time! Rebooted the Win10 laptop and still have the problem. Disabled the Comodo Firewall and it still fails.

Meanwhile, all my LibreOffice versions are working fine.

What on earth is going on?

--kiloran

I've just re-run the Simple Yahoo Price Scrape at 17:13 and it worked fine. Mine is OO 4.1.5

[kiloran]

I'm getting confused!
...
What on earth is going on?

--kiloran

In my case, to pin down the error I was getting I had to let Python exceptions filter up and be reported by LO. If your code is anything like this it may be worth a shot.

I was accessing the url in a try/except block and not propagating the specific error back to LO as below, the LO code just relied on a non 0 return value.

Code: Select all

   
   try:
     if sys.version[0:1] == '3':
        response = urllib.urlopen(url, context=ctx)
     else:
        response = urllib.urlopen(url)
     # updated for Yahoo JSON API
     raw = response.read()
     decoded_raw = raw.decode('utf-8')  # Or any other ISO format
     result = ConvertedJSON(decoded_raw)
     return result
   except IOError:
     return 0

HTH

Indeed. With all the odd things going on, I was failing to look at the basics. My code does use a Try/Except structure but I wasn't doing anything with the error code. Dumped this to a file and the error is "<urlopen error unknown url type: https>". That does ring a bell with some OpenOffice problems I was having a year or two ago, which I think were SSL related.

Thanks for putting me back on the straight and narrow

But why on earth did OO 4.1.5 work the first time I tried it and failed on subsequent attempts???

--kiloran

[kiloran]

But why on earth did OO 4.1.5 work the first time I tried it and failed on subsequent attempts???
--kiloran

I just deleted OO 4.1.5 then reinstalled it. And once again, after I had set Macro Security to Medium and opened the Yahoo Price Scrape, it worked 100% reliably. But when I closed OO and then reopened, it failed 100% of the time.

Very strange behaviour. At least it appears to be consistent so I have something to work on

--kiloran

[kyu66]

But why on earth did OO 4.1.5 work the first time I tried it and failed on subsequent attempts???
--kiloran

I just deleted OO 4.1.5 then reinstalled it. And once again, after I had set Macro Security to Medium and opened the Yahoo Price Scrape, it worked 100% reliably. But when I closed OO and then reopened, it failed 100% of the time.

Very strange behaviour. At least it appears to be consistent so I have something to work on

--kiloran

I found little concrete info on the Python/LO/OO combination with respect to SSL cert checking, hence my sledgehammer approach to forcing Python to NOT check the SSL cert by using a context- see my previous post. Caching/default certificate locations etc could all be possible areas of investigation but I deemed it too much like hard work.

[kiloran]

I'm getting confused!
...
What on earth is going on?

--kiloran

In my case, to pin down the error I was getting I had to let Python exceptions filter up and be reported by LO. If your code is anything like this it may be worth a shot.

I was accessing the url in a try/except block and not propagating the specific error back to LO as below, the LO code just relied on a non 0 return value.

Code: Select all

   
   try:
     if sys.version[0:1] == '3':
        response = urllib.urlopen(url, context=ctx)
     else:
        response = urllib.urlopen(url)
     # updated for Yahoo JSON API
     raw = response.read()
     decoded_raw = raw.decode('utf-8')  # Or any other ISO format
     result = ConvertedJSON(decoded_raw)
     return result
   except IOError:
     return 0

HTH

The issue is with definitely with OpenOffice, which uses 24-bit Python 2.7.15 (not Python 3). The problem exists on two Windows 10 laptops, with Desktop or Portable versions of OpenOffice.

I tried the following test script:

Code: Select all

try:
    import ssl
    import socket
except Exception as e:
    with open('C:/Users/User/Downloads/pydump.txt','w') as logfile:
        logfile.write(str(e) + "\n")

import urllib2

def test():
    url = 'https://query1.finance.yahoo.com/v7/finance/quote?symbols=SSE.L'
    
    try:
        ctx = ssl.create_default_context()
        ctx.check_hostname = False
        ctx.verify_mode = ssl.CERT_NONE
        req = urllib2.urlopen(url)
        response = req.read()
        
        f = open('C:/Users/User/Downloads/pydump.txt','a')
        f.write(response + "\n")
        f.close()
    except Exception as e:
        with open('C:/Users/User/Downloads/pydump.txt','a') as logfile:
            logfile.write(str(e) + "\n")

The error I get is: 'module' object has no attribute 'create_default_context'

I checked the OpenOffice version of ssl.py and sure enough there is no reference to create_default_context (LibreOffice and standalone versions of Python do have create_default_context in their ssl.py files)

My standalone installation of Python 2.7.15 is fine, as is LibreOffice and Python 3

I'm convinced it's an OpenOffice problem, with the wrong version of ssl.py packaged within it, and I'll report it as a bug. I think I could work around it, but I'll park that for the moment, I've spent far too much time on it in the past few days. I need to get back to testing and debugging the enhanced HYPTUSS that Itsallaguess and I are working on.

Many thanks for your ideas, kyu66

--kiloran

[TUK020]

HYPTUSS retrieving yields for BHP & PSN as zero or n/a.
Tried updating the Digital Look datasheet to no avail

[kiloran]

HYPTUSS retrieving yields for BHP & PSN as zero or n/a.
Tried updating the Digital Look datasheet to no avail

HYPTUSS gets yields from Webfg (ex Digital Look), not Yahoo, the subject of this thread.

BHP.... https://uk.webfg.com/equity/BHP_Group shows n/a for Forecast Yield. This is therefore correctly displayed in HYPTUSS
PSN... https://uk.webfg.com/equity/Pearson shows n/a for Forecast Yield. This is therefore correctly displayed in HYPTUSS

--kiloran

[TUK020]

HYPTUSS gets yields from Webfg (ex Digital Look), not Yahoo, the subject of this thread.

BHP.... https://uk.webfg.com/equity/BHP_Group shows n/a for Forecast Yield. This is therefore correctly displayed in HYPTUSS
PSN... https://uk.webfg.com/equity/Pearson shows n/a for Forecast Yield. This is therefore correctly displayed in HYPTUSS

--kiloran

apologies for confusion on both counts