Page 9 of 11
Re: HYPTUS v11.31 Run-time error 5
Posted: November 10th, 2017, 12:53 pm
by Itsallaguess
desmid wrote:
The offending URL inside HYPTUSS is:
Code: Select all
https://query1.finance.yahoo.com/v7/finance/quote?symbols=^FTSE,^FTAS
What do you get from a browser on that exact same device? If you can paste it here that might help.
I'd be interested to hear that answer from NeilW too.
It would also be great to hear if any other LibreOffice users who have not had the Beta working yet are now able to get the
b-g-dev3 version working OK?
http://www.filedropper.com/hyptop-upversion-b-g-dev3
That would help to see if this is an isolated issue for NeilW alone, or a wider problem with the code still.
Whilst I'm here, I do appreciate that this may not be the most efficient way to work with these issues, but I've tried my best to try to nudge things along where issues have been seen with the LibreOffice users, in the hope of giving kiloran a running start on his return.
We're really grateful for
desmid's continued input into that process.
Cheers,
Itsallaguess
Re: HYPTUS v11.31 Run-time error 5
Posted: November 10th, 2017, 1:28 pm
by spiderbill
Curious; version b-f-dev2 (on Win10 and Libre Office Version: 5.3.2.2) was working fine for me a couple of days or so ago but when I just tried it I got:
com.sun.star.uno.RuntimeException: Error during invoking function HypUpdPriPy_1 in module vnd.sun.star.tdoc:/1/Scripts/python/HypTopUp.py (<class 'UnboundLocalError'>: local variable 'data' referenced before assignment
File "C:\Program Files\LibreOffice 5\program\pythonscript.py", line 870, in invoke
ret = self.func( *args )
File "vnd.sun.star.tdoc:/1/Scripts/python/HypTopUp.py", line 1098, in HypUpdPriPy_1
File "vnd.sun.star.tdoc:/1/Scripts/python/HypTopUp.py", line 660, in createPriceDict
)
and a similar message with slightly different lines numbers on the Watchlist.
I'll download the new version and give that a try later today. Then I'll upgrade Libre Office and try both again.
cheers
Spiderbill
Re: HYPTUS v11.31 Run-time error 5
Posted: November 10th, 2017, 1:53 pm
by desmid
spiderbill wrote:Curious; version b-f-dev2 (on Win10 and Libre Office Version: 5.3.2.2) was working fine for me a couple of days or so ago but when I just tried it I got:
<class 'UnboundLocalError'>: local variable 'data' referenced before assignment
...
I'll download the new version and give that a try later today. Then I'll upgrade Libre Office and try both again.
cheers
Spiderbill
That's actually good news because it's consistently inconsistent. Must be Yahoo timeouts or sun-spots or summat. Anyway, that one's squashed in itsallaguess' latest version, b-g-dev3.
desmid
Re: HYPTUS v11.31 Run-time error 5
Posted: November 10th, 2017, 3:36 pm
by mrbrightside
Just to report the latest beta (b-g-dev3) works fine for me using LibreOffice 5.4.2.2.0+ on Fedora 27 (beta).
Thanks for an invaluable tool.
Re: HYPTUS v11.31 Run-time error 5
Posted: November 10th, 2017, 3:47 pm
by Itsallaguess
mrbrightside wrote:
Just to report the latest beta (b-g-dev3) works fine for me using LibreOffice 5.4.2.2.0+ on Fedora 27 (beta).
Great to hear. Can I please ask if you'd tried any previous versions and had any errors?
Cheers,
Itsallaguess
Re: HYPTUS v11.31 Run-time error 5
Posted: November 10th, 2017, 4:44 pm
by mrbrightside
Can I please ask if you'd tried any previous versions and had any errors?
No - I only update my HYPTUS about as often as I record my dividends (monthly) so I was blissfully unaware of all the fuss.
Re: HYPTUS v11.31 Run-time error 5
Posted: November 10th, 2017, 6:32 pm
by Breelander
Version b-g-dev3 works fine for my 29 share portfolio with LibreOffice Portable versions 4.3.5.2 and 5.4.2.2 on Windows 10 Fall Creators Update. But then so does b-f-dev2 (tested again just now). I don't see any of the problems some others are reporting.
Re: HYPTUS v11.31 Run-time error 5
Posted: November 10th, 2017, 9:16 pm
by spiderbill
desmid wrote:
That's actually good news because it's consistently inconsistent. Must be Yahoo timeouts or sun-spots or summat. Anyway, that one's squashed in itsallaguess' latest version, b-g-dev3.
If in doubt I usually blame Microsoft! As a default position it's always worked out pretty well.
Anyway I can report further.
On Libre Office Version: 5.3.2.2 the new HYPTUS version, b-g-dev3, didn't work either.
I then installed the latest Libre Office 5.4.2.2 - in the process remembering why I hadn't done it before (incomplete installation due to permissions problems) and after having to install it to a different directory as the old one was locked and required SYSTEM permissions to delete (Grrrr) - I eventually got it running.
Happy to say that b-g-dev3 now works correctly.
I'll check it regularly to make sure there is no intermittent element at play, but so far so good.
cheers
Spiderbill
Re: HYPTUS v11.31 Run-time error 5
Posted: November 10th, 2017, 11:55 pm
by mike
Using OpenOffice 4.1.4 Windows 8.1
No prices retrieved, error message "FT indices not retrieved - please try again later"
But note I don't normally use HYPTSS so not familiar with it, having used LivePrice since 2010 under OO. Hope the above is of help as all reports seem to be from LibreOffice users.
Mike
Re: HYPTUS v11.31 Run-time error 5
Posted: November 11th, 2017, 8:30 am
by torata
kiloran wrote:Hi boys and girls!
Profuse apologies for my absence, I had a family bereavement last saturday, so it was a mad panic to get the beta version of the new HYPTUSS published with very little testing and no error trapping and the like, before a 350 mile drive. {Snip}
--kiloran
Please do not apologize.
You guys are providing a free and extremely useful tool.
I'm sure we can all wait.
torata
Re: HYPTUS v11.31 Run-time error 5
Posted: November 11th, 2017, 6:19 pm
by Dochas
Hi Itsallaguess
Just tried version b-g-dev3 with LibreOffice. I get 'FT indices not retrieved. Please try later' ahead of 'Processing finished'.
Dochas
Re: HYPTUS v11.31 Run-time error 5
Posted: November 12th, 2017, 6:25 am
by torata
b-g-dev 3 works for me with a small selection of trial shares on LO 5.4.2.2 on Win 10
torata
Re: HYPTUS v11.31 Run-time error 5
Posted: November 12th, 2017, 12:56 pm
by Whatsup
Dear Kiloran /Itsallaguess
I m getting the same error message regarding indices on Mac OsX High Sierra and LibreOffice 5.4.2.2
Regards
Colin
Re: HYPTUS v11.31 Run-time error 5
Posted: November 12th, 2017, 2:13 pm
by Itsallaguess
Just wanted to say thanks to everyone who had issues with the original Beta for their continued feedback with the newer version(s), it's very much appreciated.
Cheers,
Itsallaguess
Re: HYPTUS v11.31 Run-time error 5
Posted: November 12th, 2017, 2:42 pm
by desmid
Well, hmmm, several Mac users and at least one Windows user with the new error. We need to be able to see exactly what is happening between clicking the button and allegedly getting something back from Yahoo.
I've written a new getHtml(), heavily leaning on the old one, which collects all the gubbins like HTTP response code and server header and exception codes, into a data structure that can be queried and used to produce messages. It's not quite a drop-in replacement, as the function is now a method on an object, and the object collects all the gory details.
I've tried it inside a modified copy of HYPTUSS on just the Yahoo fetches, and dumped useful text into yuuuuge message boxes, so it is possible to gather clues even without the logging, though that works too. This is one way you might use it:
Code: Select all
webAgent = WebAgent()
...
html = webAgent.fetch(url)
if webAgent.fetch_ok():
print(html)
else:
print( "Diagnostics:\n" + str(webAgent) )
if webAgent.get_response_code() == 404:
#do something with this situation
...
There's a bit more documentation inside the class.
If Kiloran or anyone is interested in trying it, here's is a test file with the present version embedded (run it with: 'python3 test_file.py' (or whatever you save it as). All the output appears in the logfile 'output.log'.
Code: Select all
###########################################################################
import logging
LOGFILE = 'output.log'
LOGFORMAT = '%(asctime)s %(levelname)5s [%(lineno)4s - %(funcName)-15s] %(message)s'
LOGLEVEL = logging.DEBUG
Logger = logging.getLogger()
Logger.setLevel(LOGLEVEL)
tmp = logging.FileHandler(LOGFILE)
tmp.setFormatter(logging.Formatter(LOGFORMAT))
Logger.addHandler(tmp)
del tmp
Logger.info("Start")
###########################################################################
import sys
try:
#Python3
from urllib.request import Request, urlopen
from urllib.error import URLError, HTTPError
from urllib.parse import urlencode
except:
#Python2
from urllib2 import Request, urlopen, URLError, HTTPError
from urllib import urlencode
Logger.info("Python %s", sys.version)
###########################################################################
try:
import ParameterDict
except:
pass
###########################################################################
class WebAgent(object):
"""
Class to access web pages. Wraps Python3 or Python2 implementation details.
Keeps a record of the request:
(1) requested url, (2) real url retrieved, (3) HTTP response code,
(4) returned header info, (5) number of tries, (6) maximum tries allowed,
(7) last request timeout, (8) last exception message, (9) the page itself.
Constructor and usage:
webAgent = WebAgent()
html = webAgent.fetch(url)
if webAgent.fetch_ok():
print(html)
else:
print( "Diagnostics:\n" + str(webAgent) )
if webAgent.get_response_code() == 404:
#do something with this situation
...
Public methods:
fetch(url) : fetches and returns the web page:
returns 'no response' if URL cannot be retrieved after preset retries
and timeouts, or is invalid.
fetched_ok() : returns True/False as fetch succeeded/failed
get_html() : returns already fetched web page;
get_response_code() : returns HTTP response code as integer (200, 404, etc.)
get_url() : returns original URL
get_real_url() : returns real URL retrieved
get_error() : returns exception/error condition
get_info() : rerturns server headers as a dict (Content-Type, etc.)
See: https://docs.python.org/3.4/howto/urllib2.html (Python3)
https://docs.python.org/2/howto/urllib2.html (Python2)
"""
Header = {
'User-Agent': 'Mozilla/5.0 AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11',
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
'Accept-Charset': 'ISO-8859-1,utf-8;q=0.7,*;q=0.3',
'Accept-Encoding': 'none',
'Accept-Language': 'en-US,en;q=0.8',
'Connection': 'keep-alive',
}
Deft_Html = 'no response'
Deft_Timeout = 10
MaxTries = 5
NoError = None
def __init__(self, paramDict=None):
self.params = paramDict
self.state = {
'url': None, #supplied url
'geturl': None, #actual url retrieved (possible redirect)
'status': None, #http response code (200, 404, etc)
'error': None, #exception raised with error message of last try
'tries': None, #number of tries
'timeout': None, #timeout of last try
'info': None, #meta info
'html': None, #the retrieved page
}
self._reset_state()
def _reset_state(self, url=None):
self.state['url'] = url
self.state['geturl'] = None
self.state['status'] = None
self.state['error'] = self.NoError
self.state['html'] = self.Deft_Html
self.state['info'] = {}
self.state['tries'] = 0
try:
self.state['timeout'] = self.params['webTimeOut']
if self.state['timeout'] < 1:
self.state['timeout'] = self.Deft_Timeout
except Exception as e:
Logger.debug(e)
self.state['timeout'] = self.Deft_Timeout
def fetch(self, url):
if not isinstance(url, str):
self.state['error'] = "URL must be a string '{}'".format(str(url))
return self.state['html']
self._reset_state(url)
self.state['tries'] = 1
while True:
Logger.debug('try {!s}/{!s}/{!s}'.format(self.state['tries'],
self.MaxTries,
self.state['timeout']))
try:
req = Request(url, headers=self.Header)
response = urlopen(req, None, self.state['timeout']) #with timeout
except HTTPError as e:
self.state['error'] = 'HTTPError: ' + str(e.code)
except URLError as e:
self.state['error'] = 'URLError: ' + str(e.reason)
except Exception as e:
self.state['error'] = 'Exception: ' + str(e)
else:
self.state['status'] = response.getcode()
self.state['geturl'] = response.geturl()
self.state['info'] = response.info()
self.state['html'] = response.read().decode('utf-8', 'ignore')
self.state['error'] = self.NoError #cleanup
break #got something
Logger.error(self.state['error'])
if self.state['tries'] >= self.MaxTries:
break #give up
self.state['timeout'] *= 2
self.state['tries'] += 1
return self.state['html']
#pretty-print the object diagnostics for print() and str() calls
def __str__(self):
triesmaxtime = '{!s}/{!s}/{!s}'.format(
self.state['tries'], self.MaxTries, self.state['timeout'])
s = (" {:<}: {!s}" * 6)[2:]
s = s.format(
'status', self.state['status'],
'tries/max/timeout', triesmaxtime,
'error', self.state['error'],
'url', self.state['url'],
'geturl', self.state['geturl'],
'info', sorted(self.state['info'].items()),
)
return s
def fetched_ok(self):
if self.state['status'] == 200: return True
if self.state['html'] != self.Deft_Html: return True
if self.state['error'] == self.NoError: return True
return False
def get_html(self): return self.state['html']
def get_response_code(self): return self.state['status']
def get_url(self): return self.state['url']
def get_real_url(self): return self.state['geturl']
def get_error(self): return self.state['error']
def get_info(self): return self.state['info']
###########################################################################
if __name__ == '__main__':
def make_url(url, values):
if url is not None and values is not None:
return url + '?' + urlencode(values)
return url
p = ParameterDict.ParameterDict(None)
o = WebAgent(p) #create once, then reuse
Logger.info(sorted(p.items()))
url = 1234
Logger.debug('')
Logger.debug('Trying: ' + str(url))
o.fetch(url)
Logger.debug(o)
Logger.debug(o.get_html())
url = 'this is garbage'
Logger.debug('')
Logger.debug('Trying: ' + url)
o.fetch(url)
Logger.debug(o)
Logger.debug(o.get_html())
url = make_url('https://valid.looking.url', [('d',4), ('e',5), ('f',6)])
Logger.debug('')
Logger.debug('Trying: ' + url)
o.fetch(url)
Logger.debug(o)
Logger.debug(o.get_html())
url = 'https://query1.finance.yahoo.com/v7/finance/quote?symbols=^FTSE,^FTAS'
Logger.debug('')
Logger.debug('Trying: ' + url)
o.fetch(url)
Logger.debug(o)
Logger.debug(o.get_html())
Logger.info('End')
###########################################################################
cheers
desmid
Re: HYPTUS v11.31 Run-time error 5
Posted: November 12th, 2017, 3:15 pm
by kiloran
I'm back! Though completely overwhelmed by all the activity in this thread and helping the family with funeral arrangements, probate, sorting out accounts and passwords, and whatever.
It'll take me an age to read through all the posts, and especially the reams of stuff written by desmid (wonderful to have you on board, by the way, I get the feeling that your Python skills are rather in advance of my own!) so I've just jumped in and updated the "Hyp Top-Up Version-b-f-dev2.ods" to "Hyp Top-Up Version-b-f-dev3.ods", available at:
http://lemonfoolfinancialsoftware.weebl ... _page.html
A brief scan over the posts does suggest to me that the problem may be related to a LibreOffice bug I reported a few months ago. The LibreOffice team were able to reproduce the bug and implement a solution, though this has not yet been published.
Note: the following is only applicable to those of you who have problems. If the dev2 version is working OK, please ignore the rest of this message.
The main code has not been changed, but I've added some debug code to help identify the problem. To use it:
- Download the file and open it
- Select the Parameters sheet and change the value in cell B4 from 0 to 1
- Select the High Yield Portfolio sheet and click on "Get Yahoo Prices"
When this runs, it will append a bunch of data into a file HYPTUSS_log.txt
Everything you do will append more data to this file, so before you do anything else, change Parameters cell B4 back to 0
Then send an email to the address at the bottom of
http://lemonfoolfinancialsoftware.weebly.com/, tell us the OS you are using, whether LibreOffice or OpenOffice, and which version, and add the HYPTUSS_log.txt file as an attachment. It would also be good if you could add your LMF username.
I'll do all I can to read all the posts and catch up, especially with desmid's contributions.
--kiloran
Re: HYPTUS v11.31 Run-time error 5
Posted: November 12th, 2017, 5:30 pm
by NeilW
No python3 on a Mac.
2.7.10 is as good as it gets
![Smile :-)](./images/smilies/icon_e_smile.gif)
Re: HYPTUS v11.31 Run-time error 5
Posted: November 12th, 2017, 5:43 pm
by NeilW
Well that confirmed my suspicions it was SSL fun. Within LibreOffice we're getting "<urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:719)>". No problem at all outside it or within Safari.
So we're back to the problem we dodged back in May...
Re: HYPTUS v11.31 Run-time error 5
Posted: November 12th, 2017, 5:47 pm
by kiloran
NeilW wrote:No python3 on a Mac.
2.7.10 is as good as it gets
![Smile :-)](./images/smilies/icon_e_smile.gif)
I'm not sure what you are referring to, Neil.
LibreOffice has built-in Python 3
OpenOffice has built-in Python 2.7
HYPTUSS should work with either (it detects which version of Python is in use, and adapts accordingly)
--kiloran
Re: HYPTUS v11.31 Run-time error 5
Posted: November 12th, 2017, 5:50 pm
by kiloran
NeilW wrote:Well that confirmed my suspicions it was SSL fun. Within LibreOffice we're getting "<urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:719)>". No problem at all outside it or within Safari.
So we're back to the problem we dodged back in May...
Where are you seeing this, Neil? I'm not sure which message you are replying to. It does sound rather like the bug I reported to LibreOffice. I have a work-around for Windows, don't know if it might apply to a Mac
--kiloran