How to parse Json the easy way
Posted: February 11th, 2018, 8:01 pm
1) This command results in an unreadable json blob!
query1.finance.yahoo.com/v7/finance/quote?symbols=VOD.L
2) The solution; ............... install jq
https://stedolan.github.io/jq/
3) Now enter this code at the command line
curl 'query1.finance.yahoo.com/v7/finance/quote?symbols=VOD.L' | jq '.'
4) Or better save this code & make it executable; invoke with filename1 VOD.L at the command line
#!/bin/bash
qu="query1.finance.yahoo.com/v7/finance/quote?symbols=$1"
curl "$qu" | jq '.quoteResponse.result[]'
5) The code below makes the data "usable" by converting it to csv format. Save the code & make it executable; invoke it with filename2 VOD.L
6) If anyone uses this data source please let us know what you did and what happened.
##### Convert Json data to Csv and save to file. Invoke with filename PRU.L or filename MSFT etc ####
#!/bin/bash
#epic="PRU.L"
epic=$1
curl "query1.finance.yahoo.com/v7/finance/quote?symbols=$epic" > /root/my-applications/Yhoo/vod
echo "Getting data from Yahoo and saving to file"
######################### Get Json data from file & filter through jq #######################
IFS=$'\n'
i=0
ar=()
while IFS= read -r line; do
#printf '%s\n' "$line"
ar[$i]=$line
let i=i+1
done <<< "$(jq '.quoteResponse.result[0]' /root/my-applications/Yhoo/vod)"
Len=${#ar[@]}
unset ar[0] # Delete [ & ] at start and end of array
unset ar[$Len-1]
#for i in ${ar[@]};do echo "$i"; done
echo "Array filled and ready for processing into csv format"
################# Convert each line into csv format ####################
n=0
for ((i=0; i<=${Len}; i++)); do
ar="$(echo "${ar}" | tr -d '[:space:]')" # Delete spaces
ar="$(echo "${ar}" | tr -d '["]')" # Delete quotes
ar="$(echo "${ar}" | tr -d '[,]')" # Delete commas
ar=${ar/:/,} # Change colons into commas
#echo "${ar}"
let n=n+1
done
################## Convert TimeStamps to Time ########################
echo "Converting TimeStamps etc to Time"
n=0
for ((i=0; i<=${Len}; i++)); do
if [[ ${ar[$i]} =~ "Timestamp" ]] || [[ ${ar[$i]} =~ "MarketTime" ]] ;then
dte=$(date -d @$(echo "${ar[$i]}" | cut -d, -f2) +"%d-%b-%y") # Generate dates from values
#echo "$dte"
key=$(echo "${ar[$i]}" | cut -d, -f1) # Get key to data pair
ar[$i]=$key,$dte
#echo "SSSSSSSSSSSSSS ${ar[$i]}"
fi
let n=n+1
done
########### Empty file & Copy data to file & open file in spreadsheet ##########
echo > /root/my-applications/Yhoo/YhCsv # Empty file
IFS=$'\n'
for i in ${ar[@]}
do
echo $i >> /root/my-applications/Yhoo/YhCsv
done
echo "Saved $epic array to YhCsv file"
gnumeric /root/my-applications/Yhoo/YhCsv # Display results in a spreadsheet
#################################### The End #####################################
# tested on GNU bash, version 3.00.16(1)-release (i486-t2-linux-gnu) & Linux Puppy Wary 5.3 & Gnumeric Spreadsheet 1.10.16. Needs jq at https://stedolan.github.io/jq/ to be installed
# File references at lines 5, 15, 49, 53 will need changing to suit you
# Run with Filename VOD.L
query1.finance.yahoo.com/v7/finance/quote?symbols=VOD.L
2) The solution; ............... install jq
https://stedolan.github.io/jq/
3) Now enter this code at the command line
curl 'query1.finance.yahoo.com/v7/finance/quote?symbols=VOD.L' | jq '.'
4) Or better save this code & make it executable; invoke with filename1 VOD.L at the command line
#!/bin/bash
qu="query1.finance.yahoo.com/v7/finance/quote?symbols=$1"
curl "$qu" | jq '.quoteResponse.result[]'
5) The code below makes the data "usable" by converting it to csv format. Save the code & make it executable; invoke it with filename2 VOD.L
6) If anyone uses this data source please let us know what you did and what happened.
##### Convert Json data to Csv and save to file. Invoke with filename PRU.L or filename MSFT etc ####
#!/bin/bash
#epic="PRU.L"
epic=$1
curl "query1.finance.yahoo.com/v7/finance/quote?symbols=$epic" > /root/my-applications/Yhoo/vod
echo "Getting data from Yahoo and saving to file"
######################### Get Json data from file & filter through jq #######################
IFS=$'\n'
i=0
ar=()
while IFS= read -r line; do
#printf '%s\n' "$line"
ar[$i]=$line
let i=i+1
done <<< "$(jq '.quoteResponse.result[0]' /root/my-applications/Yhoo/vod)"
Len=${#ar[@]}
unset ar[0] # Delete [ & ] at start and end of array
unset ar[$Len-1]
#for i in ${ar[@]};do echo "$i"; done
echo "Array filled and ready for processing into csv format"
################# Convert each line into csv format ####################
n=0
for ((i=0; i<=${Len}; i++)); do
ar="$(echo "${ar}" | tr -d '[:space:]')" # Delete spaces
ar="$(echo "${ar}" | tr -d '["]')" # Delete quotes
ar="$(echo "${ar}" | tr -d '[,]')" # Delete commas
ar=${ar/:/,} # Change colons into commas
#echo "${ar}"
let n=n+1
done
################## Convert TimeStamps to Time ########################
echo "Converting TimeStamps etc to Time"
n=0
for ((i=0; i<=${Len}; i++)); do
if [[ ${ar[$i]} =~ "Timestamp" ]] || [[ ${ar[$i]} =~ "MarketTime" ]] ;then
dte=$(date -d @$(echo "${ar[$i]}" | cut -d, -f2) +"%d-%b-%y") # Generate dates from values
#echo "$dte"
key=$(echo "${ar[$i]}" | cut -d, -f1) # Get key to data pair
ar[$i]=$key,$dte
#echo "SSSSSSSSSSSSSS ${ar[$i]}"
fi
let n=n+1
done
########### Empty file & Copy data to file & open file in spreadsheet ##########
echo > /root/my-applications/Yhoo/YhCsv # Empty file
IFS=$'\n'
for i in ${ar[@]}
do
echo $i >> /root/my-applications/Yhoo/YhCsv
done
echo "Saved $epic array to YhCsv file"
gnumeric /root/my-applications/Yhoo/YhCsv # Display results in a spreadsheet
#################################### The End #####################################
# tested on GNU bash, version 3.00.16(1)-release (i486-t2-linux-gnu) & Linux Puppy Wary 5.3 & Gnumeric Spreadsheet 1.10.16. Needs jq at https://stedolan.github.io/jq/ to be installed
# File references at lines 5, 15, 49, 53 will need changing to suit you
# Run with Filename VOD.L