Scraping the Ecowater API with Python
I recently bought a house that has well water, and the quality of the water was very poor. Thus, I had to purchase a water treatment system. I wanted something with wifi, because, you know, we live in the future. Along with a pH system, and pre- and post-filters, I ended up with an Ecowater EWS ECR3702R30, which touts:
HydroLink Plus® Wi-Fi and Smartphone App Monitoring – Wi-Fi enabled technology sends continuous and excessive water use, system error, low salt and service reminder alerts, including a water-to-drain sensing alert notifying you when an excessive amount of water is going to drain.
Kind of already knowing what I was in store for, I got in touch with Ecowater technical support, to ask “Does the HydroLink support local, non-cloud access, or any type of API?” They asked me to reach out to their Business Development Manager, who responded: “Unfortunately at this time the information can only be accessed thru the app and our cloud.”
Not to my surprise, the HydroLink system is closed, so I set out to use python to scrape their website for my data.
Once logged in, I was lucky enough to find that the websites queries a JSON endpoint to update the data displayed on the page. So it was just a matter of logging in, querying the endpoint, and then doing with the data as I pleased.
I used python’s requests
library, along with requests.Session
, to simulate
logging in. There was also a hidden form input which contributed to authenticating
with the site, so that took a few extra lines of code.
Once I have the JSON data, I am sending it off to my mosquitto
MQTT broker.
You can also find this script as a github gist
#!/usr/bin/python3
import requests
import json
import re
import paho.mqtt.client as mqtt
import paho.mqtt.publish as publish
# Regex to match the hidden input on the initial log in page
request_validation_re = re.compile(r'<input name="__RequestVerificationToken" type="hidden" value="(.*?)" />')
# The serial number of your ecowater device
dsn = { "dsn": 'serialnumber' }
# The initial form data
payload = {
"Email" : "emailaddress",
"Password" : "password",
"Remember" : 'false'
}
# The headers needed for the JSON request
headers = {
'Accept': '*/*',
'Accept-Encoding': 'gzip, deflate, br',
'Accept-Language' : 'en-US,en;q=0.5',
'X-Requested-With': 'XMLHttpRequest',
'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8',
'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:73.0) Gecko/20100101 Firefox/73.0'
}
# MQTT server details
mqtt_host = '127.0.0.1'
mqtt_port = 1883
mqtt_user = 'username'
mqtt_pass = 'password'
with requests.Session() as s:
# Initial GET request
g = s.get('https://www.wifi.ecowater.com/Site/Login')
# Grab the token from the hidden input
tokens = request_validation_re.findall(g.text)
# Add the token to the form data payload
payload['__RequestVerificationToken'] = tokens[0]
# Log in to the site
login = s.post('https://www.wifi.ecowater.com/Site/Login', data=payload)
# Add the correct Referer header
headers['Referer'] = login.url + '/' + dsn['dsn']
# Query the JSON endpoint for the data that we actually want
data = s.post('https://www.wifi.ecowater.com/Dashboard/UpdateFrequentData', data=dsn, headers=headers)
# Load the data in to json
json = json.loads(data.text)
# Delete the elements that we don't want
del json['out_of_salt']
del json['out_of_salt_days']
del json['water_units']
del json['time']
del json['recharge']
# Placeholder for each message
messages = []
# Format each piece of json data in to a mqtt 'message'
for d in json:
msg = {
'topic': 'ecowater/' + d,
'payload': json[d]
}
messages.append(msg)
# Publish the message, consisting of all of the json data
if len(messages) > 0:
publish.multiple(messages, hostname=mqtt_host, port=mqtt_port, auth={'username': mqtt_user, 'password': mqtt_pass})
Once the data is published, I am consuming it with Home Assistant, via a couple of sensors:
sensor:
- platform: mqtt
name: Well Current
state_topic: "ct/irms"
availability_topic: "checkIn/ct/well"
unit_of_measurement: 'amps'
- platform: mqtt
name: Ecowater Salt Level
state_topic: "ecowater/salt_level"
- platform: mqtt
name: Ecowater Salt Level Percent
state_topic: "ecowater/salt_level_percent"
unit_of_measurement: '%'
- platform: mqtt
name: Ecowater Water Today
state_topic: "ecowater/water_today"
unit_of_measurement: "gallons"
- platform: mqtt
name: Ecowater Water Average
state_topic: "ecowater/water_avg"
unit_of_measurement: "gallons"
- platform: mqtt
name: Ecowater Water Available
state_topic: "ecowater/water_avail"
unit_of_measurement: "gallons"
- platform: mqtt
name: Ecowater Water Flow
state_topic: "ecowater/water_flow"
unit_of_measurement: "GPM"
binary_sensor:
- platform: mqtt
name: Ecowater Online
state_topic: "ecowater/online"
payload_on: True
payload_off: False
- platform: mqtt
name: Ecowater Recharge Enabled
state_topic: "ecowater/rechargeEnabled"
payload_on: True
payload_off: False