v0.4 commit

README.md

@@ -21,25 +21,29 @@ a great [EVE Online](https://www.eveonline.com/) third-party tool for gathering
 PySpy connects to [CCP's ESI API](https://esi.evetech.net/ui/) and the
 [zKillboard API](https://github.com/zKillboard/zKillboard/wiki) and is available on Windows, macOS and Linux.
 
+In addition, PySpy uses a proprietary database which creates summary statistics for approximately 2.4 million EVE Online pilots, based on some 50 million killmails dating back to December 2007. This database is updated daily, shortly after CCP server downtime.
+
+If you enjoy using PySpy and would like to show your appreciation, please feel free to send ISK in-game to White Russsian (with 3 's'). Thank you.
+
 ## How to use PySpy
 
 1. Open PySpy.
 2. In your EVE client, select a list of characters and copy them to the clipboard (`CTRL+C` on Windows *or* `⌘+C` on macOS).
 3. Wait until PySpy is done and inspect the results.
 4. Double-click a name to open the respective zKillboard in your browser.
 
-**Note**: PySpy will save its window location, size, column sizes, sorting order and transparency (slider on bottom right) and any other settings automatically and restore them the next time you launch it. If selected in the _View Menu_, PySpy will stay on top of the EVE client so long as the game runs in *window mode*.
+**Note**: PySpy will save its window location, size, column sizes, sorting order and transparency (slider on bottom right) and any other settings automatically and restore them the next time you launch it (settings will be reset whenever you update to a new version). If selected in the _View Menu_, PySpy will stay on top of the EVE client so long as the game runs in *window mode*.
 
 ## Information Provided by PySpy
 
 ### New Dark Mode
 <p align="center">
-  <img alt="PySpy in action" src="https://github.com/WhiteRusssian/PySpy/blob/master/assets/v0.3_dark_screenshot.png?raw=true">
+  <img alt="PySpy in action" src="https://github.com/WhiteRusssian/PySpy/blob/master/assets/v0.4_dark_screenshot.png?raw=true">
 </p>
 
 ### Traditional Normal Mode
 <p align="center">
-  <img alt="PySpy in action" src="https://github.com/WhiteRusssian/PySpy/blob/master/assets/v0.3_light_screenshot.png?raw=true">
+  <img alt="PySpy in action" src="https://github.com/WhiteRusssian/PySpy/blob/master/assets/v0.4_light_screenshot.png?raw=true">
 </p>
 
 * **Character**: Character name.
@@ -53,8 +57,16 @@ PySpy connects to [CCP's ESI API](https://esi.evetech.net/ui/) and the
 * **Solo**: Ratio of solo kills over total kills.
 * **BLOPS**: Number of Black Ops Battleships (BLOPS) killed.
 * **HICs**: Number of lost Heavy Interdiction Cruisers (HIC).
+* **Last Loss**: Days since last loss.
+* **Last Kill**: Days since last kill.
+* **Avg. Attackers**: Average number of attackers per kill.
+* **Covert Cyno**: Ratio of losses where a covert cyno was fitted to total losses.
+* **Regular Cyno**: Ratio of losses where a regular cyno was fitted to total losses.
+* **Last Covert Cyno**: Ship type of most recent loss where covert cyno was fitted.
+* **Last Regular Cyno**: Ship type of most recent loss where regular cyno was fitted.
+* **Abyssal Losses**: Number of ship losses in Abyssal space.
 
-**Current Limitations**: To avoid undue strain on zKillboard's API, PySpy will run the *Kills*, *Losses*, *Last Wk*, *Solo*, *BLOPS* and *HICs* analyses only for the first 30 characters in the list.
+**Current Limitations**: To avoid undue strain on zKillboard's API, PySpy will run the *Kills*, *Losses*, *Last Wk*, *Solo*, *BLOPS* and *HICs* analyses only for the first 100 characters in the list.
 
 ## Ignore Certain Entities
 
@@ -86,9 +98,6 @@ Delete the PySpy executable and remove the following files manually:
 
 Below is a non-exhaustive list of additional features I plan to add to PySpy as and when the ESI and zKillboard APIs support them:
 
-* **Cynos**: Indicate if a character has in the past lost ships fitted with regular or covert cynos. I am currently putting together the underlying database of killmails and expect to add this feature in August.
-* **Average Attackers**: Average number of attackers across all kills a character has been involved in.
-* **Abyssal Losses**: Show number of losses in abyssal sites.
 * **Custom Highlighting**: Choose a list of characters, corproations or alliances to highlight.
 * **Standings**: Only show characters that are non-blue, i.e. neutral or hostile.
 * **Highlight New Pilots**: Highlight any pilots that have entered system since last PySpy run.
@@ -98,7 +107,7 @@ Please feel free to add a [feature request](https://github.com/WhiteRusssian/PyS
 
 ## Bug Reporting
 
-Despite PySpy's simplicity and extensive testing, you may encounter the odd bug. If so, please check if an existing [issue](https://github.com/WhiteRusssian/PySpy/issues) already describes your bug. If not, feel free to [create a new issue](https://github.com/WhiteRusssian/PySpy/issues/new?template=pyspy-bug-report.md) for your bug.
+Despite extensive testing, you may encounter the odd bug. If so, please check if an existing [issue](https://github.com/WhiteRusssian/PySpy/issues) already describes your bug. If not, feel free to [create a new issue](https://github.com/WhiteRusssian/PySpy/issues/new?template=pyspy-bug-report.md) for your bug.
 
 ## Dependencies & Acknowledgements
 

VERSION

@@ -1 +1 @@
-v0.3.1
+v0.4

__main__.py

@@ -89,13 +89,13 @@ def analyze_chars(char_names):
 background_thread = threading.Thread(
     target=watch_clpbd,
     daemon=True
-)
+    )
 background_thread.start()
 
 update_checker = threading.Thread(
     target=chkversion.chk_github_update,
     daemon=True
-)
+    )
 update_checker.start()
 
 app.MainLoop()

aboutdialog.py

@@ -25,8 +25,10 @@ def showAboutBox(parent, event=None):
     # __main__.app.PySpy.ToggleWindowStyle(wx.STAY_ON_TOP)
 
     description = """
-    PySpy is a a simple EVE Online character intel tool
-    using CCP's ESI API.
+    PySpy is an EVE Online character intel tool
+    using CCP's ESI API and a daily updated proprietary
+    database containing key statistics on approximately
+    2.4 million pilots.
 
     If you enjoy PySpy and want to show your appreciation
     to its author, you are welcome to send an ISK donation
@@ -43,7 +45,7 @@ def showAboutBox(parent, event=None):
     info = wx.adv.AboutDialogInfo()
 
     info.SetIcon(wx.Icon(config.ABOUT_ICON, wx.BITMAP_TYPE_PNG))
-    info.SetName("AboutDialog")
+    info.SetName("PySpy")
     info.SetVersion(config.CURRENT_VER)
     info.SetDescription(description)
     info.SetCopyright('(C) 2018 White Russsian')

analyze.py

@@ -19,6 +19,7 @@
 import statusmsg
 # cSpell Checker - Correct Words****************************************
 # // cSpell:words affil, zkill, blops, qsize, numid, russsian, ccp's
+# // cSpell:words records_added
 # **********************************************************************
 Logger = logging.getLogger(__name__)
 # Example call: Logger.info("Something badhappened", exc_info=True) ****
@@ -28,22 +29,36 @@ def main(char_names):
     conn, cur = db.connect_db()
     chars_found = get_char_ids(conn, cur, char_names)
     if chars_found > 0:
+        # Run Pyspy remote database query in seprate thread
+        tp = threading.Thread(
+            target=get_character_intel(conn, cur),
+            daemon=True
+            )
+        tp.start()
+
+        # Run zKill query in seprate thread
         char_ids = cur.execute(
             "SELECT char_id FROM characters ORDER BY char_name"
             ).fetchall()
         q_main = queue.Queue()
-        t = zKillStats(char_ids, q_main)
-        t.start()
+        tz = zKillStats(char_ids, q_main)
+        tz.start()
+
         get_char_affiliations(conn, cur)
         get_affil_names(conn, cur)
-        t.join()
+
+        # Join zKill thread
+        tz.join()
         zkill_stats = q_main.get()
         query_string = (
             '''UPDATE characters SET kills=?, blops_kills=?, hic_losses=?,
             week_kills=?, losses=?, solo_ratio=?, sec_status=?
             WHERE char_id=?'''
             )
         db.write_many_to_db(conn, cur, query_string, zkill_stats)
+
+        # Join Pyspy remote database thread
+        tp.join()
         output = output_list(cur)
         conn.close()
         return output
@@ -91,7 +106,7 @@ def get_char_affiliations(conn, cur):
         '''UPDATE characters SET corp_id=?, alliance_id=?, faction_id=?
         WHERE char_id=?'''
         )
-    records_added = db.write_many_to_db(conn, cur, query_string, records)
+    db.write_many_to_db(conn, cur, query_string, records)
 
 
 def get_affil_names(conn, cur):
@@ -172,18 +187,59 @@ def run(self):
         return
 
 
+def get_character_intel(conn, cur):
+    '''
+    Adds certain character killboard statistics derived from PySpy's
+    proprietary database to the local SQLite3 database.
+
+    :param `conn`: SQLite3 connection object.
+    :param `cur`: SQLite3 cursor object.
+    '''
+    char_ids = cur.execute("SELECT char_id FROM characters").fetchall()
+    char_intel = apis.post_proprietary_db(char_ids)
+    records = ()
+    for r in char_intel:
+        char_id = r["character_id"]
+        last_loss_date = r["last_loss_date"] if r["last_loss_date"] is not None else 0
+        last_kill_date = r["last_kill_date"] if r["last_kill_date"] is not None else 0
+        avg_attackers = r["avg_attackers"] if r["avg_attackers"] is not None else 0
+        covert_prob = r["covert_prob"] if r["covert_prob"] is not None else 0
+        normal_prob = r["normal_prob"] if r["normal_prob"] is not None else 0
+        last_cov_ship = r["last_cov_ship"] if r["last_cov_ship"] is not None else 0
+        last_norm_ship = r["last_norm_ship"] if r["last_norm_ship"] is not None else 0
+        abyssal_losses = r["abyssal_losses"] if r["abyssal_losses"] is not None else 0
+
+        records = records + ((
+            last_loss_date, last_kill_date, avg_attackers, covert_prob,
+            normal_prob, last_cov_ship, last_norm_ship, abyssal_losses, char_id
+            ), )
+
+    query_string = (
+        '''UPDATE characters SET last_loss_date=?, last_kill_date=?,
+        avg_attackers=?, covert_prob=?, normal_prob=?,
+        last_cov_ship=?, last_norm_ship=?, abyssal_losses=?
+        WHERE char_id=?'''
+        )
+    db.write_many_to_db(conn, cur, query_string, records)
+
+
 def output_list(cur):
     query_string = (
         '''SELECT
         ch.char_id, ch.faction_id, ch.char_name, co.id, co.name, al.id,
         al.name, fa.name, ac.numid, ch.week_kills, ch.kills, ch.blops_kills,
-        ch.hic_losses, ch.losses, ch.solo_ratio, ch.sec_status
+        ch.hic_losses, ch.losses, ch.solo_ratio, ch.sec_status,
+        ch.last_loss_date, ch.last_kill_date,
+        ch.avg_attackers, ch.covert_prob, ch.normal_prob,
+        IFNULL(cs.name,'-'), IFNULL(ns.name,'-'), ch.abyssal_losses
         FROM characters AS ch
         LEFT JOIN alliances AS al ON ch.alliance_id = al.id
         LEFT JOIN corporations AS co ON ch.corp_id = co.id
         LEFT JOIN factions AS fa ON ch.faction_id = fa.id
-        LEFT JOIN (SELECT alliance_id, COUNT(alliance_id) AS numid FROM characters GROUP BY alliance_id) AS ac ON
-        ch.alliance_id = ac.alliance_id
+        LEFT JOIN (SELECT alliance_id, COUNT(alliance_id) AS numid FROM characters GROUP BY alliance_id)
+            AS ac ON ch.alliance_id = ac.alliance_id
+        LEFT JOIN ships AS cs ON ch.last_cov_ship = cs.id
+        LEFT JOIN ships AS ns ON ch.last_norm_ship = ns.id
         ORDER BY ch.char_name'''
         )
     return cur.execute(query_string).fetchall()

apis.py

@@ -2,8 +2,8 @@
 # MIT licensed
 # Copyright (c) 2018 White Russsian
 # Github: <https://github.com/WhiteRusssian/PySpy>**********************
-''' This module provides connectivity to CCP's ESI API and to zKillboard's
-API.
+''' This module provides connectivity to CCP's ESI API, to zKillboard's
+API and to PySpy's own proprietary RESTful API.
 '''
 # **********************************************************************
 import json
@@ -23,6 +23,9 @@
 # Example call: Logger.info("Something badhappened", exc_info=True) ****
 
 
+# ESI Status
+# https://esi.evetech.net/ui/?version=meta#/Meta/get_status
+
 def post_req_ccp(esi_path, json_data):
     url = "https://esi.evetech.net/latest/" + esi_path + \
         "?datasource=tranquility"
@@ -143,3 +146,98 @@ def run(self):
             sec_status, self._char_id]
             )
         return
+
+
+def post_proprietary_db(character_ids):
+    '''
+    Query PySpy's proprietary kill database for the character ids
+    provided as a list or tuple of integers. Returns a JSON containing
+    one line per character id.
+
+    :param `character_ids`: List or tuple of character ids as integers.
+    :return: JSON dictionary containing certain statistics for each id.
+    '''
+    url = "http://pyspy.pythonanywhere.com" + "/character_intel/" + "v1/"
+    headers = {
+        "Accept-Encoding": "gzip",
+        "User-Agent": "PySpy, Author: White Russsian, https://github.com/WhiteRusssian/PySpy"
+        }
+    # Character_ids is a list of tuples, which needs to be converted to dict
+    # with list as value.
+    character_ids = {"character_ids": character_ids}
+    try:
+        r = requests.post(url, headers=headers, json=character_ids)
+    except requests.exceptions.ConnectionError:
+        Logger.info("No network connection.", exc_info=True)
+        statusmsg.push_status(
+            "NETWORK ERROR: Check your internet connection and firewall settings."
+            )
+        time.sleep(5)
+        return "network_error"
+    if r.status_code != 200:
+        server_msg = json.loads(r.text)["error"]
+        Logger.info(
+            "PySpy server returned error code: " +
+            str(r.status_code) + ", saying: " + server_msg, exc_info=True
+            )
+        statusmsg.push_status(
+            "PYSPY SERVER ERROR: " + str(r.status_code) + " (" + server_msg + ")"
+            )
+        return "server_error"
+    return r.json()
+
+
+def get_ship_data():
+    '''
+    Produces a list of ship id and ship name pairs for each ship in EVE
+    Online, using ESI's universe/names endpoint.
+
+    :return: List of lists containing ship ids and related ship names.
+    '''
+    all_ship_ids = get_all_ship_ids()
+    if not isinstance(all_ship_ids, (list, tuple)) or len(all_ship_ids) < 1:
+        Logger.error("[get_ship_data] No valid ship ids provided.", exc_info=True)
+        return
+
+    url = "https://esi.evetech.net/v2/universe/names/?datasource=tranquility"
+    json_data = json.dumps(all_ship_ids)
+    try:
+        r = requests.post(url, json_data)
+    except requests.exceptions.ConnectionError:
+        Logger.error("[get_ship_data] No network connection.", exc_info=True)
+        return "network_error"
+    if r.status_code != 200:
+        server_msg = json.loads(r.text)["error"]
+        Logger.error(
+            "[get_ship_data] CCP Servers returned error code: " +
+            str(r.status_code) + ", saying: " + server_msg, exc_info=True
+            )
+        return "server_error"
+    ship_data = list(map(lambda r: [r['id'], r['name']], r.json()))
+    return ship_data
+
+
+def get_all_ship_ids():
+    '''
+    Uses ESI's insurance/prices endpoint to get all available ship ids.
+
+    :return: List of ship ids as integers.
+    '''
+    url = "https://esi.evetech.net/v1/insurance/prices/?datasource=tranquility"
+
+    try:
+        r = requests.get(url)
+    except requests.exceptions.ConnectionError:
+        Logger.error("[get_ship_ids] No network connection.", exc_info=True)
+        return "network_error"
+    if r.status_code != 200:
+        server_msg = json.loads(r.text)["error"]
+        Logger.error(
+            "[get_ship_ids] CCP Servers at returned error code: " +
+            str(r.status_code) + ", saying: " + server_msg, exc_info=True
+            )
+        return "server_error"
+
+    ship_ids = list(map(lambda r: str(r['type_id']), r.json()))
+    Logger.info("[get_ship_ids] Number of ship ids found: " + str(len(ship_ids)))
+    return ship_ids

chkversion.py

@@ -29,13 +29,16 @@ def chk_github_update():
         # Get latest version available on GitHub
         GIT_URL = "https://api.github.com/repos/WhiteRusssian/PySpy/releases/latest"
         try:
-            latest_ver = requests.get(GIT_URL).json()["tag_name"]
+            # verify=False to avoid certificate errors. This is not critical.
+            latest_ver = requests.get(GIT_URL, verify=False).json()["tag_name"]
             Logger.info(
                 "You are running " + CURRENT_VER + " and " +
                 latest_ver + " is the latest version available on GitHub."
                 )
+            config.OPTIONS_OBJECT.Set("last_update_check", datetime.date.today())
+            if latest_ver != CURRENT_VER:
+                wx.CallAfter(__main__.app.PySpy.updateAlert, latest_ver, CURRENT_VER)
         except:
             Logger.info("Could not check GitHub for potential available updates.")
-        config.OPTIONS_OBJECT.Set("last_update_check", datetime.date.today())
-        if latest_ver != CURRENT_VER:
-            wx.CallAfter(__main__.app.PySpy.updateAlert, latest_ver, CURRENT_VER)
+
+