Merge pull request #796 from guydavis/integration

Chia 1.6.1

.github/workflows/develop-chia.yaml

@@ -44,7 +44,7 @@ jobs:
             "MACHINARIS_STREAM=develop"
             "CHIADOG_BRANCH=dev"
             "CHIA_BRANCH=release/1.6.1"
-            "BLADEBIT_BRANCH=2.0.0-beta1"
+            "BLADEBIT_BRANCH=develop"
           tags: |
             ${{ secrets.DOCKERHUB_USERNAME }}/machinaris:develop
             ghcr.io/${{ secrets.DOCKERHUB_USERNAME }}/machinaris:develop

.github/workflows/develop-chives.yaml

@@ -43,7 +43,7 @@ jobs:
             "UBUNTU_VER=focal"
             "MACHINARIS_STREAM=develop"
             "CHIADOG_BRANCH=dev"
-            "CHIVES_BRANCH=1.3.5"
+            "CHIVES_BRANCH=1.5.1"
           tags: |
             ${{ secrets.DOCKERHUB_USERNAME }}/machinaris-chives:develop
             ghcr.io/${{ secrets.DOCKERHUB_USERNAME }}/machinaris-chives:develop

.github/workflows/main-chives.yaml

@@ -43,7 +43,7 @@ jobs:
           build-args: |
             "UBUNTU_VER=focal"
             "MACHINARIS_STREAM=latest"
-            "CHIVES_BRANCH=1.3.5"
+            "CHIVES_BRANCH=1.5.1"
           tags: |
             ${{ secrets.DOCKERHUB_USERNAME }}/machinaris-chives:latest
             ${{ secrets.DOCKERHUB_USERNAME }}/machinaris-chives:v${{ github.event.inputs.version }}

.github/workflows/test-chia.yaml

@@ -44,7 +44,6 @@ jobs:
             "MACHINARIS_STREAM=test"
             "CHIADOG_BRANCH=dev"
             "CHIA_BRANCH=release/1.6.1"
-            "BLADEBIT_BRANCH=2.0.0-beta1"
           tags: |
             ${{ secrets.DOCKERHUB_USERNAME }}/machinaris:test
             ghcr.io/${{ secrets.DOCKERHUB_USERNAME }}/machinaris:test

.github/workflows/test-chives.yaml

@@ -43,7 +43,7 @@ jobs:
             "UBUNTU_VER=focal"
             "MACHINARIS_STREAM=test"
             "CHIADOG_BRANCH=dev"
-            "CHIVES_BRANCH=1.3.5"
+            "CHIVES_BRANCH=1.5.1"
           tags: |
             ${{ secrets.DOCKERHUB_USERNAME }}/machinaris-chives:test
             ghcr.io/${{ secrets.DOCKERHUB_USERNAME }}/machinaris-chives:test

CHANGELOG.md

@@ -2,13 +2,30 @@
 
 All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
-## [0.8.5] - 2022-10-?
+## [Unreleased]
+
+## [0.8.5] - 2022-11-03
+### Added
  - Wallets page - Claim Rewards button offers portable plot reward recovery after one week has elapsed. (EARLY BETA!)
  - Chart memory usage per container (GiB) as well as total host memory usage (%) for OS and all apps.
  - Enhanced Forktools to optionally decrease a blockchain's full_node process count, which greatly limits memory usage.
  - Improve Plotting page to display configured tmp, dst, and archiving directories before starting to plot. Support `site_path_filter` for archive folders under `site_root`.
- - Fixes: Enhance 'NFT Reward Recovery' tool to support v2 databases. Fix for invalid Chiadog harvester alerts. Fix for bladebit diskplot mode (NOTE: ramplot mode under development again by original author.)
- - Update: [Chia](https://github.com/Chia-Network/chia-blockchain/releases/tag/1.6.1) to v1.6.1, [Chinilla](https://github.com/Chinilla/chinilla-blockchain/releases/tag/1.3.0 to v1.3.0, [Littlelambocoin](https://github.com/BTCgreen-Network/littlelambocoin-blockchain/releases/tag/1.6.1) to v1.6.1, [Maize](https://github.com/Maize-Network/maize-blockchain/releases/tag/1.6.0) to v1.6.0, [MMX](https://github.com/madMAx43v3r/mmx-node) to `testnet8`.
+  - Bladebit support for new `ramplot` mode (needs 416 GB RAM) 
+### Changed
+ - Enhance 'NFT Reward Recovery' tool to support v2 databases. 
+ - Fixes for invalid Chiadog harvester alerts. 
+ - Fixes for bladebit `diskplot` mode (less than 416 GB RAM, needs 400+GB of tmp disk) 
+### Updated
+ - [Bladebit](https://github.com/Chia-Network/bladebit/releases/tag/v2.0.0) to v2.0.0
+ - [Chia](https://github.com/Chia-Network/chia-blockchain/releases/tag/1.6.1) to v1.6.1
+ - [Chinilla](https://github.com/Chinilla/chinilla-blockchain/releases/tag/1.3.0 to v1.3.0
+ - [Littlelambocoin](https://github.com/BTCgreen-Network/littlelambocoin-blockchain/releases/tag/1.6.1) to v1.6.1
+ - [Maize](https://github.com/Maize-Network/maize-blockchain/releases/tag/1.6.0) to v1.6.0
+ - [MMX](https://github.com/madMAx43v3r/mmx-node) to `testnet8`.
+ - [Tad](https://github.com/BTCgreen-Network/tad-blockchain/releases/tag/1.6.0) to v1.6.0
+### Known Issues
+ - Incorrect reward recovery calculations for some blockchains.  Please use AllTheBlocks site if this affects you.
+ - Chia CLI broke `chia plots check`, affecting new plots on Farming page.  Still investigating...
 
 ## [0.8.4] - 2022-09-21
 - Scaling-Down: Optional mode where wallets are synced daily, not run 24/7.  Saves ~35% memory so smaller farmers can farm more blockchains concurrently on the same machine. See Wallets page, top-right Settings.

CREDITS.md

@@ -100,7 +100,8 @@ A big thanks to all that contributed with dev and test including:
 * @fabriziocacicia (Fabrizio Cacicia)
 * @bdeprez (Bernie Deprez)
 * Gnomuz
-* borifrmr
+* borifrmr (aka Inabon)
+* DeathandDestruction
 
 ## Trademark Notice
 CHIA NETWORK INC, CHIA™, the CHIA BLOCKCHAIN™, the CHIA PROTOCOL™, CHIALISP™ and the “leaf Logo” (including the leaf logo alone when it refers to or indicates Chia), are trademarks or registered trademarks of Chia Network, Inc., a Delaware corporation.  *There is no affliation between this Machinaris project and the main Chia Network project.*

api/commands/plotman_cli.py

@@ -240,7 +240,7 @@ def find_plotting_job_log(plot_id):
         try:
             if filename.endswith(".log") and not filename.startswith('plotman.') and not filename.startswith('archiver.'):
                 with open(os.path.join(str(dir_path), filename)) as logfile:
-                    for line in itertools.islice(logfile, 0, 35):
+                    for line in itertools.islice(logfile, 0, 50):
                         if plot_id in line:
                             return os.path.join(str(dir_path), filename)
                 continue

api/commands/rewards.py

@@ -29,7 +29,8 @@ def reward_recovery(wallet_id, launcher_id, pool_contract_address):
         if blockchain in BLOCKCHAINS_WITH_PLOTNFT_CLAIM_CMD:
             for wallet in db.session.query(w.Wallet).filter(w.Wallet.blockchain==blockchain).order_by(w.Wallet.blockchain).all():
                 for wallet_num in wallet.wallet_nums():
-                    execute_plotnft_claim_recovery(wallet_num)
+                    if wallet.get_wallet_type(wallet_num) == 'POOLING_WALLET':
+                        execute_plotnft_claim_recovery(wallet_num)
         else: # Try old FD-CLI recovery instead
             execute_fd_cli_recovery(wallet_id, launcher_id, pool_contract_address)
 

api/gunicorn.conf.py

@@ -50,8 +50,9 @@ def on_starting(server):
 
     # Status for both farmers and harvesters (includes fullnodes)
     if globals.farming_enabled() or globals.harvesting_enabled():
-        scheduler.add_job(func=status_challenges.update, name="status_challenges", trigger='interval', seconds=JOB_FREQUENCY, jitter=JOB_JITTER)
-        scheduler.add_job(func=status_alerts.update, name="status_alerts", trigger='interval', seconds=JOB_FREQUENCY, jitter=JOB_JITTER) 
+        if not 'mmx' in globals.enabled_blockchains():
+            scheduler.add_job(func=status_challenges.update, name="status_challenges", trigger='interval', seconds=JOB_FREQUENCY, jitter=JOB_JITTER)
+            scheduler.add_job(func=status_alerts.update, name="status_alerts", trigger='interval', seconds=JOB_FREQUENCY, jitter=JOB_JITTER) 
         scheduler.add_job(func=log_rotate.execute, name="log_rotate", trigger='cron', minute=0)  # Hourly
 
     if globals.farming_enabled() and 'chia' in globals.enabled_blockchains():  # For now, only Chia fullnodes
@@ -92,7 +93,7 @@ def on_starting(server):
         scheduler.add_job(func=stats_balances.collect, name="stats_balances", trigger='cron', minute=0)  # Hourly
 
     # Testing only
-    #scheduler.add_job(func=status_plotnfts.update, name="status_plotnfts", trigger='interval', seconds=10) # Test immediately
+    #scheduler.add_job(func=plots_check.execute, name="plots_check", trigger='interval', seconds=60) # Test immediately
     #scheduler.add_job(func=stats_effort.collect, name="stats_effort", trigger='interval', seconds=10) # Test immediately
     #scheduler.add_job(func=stats_balances.collect, name="stats_balances", trigger='interval', seconds=10) # Test immediately
     #scheduler.add_job(func=websvcs.get_chain_statuses, name="get_chain_statuses", trigger='interval', seconds=10) # Test immediately

api/models/mmx.py

@@ -60,9 +60,11 @@ def calc_mmx_etw(self, netspace_size, plots_size):
         #app.logger.info("MMX PLOTS_SIZE: {0}".format(plots_size_gib))
         netspace_size_gib = converters.str_to_gibs(netspace_size)
         #app.logger.info("MMX NETSPACE_SIZE: {0}".format(netspace_size_gib))
-        etw_mins = round(netspace_size_gib / plots_size_gib /  MMX_BLOCKS_PER_MIN)
-        #app.logger.info("MMX ETW_MINS: {0}".format(etw_mins))
-        return converters.format_minutes(etw_mins)
+        if plots_size_gib > 0:
+            etw_mins = round(netspace_size_gib / plots_size_gib /  MMX_BLOCKS_PER_MIN)
+            #app.logger.info("MMX ETW_MINS: {0}".format(etw_mins))
+            return converters.format_minutes(etw_mins)
+        return "" # Zero plots, so basically never
 
 class HarvesterSummary:
 

api/schedules/plots_check.py

@@ -112,7 +112,7 @@ def request_analyze(plot_file, workers):
     return [None, None, None]
 
 def set_check_status(workers, status, plot):
-    #app.logger.info("Checking for plot check of {0}".format(plot.plot_id))
+    app.logger.debug("Checking for plot check of {0}".format(plot.plot_id))
     check_log = CHECK_LOGS + '/' + plot.plot_id[:8] + '.log'
     check_status = None
     requested_status = False
@@ -155,20 +155,20 @@ def set_check_status(workers, status, plot):
 def request_check(plot, workers):
     # Don't know which harvester might have the plot result so try them in-turn
     for harvester in workers:
-        #app.logger.info("{0}:{1} - {2} - {3}".format(harvester.hostname, harvester.port, harvester.blockchain, harvester.mode))
+        app.logger.debug("{0}:{1} - {2} - {3}".format(harvester.hostname, harvester.port, harvester.blockchain, harvester.mode))
         if harvester.mode == 'fullnode' or 'harvester' in harvester.mode:
             if harvester.latest_ping_result != "Responding":
-                #app.logger.info("Skipping check call to {0} as last ping was: {1}".format( \
-                #    harvester.hostname, harvester.latest_ping_result))
+                app.logger.debug("Skipping check call to {0} as last ping was: {1}".format( \
+                    harvester.hostname, harvester.latest_ping_result))
                 continue
             if harvester.hostname != plot.hostname or harvester.blockchain != plot.blockchain:
-                #app.logger.info("Skipping check call to {0} ({1}) for plot on {2} ({3})".format( \
-                #    harvester.hostname, harvester.blockchain, plot.hostname, plot.blockchain))
+                app.logger.debug("Skipping check call to {0} ({1}) for plot on {2} ({3})".format( \
+                    harvester.hostname, harvester.blockchain, plot.hostname, plot.blockchain))
                 continue
             try:
                 app.logger.debug("Trying {0}:{1} for plot check....".format(harvester.hostname, harvester.port))
                 payload = {"service":"farming", "action":"check", "plot_file": plot.dir + '/' + plot.file }
-                response = utils.send_worker_post(harvester, "/analysis/", payload, debug=False)
+                response = utils.send_worker_post(harvester, "/analysis/", payload, debug=True)
                 if response.status_code == 200:
                     return [harvester.hostname, harvester.displayname, response.content.decode('utf-8')]
                 elif response.status_code == 404:

api/translations/de_DE/LC_MESSAGES/messages.po

@@ -7,7 +7,7 @@ msgstr ""
 "Project-Id-Version: Machinaris VERSION\n"
 "Report-Msgid-Bugs-To: https://github.com/guydavis/machinaris"
 "/issuesCreation-Date: 2022-02-16 15:00-0700\n"
-"POT-Creation-Date: 2022-10-22 06:07-0600\n"
+"POT-Creation-Date: 2022-10-26 09:06-0600\n"
 "PO-Revision-Date: 2022-02-15 14:29-0700\n"
 "Last-Translator: ApfelBirneKreis\n"
 "Language: de\n"

api/translations/fr_FR/LC_MESSAGES/messages.po

@@ -7,7 +7,7 @@ msgstr ""
 "Project-Id-Version: Machinaris VERSION\n"
 "Report-Msgid-Bugs-To: https://github.com/guydavis/machinaris"
 "/issuesCreation-Date: 2022-02-16 15:00-0700\n"
-"POT-Creation-Date: 2022-10-22 06:07-0600\n"
+"POT-Creation-Date: 2022-10-26 09:06-0600\n"
 "PO-Revision-Date: 2022-02-15 14:29-0700\n"
 "Last-Translator: Guy Davis\n"
 "Language: fr\n"

api/translations/it_IT/LC_MESSAGES/messages.po

@@ -7,7 +7,7 @@ msgstr ""
 "Project-Id-Version: Machinaris VERSION\n"
 "Report-Msgid-Bugs-To: https://github.com/guydavis/machinaris"
 "/issuesCreation-Date: 2022-02-16 15:00-0700\n"
-"POT-Creation-Date: 2022-10-22 06:07-0600\n"
+"POT-Creation-Date: 2022-10-26 09:06-0600\n"
 "PO-Revision-Date: 2022-02-15 14:29-0700\n"
 "Last-Translator: Fabrizio Cacicia\n"
 "Language: it\n"

api/translations/nl_NL/LC_MESSAGES/messages.po

@@ -7,7 +7,7 @@ msgstr ""
 "Project-Id-Version: Machinaris VERSION\n"
 "Report-Msgid-Bugs-To: https://github.com/guydavis/machinaris/issuesPOT-"
 "Creation-Date: 2022-03-13 10:30-0600\n"
-"POT-Creation-Date: 2022-10-22 06:07-0600\n"
+"POT-Creation-Date: 2022-10-26 09:06-0600\n"
 "PO-Revision-Date: 2022-03-13 10:29-0600\n"
 "Last-Translator: Bernie Deprez\n"
 "Language: nl_NL\n"

api/translations/pt_PT/LC_MESSAGES/messages.po

@@ -7,7 +7,7 @@ msgstr ""
 "Project-Id-Version: Machinaris VERSION\n"
 "Report-Msgid-Bugs-To: https://github.com/guydavis/machinaris"
 "/issuesCreation-Date: 2022-02-16 15:00-0700\n"
-"POT-Creation-Date: 2022-10-22 06:07-0600\n"
+"POT-Creation-Date: 2022-10-26 09:06-0600\n"
 "PO-Revision-Date: 2022-02-13 13:45-0700\n"
 "Last-Translator: Antonio Casqueiro\n"
 "Language: pt\n"

api/translations/zh/LC_MESSAGES/messages.po

@@ -7,7 +7,7 @@ msgstr ""
 "Project-Id-Version: Machinaris VERSION\n"
 "Report-Msgid-Bugs-To: https://github.com/guydavis/machinaris"
 "/issuesCreation-Date: 2022-02-16 15:00-0700\n"
-"POT-Creation-Date: 2022-10-22 06:07-0600\n"
+"POT-Creation-Date: 2022-10-26 09:06-0600\n"
 "PO-Revision-Date: 2022-02-15 14:29-0700\n"
 "Last-Translator: Guy Davis\n"
 "Language: zh\n"

common/config/blockchains.json

@@ -281,7 +281,7 @@
         "farmer_port": 11330,
         "worker_port": 8940,
         "mojos_per_coin": 1000000,
-        "blocks_per_day": 10800,
+        "blocks_per_day": 8640,
         "git_url": "https://github.com/madMAx43v3r/mmx-node",
         "discord_url": "https://discord.gg/pQwkebKnPB",
         "website_url": "https://github.com/madMAx43v3r/mmx-node"

common/config/globals.py

@@ -90,10 +90,10 @@ def load():
     fullnode_db_version = load_fullnode_db_version()
     if fullnode_db_version:
         cfg['fullnode_db_version'] = fullnode_db_version
-    if cfg['enabled_blockchains'][0] == 'mmx':
-        cfg['mmx_reward'] = gather_mmx_reward()
     if cfg['machinaris_mode'] == 'fullnode':
         cfg['wallet_status'] = "running" if wallet_running() else "paused"
+        if cfg['enabled_blockchains'][0] == 'mmx':
+            cfg['mmx_reward'] = gather_mmx_reward()
     return cfg
 
 def load_blockchain_info(blockchain, key):
@@ -221,6 +221,8 @@ def strip_data_layer_msg(lines):
 last_blockchain_version = None
 last_blockchain_version_load_time = None
 def load_blockchain_version(blockchain):
+    if blockchain == 'mmx':
+        return "recent" # Author didn't bother to version his binaries
     chia_binary = get_blockchain_binary(blockchain)
     global last_blockchain_version
     global last_blockchain_version_load_time
@@ -243,6 +245,8 @@ def load_blockchain_version(blockchain):
             # Chia version with .dev is actually one # to high, never fixed by Chia team...
             # See: https://github.com/Chia-Network/chia-blockchain/issues/5655
             sem_ver = last_blockchain_version.split('.')
+            if 'rc' in sem_ver[2]: # Strip out 'rcX' if found.
+                sem_ver[2] = sem_ver[2][:sem_ver[2].index('rc')]
             last_blockchain_version = sem_ver[0] + '.' + \
                 sem_ver[1] + '.' + str(int(sem_ver[2])-1)
         elif '.dev' in last_blockchain_version:
@@ -436,7 +440,6 @@ def load_fullnode_db_version():
     except:
         logging.info(traceback.format_exc())
     fullnode_db_version_load_time = datetime.datetime.now()
-    logging.info("Found neither!")
     return fullnode_db_version
 
 def get_disks(disk_type):

common/models/wallets.py

@@ -29,6 +29,17 @@ def wallet_nums(self):
                 wallet_nums.append(line.split(':')[1].strip())
         return wallet_nums
 
+    def get_wallet_type(self, wallet_num):
+        wallet_type = None
+        for line in self.details.split('\n'):
+            if "-Type:" in line:
+                wallet_type = line.split(':')[1].strip()
+            elif "Wallet ID:" in line:
+                current_wallet_num = line.split(':')[1].strip()
+                if wallet_num == current_wallet_num:
+                    return wallet_type
+        return None
+
     def is_synced(self):
         for line in self.details.split('\n'):
             if line.strip().startswith("Sync status: Synced"):

config/plotman.sample.yaml

@@ -44,13 +44,10 @@ directories:
 
 # See: https://github.com/guydavis/machinaris/wiki/Plotman#archiving
 #archiving:
-        #target: rsyncd
+        #target: local_rsync
         #env:
-                #site_root: /mnt/disks
-                #user: root
-                #host: aragorn
-                #rsync_port: 12000
-                #site: disks
+                #site_root: /
+                #site_path_filter: plots
 
 # Plotting scheduling parameters
 scheduling:
@@ -106,16 +103,8 @@ plotting:
         # If you enable 'madmax', plot in *sequence* with very low tmpdir_max_jobs and global_max_jobs
         type: madmax
 
-        chia:
-                # The stock plotter: https://github.com/Chia-Network/chia-blockchain
-                k: 32                # k-size of plot, leave at 32 most of the time
-                e: False             # Disable bitfield back sorting (default is True) 
-                n_threads: 2         # Threads per job
-                n_buckets: 128       # Number of buckets to split data into
-                job_buffer: 3389     # Per job memory
-
         madmax:
-                # madMAx plotter: https://github.com/madMAx43v3r/chia-plotter
+                # madMAx plotter; see https://github.com/guydavis/machinaris/wiki/MadMax
                 k: 32                 # Default is 32 for Chia, but can do 33 or 34. Use 29,30,31 for Chives
                 n_threads: 4          # Default is 4, SLOWLY crank up if you have many cores
                 n_buckets: 256        # Default is 256
@@ -124,8 +113,16 @@ plotting:
                 network_port: 8444    # Default is 8444, but use 9699 for Chives
 
         bladebit:
-                # BladeBit plotter: https://github.com/Chia-Network/bladebit/
+                # BladeBit plotter; see https://github.com/guydavis/machinaris/wiki/Bladebit
                 threads: 2            # Threads per job
-                no_numa: false        # Whether to disable NUMA memory support or not
-                diskplot: True        # Whether to plot to tmp1 (and tmp2) disks or only RAM (false)
+                no_numa: true         # Whether to disable NUMA memory support or not
+                diskplot: true        # Whether to plot to tmp1 (and tmp2) disks or only RAM (false)
                 diskplot_cache:   4GB # If diskplot, amount of RAM to allocate to caching disk writes
+
+        chia:
+                # The stock plotter; see https://github.com/guydavis/machinaris/wiki/Chia#plotting
+                k: 32                # k-size of plot, leave at 32 most of the time
+                e: False             # Disable bitfield back sorting (default is True) 
+                n_threads: 2         # Threads per job
+                n_buckets: 128       # Number of buckets to split data into
+                job_buffer: 3389     # Per job memory

docker/dockerfile

@@ -8,7 +8,7 @@ ARG PLOTMAN_BRANCH=development
 ARG CHIADOG_BRANCH=main
 ARG FDCLI_BRANCH=master
 ARG FORKTOOLS_BRANCH=testing
-ARG BLADEBIT_BRANCH=2.0.0-beta1
+ARG BLADEBIT_BRANCH=master
 
 ARG APPLE_BRANCH
 ARG BPX_BRANCH