Merge pull request #223 from guydavis/integration

v0.5.2 release

.github/workflows/develop.yaml

@@ -39,7 +39,7 @@ jobs:
           platforms: linux/amd64
           push: true
           build-args: |
-            "CHIA_BRANCH=1.2.2"
+            "CHIA_BRANCH=1.2.3"
             "FLAX_BRANCH=main"
           tags: |
             ${{ secrets.DOCKERHUB_USERNAME }}/machinaris:develop

.github/workflows/main.yaml

@@ -37,10 +37,10 @@ jobs:
         uses: docker/build-push-action@v2
         with:
           context: .
-          platforms: linux/amd64
+          platforms: linux/amd64,linux/arm64
           push: true
           build-args: |
-            "CHIA_BRANCH=1.2.2"
+            "CHIA_BRANCH=1.2.3"
             "FLAX_BRANCH=main"
           tags: |
             ${{ secrets.DOCKERHUB_USERNAME }}/machinaris:latest

.github/workflows/test.yaml

@@ -1,10 +1,9 @@
 name: test
 
-on: 
-  workflow_dispatch:
-    inputs:
-      version:
-        description: 'Test Version' 
+on:
+  push:
+    branches:
+      - 'integration'
 
 jobs:
   docker:
@@ -37,10 +36,10 @@ jobs:
         uses: docker/build-push-action@v2
         with:
           context: .
-          platforms: linux/amd64
+          platforms: linux/amd64,linux/arm64
           push: true
           build-args: |
-            "CHIA_BRANCH=1.2.2"
+            "CHIA_BRANCH=1.2.3"
             "FLAX_BRANCH=main"
           tags: |
             ${{ secrets.DOCKERHUB_USERNAME }}/machinaris:test

CHANGELOG.md

@@ -2,11 +2,24 @@
 
 All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.5.2] - 2021-08-13
+
+- Machinaris - Docker images now available for [Apple M1](https://github.com/guydavis/machinaris/issues/43) and [Raspberry Pi OS](https://github.com/guydavis/machinaris/issues/155) architectures. 
+- Chiadog - Update to new v0.7.0 to [support for parsing partials and solo blocks](https://github.com/martomi/chiadog/pull/268).
+- Chia - Update to patch release of 1.2.3.  See their [changelog for details](https://github.com/Chia-Network/chia-blockchain/releases/tag/1.2.3).
+- Flax - Update to version 0.1.1.  See their [changelog for details](https://github.com/Flax-Network/flax-blockchain/releases/tag/0.1.1).
+- TrueNAS - Support for Machinaris deployment via helm chart. [Issue #78](https://github.com/guydavis/machinaris/issues/78) - Big thanks to @kmoore134 for this!
+- Machinaris - Daily Farming Summary now available on Farming page for both Chia and Flax.  Add new plot type column.
+- Machinaris - Pools - Show each Pool's status including link to your pool provider.  List pool point events on Summary page.
+- Machinaris - Workers - Use hostname for Worker display name, even when using IP addresses behind the scenes. Also show versions. Automated harvester setup. 
+- Machinaris - Connections page has link to test your router port forward for farming.
+- Machinaris - New [public website](http://www.machinaris.app) with launch Wizard for generating first Docker run/compose of Machinaris.
+
 ## [0.5.1] - 2021-07-22
 
 - Wizard on Workers page to create a Docker run/compose based on your settings. [Issue #97](https://github.com/guydavis/machinaris/issues/97)
-- Update to patch release of Chia 1.2.2, including a fix for harvester cache updates.  See their [changelog for details](https://github.com/Chia-Network/chia-blockchain/releases/tag/1.2.2).
 - Latest Madmax plotter with support for n_buckets3 and n_rmulti2 settings in Plotman.
+- Update to patch release of Chia 1.2.2, including a fix for harvester cache updates.  See their [changelog for details](https://github.com/Chia-Network/chia-blockchain/releases/tag/1.2.2).
 
 ## [0.5.0] - 2021-07-09
 

CREDITS.md

@@ -52,6 +52,7 @@ A big thanks to all that contributed with dev and test including:
 * barkcollar
 * dartec
 * elexx
+* danicraftscz
 
 
 ## Trademark Notice

README.md

@@ -1,10 +1,10 @@
 # machinaris
 
-A Docker image for plotting and farming the Chia™ cryptocurrency on [one computer](https://github.com/guydavis/machinaris/wiki/Docker) or across [many](https://github.com/guydavis/machinaris/wiki/Workers). Now with [official Pool support](https://github.com/guydavis/machinaris/wiki/Pooling)!
+A Docker image for plotting and farming the Chia™ cryptocurrency on [one computer](https://github.com/guydavis/machinaris/wiki/Docker) or across [many](https://github.com/guydavis/machinaris/wiki/Workers).  Try the easy install using [Launch Wizard](https://machinaris.app).
 
 ![Home](https://raw.githubusercontent.com/guydavis/machinaris-unraid/master/docs/img/machinaris_home.png)
 
-To get started with Machinaris, follow an install guide for your platform: [Windows](https://github.com/guydavis/machinaris/wiki/Windows), [Linux](https://github.com/guydavis/machinaris/wiki/Linux), [Macintosh](https://github.com/guydavis/machinaris/wiki/MacOS), [Unraid](https://github.com/guydavis/machinaris/wiki/Unraid), and [others](https://github.com/guydavis/machinaris/wiki/Docker).
+For details, see your particular platform: [Windows](https://github.com/guydavis/machinaris/wiki/Windows), [Linux](https://github.com/guydavis/machinaris/wiki/Linux), [Macintosh](https://github.com/guydavis/machinaris/wiki/MacOS), [Unraid](https://github.com/guydavis/machinaris/wiki/Unraid), and [others](https://github.com/guydavis/machinaris/wiki/Docker).
 
 ## Plotting View
 

VERSION

@@ -1 +1 @@
-0.5.1
+0.5.2

api/commands/chia_cli.py

@@ -2,6 +2,7 @@
 # CLI interactions with the chia binary.
 #
 
+import asyncio
 import datetime
 import os
 import pexpect
@@ -313,3 +314,11 @@ def check_plots(first_load):
         ansi_escape = re.compile(r'\x1B(?:[@-Z\\-_]|\[[0-?]*[ -/]*[@-~])')
         proc = Popen(['tail', '-n', str(MAX_LOG_LINES), output_file], stdout=PIPE)
         return  class_escape.sub('', ansi_escape.sub('', proc.stdout.read().decode("utf-8")))
+
+def get_pool_login_link(launcher_id):
+    try:
+        stream = os.popen("chia plotnft get_login_link -l {0}".format(launcher_id))
+        return stream.read()
+    except Exception as ex:
+        app.logger.info("Failed to get_login_link: {0}".format(str(ex)))
+    return ""

api/commands/log_parser.py

@@ -27,6 +27,9 @@
 # Roughly 1 minutes worth of challenges
 CHALLENGES_TO_LOAD = 8
 
+# Most recent partial proofs, actually double as 2 log lines per partial
+PARTIALS_TO_LOAD = 50
+
 # When reading tail of a log, only send this many lines
 MAX_LOG_LINES = 250
 
@@ -56,6 +59,34 @@ def recent_challenges(blockchain):
     # app.logger.debug(challenges)
     return challenges
 
+def recent_partials(blockchain):
+    log_file = CHIA_LOG
+    if blockchain == 'flax':
+        log_file = FLAX_LOG
+    if not os.path.exists(log_file):
+        app.logger.debug(
+            "Skipping partials parsing as no such log file: {0}".format(log_file))
+        return []
+    rotated_log_file = ''
+    if os.path.exists(log_file + '.1'):
+        rotated_log_file = log_file + '.1'
+    proc = Popen("grep -h --text -C1 -i partial {0} {1} | tail -n {2}".format(rotated_log_file, log_file, PARTIALS_TO_LOAD),
+                 stdout=PIPE, stderr=PIPE, shell=True)
+    try:
+        outs, errs = proc.communicate(timeout=90)
+    except TimeoutExpired:
+        proc.kill()
+        proc.communicate()
+        abort(500, description="The timeout is expired!")
+    if errs:
+        app.logger.error(errs.decode('utf-8'))
+        abort(500, description=errs.decode('utf-8'))
+    cli_stdout = outs.decode('utf-8')
+    #app.logger.debug("Partials grep: {0}".format(cli_stdout))
+    partials = log.Partials(cli_stdout.splitlines())
+    # app.logger.debug(partials)
+    return partials
+
 
 def find_plotting_job_log(plot_id):
     dir_path = '/root/.chia/plotman/logs'

api/gunicorn.conf.py

@@ -9,32 +9,37 @@ def on_starting(server):
     from api import app
     from api.schedules import status_worker, status_farm, status_plotting, \
         status_plots, status_challenges, status_wallets, status_blockchains, \
-        status_connections, status_keys, status_alerts, status_controller, status_plotnfts
+        status_connections, status_keys, status_alerts, status_controller, \
+        status_plotnfts, status_points, status_pools, status_partials
     from api.schedules import stats_disk, stats_farm
 
     scheduler = BackgroundScheduler()
 
     # Statistics gathering locally
-    scheduler.add_job(func=stats_farm.collect, trigger='cron', minute=0)  # Hourly
-    scheduler.add_job(func=stats_disk.collect, trigger='cron', minute="*/5") # Every 5 minutes
+    scheduler.add_job(func=stats_farm.collect, name="stats_farm", trigger='cron', minute=0)  # Hourly
+    scheduler.add_job(func=stats_disk.collect, name="stats_disk", trigger='cron', minute="*/5") # Every 5 minutes
 
     # Testing only
     #scheduler.add_job(func=stats_farm.collect, trigger='interval', seconds=10) # Test immediately
     #scheduler.add_job(func=stats_disk.collect, trigger='interval', seconds=10) # Test immediately
 
     # Status gathering - reported via API
-    scheduler.add_job(func=status_challenges.update, trigger='interval', seconds=5)
-    scheduler.add_job(func=status_worker.update, trigger='interval', seconds=60, jitter=30) 
-    scheduler.add_job(func=status_controller.update, trigger='interval', seconds=60, jitter=30) 
-    scheduler.add_job(func=status_farm.update, trigger='interval', seconds=60, jitter=30) 
-    scheduler.add_job(func=status_plotting.update, trigger='interval', seconds=60, jitter=30) 
-    scheduler.add_job(func=status_plots.update, trigger='interval', seconds=60, jitter=30)  
-    scheduler.add_job(func=status_wallets.update, trigger='interval', seconds=60, jitter=30) 
-    scheduler.add_job(func=status_plotnfts.update, trigger='interval', seconds=60, jitter=30) 
-    scheduler.add_job(func=status_blockchains.update, trigger='interval', seconds=60, jitter=30) 
-    scheduler.add_job(func=status_connections.update, trigger='interval', seconds=60, jitter=30) 
-    scheduler.add_job(func=status_keys.update, trigger='interval', seconds=60, jitter=30) 
-    scheduler.add_job(func=status_alerts.update, trigger='interval', seconds=60, jitter=30) 
+    scheduler.add_job(func=status_challenges.update, name="challenges", trigger='interval', seconds=5)
+    scheduler.add_job(func=status_worker.update, name="workers", trigger='interval', seconds=60, jitter=30) 
+    scheduler.add_job(func=status_controller.update, name="controller", trigger='interval', seconds=60, jitter=30) 
+    scheduler.add_job(func=status_farm.update, name="farms", trigger='interval', seconds=60, jitter=30) 
+    scheduler.add_job(func=status_plotting.update, name="plottings", trigger='interval', seconds=60, jitter=30) 
+    scheduler.add_job(func=status_plots.update, name="plots", trigger='interval', seconds=60, jitter=30)  
+    scheduler.add_job(func=status_wallets.update, name="wallets", trigger='interval', seconds=60, jitter=30) 
+    scheduler.add_job(func=status_plotnfts.update, name="plotnfts", trigger='interval', seconds=60, jitter=30) 
+    scheduler.add_job(func=status_blockchains.update, name="blockchains", trigger='interval', seconds=60, jitter=30) 
+    scheduler.add_job(func=status_connections.update, name="connections", trigger='interval', seconds=60, jitter=30) 
+    scheduler.add_job(func=status_keys.update, name="keys", trigger='interval', seconds=60, jitter=30) 
+    scheduler.add_job(func=status_alerts.update, name="alerts", trigger='interval', seconds=60, jitter=30) 
+    scheduler.add_job(func=status_pools.update, name="pools", trigger='interval', seconds=60, jitter=30)
+    scheduler.add_job(func=status_partials.update, name="partials", trigger='interval', seconds=60, jitter=30) 
+
+    #scheduler.add_job(func=status_points.update, name="points", trigger='interval', seconds=10, jitter=0) 
     app.logger.debug("Starting background scheduler...")
     scheduler.start()
 

api/migrations/README

@@ -5,7 +5,8 @@ As developer, modify models, then run this to generate a new migration and commi
 
 ```
 cd /code/machinaris/api
-FLASK_APP=__init__.py flask db migrate 
+FLASK_APP=__init__.py flask db migrate
+chown -R 1000.users .
 ```
 This creates migration based on current model.
 

api/migrations/versions/5ea0ec1d8711_.py

@@ -0,0 +1,62 @@
+"""empty message
+
+Revision ID: 5ea0ec1d8711
+Revises: bdfe8db75307
+Create Date: 2021-07-29 17:21:19.447778
+
+"""
+from alembic import op
+import sqlalchemy as sa
+
+
+# revision identifiers, used by Alembic.
+revision = '5ea0ec1d8711'
+down_revision = 'bdfe8db75307'
+branch_labels = None
+depends_on = None
+
+
+def upgrade(engine_name):
+    globals()["upgrade_%s" % engine_name]()
+
+
+def downgrade(engine_name):
+    globals()["downgrade_%s" % engine_name]()
+
+
+
+
+
+def upgrade_():
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.create_table('pools',
+    sa.Column('unique_id', sa.String(length=255), nullable=False),
+    sa.Column('hostname', sa.String(length=255), nullable=False),
+    sa.Column('blockchain', sa.String(length=64), nullable=True),
+    sa.Column('launcher_id', sa.String(length=255), nullable=False),
+    sa.Column('pool_state', sa.String(), nullable=False),
+    sa.Column('updated_at', sa.String(length=64), nullable=False),
+    sa.PrimaryKeyConstraint('unique_id')
+    )
+    op.add_column('workers', sa.Column('displayname', sa.String(length=255), nullable=True))
+    # ### end Alembic commands ###
+
+
+def downgrade_():
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.drop_column('workers', 'displayname')
+    op.drop_table('pools')
+    # ### end Alembic commands ###
+
+
+def upgrade_stats():
+    # ### commands auto generated by Alembic - please adjust! ###
+    pass
+    # ### end Alembic commands ###
+
+
+def downgrade_stats():
+    # ### commands auto generated by Alembic - please adjust! ###
+    pass
+    # ### end Alembic commands ###
+

api/migrations/versions/ce840f016302_.py

@@ -0,0 +1,52 @@
+"""empty message
+
+Revision ID: ce840f016302
+Revises: f529fc25f0d6
+Create Date: 2021-08-11 10:15:42.883128
+
+"""
+from alembic import op
+import sqlalchemy as sa
+
+
+# revision identifiers, used by Alembic.
+revision = 'ce840f016302'
+down_revision = 'f529fc25f0d6'
+branch_labels = None
+depends_on = None
+
+
+def upgrade(engine_name):
+    globals()["upgrade_%s" % engine_name]()
+
+
+def downgrade(engine_name):
+    globals()["downgrade_%s" % engine_name]()
+
+
+
+
+
+def upgrade_():
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.add_column('pools', sa.Column('login_link', sa.String(), nullable=True))
+    # ### end Alembic commands ###
+
+
+def downgrade_():
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.drop_column('pools', 'login_link')
+    # ### end Alembic commands ###
+
+
+def upgrade_stats():
+    # ### commands auto generated by Alembic - please adjust! ###
+    pass
+    # ### end Alembic commands ###
+
+
+def downgrade_stats():
+    # ### commands auto generated by Alembic - please adjust! ###
+    pass
+    # ### end Alembic commands ###
+