Skip to content

Commit

Permalink
Merge pull request #133 from NREL/cleanup
Browse files Browse the repository at this point in the history 
cleanup readme and rework BCL tasks
  • Loading branch information
@nllong
nllong authored Oct 4, 2021
2 parents cd4d9f1 + 4fa3a75 commit 4c61f87
Show file tree
Hide file tree
Showing 2 changed files with 19 additions and 213 deletions.
31 changes: 18 additions & 13 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -10,10 +10,10 @@ Derivative extension gems should include this gem to access common functionality
* OpenStudio CLI functionality such as list_measures and update_measures
* adding documentation and license files to measures
* adding core resource files to measures
* pushing measures to BCL
* correct structure for contributing content to the Building Component Library (BCL)

Extension gems will contain a small group of related measures. Each extension gem will be the unique location to find these measures, and will be responsible for
testing and maintaining the measures as well as pushing them to BCL.
testing and maintaining the measures as well as indexing them on BCL.

## Usage

Expand All @@ -31,7 +31,7 @@ Each OpenStudio extension gem should define its own module name to ensure that t

## Installation

To use the latest version of this and other extension gems, you will need Ruby 2.7.x and OpenStudio 3.1.0 or greater.
To use the latest version of this and other extension gems, you will need Ruby 2.7.x and OpenStudio 3.1.0 or greater. For earlier versions, view the [compatibility matrix](#compatibility-matrix) below.

### Windows Installation
Install Ruby with Devkit using the [RubyInstaller](https://rubyinstaller.org/downloads/archives/) for [Ruby 2.7.2 (x64)](
Expand Down Expand Up @@ -81,6 +81,14 @@ the OpenStudio Application you just downloaded (replace 3.1.0 with the version y
```
export RUBYLIB=/Applications/OpenStudio-3.1.0/Ruby
```
## Compatibility Matrix

|OpenStudio Extension Gem|OpenStudio|Ruby|
|:--------------:|:----------:|:--------:|
| 0.4.0 - 0.4.4 | 3.2 | 2.7 |
| 0.3.0 - 0.3.2 | 3.1 | 2.5 |
| 0.2.0 - 0.2.6 | 3.0 | 2.5 |
| 0.1.6 and below | 2.9 and below | 2.2.4 |


## Contents
Expand Down Expand Up @@ -139,18 +147,15 @@ Note that this folder is for 'core' functionality; if a measure's requires a new
Having a single repository for all measures, such as the OpenStudio-measures repo, can be cumbersome to test and keep up to date.
In this new framework, each extension gem will contain one or more related measures. The gem will be the new 'home' of these measures, and the repo owner will be responsible for testing and keeping the measures up to date.

In the short term, in order to preserve the PAT/OS App functionality, resource files will still be copied directly into the measures, and these measures will be pushed to BCL.
In the short term, in order to preserve the PAT/OS App functionality, resource files will still be copied directly into the measures, and these measures will be indexed on the BCL.

### Rake Tasks

Common Rake Tasks that are available to derivative extension gems include:

| Rake Task | Description |
| --------- | ----------- |
| openstudio:bcl:test_login | Test BCL login |
| openstudio:bcl:search_measures | Search BCL |
| openstudio:bcl:stage[reset] | Copy the measures/components to a staging location |
| openstudio:bcl:push | Upload measures from the staging location |
| openstudio:change_log[start_date,end_date,apikey] | Print the change log from GitHub. Specify dates in yyyy-mm-dd format |
| openstudio:list_measures | List all measures in the calling gem |
| openstudio:measures:add_license | Add License File to measures in the calling gem |
Expand Down Expand Up @@ -210,12 +215,8 @@ The OpenStudio-extension gem can be used to easily initialize a new derivative e
* All files and classes should have underscores (no dashes) and (lowercase snake_case)
* Dashes should be used in module names

### Pushing to BCL
Use the rake tasks listed above to stage and push measures to BCL.
**Note of warning**: Use caution when pushing to the BCL. Public and private measures could be pushed to BCL from within an extension gem, even though the extension gem repo may be private.

TODO: Add warning to rake task and lists what gems will be pushed
TODO: Check that license files, etc. are present in each measure before pushing to BCL
### Contributing measures to the BCL
Register your extension gem repo with the [BCL Manifest](https://github.com/BuildingComponentLibrary/bcl-manifest) and [follow the instructions](https://github.com/BuildingComponentLibrary/bcl-manifest#contribute-content) to contribute content to the BCL.

## Include in a project

Expand All @@ -234,6 +235,10 @@ Or install it yourself as:
$ gem install 'openstudio-extension'
# Contributing
Please review the [OpenStudio Contribution Policy](https://openstudio.net/openstudio-contribution-policy) if you would like to contribute code to this gem.
# Releasing the gem
* Update CHANGELOG.md
Expand Down
201 changes: 1 addition & 200 deletions lib/openstudio/extension/rake_task.rb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -147,19 +147,12 @@ def setup_subtasks(name)
end

namespace 'bcl' do
desc 'Test BCL login'
task :test_login do
puts 'test BCL login'
bcl = ::BCL::ComponentMethods.new
bcl.login
end


# for custom search, populate env var: bcl_search_keyword
desc 'Search BCL'
task :search_measures do
puts 'test search BCL'
bcl = ::BCL::ComponentMethods.new
bcl.login

# check for env var specifying keyword first
if ENV['bcl_search_keyword']
Expand All @@ -176,198 +169,6 @@ def setup_subtasks(name)
puts(res[:measure][:name]).to_s
end
end

# to call with argument: "openstudio:bcl:stage[true]" (true = remove existing staged content)
desc 'Copy the measures/components to a location that can be uploaded to BCL'
task :stage, [:reset] do |t, args|
puts 'Staging measures for BCL'
# initialize BCL and login
bcl = ::BCL::ComponentMethods.new
bcl.login

# process reset options: true to clear out old staged content
options = { reset: false }
if args[:reset].to_s == 'true'
options[:reset] = true
end

# ensure staged dir exists
FileUtils.mkdir_p(@staged_path)

# delete existing tarballs if reset is true
if options[:reset]
puts 'Deleting existing staged content'
FileUtils.rm_rf(Dir.glob("#{@staged_path}/*"))
end

# create new and existing directories
FileUtils.mkdir_p("#{@staged_path}/update")
FileUtils.mkdir_p("#{@staged_path}/push/component")
FileUtils.mkdir_p("#{@staged_path}/push/measure")

# keep track of noop, update, push
noops = 0
new_ones = 0
updates = 0

# get all content directories to process
dirs = Dir.glob("#{@measures_dir}/*")

dirs.each do |dir|
next if dir.include?('Rakefile') || File.basename(dir) == 'staged'

current_d = Dir.pwd
content_name = File.basename(dir)
puts '', '---'
puts "Generating #{content_name}"

Dir.chdir(dir)

# figure out whether to upload new or update existing
files = Pathname.glob('**/*')
uuid = nil
vid = nil
content_type = 'measure'

paths = []
files.each do |file|
# don't tar tests/outputs directory
next if file.to_s.start_with?('tests/output') # From measure testing process
next if file.to_s.start_with?('tests/test') # From openstudio-measure-tester-gem
next if file.to_s.start_with?('tests/coverage') # From openstudio-measure-tester-gem
next if file.to_s.start_with?('test_results') # From openstudio-measure-tester-gem

paths << file.to_s
if file.to_s =~ /^.{0,2}component.xml$/ || file.to_s =~ /^.{0,2}measure.xml$/
if file.to_s.match?(/^.{0,2}component.xml$/)
content_type = 'component'
end
# extract uuid and vid
uuid, vid = bcl.uuid_vid_from_xml(file)
end
end
puts "UUID: #{uuid}, VID: #{vid}"

# NOTE: if uuid is missing, will assume new content
action = bcl.search_by_uuid(uuid, vid)
puts "#{content_name} ACTION TO TAKE: #{action}"
# new content functionality needs to know if measure or component. update is agnostic.
case action
when 'noop' # ignore up-to-date content
puts " - WARNING: local #{content_name} uuid and vid match BCL... no update will be performed"
noops += 1
next
when 'update'
# puts "#{content_name} labeled as update for BCL"
destination = "#{@staged_path}/#{action}/#{content_name}.tar.gz"
updates += 1
when 'push'
# puts "#{content_name} labeled as new content for BCL"
destination = "#{@staged_path}/#{action}/#{content_type}/#{content_name}.tar.gz"
new_ones += 1
end

puts "destination: #{destination}"

# copy over only if 'reset_receipts' is set to TRUE. otherwise ignore if file exists already
if File.exist?(destination)
if options[:reset]
FileUtils.rm(destination)
::BCL.tarball(destination, paths)
else
puts "*** WARNING: File #{content_name}.tar.gz already exists in staged directory... keeping existing file. To overwrite, set reset_receipts arg to true ***"
end
else
::BCL.tarball(destination, paths)
end
Dir.chdir(current_d)
end
puts '', "****STAGING DONE**** #{new_ones} new content, #{updates} updates, #{noops} skipped (already up-to-date on BCL)", ''
end

desc 'Upload measures from the specified location.'
task :push do
puts 'Push measures to BCL'

# initialize BCL and login
bcl = ::BCL::ComponentMethods.new
bcl.login
reset = false

total_count = 0
successes = 0
errors = 0
skipped = 0

# grab all the new measure and component tar files and push to bcl
['measure', 'component'].each do |content_type|
items = []
paths = Pathname.glob(@staged_path.to_s + "/push/#{content_type}/*.tar.gz")
paths.each do |path|
# puts path
items << path.to_s
end

items.each do |item|
puts item.split('/').last
total_count += 1

receipt_file = "#{File.dirname(item)}/#{File.basename(item, '.tar.gz')}.receipt"
if !reset && File.exist?(receipt_file)
skipped += 1
puts 'SKIP: receipt file found'
next
end

valid, res = bcl.push_content(item, true, "nrel_#{content_type}")
if valid
successes += 1
else
errors += 1
if res.key?(:error)
puts " ERROR MESSAGE: #{res[:error]}"
else
puts "ERROR: #{res.inspect.chomp}"
end
end
puts '', '---'
end
end

# grab all the updated content (measures and components) tar files and push to bcl
items = []
paths = Pathname.glob("#{@staged_path}/update/*.tar.gz")
paths.each do |path|
# puts path
items << path.to_s
end
items.each do |item|
puts item.split('/').last
total_count += 1

receipt_file = "#{File.dirname(item)}/#{File.basename(item, '.tar.gz')}.receipt"
if !reset && File.exist?(receipt_file)
skipped += 1
puts 'SKIP: receipt file found'
next
end

valid, res = bcl.update_content(item, true)
if valid
successes += 1
else
errors += 1
if res.key?(:error)
puts " ERROR MESSAGE: #{res[:error]}"
else
puts "ERROR MESSAGE: #{res.inspect.chomp}"
end
end
puts '', '---'
end

puts "****UPLOAD DONE**** #{total_count} total, #{successes} success, #{errors} failures, #{skipped} skipped"
end
end
end
end
Expand Down

0 comments on commit 4c61f87

Please sign in to comment.