Making changes for ForemanAzureRM with azure-sdk-for-ruby #36

apuntamb · 2019-06-15T07:37:38Z

This PR is a WIP task. It deals with making changes to the foreman_azure_rm code base in alignment with the azure sdk for ruby. This PR is the first step towards using the azure-sdk instead of fog libraries.
This PR is expected to have couple of changes either with respect to functionalities or refactoring during it's review phase.
This PR shall allow user to:

Create an AzureRM compute resource.
Test Connection.
Create Images required for creating OS in AzureRM.
Create a Host to be able to deploy on AzureRM compute resource.

Although a host and other resources such as OSdisk, NIC, PIP and VM will be created and seen on the Azure portal, yet, the orchestration steps from Foreman are still in progress to be successful.

apuntamb · 2019-06-28T05:05:12Z

As an update, the PR now completes the orchestration steps successfully with both User data and Finish template approach along with password and ssh-key based authentication mechanisms. However, the PR is still in WIP state as the code still needs refining.

lib/foreman_azure_rm/azure_sdk_adapter.rb

vijay8451 · 2019-07-10T09:50:49Z

@apuntamb

QE Testing status:

Pass:

Azure CR can be created
Azure CR can be Updated
Azure CR can be deleted
'Test Connection' to Azure successful
Azure CR VMs can be displayed
Azure CR Compute Profile can be displayed
Azure CR image can be created
Azure CR image can be updated
Azure CR image can be deleted
Image with 'user_data' check enabled is created
Able to list the created image under CR

Fail/Blocked:

VM is being provisioned using CR. => Azure subnet does not display the list of available subnet in selected Azure Region and it is blocking the create host :

Note: Will test more scenarios around it once above one unblocked for me.

ShimShtein

We still need to make sure the tests are running correctly. Right now Travis fails to load the tests:

/home/travis/.rvm/gems/ruby-2.3.8/gems/factory_bot-5.0.2/lib/factory_bot/definition_proxy.rb:97:in `method_missing': undefined method 'provider' in 'azure_cr' factory (NoMethodError)
	from /home/travis/build/theforeman/foreman_azure_rm/test/factories/azure.rb:3:in `block (2 levels) in <top (required)>'
	from /home/travis/.rvm/gems/ruby-2.3.8/gems/factory_bot-5.0.2/lib/factory_bot/syntax/default.rb:18:in `instance_eval'
	from /home/travis/.rvm/gems/ruby-2.3.8/gems/factory_bot-5.0.2/lib/factory_bot/syntax/default.rb:18:in `factory'
	from /home/travis/build/theforeman/foreman_azure_rm/test/factories/azure.rb:2:in `block in <top (required)>'
	from /home/travis/.rvm/gems/ruby-2.3.8/gems/factory_bot-5.0.2/lib/factory_bot/syntax/default.rb:49:in `instance_eval'
	from /home/travis/.rvm/gems/ruby-2.3.8/gems/factory_bot-5.0.2/lib/factory_bot/syntax/default.rb:49:in `run'
	from /home/travis/.rvm/gems/ruby-2.3.8/gems/factory_bot-5.0.2/lib/factory_bot/syntax/default.rb:7:in `define'
	from /home/travis/build/theforeman/foreman_azure_rm/test/factories/azure.rb:1:in `<top (required)>'
	from /home/travis/.rvm/gems/ruby-2.3.8/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:281:in `load'
	from /home/travis/.rvm/gems/ruby-2.3.8/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:281:in `block in load'
	from /home/travis/.rvm/gems/ruby-2.3.8/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:253:in `load_dependency'
	from /home/travis/.rvm/gems/ruby-2.3.8/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:281:in `load'
	from /home/travis/.rvm/gems/ruby-2.3.8/gems/factory_bot-5.0.2/lib/factory_bot/find_definitions.rb:20:in `block (2 levels) in find_definitions'
	from /home/travis/.rvm/gems/ruby-2.3.8/gems/factory_bot-5.0.2/lib/factory_bot/find_definitions.rb:19:in `each'
	from /home/travis/.rvm/gems/ruby-2.3.8/gems/factory_bot-5.0.2/lib/factory_bot/find_definitions.rb:19:in `block in find_definitions'
	from /home/travis/.rvm/gems/ruby-2.3.8/gems/factory_bot-5.0.2/lib/factory_bot/find_definitions.rb:15:in `each'
	from /home/travis/.rvm/gems/ruby-2.3.8/gems/factory_bot-5.0.2/lib/factory_bot/find_definitions.rb:15:in `find_definitions'
	from /home/travis/.rvm/gems/ruby-2.3.8/gems/factory_bot-5.0.2/lib/factory_bot/reload.rb:6:in `reload'
	from /home/travis/build/theforeman/foreman_azure_rm/test/test_plugin_helper.rb:6:in `<top (required)>'
	from /home/travis/build/theforeman/foreman_azure_rm/test/functional/azure_rm_test.rb:1:in `require_relative'
	from /home/travis/build/theforeman/foreman_azure_rm/test/functional/azure_rm_test.rb:1:in `<top (required)>'
	from /home/travis/.rvm/gems/ruby-2.3.8/gems/rake-12.3.2/lib/rake/rake_test_loader.rb:17:in `require'
	from /home/travis/.rvm/gems/ruby-2.3.8/gems/rake-12.3.2/lib/rake/rake_test_loader.rb:17:in `block in <main>'
	from /home/travis/.rvm/gems/ruby-2.3.8/gems/rake-12.3.2/lib/rake/rake_test_loader.rb:5:in `select'
	from /home/travis/.rvm/gems/ruby-2.3.8/gems/rake-12.3.2/lib/rake/rake_test_loader.rb:5:in `<main>'

app/models/foreman_azure_rm/azure_rm_compute.rb

ShimShtein · 2019-07-10T18:03:13Z

app/models/foreman_azure_rm/azure_rm_compute.rb

+    end
+
+    def vm_size=(setvsize)
+      @azure_vm.hardware_profile.vm_size = setvsize


We probably need to initialize hardware_profile in case it was not initialized before . Especially when setting up an empty instance.

app/models/foreman_azure_rm/azure_rm_compute.rb

ShimShtein · 2019-07-10T18:16:36Z

app/models/foreman_azure_rm/azure_rm_compute.rb

+      @azure_vm.hardware_profile.vm_size = setvsize
+    end
+
+    def platform


It's the same comment for all properties here: we should implement all getters either by using delegate or by implementing them manually.

Now that we are defining only foreman specific methods in AzureRMCompute class, hence removing the other methods and added delegation for id and name methods.

ShimShtein · 2019-07-10T18:17:27Z

app/models/foreman_azure_rm/azure_rm_compute.rb

+    end
+
+    def os_disk_size=(setosdisk)
+      @azure_vm.os_disk.disk_size_gb = setosdisk


We need to initialize os_disk in case it is not initialized before.

app/models/foreman_azure_rm/azure_rm_compute.rb

foreman_azure_rm.gemspec

ShimShtein · 2019-07-10T18:29:38Z

lib/foreman_azure_rm/azure_sdk_adapter.rb

+      # List all VMs in a resource group
+      virtual_machines = compute_client.virtual_machines.list(rg_name)
+      rg_vms = []
+      virtual_machines.each do |vm|


I think we can return virtual_machines directly.

apuntamb · 2019-07-11T07:14:12Z

@vijay8451 Are you selecting the Image first from Operating System tab or directly visiting Virtual Machine or Network Interfaces tab?

vijay8451 · 2019-07-11T07:17:52Z

@vijay8451 Are you selecting the Image first from Operating System tab or directly visiting Virtual Machine or Network Interfaces tab?

@apuntamb it is like this:
Create Host => OS tab => VM tab to select minimal details => Interface to select Domain and subnet

apuntamb · 2019-07-11T12:32:23Z

@apuntamb it is like this:
Create Host => OS tab => VM tab to select minimal details => Interface to select Domain and subnet

@vijay8451 Now, it should work as expected. I've pushed the changes.

vijay8451 · 2019-07-12T11:07:42Z

@vijay8451 Now, it should work as expected. I've pushed the changes.

@apuntamb I am able to list the subnet and same is showing over Azure Portal 👍

apuntamb · 2019-07-12T12:51:19Z

test/functional/azure_rm_test.rb

    test "should create a compute resource and return edit page" do
-      test_client = Fog::Resources::AzureRM.new(
+      test_sdk = ForemanAzureRM::AzureSDKAdapter.new(


@ShimShtein Do you think this test_sdk object should be defined outside the test so that other tests would use it too? Can we do:

test_sdk = ForemanAzureRM::AzureSDKAdapter.new( tenant_id: '', client_id: '', client_secret: '', subscription_id: '' ) test "should create a compute resource and return edit page" do ....... end

You can make the sdk mock available in all tests through a class variable in setup block:

class ComputeResourcesControllerTest setup do @test_sdk = mock('test_sdk') #creates an empty object ForemanAzureRM::AzureRM.any_instance.stubs(:sdk).returns(test_sdk) @test_resource_client = mock('resource_client') test_sdk.stubs(:resource_client).returns(test_resource_client) end test 'do some tests' @test_resource_client.stubs(:rgs).returns([]) # ... end test 'do other tests' @test_resource_client.stubs(:rgs).returns(['a', 'b', 'c']) # ... end end

app/models/foreman_azure_rm/azure_rm.rb

ShimShtein · 2019-07-15T05:56:15Z

app/models/foreman_azure_rm/azure_rm.rb

+        response = sdk.create_or_update_vm(vm_hash[:resource_group], vm_hash[:name], vm_create_params)
+
+      rescue MsRestAzure::AzureOperationError => e
+        puts "**********FAILURE!!!***********"


I don't think we should ignore Azure exceptions here

app/models/foreman_azure_rm/azure_rm.rb

ShimShtein · 2019-07-15T06:15:00Z

test/functional/azure_rm_test.rb

@@ -4,19 +4,18 @@ module ForemanAzureRM
  class ComputeResourcesControllerTest < ActionController::TestCase
    tests ::ComputeResourcesController

-    setup { Fog.mock! }
-    teardown { Fog.unmock! }
-
    test "should create a compute resource and return edit page" do


The name of the test is a bit misleading here, it does not create a compute resource. I displays valid edit form for existing compute resource.

ShimShtein · 2019-07-15T06:19:58Z

test/functional/azure_rm_test.rb

+      ForemanAzureRM::AzureRM.any_instance.stubs(:sdk).returns(test_sdk)
+      test_resource_client = Object.new
+      test_sdk.stubs(:resource_client).returns(test_resource_client)
+      test_resource_client.stubs(:rgs).returns([])


Is it a common case that resource groups would return an empty array? If it is, I would suggest adding another test with a non-empty array and validates that we are displaying the groups correctly. If this is an edge case - we can probably modify the current test to work with non-empty array.

ShimShtein · 2019-07-15T06:45:24Z

test/functional/azure_rm_test.rb

    test "should create a compute resource and return edit page" do
-      test_client = Fog::Resources::AzureRM.new(
+      test_sdk = ForemanAzureRM::AzureSDKAdapter.new(


You can make the sdk mock available in all tests through a class variable in setup block:

class ComputeResourcesControllerTest setup do @test_sdk = mock('test_sdk') #creates an empty object ForemanAzureRM::AzureRM.any_instance.stubs(:sdk).returns(test_sdk) @test_resource_client = mock('resource_client') test_sdk.stubs(:resource_client).returns(test_resource_client) end test 'do some tests' @test_resource_client.stubs(:rgs).returns([]) # ... end test 'do other tests' @test_resource_client.stubs(:rgs).returns(['a', 'b', 'c']) # ... end end

vijay8451 · 2019-07-15T07:19:38Z

@apuntamb
QE Test status:

Fails:

Azure CR(Infrastructure => CR=> Virtual Machines) VM power fails as:
Issue => https://projects.theforeman.org/issues/27305
VM directly (Hosts => All Hosts) power off fails as:
Issue => https://projects.theforeman.org/issues/27304
VM edit fails as :
Issue => https://projects.theforeman.org/issues/27303

vijay8451 · 2019-07-22T10:39:49Z

@apuntamb
QE tests status:

Pass:

Able to provisoin VM using Finish template and status shows installed.
Able to login with user added/specified during Host create.

Fails/Issues:

Bug #27365 (Create host with username+SSHKey fails with error 'undefined method `id' for nil:NilClass')
Bug #26499 (on Foreman UI=> VM properties widget is overlapping)
Bug #26483 (VM 'Platform/Region/Vm size/' value always shows 'N/A' after VM create.)
Bug #27364 (Azure Region list shows less than from portal list)

apuntamb · 2019-07-22T12:27:36Z

UPDATE:
In this latest commit, there is no explicit require statement in any file. So, I assume for testing/reviewing, the gems viz. azure_mgmt_compute, resources, network and storage should be added to the foreman's bundler.d/Gemfile.local before starting the server.

app/models/foreman_azure_rm/azure_rm_compute.rb

ShimShtein

@apuntamb, we need to make Travis green again before we merge this PR.

Issue created: #37 to track one major refactor.
Probably will add a lot of smaller refactoring issues later.

@vijay8451 Could you please make a round of testing before I merge the PR? I don't see any major changes to the code anymore. There is only the issue with Travis, but it shouldn't make any difference as for testing.

vijay8451 · 2019-08-12T06:53:01Z

@ShimShtein @apuntamb Thanks for the update. I'll start testing part on below but it would great to have update on:
What are the Azure redmine issues fixed under recent reflector ?

apuntamb · 2019-08-12T06:55:55Z

@ShimShtein thanks for the review. I'll start fixing Travis errors now, let's see why it goes red. Also, about the refactor issue you've created, I'll now take a look at it.

apuntamb · 2019-08-12T06:57:14Z

@vijay8451 I'll start marking redmine issues (if any) that you should take a look while testing.

apuntamb · 2019-08-12T07:02:19Z

@vijay8451 please take a look at the #27271 tracker that has issues linked which are ready for testing.

apuntamb · 2019-08-13T10:13:11Z

@vijay8451 For your reference, this PR will cover the following use cases:

Create an AzureRM compute resource.
Test Connection.
Create Images required for creating OS in AzureRM.
Create a Host to be able to deploy on AzureRM compute resource.
Create a single NIC, single disk (the default vm size that is selected), create public/private IP for the VM.
Create a VM with 'password' or 'sshkey'.
Create a VM with or without 'script command'. Eg: "touch ~/hello.txt".
Destroy VM.
List all the VMs under VM tab on Compute Resources page.

vijay8451 · 2019-08-14T08:44:18Z

@apuntamb @ShimShtein

QE Test Status:

Pass:

Azure CR can be created. => Pass
Azure CR can be Updated => Pass
Azure CR can be deleted. => Pass
'Test Connection' to Azure successful. => Pass
Azure CR image can be created => Pass
Azure CR image can be updated => Pass
Azure CR image can be deleted => Pass
Image with 'user_data' check enabled is created. => Pass
Able to list the created image under CR. => Pass
VM is being provisioned using CR. => Pass
Able to list the Azure Region/Resource Group/VM Size/Platform/Subnet. => Partial Pass[Pending issue: https://projects.theforeman.org/issues/27364]
VM provisioned is listed in CR. => Pass
Hostname in provisioned VM is set correctly. => Pass
Provisioned VM can be deleted. => Pass
Post-install scripts are being executed on provisioned VM. => Pass
SSH Keys of user are being added to the provisioned VM.
External IP is being associated to provisioned VM. => Pass
VM should be provisioned with 'User-data template/cloud-init' and shows' Installed' status on Satellite => Partial Pass[facing some issue around User data]
VM should be provisioned with 'Finish template' and shows'Installed' status on Satellite => Pass
VM should provisioned with username+password => Pass
VM should provisioned with username+SSH-Key => Pass

New Issues/Failed tests:

Two times ask for non-root user details : https://projects.theforeman.org/issues/27618
Compute Resourse: Name difference: https://projects.theforeman.org/issues/27619
Compute Resourse: VM list table missing Label 'Action' on top of 'Power on/Off delete' Button : https://projects.theforeman.org/issues/27620
Click on 'Compute profile' fails with undefined method `reflect_on_association' for ForemanAzureRM::AzureRMCompute:Class: https://projects.theforeman.org/issues/27621
After edit the created Image provided password gets blank: https://projects.theforeman.org/issues/27622
Managed Image ID or Marketplace URN does not have validation to check provided URN is real exists or not.: https://projects.theforeman.org/issues/27623
VM can power off/on. => Failed due to already exists Bugs[https://projects.theforeman.org/issues/27304, https://projects.theforeman.org/issues/27305]
VM Edit: Able to click on edit however 'Virtual Machine' and 'Interface' shows values deselected : https://projects.theforeman.org/issues/27303

Others:

Requires to comment line 15(Faraday::default_adapter=:excon) in lib/foreman_azure_rm/engine.rb to create CR.

Open Questions/Queries:

Why custom URI script running through root user only ? as it's excepted to be run through non-root user created only ..
Why UserData/Cloud-init created VM has Compute Resource ssh-key copied to root and nonroot users authorized_keys ? and which one is expected one?
Currently userdata template is not executing on created VM.

vijay8451 · 2019-08-14T08:56:55Z

test/factories/azure.rb

+    add_attribute(:user) { 'azurermuser' }
+    add_attribute(:password) { 'azurermpassword' }
+    add_attribute(:url) { 'http://azurerm.example.com' }
+    add_attribute(:uuid) { 'azurermuuid' }


It would be much helpful , if use the mock data format as using in actually:

i.e. add_attribute(:uuid) { 'a0a086e2-da7a-4059-aafe-9cc07cc519cd' }

same for user, password, url.

apuntamb · 2019-08-15T10:46:12Z

@vijay8451 Thanks for bringing the issues into our notice.
I would like you to test User data template with the latest commit. It is working for me. Let me know if you still face any issues.

vijay8451 · 2019-08-19T11:05:25Z

@apuntamb

Below are working now:

Userdata
URI script running through non-root user

Below is an open one only:

Why UserData/Cloud-init created VM has Compute Resource ssh-key copied to root and nonroot users authorized_keys ? and which one is expected one?

Note: Just to clear, other reported issues are required to fix as well but currently not a blocker for this PR.

ShimShtein · 2019-08-19T12:23:01Z

Thanks, @apuntamb and @vijay8451!
I am confident enough to merge this new version. Now we have a lot of work in fixing bugs and adding missing functionality, but at least we have a good place to start with.

ShimShtein · 2019-08-19T12:33:00Z

@vijay8451, about the ssh keys issue: from what I see, if you set up admin user instead of root, it does put the certificate in both, but it disables ssh login by the .sss/auth_keys file. It creates a no-login session for root user:

no-port-forwarding,no-agent-forwarding,no-X11-forwarding,command="echo 'Please login as the user \"azureuser\" rather than the user \"root\".';echo;sleep 10" ssh-rsa ...

I think it's functionality that is provided by Azure.

vijay8451 · 2019-08-19T12:37:44Z

@ShimShtein thanks ..
ACK From QE ...

apuntamb requested a review from ShimShtein June 15, 2019 07:37

apuntamb added the WIP label Jun 15, 2019

apuntamb force-pushed the new_azurerm_codebase branch 3 times, most recently from 18d66a4 to fb0a65b Compare June 27, 2019 15:24

apuntamb force-pushed the new_azurerm_codebase branch 3 times, most recently from ee7c325 to a407146 Compare July 3, 2019 13:14

ShimShtein requested changes Jul 3, 2019

View reviewed changes

apuntamb force-pushed the new_azurerm_codebase branch 2 times, most recently from 5861fa1 to c3372fe Compare July 4, 2019 07:42

ShimShtein requested changes Jul 10, 2019

View reviewed changes

apuntamb force-pushed the new_azurerm_codebase branch 2 times, most recently from 7545f94 to fd13ec6 Compare July 11, 2019 12:18

apuntamb commented Jul 12, 2019

View reviewed changes

ShimShtein requested changes Jul 15, 2019

View reviewed changes

apuntamb force-pushed the new_azurerm_codebase branch from fd13ec6 to 4fb4ddf Compare July 19, 2019 13:31

apuntamb force-pushed the new_azurerm_codebase branch 2 times, most recently from e909759 to cf8f40f Compare July 22, 2019 12:17

apuntamb commented Jul 22, 2019

View reviewed changes

app/models/foreman_azure_rm/azure_rm_compute.rb Show resolved Hide resolved

apuntamb force-pushed the new_azurerm_codebase branch 3 times, most recently from 62eb81f to ae083ef Compare August 6, 2019 16:17

ShimShtein approved these changes Aug 11, 2019

View reviewed changes

apuntamb force-pushed the new_azurerm_codebase branch from ae083ef to 56758ed Compare August 13, 2019 10:40

vijay8451 reviewed Aug 14, 2019

View reviewed changes

Making changes for ForemanAzureRM with azure-sdk-for-ruby

5c4500d

apuntamb force-pushed the new_azurerm_codebase branch from 56758ed to 5c4500d Compare August 14, 2019 11:21

ShimShtein approved these changes Aug 19, 2019

View reviewed changes

ShimShtein merged commit 590fb36 into theforeman:master Aug 19, 2019

apuntamb mentioned this pull request Aug 19, 2019

Update README.md with azure-sdk changes #38

Merged

apuntamb removed the WIP label Aug 26, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Making changes for ForemanAzureRM with azure-sdk-for-ruby #36

Making changes for ForemanAzureRM with azure-sdk-for-ruby #36

apuntamb commented Jun 15, 2019

apuntamb commented Jun 28, 2019

vijay8451 commented Jul 10, 2019 •

edited

Loading

ShimShtein left a comment

ShimShtein Jul 10, 2019

ShimShtein Jul 10, 2019

apuntamb Jul 22, 2019

ShimShtein Jul 10, 2019

ShimShtein Jul 10, 2019

apuntamb commented Jul 11, 2019

vijay8451 commented Jul 11, 2019

apuntamb commented Jul 11, 2019 •

edited

Loading

vijay8451 commented Jul 12, 2019 •

edited

Loading

apuntamb Jul 12, 2019 •

edited

Loading

ShimShtein Jul 15, 2019

ShimShtein Jul 15, 2019

ShimShtein Jul 15, 2019

ShimShtein Jul 15, 2019

ShimShtein Jul 15, 2019

vijay8451 commented Jul 15, 2019 •

edited

Loading

vijay8451 commented Jul 22, 2019

apuntamb commented Jul 22, 2019

ShimShtein left a comment

vijay8451 commented Aug 12, 2019

apuntamb commented Aug 12, 2019

apuntamb commented Aug 12, 2019

apuntamb commented Aug 12, 2019

apuntamb commented Aug 13, 2019 •

edited

Loading

vijay8451 commented Aug 14, 2019 •

edited

Loading

vijay8451 Aug 14, 2019 •

edited

Loading

apuntamb commented Aug 15, 2019

vijay8451 commented Aug 19, 2019

ShimShtein commented Aug 19, 2019

ShimShtein commented Aug 19, 2019

vijay8451 commented Aug 19, 2019

Making changes for ForemanAzureRM with azure-sdk-for-ruby #36

Making changes for ForemanAzureRM with azure-sdk-for-ruby #36

Conversation

apuntamb commented Jun 15, 2019

apuntamb commented Jun 28, 2019

vijay8451 commented Jul 10, 2019 • edited Loading

ShimShtein left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

apuntamb commented Jul 11, 2019

vijay8451 commented Jul 11, 2019

apuntamb commented Jul 11, 2019 • edited Loading

vijay8451 commented Jul 12, 2019 • edited Loading

apuntamb Jul 12, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vijay8451 commented Jul 15, 2019 • edited Loading

vijay8451 commented Jul 22, 2019

apuntamb commented Jul 22, 2019

ShimShtein left a comment

Choose a reason for hiding this comment

vijay8451 commented Aug 12, 2019

apuntamb commented Aug 12, 2019

apuntamb commented Aug 12, 2019

apuntamb commented Aug 12, 2019

apuntamb commented Aug 13, 2019 • edited Loading

vijay8451 commented Aug 14, 2019 • edited Loading

vijay8451 Aug 14, 2019 • edited Loading

Choose a reason for hiding this comment

apuntamb commented Aug 15, 2019

vijay8451 commented Aug 19, 2019

ShimShtein commented Aug 19, 2019

ShimShtein commented Aug 19, 2019

vijay8451 commented Aug 19, 2019

vijay8451 commented Jul 10, 2019 •

edited

Loading

apuntamb commented Jul 11, 2019 •

edited

Loading

vijay8451 commented Jul 12, 2019 •

edited

Loading

apuntamb Jul 12, 2019 •

edited

Loading

vijay8451 commented Jul 15, 2019 •

edited

Loading

apuntamb commented Aug 13, 2019 •

edited

Loading

vijay8451 commented Aug 14, 2019 •

edited

Loading

vijay8451 Aug 14, 2019 •

edited

Loading