Scanner changes #1

nitin-deamon · 2021-08-03T06:59:02Z

There are 2 commits in this PR.

Commit should go to azcopy repo as it export the different APIs.
Commit where a sample scanner.go is there which utilize these exported APIs to give scanner functionality. We are utilizing azcopy creation of crawler code which is linked to traverser interface depending on type of source directory.

Need to try with /dev/NULL as destination if it works we can provide dummy destination to /dev/NULL.
Right now i tried with giving src-> local filesystem, dest-> blob and vice-versa.

- Changes here to export function from cmd package - Changes to use ste/concurrency to load environment variable.

- This file has build in function which init enumerator and cook the structure required for enumerator. - To scan call function Scanner(src, dest) with source and destination folder. Destination here can be dummy like blob where you want to copy the content. As of now azcopy use enum and enumerator require destination folder. So we need to provide dummy destination folder.

This is a sample code , for test purpose created this file under new folder scanner. Otherwise it will be part of go routine which call scanner. - This file has build in function which init enumerator and cook the structure required for enumerator. - To scan call function Scanner(src, dest) with source and destination folder. Destination here can be dummy like blob where you want to copy the content. As of now azcopy use enum and enumerator require destination folder. So we need to provide dummy destination folder.

shalinijoshi19 · 2021-08-03T14:35:14Z

@nitin-deamon could you also include @johnmic in these PRs so he can build some context etc? Thanks!

nitin-deamon · 2021-08-03T15:51:16Z

@nitin-deamon could you also include @johnmic in these PRs so he can build some context etc? Thanks!

done @shalinijoshi19

zezha-msft · 2021-08-04T08:02:27Z

Could you please add mohsha-msft nakulkar-msft adreed-msft as reviewers and to this repo? Thanks

nitin-deamon · 2021-08-04T10:46:04Z

Could you please add mohsha-msft nakulkar-msft adreed-msft as reviewers and to this repo? Thanks

Hi @zezha-msft I added them to repo. Once they accept the invitation , i will added them as reviewer.

- Variable case change required for export some functionality, cause testcase broken. - This patch fix broken testcases.

zezha-msft

To set the scope for the refactoring, we should clarify the intended usage. There are a couple of choices:

Put in raw inputs with rawCopyCmdArgs and invoke the whole execution chain, this is essentially running AzCopy in the same process. Doing so means using AzCopy's schema for saving job plan files and job tracking. This is probably not what you want since the lifecycle manager will call os.exit when the job finishes.
Leverage the existing enumerator to traverse, filter, and process stored objects in a customized way that you'd like. Doing so requires all the constructs to be exported. Several of them are missing right now.
Only leverage certain traverser, filter to build your own enumerator or different concept entirely.

zezha-msft · 2021-08-11T07:04:50Z

cmd/zc_enumerator.go

@@ -46,7 +46,7 @@ import (
 // represent a local or remote resource object (ex: local file, blob, etc.)
 // we can add more properties if needed, as this is easily extensible
 // ** DO NOT instantiate directly, always use newStoredObject ** (to make sure its fully populated and any preprocessor method runs)
-type storedObject struct {
+type StoredObject struct {
 	name             string


Were these properties missed?

No Ze, Till now sample code not using any of its properties. May be as i add more functionality i need those properties. Then i will export them. Or i can write function in cmd which work on these properties.

zezha-msft · 2021-08-11T07:07:43Z

scanner/scanner.go

@@ -0,0 +1,361 @@
+// Copyright © 2017 Microsoft <nitinsingla@microsoft.com>


Minor: please remove the date 2017, for new files we don't need to put a date.

sure got it Ze, Thanks

zezha-msft · 2021-08-11T07:08:08Z

scanner/scanner.go

+package scanner
+
+import (
+        "fmt"


Please set up go fmt to run on file save in your IDE.

sure Ze, i modified the file using vi, let me start using VSC and add go fmt to run on file save.

zezha-msft · 2021-08-11T07:13:32Z

scanner/scanner.go

+		return nil
+	}
+
+	return cmd.NewCopyEnumerator(traverser, filters, processor, finalizer), nil


why not use the existing initEnumerator function?

The reason for not using initEnumerator function is we want Processing function and Finalizer Function should work according to our requirement. That diverge lead me to use initResourceEnumerator and create our own initEnumerator function.

zezha-msft · 2021-08-11T07:14:44Z

cmd/copyEnumeratorInit.go

@@ -26,73 +26,73 @@ type BucketToContainerNameResolver interface {
 	ResolveName(bucketName string) (string, error)
 }

-func (cca *cookedCopyCmdArgs) initEnumerator(jobPartOrder common.CopyJobPartOrderRequest, ctx context.Context) (*copyEnumerator, error) {
-	var traverser resourceTraverser
+func (cca *CookedCopyCmdArgs) initEnumerator(jobPartOrder common.CopyJobPartOrderRequest, ctx context.Context) (*CopyEnumerator, error) {


This should be useful for you too I think

Yes Ze, as i mentioned right now thought process is to create our own enumerator. So that it will not effect azcopy code base. I am trying not to change azcopy functionality as of now.

zezha-msft · 2021-08-11T07:15:50Z

cmd/zc_enumerator.go

@@ -587,14 +587,14 @@ type preFilterProvider interface {
 type syncEnumerator struct {


I think these enumerators should be useful too.

Sync Enumerator may be not , we are trying to scan source only on the basis of Last modified time. As per initial thought we don't want to scan destination, try to keep some meta-data on premise. Which help in finding delta.

shalinijoshi19 · 2021-08-20T14:19:37Z

There are 2 commits in this PR.

Commit should go to azcopy repo as it export the different APIs.

Commit where a sample scanner.go is there which utilize these exported APIs to give scanner functionality. We are utilizing azcopy creation of crawler code which is linked to traverser interface depending on type of source directory.

Need to try with /dev/NULL as destination if it works we can provide dummy destination to /dev/NULL.
Right now i tried with giving src-> local filesystem, dest-> blob and vice-versa.

@nitin-deamon do we have an updated set of commits at this point already? My assumption is we were going to split this so that #1 could be a new PR in the azcopy repo and #2 would be merged into our msazure instance?

shalinijoshi19

Hey @nitin-deamon when do you think we'd be able to close on this one? Are there more changes for us to look at since?

nitin-deamon · 2021-08-23T04:09:32Z

Hey @nitin-deamon when do you think we'd be able to close on this one? Are there more changes for us to look at since?

@shalinijoshi19 Already created the branch in azcopy repo, and cherry pick the changes to it. I need to push changes to our Repo.

Add Azure Arc server VMs support

nitin-deamon added 3 commits August 2, 2021 04:07

This changes required to make scanner separate module.

c8de675

- Changes here to export function from cmd package - Changes to use ste/concurrency to load environment variable.

nitin-deamon requested review from barooah, shalinijoshi19, zezha-msft and linuxsmiths August 3, 2021 06:59

nitin-deamon requested a review from johnmic August 4, 2021 04:29

Fix cmd testcases

33dff61

- Variable case change required for export some functionality, cause testcase broken. - This patch fix broken testcases.

nitin-deamon requested review from mohsha-msft and nakulkar-msft August 11, 2021 03:54

zezha-msft reviewed Aug 11, 2021

View reviewed changes

shalinijoshi19 requested changes Aug 20, 2021

View reviewed changes

nitin-deamon pushed a commit that referenced this pull request Oct 4, 2021

Merge pull request #1 from Strikerzee/feature/arcvmsupport

f64ac4c

Add Azure Arc server VMs support

linuxsmiths closed this Mar 7, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Scanner changes #1

Scanner changes #1

nitin-deamon commented Aug 3, 2021

shalinijoshi19 commented Aug 3, 2021

nitin-deamon commented Aug 3, 2021

zezha-msft commented Aug 4, 2021

nitin-deamon commented Aug 4, 2021

zezha-msft left a comment

zezha-msft Aug 11, 2021

nitin-deamon Aug 11, 2021

zezha-msft Aug 11, 2021

nitin-deamon Aug 11, 2021

zezha-msft Aug 11, 2021

nitin-deamon Aug 11, 2021

zezha-msft Aug 11, 2021

nitin-deamon Aug 11, 2021

zezha-msft Aug 11, 2021

nitin-deamon Aug 11, 2021

zezha-msft Aug 11, 2021

nitin-deamon Aug 11, 2021

shalinijoshi19 commented Aug 20, 2021

shalinijoshi19 left a comment

nitin-deamon commented Aug 23, 2021

		@@ -0,0 +1,361 @@
		// Copyright © 2017 Microsoft <nitinsingla@microsoft.com>

		@@ -587,14 +587,14 @@ type preFilterProvider interface {
		type syncEnumerator struct {

Scanner changes #1

Scanner changes #1

Conversation

nitin-deamon commented Aug 3, 2021

shalinijoshi19 commented Aug 3, 2021

nitin-deamon commented Aug 3, 2021

zezha-msft commented Aug 4, 2021

nitin-deamon commented Aug 4, 2021

zezha-msft left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shalinijoshi19 commented Aug 20, 2021

shalinijoshi19 left a comment

Choose a reason for hiding this comment

nitin-deamon commented Aug 23, 2021