Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Full support for dbt deps on Windows #778

Closed
drewbanin opened this issue May 22, 2018 · 7 comments
Closed

Full support for dbt deps on Windows #778

drewbanin opened this issue May 22, 2018 · 7 comments
Labels
enhancement New feature or request windows Everyone's favorite OS that's sometimes a little weird
Milestone
Grace Kelly

Comments

@drewbanin
Copy link
Contributor

OSError: [WinError 17] The system cannot move the file to a different disk drive: 'C:\Users\UserName\AppData\Local\Temp\dbt-downloads\75cce4837e0fd73e2934f79b9a5238a9' -> 'dbt_modules\snowplow'

This can happen on Windows if the temp dir is on a different drive than the dbt project dir
@ryanmcdonnell
Copy link

Windows 10 Pro version 1803
Python v3.6.3
dbt 0.10.1

packages.yml:

packages:
  - git: "https://github.com/fishtown-analytics/snowplow.git"
    revision: 0.3.8
  - git: "https://github.com/fishtown-analytics/dbt-utils.git"
    revision: 0.1.7
  - git: "https://github.com/fishtown-analytics/bing-ads.git"
    revision: 0.0.1

dbt.log:

2018-05-22 10:45:21,096: Tracking: tracking
2018-05-22 10:45:21,096: Sending event: {'category': 'dbt', 'action': 'invocation', 'label': 'start', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x000001C92DD12A90>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x000001C92DD12FD0>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x000001C92DD12C88>]}
2018-05-22 10:45:21,489: Executing "git clone --depth 1 https://github.com/fishtown-analytics/snowplow.git 75cce4837e0fd73e2934f79b9a5238a9"
2018-05-22 10:45:22,609: STDOUT: "b''"
2018-05-22 10:45:22,609: STDERR: "b"Cloning into '75cce4837e0fd73e2934f79b9a5238a9'...\n""
2018-05-22 10:45:22,609: Pulling new dependency 75cce4837e0fd73e2934f79b9a5238a9.
2018-05-22 10:45:22,610: Executing "git rev-parse HEAD"
2018-05-22 10:45:22,641: STDOUT: "b'f77428b270d28c31c776ecd41b5825c96b1521e0\n'"
2018-05-22 10:45:22,641: STDERR: "b''"
2018-05-22 10:45:22,641:   Checking out branch 0.3.8.
2018-05-22 10:45:22,641: Executing "git remote set-branches origin 0.3.8"
2018-05-22 10:45:22,680: STDOUT: "b''"
2018-05-22 10:45:22,680: STDERR: "b''"
2018-05-22 10:45:22,680: Executing "git fetch --tags --depth 1 origin 0.3.8"
2018-05-22 10:45:23,394: STDOUT: "b''"
2018-05-22 10:45:23,394: STDERR: "b'From https://github.com/fishtown-analytics/snowplow\n * tag               0.3.8      -> FETCH_HEAD\n * [new tag]         0.1.0      -> 0.1.0\n * [new tag]         0.2.0      -> 0.2.0\n * [new tag]         0.2.1      -> 0.2.1\n * [new tag]         0.2.2      -> 0.2.2\n * [new tag]         0.2.3      -> 0.2.3\n * [new tag]         0.3.0      -> 0.3.0\n * [new tag]         0.3.1      -> 0.3.1\n * [new tag]         0.3.2      -> 0.3.2\n * [new tag]         0.3.3      -> 0.3.3\n * [new tag]         0.3.4      -> 0.3.4\n * [new tag]         0.3.5      -> 0.3.5\n * [new tag]         0.3.6      -> 0.3.6\n * [new tag]         0.3.7      -> 0.3.7\n * [new tag]         0.3.8      -> 0.3.8\n * [new tag]         0.3.9      -> 0.3.9\n * [new tag]         0.4.0      -> 0.4.0\n'"
2018-05-22 10:45:23,394: Executing "git tag --list"
2018-05-22 10:45:23,432: STDOUT: "b'0.1.0\n0.2.0\n0.2.1\n0.2.2\n0.2.3\n0.3.0\n0.3.1\n0.3.2\n0.3.3\n0.3.4\n0.3.5\n0.3.6\n0.3.7\n0.3.8\n0.3.9\n0.4.0\n'"
2018-05-22 10:45:23,432: STDERR: "b''"
2018-05-22 10:45:23,432: Executing "git reset --hard tags/0.3.8"
2018-05-22 10:45:23,530: STDOUT: "b'HEAD is now at 1499581 Merge pull request #18 from fishtown-analytics/feature/fivetran\n'"
2018-05-22 10:45:23,531: STDERR: "b''"
2018-05-22 10:45:23,531: Executing "git rev-parse HEAD"
2018-05-22 10:45:23,562: STDOUT: "b'14995815b385568d834db7cd496865709a0501da\n'"
2018-05-22 10:45:23,562: STDERR: "b''"
2018-05-22 10:45:23,562:   Checked out at 1499581.
2018-05-22 10:45:23,574: Executing "git clone --depth 1 https://github.com/fishtown-analytics/dbt-utils.git 859fd0fa21176efee375a447937f7a46"
2018-05-22 10:45:24,426: STDOUT: "b''"
2018-05-22 10:45:24,426: STDERR: "b"Cloning into '859fd0fa21176efee375a447937f7a46'...\n""
2018-05-22 10:45:24,426: Pulling new dependency 859fd0fa21176efee375a447937f7a46.
2018-05-22 10:45:24,426: Executing "git rev-parse HEAD"
2018-05-22 10:45:24,457: STDOUT: "b'6117edf2720b38893f7520e016c9a1a54e88c6b0\n'"
2018-05-22 10:45:24,457: STDERR: "b''"
2018-05-22 10:45:24,457:   Checking out branch 0.1.7.
2018-05-22 10:45:24,457: Executing "git remote set-branches origin 0.1.7"
2018-05-22 10:45:24,498: STDOUT: "b''"
2018-05-22 10:45:24,498: STDERR: "b''"
2018-05-22 10:45:24,498: Executing "git fetch --tags --depth 1 origin 0.1.7"
2018-05-22 10:45:25,704: STDOUT: "b''"
2018-05-22 10:45:25,704: STDERR: "b'From https://github.com/fishtown-analytics/dbt-utils\n * tag               0.1.7      -> FETCH_HEAD\n * [new tag]         0.0.1      -> 0.0.1\n * [new tag]         0.1.0      -> 0.1.0\n * [new tag]         0.1.1      -> 0.1.1\n * [new tag]         0.1.2      -> 0.1.2\n * [new tag]         0.1.3      -> 0.1.3\n * [new tag]         0.1.4      -> 0.1.4\n * [new tag]         0.1.5      -> 0.1.5\n * [new tag]         0.1.6      -> 0.1.6\n * [new tag]         0.1.7      -> 0.1.7\n * [new tag]         0.1.8      -> 0.1.8\n * [new tag]         0.1.9      -> 0.1.9\n'"
2018-05-22 10:45:25,704: Executing "git tag --list"
2018-05-22 10:45:25,742: STDOUT: "b'0.0.1\n0.1.0\n0.1.1\n0.1.10\n0.1.2\n0.1.3\n0.1.4\n0.1.5\n0.1.6\n0.1.7\n0.1.8\n0.1.9\n'"
2018-05-22 10:45:25,742: STDERR: "b''"
2018-05-22 10:45:25,742: Executing "git reset --hard tags/0.1.7"
2018-05-22 10:45:25,872: STDOUT: "b'HEAD is now at 331df26 Merge pull request #54 from fishtown-analytics/feature/last-day\n'"
2018-05-22 10:45:25,872: STDERR: "b''"
2018-05-22 10:45:25,872: Executing "git rev-parse HEAD"
2018-05-22 10:45:25,903: STDOUT: "b'331df26aa543807a8721f243835e0bbeaaf931e5\n'"
2018-05-22 10:45:25,904: STDERR: "b''"
2018-05-22 10:45:25,904:   Checked out at 331df26.
2018-05-22 10:45:25,914: Executing "git clone --depth 1 https://github.com/fishtown-analytics/bing-ads.git 3ef5c85fd464f489424563cc23940fed"
2018-05-22 10:45:26,671: STDOUT: "b''"
2018-05-22 10:45:26,671: STDERR: "b"Cloning into '3ef5c85fd464f489424563cc23940fed'...\n""
2018-05-22 10:45:26,671: Pulling new dependency 3ef5c85fd464f489424563cc23940fed.
2018-05-22 10:45:26,671: Executing "git rev-parse HEAD"
2018-05-22 10:45:26,703: STDOUT: "b'675c0adb5d9f7085c34f1af8c10751cc2628ad4a\n'"
2018-05-22 10:45:26,703: STDERR: "b''"
2018-05-22 10:45:26,703:   Checking out branch 0.0.1.
2018-05-22 10:45:26,703: Executing "git remote set-branches origin 0.0.1"
2018-05-22 10:45:26,745: STDOUT: "b''"
2018-05-22 10:45:26,745: STDERR: "b''"
2018-05-22 10:45:26,745: Executing "git fetch --tags --depth 1 origin 0.0.1"
2018-05-22 10:45:27,328: STDOUT: "b''"
2018-05-22 10:45:27,328: STDERR: "b'From https://github.com/fishtown-analytics/bing-ads\n * tag               0.0.1      -> FETCH_HEAD\n'"
2018-05-22 10:45:27,329: Executing "git tag --list"
2018-05-22 10:45:27,363: STDOUT: "b'0.0.1\n'"
2018-05-22 10:45:27,363: STDERR: "b''"
2018-05-22 10:45:27,363: Executing "git reset --hard tags/0.0.1"
2018-05-22 10:45:27,425: STDOUT: "b'HEAD is now at 675c0ad updated data types for keyword performance\n'"
2018-05-22 10:45:27,425: STDERR: "b''"
2018-05-22 10:45:27,425: Executing "git rev-parse HEAD"
2018-05-22 10:45:27,457: STDOUT: "b'675c0adb5d9f7085c34f1af8c10751cc2628ad4a\n'"
2018-05-22 10:45:27,457: STDERR: "b''"
2018-05-22 10:45:27,457:   Checked out at 675c0ad.
2018-05-22 10:45:27,468: Installing https://github.com/fishtown-analytics/snowplow.git@0.3.8
2018-05-22 10:45:27,469: Executing "git clone --depth 1 https://github.com/fishtown-analytics/snowplow.git 75cce4837e0fd73e2934f79b9a5238a9"
2018-05-22 10:45:27,499: STDOUT: "b''"
2018-05-22 10:45:27,499: STDERR: "b"fatal: destination path '75cce4837e0fd73e2934f79b9a5238a9' already exists and is not an empty directory.\n""
2018-05-22 10:45:27,499: Updating existing dependency 75cce4837e0fd73e2934f79b9a5238a9.
2018-05-22 10:45:27,499: Executing "git rev-parse HEAD"
2018-05-22 10:45:27,528: STDOUT: "b'14995815b385568d834db7cd496865709a0501da\n'"
2018-05-22 10:45:27,529: STDERR: "b''"
2018-05-22 10:45:27,529:   Checking out branch 0.3.8.
2018-05-22 10:45:27,529: Executing "git remote set-branches origin 0.3.8"
2018-05-22 10:45:27,568: STDOUT: "b''"
2018-05-22 10:45:27,568: STDERR: "b''"
2018-05-22 10:45:27,568: Executing "git fetch --tags --depth 1 origin 0.3.8"
2018-05-22 10:45:28,169: STDOUT: "b''"
2018-05-22 10:45:28,169: STDERR: "b'From https://github.com/fishtown-analytics/snowplow\n * tag               0.3.8      -> FETCH_HEAD\n'"
2018-05-22 10:45:28,169: Executing "git tag --list"
2018-05-22 10:45:28,207: STDOUT: "b'0.1.0\n0.2.0\n0.2.1\n0.2.2\n0.2.3\n0.3.0\n0.3.1\n0.3.2\n0.3.3\n0.3.4\n0.3.5\n0.3.6\n0.3.7\n0.3.8\n0.3.9\n0.4.0\n'"
2018-05-22 10:45:28,207: STDERR: "b''"
2018-05-22 10:45:28,207: Executing "git reset --hard tags/0.3.8"
2018-05-22 10:45:28,272: STDOUT: "b'HEAD is now at 1499581 Merge pull request #18 from fishtown-analytics/feature/fivetran\n'"
2018-05-22 10:45:28,272: STDERR: "b''"
2018-05-22 10:45:28,272: Executing "git rev-parse HEAD"
2018-05-22 10:45:28,303: STDOUT: "b'14995815b385568d834db7cd496865709a0501da\n'"
2018-05-22 10:45:28,303: STDERR: "b''"
2018-05-22 10:45:28,303:   Already at 1499581, nothing to do.
2018-05-22 10:45:28,733: Sending event: {'category': 'dbt', 'action': 'invocation', 'label': 'end', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x000001C92DE227B8>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x000001C92DE22BA8>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x000001C92DE2BAC8>]}
2018-05-22 10:45:29,104: Encountered an error:
2018-05-22 10:45:29,105: [WinError 5] Access is denied: 'C:\\Users\\Ryan\\AppData\\Local\\Temp\\dbt-downloads\\75cce4837e0fd73e2934f79b9a5238a9\\.git\\objects\\05\\f1220c8c8d13e365deb6c38e54d3d9ef9e8c0c'
2018-05-22 10:45:29,110: Traceback (most recent call last):
  File "c:\program files\python36\lib\shutil.py", line 544, in move
    os.rename(src, real_dst)
OSError: [WinError 17] The system cannot move the file to a different disk drive: 'C:\\Users\\Ryan\\AppData\\Local\\Temp\\dbt-downloads\\75cce4837e0fd73e2934f79b9a5238a9' -> 'dbt_modules\\snowplow'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c:\program files\python36\lib\site-packages\dbt\main.py", line 41, in main
    results, succeeded = handle_and_check(args)
  File "c:\program files\python36\lib\site-packages\dbt\main.py", line 85, in handle_and_check
    task, res = run_from_args(parsed)
  File "c:\program files\python36\lib\site-packages\dbt\main.py", line 139, in run_from_args
    results = run_from_task(task, proj, parsed)
  File "c:\program files\python36\lib\site-packages\dbt\main.py", line 147, in run_from_task
    result = task.run()
  File "c:\program files\python36\lib\site-packages\dbt\task\deps.py", line 424, in run
    package.install(self.project)
  File "c:\program files\python36\lib\site-packages\dbt\task\deps.py", line 228, in install
    shutil.move(self._checkout(project), dest_path)
  File "c:\program files\python36\lib\shutil.py", line 556, in move
    rmtree(src)
  File "c:\program files\python36\lib\shutil.py", line 494, in rmtree
    return _rmtree_unsafe(path, onerror)
  File "c:\program files\python36\lib\shutil.py", line 384, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  File "c:\program files\python36\lib\shutil.py", line 384, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  File "c:\program files\python36\lib\shutil.py", line 384, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  File "c:\program files\python36\lib\shutil.py", line 389, in _rmtree_unsafe
    onerror(os.unlink, fullname, sys.exc_info())
  File "c:\program files\python36\lib\shutil.py", line 387, in _rmtree_unsafe
    os.unlink(fullname)
PermissionError: [WinError 5] Access is denied: 'C:\\Users\\Ryan\\AppData\\Local\\Temp\\dbt-downloads\\75cce4837e0fd73e2934f79b9a5238a9\\.git\\objects\\05\\f1220c8c8d13e365deb6c38e54d3d9ef9e8c0c'

@atharvai
Copy link

atharvai commented Jun 5, 2018

I spoke with Drew and suggested a workaround being able to override the temp dir. Of course this is no fix or solution but general good idea of having it configurable

@drewbanin drewbanin added bug Something isn't working enhancement New feature or request and removed bug Something isn't working labels Jun 12, 2018
@drewbanin
Copy link
Contributor Author

Combining issue #766

dbt deps - symlink fails on windows (symbolic link privilege not held)

Via slack:

2018-05-09 15:52:04,681: Tracking: do not track
2018-05-09 15:52:04,692: Installing ../
2018-05-09 15:52:04,693:   Creating symlink to local dependency.
2018-05-09 15:52:04,693: Encountered an error:
2018-05-09 15:52:04,693: symbolic link privilege not held
2018-05-09 15:52:04,695: Traceback (most recent call last):
  File "c:\users\bene\appdata\local\programs\python\python36-32\lib\site-packages\dbt\main.py", line 41, in main
    results, succeeded = handle_and_check(args)
  File "c:\users\bene\appdata\local\programs\python\python36-32\lib\site-packages\dbt\main.py", line 85, in handle_and_check
    task, res = run_from_args(parsed)
  File "c:\users\bene\appdata\local\programs\python\python36-32\lib\site-packages\dbt\main.py", line 139, in run_from_args
    results = run_from_task(task, proj, parsed)
  File "c:\users\bene\appdata\local\programs\python\python36-32\lib\site-packages\dbt\main.py", line 147, in run_from_task
    result = task.run()
  File "c:\users\bene\appdata\local\programs\python\python36-32\lib\site-packages\dbt\task\deps.py", line 424, in run
    package.install(self.project)
  File "c:\users\bene\appdata\local\programs\python\python36-32\lib\site-packages\dbt\task\deps.py", line 272, in install
    dbt.clients.system.make_symlink(src_path, dest_path)
  File "c:\users\bene\appdata\local\programs\python\python36-32\lib\site-packages\dbt\clients\system.py", line 106, in make_symlink
    return os.symlink(source, link_path)
OSError: symbolic link privilege not held
  • Windows 10 Pro (developer mode enabled, can call mklink without elevated privileges)
  • Using powershell
  • dbt 10.0.0
  • It works in an admin shell

We should either fix this, or add an option to not create a symlink

Via Ben Edwards in Slack:

some other research that I have done, python's os.symlink will never work in windows unless run by an elevated user, even if the regular user can create symlinks (by enabling developer mode). The check in system.py only checks if the os supports symlink, which windows does, you just cant use it. A possible workaround I've seen is to detect windows, start a shell and call mklink from there.

@drewbanin drewbanin changed the title Error running dbt deps on windows Full support for dbt deps on Windows Jun 28, 2018
@drewbanin drewbanin mentioned this issue Jun 28, 2018
Closed
@drewbanin drewbanin added the windows Everyone's favorite OS that's sometimes a little weird label Jun 28, 2018
@drewbanin drewbanin mentioned this issue Aug 1, 2018
Closed
@brian-bk
Copy link

brian-bk commented Aug 1, 2018

Could just set the TMPDIR env var

@atharvai
Copy link

Here are some investigation scenarios that show that if the TMP dir and dbt project dir are on the OS drive the dbt deps is successful but not if they are on different drives https://gist.github.com/atharvai/822ef904c29bbd304bac07bb37233dab

@jthandy jthandy added this to the Stephen Girard milestone Oct 25, 2018
@drewbanin drewbanin modified the milestones: Stephen Girard, Grace Kelly Nov 2, 2018
@beckjake beckjake mentioned this issue Nov 6, 2018
Merged
beckjake added a commit that referenced this issue Nov 14, 2018
@beckjake
Merge pull request #1110 from fishtown-analytics/fix/dbt-deps-issues
735ff88 
Fix dbt deps issues (#778 #994 #895)
@beckjake
Copy link
Contributor

fixed in #1110

@chapmanjacobd
Copy link

chapmanjacobd commented Nov 19, 2021 •
edited
Loading

I think this is the reason dbt has trouble installing deps on WSL2... is there a way to skip using the symlinks? when you only have 1 or 2 dbt projects it causes much much more headaches than the few MBs it saves

If dbt deps runs on the Linux side for some reason it's unable to access the files that it creates

If dbt deps is run in the Windows side then linux can't access the modules unless you copy the files outside the folder then delete the hardlink and reboot WSL then create a normal dbt_modules folder and copy the files back in. So there is a workaround if one is technically minded but for a new user this might be unknown. I feel like more people use WSL than the Windows CLI but I might be mistaken

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request windows Everyone's favorite OS that's sometimes a little weird
Projects
None yet
Milestone
Grace Kelly
Development

No branches or pull requests
7 participants
@beckjake @drewbanin @ryanmcdonnell @jthandy @chapmanjacobd @brian-bk @atharvai