Editor Autocomplete

[jox81]

Hi, I just followed your explanation and all works well !

Only one thing, I enabled pyLint in vs Code and it warns me it doesn't know bpy :

E0401:Unable to import 'bpy'

So autocompletion doesn't work well.

Have you got a tips to bypass this problem ?

Thank you !

[AlansCodeLog]

Yes, I have heard of several ways to get autocomplete working. One is building blender as a python module (see Blender as a Module and Building Blender as Module). Another probably easier option is getting/making a pyprefdef file (e.g. mutantbob/pycharm-blender) which are just stubs of the api, should work regardless of the editor. I've also found this: nutti/fake-bpy-module, not sure if it's also generating stubs or doing something else.

I have not had time to test and confirm they work and document everything. Will do when I have some time. If anything is particularly confusing I will also make a video.

[TylerGubala]

I was just able to build blender as a module.

The make process is fairly straightforward despite there being some gotchas.

Generally the process is

1. Get Blender Sources from git & svn
2. (In a command prompt) make bpy 2017

I am close to having a setup.py (for pip) which will do this automatically in a temp directory, then copy to the current site-packages directory.

Thinking about your comment the other day, this would probably be the way to go; what you said made a lot of sense. I'll open an issue for further discussion, but it will be less of an issue, more like a feature request.

[AlansCodeLog]

fairly straightforward despite there being some gotchas

Bit of an understatement...

Just finally got this working and having never done this before, as I suspected, at least for windows, it's not very straightforward, the information is scattered across guides, some I had to find elsewhere, like the fact that you need to be sure to install the C++ workload (or I'm not sure what the equivalent would be for VS 2015? anybody know?). And if you already have the Visual Studio C++ Build Tools installed, but not the IDE, as I did, the way the make.bat is written it can't detect it. I did eventually figure it out though, had to manually add set MSVC_VS_DIR=C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools at line 183 right before it calls it. Not sure if there's an easier way to do this?

Also if you configure directly from the cmake gui (which will detect the build tools without problems), it's not very obvious how to build it. I eventually figured it out, but it's easier to do make bpy 2017 nobuild which will create a rebuild.bat then modify the build with cmake.

Also the build is in bin\Release instead of just bin like the documentation. Not sure why? As for the installing, since I use Anaconda and my python path is not the default, I ended up writing a quick batch script to handle the "install" copying.

REM To use: Copy to build folder. Run "install YOURPYTHONPATH PYTHONVERSION BLENDERVERSION" e.g. "install C:\Python36 36 2.79"
REM No backslash after path.

set pypath=%1
set pyversion=%2
set version=%3

copy bin\Release\bpy.pyd %pypath%\Lib\site-packages\
copy bin\Release\*.dll %pypath%\Lib\site-packages\
del %pypath%\Lib\site-packages\python%pyversion%.dll
xcopy /E bin\Release\%version% %pypath%\%version%\

It's not too much of a pain to run everything now that it's setup, though I could probably make an easy update script to pull in changes and rebuild.

When I have some time I will test how the other methods compare to this. This isn't really ideal if you're only interested in autocomplete. Although I do wonder, are the dlls even needed for autocomplete? because if they aren't it would be fairly trivial to have an occasional build run to package the bpy.pyd and the .py files.

@TylerGubala Would be interested in seeing the setup.py you've come up with.

[TylerGubala]

@AlansCodeLog Hi,

I have no real issues installing or making the blender pyd from sources, as you said there are some issues with the documentation, looks like that hasn't been updated in a long time, but there are open and resolved issues that have led me on the path of success (I was having trouble because I was on a venv trying to reproduce the steps)

I've created a setup.py for bpy.pyd/so over in https://github.com/TylerGubala/blenderpy but is very initial/ experimental.

I am having much more of an issue finding specific documentation to let me create a valid setuptools distributable than building bpy from sources. My current concern is that the way I have it set up now, if someone were to run the setup script, I have a bunch of python files that control the build process. It's by no means perfect and needs a second pair of eyes.

The real trouble is, my python files that control the build process... I only want those alive as part of the lifecycle for the setup program... after that they should go POOF (be deleted in some way...? I don't have too much experience writing a setup.py in this way) because the .so or pyd (depending on the platform) really should be the true module, and that has to be built.

Furthermore, I'm having some trouble figuring out how to specify the files so that when it uploads to pypi, pypi will only get the python scripts that control the build process, that way when a user grabs it, the dependencies are installed from requirements.txt, they get the automated build python scripts, setup.py runs, and they get the bpy package installed into their site packages.

Last thing to note: what will happen when the user runs "uninstall blenderpy"? How can I make it know to grab the binaries that I sneakily placed in the user's site-packages?

Like I said, this definitely is far from perfect. I need some help in that regard, but as a proof of concept I'd like another set of eyes to critique.

So that's where I am with that.

As far as making a setup script for your blender debugger for vscode, I'm kind of wondering how much value would be added. As you said the setup really isn't that complicated but I have a few ideas in my mind that I will publish different forks for probably this weekend if all goes well.

One idea would be to use python's requests library to download the get-pip script into a temporary file and subprocess.call blender's bundled python to run that temp file, then to change the blender-debugger-for-vs-code to read ptvsd from the blender site-packages dir (haven't actually looked at how your code is doing it currently, got sidetracked with the automated build script for bpy). Additionally maybe there could be an additional option in the setup script to create a venv for you with bpy already installed.

Another idea would just be to literally perform all the steps that you performed in the youtube video, but instead of opening blender to install the addon just extract it to the addons directory (more research on my end is needed)

I have things to look up still for certain.

And if you already have the Visual Studio C++ Build Tools installed, but not the IDE, as I did, the way the make.bat is written it can't detect it. I did eventually figure it out though, had to manually add set MSVC_VS_DIR=C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools at line 183 right before it calls it. Not sure if there's an easier way to do this?

I wasn't aware of the limitation. I have the IDE so maybe it "just works" for me but I'd need to look; I call

make bpy 2017 and it works but if that isn't working for you that's going to be an issue that I'll need to track down in my setup script as well...

EDIT: There are some things that are broken that I'm trying to deterministically fix now, but it involves some windows registry hackery, and possibly I will just fall back on the directory if I can find it based on the issue that you were having.

[TylerGubala]

Wow.

After a lot of tomfoolery I'm able to get my setup.py to build blender from sources.

Side note: this is actually a going to represent the sdist version of the bpy.so/pyd in my initial thoughts? I'm still learning a lot about setuputils and pip as packaging solutions and would really appreciate any help or recommendations.

Taking this into consideration I think that supplying bdist_wheels for most known platforms would be better than asking users to go through the whole build process from start to finish, which may not be something they want considering it requires downloading all Blender sources (on a potentially slow connection) and having VC build tools installed.

I just committed something that might fix the build process on Windows. Of course if it suits you better to just have the polyfill/autocomplete file, then going with that makes sense. I'll try to make more improvements over the week

[AlansCodeLog]

@TylerGubala It is an interesting approach, but I can't say I understand much of what's going on except the general idea (correct me if I'm wrong: a python module that downloads Blender, builds it as a module, and installs the build correctly), it's a bit over my head. Sorry.

I don't personally see much use for it now that I have downloaded everything and got it working, but it's still interesting. I have considered setting up a CI build for Blender as a Module, but I have too many side projects as it is and the better approach seems to be generating python stubs.

...then to change the blender-debugger-for-vs-code to read ptvsd from the blender site-packages dir (haven't actually looked at how your code is doing it currently...

When the addon is activated, the field for the path to ptvsd has a default function that runs and executes where python (it tries each OS equivalent, something that could be improved), and as a last fail safe looks to see if python is in the path, then uses the result to determine the relative path to site-packages.

I wasn't aware of the limitation. I have the IDE so maybe it "just works" for me but I'd need to look; I call

make bpy 2017 and it works but if that isn't working for you that's going to be an issue that I'll need to track down in my setup script as well...

What happened to me was a specific case, I was kind of anticipating it. For most people it should not happen, and a simplified version of the build instructions will do, but it is useful to know (for if you have the build tools or don't care to install Visual Studio just the build tools).

What happens is that in the make.bat file, it tries to find Visual Studio by checking the registry. I think it cannot find just the build tools because it's only checking for the actual program's registry keys? not 100% sure, I would need to investigate if it's possible to determine the location of build tools from the registry. The point is the current make.bat file cannot, which would be useful because as I mentioned people might have already installed them for one reason or another (e.g. I have 2015 build tools installed for building native node.js modules) and they are by far the biggest thing you need to download (~5GB!). As I mentioned though, the make.bat can be edited to manually specify the build tool's path, although one other thing I just found when testing this with my 2015 build tools is that MSBuild was missing from PATH. This can quickly be fixed by adding set PATH=%PATH%;YOURMSBUILDDIR. After that it worked like a charm.

Notes Regarding Python Stubs

I have taken the day to test the two repos I mentioned above. I thought fake-bpy-module would be the better solution as it was newer but although both projects are creating a similar end result (python stubs), they do it differently.

pycharm-blender is just a python script which when run by blender goes through the api and records it so to speak. Autocomplete with it seems to be working. I would like to compare it how the module does, but see below. The way it works looks promising though. It might be possible to include a modified version of it or write something like it so that stubs can be generated from this addon (or another, technically the stubs are editor agnostic) and we can kill two birds with one stone.

fake-bpy-module was actually designed to be used with the blender source files. Did not realize this at first glance. It's instructions are a little confusing, but I eventually got it working. It's taking the docs in the source files and generating stubs from those directly instead of directly from Blender if I understand correctly. But although it does autocomplete, I'm not getting any definitions. They seem to be empty. I did get some errors when running it though, so maybe it was me. Will have to test further.

Also it must be noted for these two options that you have to set the path manually in VS Code, there's a setting called python.autoComplete.extraPaths. I initially tried to put them in site-packages, but that just produced weird results.

Now I wanted to compare how these did to the module I compiled but now I can't get it to work. The autocomplete just gets stuck on Loading... Occasionally it works, but mostly it doesn't. Ugh... Anybody had this problem? I have tried reinstalling the Python extension in VS Code and setting python.autoComplete.preloadModules but no luck.

[TylerGubala]

Now I wanted to compare how these did to the module I compiled but now I can't get it to work. The autocomplete just gets stuck on Loading... Occasionally it works, but mostly it doesn't. Ugh... Anybody had this problem? I have tried reinstalling the Python extension in VS Code and setting python.autoComplete.preloadModules but no luck.

Sorry are you talking about the built blender as a python module?

I can get my code completion to work (albeit it takes about 10s for the autocompletion list to populate) in a venv that I have pip install bpy'd into.

However please if you are doing this do it in a venv. Pip does not track these files yet... I am working on a way to have pip track these files but for now it must be done in a venv otherwise you will have to remove these manually if you plan on uninstalling.

[AlansCodeLog]

@TylerGubala Yes, I built blender as a module manually (as I said, sorry I didn't try your script, but I had already downloaded everything, took ages), and installed it (in a conda env), but it was taking way more than 10 seconds for me, sometimes it wouldn't complete at all.

So I tried it again just now though with a clean conda env and a newer version of VS Code and now it's kind of usable with a 3-5 second delay. Not sure what was happening before..

But the preloadModules option doesn't seem to make any different. Somebody has already filed an issue over at vscode-python (#1704) so hopefully this will eventually get fixed.

As for how it compares to pycharm-blender, it's much better, pycharm-blender seems to be missing a lot of parts. For example, bpy.props doesn't exist. Not sure if that's supposed to happen, or the code to handle those sections are old. I would have to take some time to look at how it works.

So building blender as a module seems to be the way to go for now. I will add this to the documentation as soon as I can. Also the "Building Blender as a Python Module" documentation no longer exists on the wiki, gotta ask what that's about. If anybody wants to have a look at it, I was able to find it by searching the link above in the wayback machine.

[TylerGubala]

I'm just glad to hear it's working. Maybe pycharm-blender can be improved upon as well.

Not sure what's going on in the docs; odd to see that section of the documentation removed...

I was able to get backups of the old documentation usable through wikitaxi, if you want it.

[AlansCodeLog]

Yes, I'm sure it's just old code but I don't have time to look at it for now.

Yes it's weird, I think it was just a mistake from moving the wiki. I'm going to wait a little bit, maybe they're still transferring stuff, then ask.

[vabrador]

Putting this here in case it helps others trying to get intellisense going for bpy.

@Korchy has a handy guide for setting up autocomplete in VSCode for Blender 2.8 that I was successfully able to follow on Windows. https://github.com/Korchy/blender_autocomplete
There's a more detailed guide linked in the repo's readme.

It's not great to have to manually follow this process after installing this extension, but it's not too difficult to follow the important steps for a given project: Download the fake bpy stubs, then modify your (likely project-local) settings.json to add the stubs as an extra autocomplete path.

[ChaoJia]

For me the fake-bpy-module pip install fake-bpy-module-2.80 just works, no need to tweak settings.json. But sometimes has to modify the fake-bpy,
e.g. register_classes_factory(classes) has to return register, unregister in file ${python_home}/Lib/site-packages/bpy/utils/__init__.py