Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 14 additions & 24 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -190,6 +190,8 @@ be located by python.
Development of tensorflow-io on Linux is similiar to development on macOS. The required packages
are gcc, g++, git, bazel, and python 3. Newer versions of gcc or python than default system installed
versions might be required though.
For instructions how to configure Visual Studio code to be able to build and debug TensorFlow I/O see
https://github.com/tensorflow/io/blob/master/docs/vscode.md

##### Ubuntu 18.04/20.04

Expand Down Expand Up @@ -304,7 +306,7 @@ TFIO_DATAPATH=bazel-bin python3 -m pip install .
```
with `TFIO_DATAPATH=bazel-bin` passed for the same readon.

Note installing with `-e` is different from the above. The
Note installing with `-e` is different from the above. The
```
TFIO_DATAPATH=bazel-bin python3 -m pip install -e .
```
Expand Down Expand Up @@ -358,24 +360,6 @@ $ bash -x -e tests/test_kafka/kafka_test.sh start kafka
$ bash -x -e tests/test_kinesis/kinesis_test.sh start kinesis
```

#### Running Python and Bazel Style Checks

Style checks for Python and Bazel can be run with the following commands
(docker has to be available):

```sh
$ bash -x -e .travis/lint.sh
```

In case there are any Bazel style errors, the following command could be invoked
to fix and Bazel style issues:

```sh
$ docker run -i -t --rm -v $PWD:/v -w /v --net=host golang:1.12 bash -x -e -c 'go get github.com/bazelbuild/buildtools/buildifier && buildifier $(find . -type f \( -name WORKSPACE -or -name BUILD -or -name *.BUILD \))'
```

After the command is run, any Bazel files with style issues will have been modified and corrected.

### R

We provide a reference Dockerfile [here](R-package/scripts/Dockerfile) for you
Expand Down Expand Up @@ -426,22 +410,28 @@ If the system have docker installed, then the following command
will automatically build manylinux2010 compatible whl package:

```sh
bash -x -e .travis/python.release.sh
ls dist/*
for f in dist/*.whl; do
docker run -i --rm -v $PWD:/v -w /v --net=host quay.io/pypa/manylinux2010_x86_64 bash -x -e /v/tools/build/auditwheel repair --plat manylinux2010_x86_64 $f
done
sudo chown -R $(id -nu):$(id -ng) .
ls wheelhouse/*
```

It takes some time to build, but once complete, there will be python
`2.7`, `3.5`, `3.6`, `3.7` compatible whl packages available in `wheelhouse`
`3.5`, `3.6`, `3.7` compatible whl packages available in `wheelhouse`
directory.

On macOS, the same command could be used though the script expect `python` in shell
and will only generate a whl package that matches the version of `python` in shell. If
you want to build a whl package for a specific python then you have to alias this version
of python to `python` in shell.
of python to `python` in shell. See [.github/workflows/build.yml](.github/workflows/build.yml)
Auditwheel step for instructions how to do that.

Note the above command is also the command we use when releasing packages for Linux and macOS.

TensorFlow I/O uses both Travis CI and Google CI (Kokoro) for continuous integration.
Travis CI is used for macOS build and test. Kokoro is used for Linux build and test.
TensorFlow I/O uses both GitHub Workflows and Google CI (Kokoro) for continuous integration.
GitHub Workflows is used for macOS build and test. Kokoro is used for Linux build and test.
Again, because of the manylinux2010 requirement, on Linux whl packages are always
built with Ubuntu 16.04 + Developer Toolset 7. Tests are done on a variatiy of systems
with different python version to ensure a good coverage:
Expand Down
2 changes: 2 additions & 0 deletions docs/_toc.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -16,3 +16,5 @@
toc:
- title: Overview
path: /io/overview
- title: "Configuring Visual Studio Code"
path: /io/vscode
Binary file added docs/images/vscode_debugger.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
135 changes: 135 additions & 0 deletions docs/vscode.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,135 @@
# Configuring Visual Studio Code

Visual Studio is a free code editor, which runs on the macOS, Linux, and Windows operating systems.

It has nice tooling for Python and C++ development, visual debugger, git integration, and many more
useful features. It is a great editor to use for TensorFlow IO development, but it takes some effort
to configure it properly. VSCode configuration is very flexible, it allows compiling project using
bazel, and running code under Python and C++ debuggers. This manual is for Linux, other OSes
might have specifics, but approach should be similar.


## Extensions
To install an extension click the extensions view icon (Extensions) on the Sidebar, or use the shortcut Ctrl+Shift+X.
Then searh for keyword below.

- [C/C++](https://marketplace.visualstudio.com/items?itemName=ms-vscode.cpptools) - Official C++ extension from Microsoft
- [Python](https://marketplace.visualstudio.com/items?itemName=ms-python.python) - Official Python extension from Microsoft
- [Python Extension Pack](https://marketplace.visualstudio.com/items?itemName=donjayamanne.python-extension-pack) - another useful extension for Python development

## Compiling projects
TensorFlow IO is compiled using bazel build command:

```sh
bazel build -s --verbose_failures --compilation_mode dbg //tensorflow_io/...
```

See project [README](https://github.com/tensorflow/io#ubuntu-18042004) file for details on how to setup development environment in Ubuntu.
--compilation_mode dbg flag here indicates that produced binary should have debug symbols.
Once you can compile project from command line, you can also configure VSCode to be able to invoke same command.

Open View->Command Pallete (Ctrl+Shift+P) and start typing: "Tasks: Configure Build Task".
If you are doing this for the first time, editor is going to suggest creating tasks.json file.
Once you have it, paste following json:

```jsonc
{
"version": "2.0.0",
"tasks": [
{
"label": "Build TF.IO (Debug)",
"type": "shell",
"command": "bazel build -s --verbose_failures --compilation_mode dbg //tensorflow_io/...",
"group": {
"kind": "build",
"isDefault": true
},
"problemMatcher": []
}
]
}
```

Now, you can press "Ctrl+Shift+B", and VSCode is going to use the command above to build the project.
It uses its own terminal window, where all links are clickable. So when compilation error occurs, you can
just click the link, and editor will open corresponding file and navigate to the line.

## Debugging projects
Debugging Python code is trivial, follow official documentation to figure out how to configure VSCode to enable that: https://code.visualstudio.com/docs/python/debugging
Debugging C++ code requires GDB to be installed on your system.
If you have a bq_sample_read.py python script that is using tensorflow-io library that is normally
executed like:
```sh
python3 bq_sample_read.py --gcp_project_id=...
```

In order to execute it under GDB, run following:
```sh
gdb -ex r --args python3 bq_sample_read.py --gcp_project_id=...
```

If application crashes in C++ code, you can run ```backtrace``` in GDB console to get stacktrace.

VSCode also has GDB debugger support, it allows adding breakpoints, see values of variables and step through the code.
To add debug configuration press the Debug View icon (Debug) on the Sidebar, or use the shortcut Ctrl+Shift+D. Here press the little down arrow next to the play button and select "Add Configuration...".
It will create launch.json file, add following config here:

```jsonc
{
"name": "(gdb) Launch",
"type": "cppdbg",
"request": "launch",
"program": "/usr/bin/python3",
"args": ["bq_sample_read.py", "--gcp_project_id=..."],
"stopAtEntry": false,
"cwd": "${workspaceFolder}",
"environment": [
{
/* path to your bazel-bin folder */
"name": "TFIO_DATAPATH",
"value": "/usr/local/google/home/io/bazel-bin"
},
{
/* other env variables to use */
"name": "GOOGLE_APPLICATION_CREDENTIALS",
"value": "..."
}
],
"externalConsole": false,
"MIMode": "gdb",
"setupCommands": [
{
"description": "Enable pretty-printing for gdb",
"text": "-enable-pretty-printing",
"ignoreFailures": true
}
]
}
```

If everything is configured correctly, you should be able to do Run -> Start Debugging (F5) or Run -> Run Without Debugging (Ctrl + F5). This will run your code under debugger:

![VSCode debugger](./images/vscode_debugger.png)

One other thing worth doing to simplify debugging experience is configuting GDB to skip standard C++ libraries, so you don't step into code you don't care about. In order to do this, create ```~/.gdbinit``` file with following content:
```
skip -gfi /usr/include/c++/*/*/*
skip -gfi /usr/include/c++/*/*
skip -gfi /usr/include/c++/*
```

## Formatting files
You can always reformat C++ or Python file by Right Click -> Format Document (Ctrl + Shift + I), but VSCode uses different style conention. Luckily it is easy to change.

For Python formatting, see https://donjayamanne.github.io/pythonVSCodeDocs/docs/formatting/

To configure C++ formatter, do following:

- Go Preferences -> Settings
- Search C_Cpp.clang_format_fallbackStyle
- Modify the file:setting.json directly
- Add following

```
"C_Cpp.clang_format_fallbackStyle": "{ BasedOnStyle: Google}"
```