Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

improved linux and windows build scripts, updated readme #348

Merged
merged 13 commits into from
Jul 24, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
64 changes: 38 additions & 26 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ A pipeline framework for developing video and image processing applications. Sup
Learn more about ApraPipes here https://apra-labs.github.io/ApraPipes.

## Build status
Aprapipes is automatically built and tested on Ubuntu (18.04 and 20.04), Jetson Boards (Jetpack 4.4) and Windows (11) x64 Visual Studio 2017 Community.
Aprapipes is automatically built and tested on Ubuntu (18.04 and 20.04), Jetson Boards (Jetpack 4.4) and Windows (11) x64 Visual Studio 2019 Community.
|OS|Version|With Cuda|Tests|Status|
|--|-------|---------|------|------|
|Windows|2019|No|[![Test Results](https://gist.githubusercontent.com/kumaakh/f80af234a4aabedc69af3ee197f66944/raw/badge_Windows.svg)](https://gist.githubusercontent.com/kumaakh/f80af234a4aabedc69af3ee197f66944/raw/badge_Windows.svg)|[![CI-Win-NoCUDA](https://github.com/Apra-Labs/ApraPipes/actions/workflows/CI-Win-NoCUDA.yml/badge.svg)](https://github.com/Apra-Labs/ApraPipes/actions/workflows/CI-Win-NoCUDA.yml)|
Expand Down Expand Up @@ -40,22 +40,24 @@ Aprapipes is automatically built and tested on Ubuntu (18.04 and 20.04), Jetson
<details>
<summary>Requirements</summary>

### Prerequisites

### Visual Studio
* Install Visual Studio 2019 Community
* Install Desktop development C++
* .NET Desktop development
* Universal Windows Development Platform

### Cuda
* Create an account on developer.nvidia.com if you're not already a member. Note : Otherwise the next step will show HTTP 404/403 error.
* Windows 10/11 : [Cuda Toolkit 10.2](https://developer.nvidia.com/cuda-10.2-download-archive?target_os=Windows&target_arch=x86_64&target_version=10&target_type=exelocal) or [CUDA Toolkit 11.8](https://developer.nvidia.com/cuda-11-8-0-download-archive?target_os=Windows).

### Cudnn
* Download [Cudnn](https://developer.nvidia.com/rdp/cudnn-archive#a-collapse765-102) and extract files where cuda is installed. Note: Please be aware that this process requires some effort. Here are the necessary steps:
* Download the correct zip file matching your cuda version. _Do not download the exe/installer/deb package._
* Windows:
* Download [this file](https://developer.nvidia.com/compute/cudnn/secure/8.3.2/local_installers/10.2/cudnn-windows-x86_64-8.3.2.44_cuda10.2-archive.zip).

### Prerequisites
* Install Visual Studio 2019 Community
* Install Desktop development C++
* .NET Desktop development
* Universal Windows Development Platform

* Download the correct zip file matching your cuda version. _Do not download the exe/installer/deb package._
* Windows:
* Download [this file](https://developer.nvidia.com/compute/cudnn/secure/8.3.2/local_installers/10.2/cudnn-windows-x86_64-8.3.2.44_cuda10.2-archive.zip).

* Clone with submodules and LFS.
```
git clone --recursive https://github.com/Apra-Labs/ApraPipes.git
Expand All @@ -66,9 +68,9 @@ Aprapipes is automatically built and tested on Ubuntu (18.04 and 20.04), Jetson
<details>
<summary>Build</summary>

### Build Without Cuda
Open PowerShell as an administrator and execute the following commands

### Build Without Cuda
If your windows system does not have an NVIDIA GPU use this script
```
build_windows_no_cuda.bat
Expand All @@ -77,7 +79,10 @@ Aprapipes is automatically built and tested on Ubuntu (18.04 and 20.04), Jetson
```
build_windows_cuda.bat
```

### To Build With Documentation
```
build_windows_cuda.bat --build-doc
```
</details>

<details>
Expand Down Expand Up @@ -114,6 +119,8 @@ Aprapipes is automatically built and tested on Ubuntu (18.04 and 20.04), Jetson
<details>
<summary>Requirements</summary>

### Prerequisites

### Cuda
* Create an account on developer.nvidia.com if you're not already a member. Note : Otherwise the next step will show HTTP 404/403 error.
* Ubuntu 18.04/20.04:
Expand All @@ -125,7 +132,8 @@ Aprapipes is automatically built and tested on Ubuntu (18.04 and 20.04), Jetson
* Linux:
* Download [this file](https://developer.nvidia.com/compute/cudnn/secure/8.3.2/local_installers/10.2/cudnn-linux-x86_64-8.3.2.44_cuda10.2-archive.tar.xz)

### Prerequisites
* Clone with submodules and LFS.
```
git clone --recursive https://github.com/Apra-Labs/ApraPipes.git
```

Expand All @@ -141,11 +149,15 @@ Aprapipes is automatically built and tested on Ubuntu (18.04 and 20.04), Jetson
### Build Without Cuda
If your windows system does not have an NVIDIA GPU use this script
```
sudo ./build_linux_no_cuda.sh
./build_linux_no_cuda.sh
```
### Build With Cuda
```
sudo ./build_linux_cuda.sh
./build_linux_cuda.sh
```
### To Build With Documentation
```
./build_linux_cuda.sh --build-doc
```

Build can take ~2 hours depending on the machine configuration.
Expand Down Expand Up @@ -200,11 +212,14 @@ Aprapipes is automatically built and tested on Ubuntu (18.04 and 20.04), Jetson
```
chmod +x build_jetson.sh
```
* ApraPipes builds CUDA version on Jerson Boads.
* ApraPipes builds CUDA version on Jetson Boards.
```
sudo ./build_jetson.sh
./build_jetson.sh
```
* To Build With Documentation
```
./build_jetson.sh --build-doc
```

Build can take ~12 hours on Jetson Nano.
Note: Jetson build can also be done using Ubuntu 18.04 x86_64 Laptop via cross compilation.
</details>
Expand Down Expand Up @@ -290,13 +305,10 @@ This build will be fairly fast (~10 mins) as entire vcpkg cache comes down with
git submodule update --init --recursive
```
## Update Documentation
After making changes to the documentation located in the /docs/source folder, it's essential to regenerate the documentation by following the provided steps. Once regenerated, commit the new content to ensure the latest documentation is up-to-date.
To update documentation, refer to Documentation Guidelines in the [Contribution-Guidelines](https://github.com/Apra-Labs/ApraPipes/wiki/Contribution-Guidelines).

### To regenerate documentation
Run,
```
./build_documentation.sh
```
To build docs
apt-install get python-sphinx
pip install sphinx-rtd-theme
cd docs
make html
```
42 changes: 38 additions & 4 deletions build_jetson.sh
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
apt-get install clang-format
sudo apt-get install clang-format
clang-format -style=llvm -dump-config > .clang-format
if ! command -v pip &> /dev/null; then
# If pip is not available, download and install pip
Expand All @@ -9,10 +9,44 @@ pip install pre-commit
pre-commit install

chmod +x build_scripts/build_dependencies_jetson_cuda.sh
./build_scripts/build_dependencies_jetson_cuda.sh
sudo ./build_scripts/build_dependencies_jetson_cuda.sh

chmod +x build_documentation.sh
./build_documentation.sh
if nvcc --version; then
USER_NAME=$(whoami)
TARGET_USER="$USER_NAME"
TARGET_HOME=$(eval echo ~$TARGET_USER)

# Append lines to the target user's ~/.bashrc
if ! grep -qxF 'export VCPKG_FORCE_SYSTEM_BINARIES=1' $TARGET_HOME/.bashrc; then
echo 'export VCPKG_FORCE_SYSTEM_BINARIES=1' | sudo -u $TARGET_USER tee -a $TARGET_HOME/.bashrc
echo "VCPKG_FORCE_SYSTEM_BINARIES flag added in .bashrc"
else
echo "VCPKG_FORCE_SYSTEM_BINARIES flag already exists in .bashrc"
fi

if ! grep -qxF 'export PATH=/usr/local/cuda/bin${PATH:+:${PATH}}' $TARGET_HOME/.bashrc; then
echo 'export PATH=/usr/local/cuda/bin${PATH:+:${PATH}}' | sudo -u $TARGET_USER tee -a $TARGET_HOME/.bashrc
echo "CUDA Binary Path added to .bashrc"
else
echo "CUDA Binary Path already exists in .bashrc"
fi

if ! grep -qxF 'export LD_LIBRARY_PATH=/usr/local/cuda/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}' $TARGET_HOME/.bashrc; then
echo 'export LD_LIBRARY_PATH=/usr/local/cuda/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}' | sudo -u $TARGET_USER tee -a $TARGET_HOME/.bashrc
echo "CUDA Library Path added to .bashrc"
else
echo "CUDA Library Path already exists in .bashrc"
fi

echo "Appended paths to ~/.bashrc and saved changes."
source ~/.bashrc
echo "Reloaded ~/.bashrc"
fi

if [[ $1 == "--build-doc" ]]; then
chmod +x build_documentation.sh
./build_documentation.sh
fi

cd vcpkg
./bootstrap-vcpkg.sh
Expand Down
53 changes: 50 additions & 3 deletions build_linux_cuda.sh
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
apt-get install clang-format
sudo apt-get install clang-format
clang-format -style=llvm -dump-config > .clang-format
if ! command -v pip &> /dev/null; then
# If pip is not available, download and install pip
Expand All @@ -11,8 +11,55 @@ pre-commit install
chmod +x build_scripts/build_dependencies_linux_cuda.sh
sudo ./build_scripts/build_dependencies_linux_cuda.sh

chmod +x build_documentation.sh
./build_documentation.sh
if ! sudo nvcc --version &>/dev/null; then
USER_NAME=$(whoami)
cudnn_archives="/home/$USER_NAME/Downloads/cudnn-*.tar.xz"

for archive in $cudnn_archives; do
if [ -e "$archive" ]; then
extracted_folder="/home/$USER_NAME/Downloads/$(basename "$archive" .tar.xz)"
if [ ! -d "$extracted_folder" ]; then
echo "Extracting $archive..."
tar xf "$archive" -C "/home/$USER_NAME/Downloads/"
else
echo "Archive already extracted: $extracted_folder"
fi
fi
done

echo "Copying files..."
sudo cp -r /home/$USER_NAME/Downloads/cudnn-*/include/* /usr/local/cuda/include/
sudo cp -r /home/$USER_NAME/Downloads/cudnn-*/lib/* /usr/local/cuda/lib64/

TARGET_USER="$USER_NAME"
TARGET_HOME=$(eval echo ~$TARGET_USER)

# Append lines to the target user's ~/.bashrc
if ! grep -qxF 'export PATH=/usr/local/cuda/bin:${PATH}' $TARGET_HOME/.bashrc; then
echo 'export PATH=/usr/local/cuda/bin:${PATH}' | sudo -u $TARGET_USER tee -a $TARGET_HOME/.bashrc
echo "CUDA Binary Path added to .bashrc"
else
echo "CUDA Binary Path already exists in .bashrc"
fi

if ! grep -qxF 'export LD_LIBRARY_PATH=/usr/local/cuda/lib64:${LD_LIBRARY_PATH}' $TARGET_HOME/.bashrc; then
echo 'export LD_LIBRARY_PATH=/usr/local/cuda/lib64:${LD_LIBRARY_PATH}' | sudo -u $TARGET_USER tee -a $TARGET_HOME/.bashrc
echo "CUDA Library Path added to .bashrc"
else
echo "CUDA Library Path already exists in .bashrc"
fi

# Reload .bashrc
source ~/.bashrc

echo "Appended line to ~/.bashrc and saved changes."
echo "Reloaded ~/.bashrc"
fi

if [[ $1 == "--build-doc" ]]; then
chmod +x build_documentation.sh
./build_documentation.sh
fi

cd vcpkg
./bootstrap-vcpkg.sh
Expand Down
10 changes: 6 additions & 4 deletions build_linux_no_cuda.sh
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
apt-get install clang-format
sudo apt-get install clang-format
clang-format -style=llvm -dump-config > .clang-format
if ! command -v pip &> /dev/null; then
# If pip is not available, download and install pip
Expand All @@ -9,13 +9,15 @@ pip install pre-commit
pre-commit install

chmod +x build_scripts/build_dependencies_linux_no_cuda.sh
./build_scripts/build_dependencies_linux_no_cuda.sh
sudo ./build_scripts/build_dependencies_linux_no_cuda.sh

chmod +x base/fix-vcpkg-json.sh
./base/fix-vcpkg-json.sh true false false

chmod +x build_documentation.sh
./build_documentation.sh
if [[ $1 == "--build-doc" ]]; then
chmod +x build_documentation.sh
./build_documentation.sh
fi

cd vcpkg
./bootstrap-vcpkg.sh
Expand Down
11 changes: 0 additions & 11 deletions build_scripts/build_dependencies_jetson_cuda.sh
Original file line number Diff line number Diff line change
Expand Up @@ -30,15 +30,4 @@ if [ ! -d "/usr/local/cuda/include" ] || [ ! -d "/usr/local/cuda/lib64" ]; then
exit 1
fi

if nvcc --version; then
TARGET_USER="$SUDO_USER"
TARGET_HOME=$(eval echo ~$TARGET_USER)
echo 'export VCPKG_FORCE_SYSTEM_BINARIES=1' | sudo -u $TARGET_USER tee -a $TARGET_HOME/.bashrc
echo 'export PATH=/usr/local/cuda/bin${PATH:+:${PATH}}' | sudo -u $TARGET_USER tee -a $TARGET_HOME/.bashrc
echo 'export LD_LIBRARY_PATH=/usr/local/cuda/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}' | sudo -u $TARGET_USER tee -a $TARGET_HOME/.bashrc
echo "Appended paths to ~/.bashrc and saved changes."
source ~/.bashrc
echo "Reloaded ~/.bashrc"
fi

echo "Dependencies verified and installed successfully."
31 changes: 1 addition & 30 deletions build_scripts/build_dependencies_linux_cuda.sh
Original file line number Diff line number Diff line change
Expand Up @@ -47,33 +47,4 @@ if [ ! -d "/usr/local/cuda/include" ] || [ ! -d "/usr/local/cuda/lib64" ]; then
exit 1
fi

if ! nvcc --version &>/dev/null; then
userName=$(whoami)
cudnn_archives="/home/$userName/Downloads/cudnn-*.tar.xz"

for archive in $cudnn_archives; do
if [ -e "$archive" ]; then
echo "Extracting $archive..."
tar xf "$archive" -C /home/$userName/Downloads/
fi
done

echo "Copying files..."
cp -r /home/$userName/Downloads/cudnn-*/include/* /usr/local/cuda/include/
cp -r /home/$userName/Downloads/cudnn-*/lib/* /usr/local/cuda/lib64/

TARGET_USER="$SUDO_USER"
TARGET_HOME=$(eval echo ~$TARGET_USER)

# Append lines to the target user's ~/.bashrc
echo 'export PATH=/usr/local/cuda/bin:${PATH}' | sudo -u $TARGET_USER tee -a $TARGET_HOME/.bashrc
echo 'export LD_LIBRARY_PATH=/usr/local/cuda/lib64:${LD_LIBRARY_PATH}' | sudo -u $TARGET_USER tee -a $TARGET_HOME/.bashrc

# Reload .bashrc
source $TARGET_HOME/.bashrc

echo "Appended line to ~/.bashrc and saved changes."
echo "Reloaded ~/.bashrc"
fi

echo "Dependencies verified and installed successfully."
echo "Dependencies verified and installed successfully."
4 changes: 2 additions & 2 deletions build_scripts/build_dependencies_linux_no_cuda.sh
Original file line number Diff line number Diff line change
Expand Up @@ -19,8 +19,8 @@ done
# If there are missing dependencies, install them
if [ "${#missing_dependencies[@]}" -gt 0 ]; then
echo "Installing missing dependencies..."
apt-get update -qq
apt-get -y install "${missing_dependencies[@]}"
apt-get update -qq
apt-get -y install "${missing_dependencies[@]}"
fi

# Install Meson if not present
Expand Down
12 changes: 8 additions & 4 deletions build_scripts/build_dependencies_windows_cuda.ps1
Original file line number Diff line number Diff line change
Expand Up @@ -22,10 +22,14 @@ if (-not (Test-Path "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\$cudaVer
} else {
$userName = $env:UserName
$zipFilePath = "C:\Users\$userName\Downloads\cudnn-*.zip"

Write-Host "Extracting zip file..."

Expand-Archive -Path $zipFilePath -DestinationPath C:\Users\$userName\Downloads\ -Force
$extractedPath = "C:\Users\$userName\Downloads\cudnn-*\"

if (-not (Test-Path $extractedPath)) {
Write-Host "Extracting zip file..."
Expand-Archive -Path $zipFilePath -DestinationPath "C:\Users\$userName\Downloads\" -Force
} else {
Write-Host "Already extracted files found."
}

Write-Host "Copying files..."
Copy-Item -Path "C:\Users\$userName\Downloads\cudnn-*\include\*.h" -Destination "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\$cudaVersion\include\" -Recurse
Expand Down
Loading
Loading