Skip to content

Commit

Permalink
Doc: update the building guide for MacOS
Browse files Browse the repository at this point in the history 
Our team has thoroughly reviewed and significantly improved the
documentation for the macOS system. We have left no stone unturned
in our efforts to ensure that our instructions for setting up the
Mac development environment, compiling and installing cbdb, and
verifying the cluster are crystal clear. We have conducted
extensive testing on version 13.4+ on various chips, including
intel, M1, and M2, and have confirmed that all of them are capable
of flawlessly executing the aforementioned functions.
  • Loading branch information
@Baowen648
Baowen648 committed Sep 27, 2023
1 parent d89fc09 commit 80b2191
Showing 1 changed file with 109 additions and 130 deletions.
239 changes: 109 additions & 130 deletions readmes/README.macOS.md
View file
Original file line number Diff line number Diff line change
@@ -1,153 +1,132 @@
We've confirmed that these steps work on a brand new installation of macOS
Sierra.
# Compile and Install Cloudberry Database on macOS

## 1. Install needed dependencies
This document shares how to build, compile, and install Cloudberry Database on macOS for development purposes. Follow the steps below.

Execute the following, this script will start by caching the `sudo` password:
According to our test, these steps work well on macOS Ventura 13.4+ with both Intel and Apple silicon processors (M1 or M2). If you have an older version of macOS, upgrading is recommended.

```bash
./README.macOS.bash
source ~/.bash_profile
```

Note: This will install Homebrew if missing

## 2. Allow ssh to use the version of python in path, not the system python

Execute the following in your terminal:

```bash
mkdir -p "$HOME/.ssh"

cat >> ~/.bash_profile << EOF
# Allow ssh to use the version of python in path, not the system python
# BEGIN SSH agent
# from http://stackoverflow.com/questions/18880024/start-ssh-agent-on-login/18915067#18915067
SSH_ENV="\$HOME/.ssh/environment"
# Refresh the PATH per new session
sed -i .bak '/^PATH/d' \${SSH_ENV}
echo "PATH=\$PATH" >> \${SSH_ENV}
function start_agent {
echo "Initialising new SSH agent..."
/usr/bin/ssh-agent | sed 's/^echo/#echo/' > "\${SSH_ENV}"
echo succeeded
chmod 600 "\${SSH_ENV}"
source "\${SSH_ENV}" > /dev/null
/usr/bin/ssh-add;
}
# Source SSH settings, if applicable
if [ -f "\${SSH_ENV}" ]; then
. "\${SSH_ENV}" > /dev/null
ps -ef | grep \${SSH_AGENT_PID} 2>/dev/null | grep ssh-agent$ > /dev/null || {
start_agent;
}
else
start_agent;
fi
[ -f ~/.bashrc ] && source ~/.bashrc
# END SSH agent
EOF
```

Then bring `.bash_profile` into effect and allow ssh to use the version of python in path, not the system python.

```bash
source ~/.bash_profile

sudo tee -a /etc/ssh/sshd_config << EOF
# Allow ssh to use the version of python in path, not the system python
PermitUserEnvironment yes
EOF
```

## 3. Enable ssh to localhost without requiring a password

Currently, the GPDB utilities require that it be possible to `ssh` to localhost
without using a password. To verify this, the following command should succeed
without erroring, or requiring you to enter a password.

```bash
ssh <hostname of your machine> # e.g., ssh briarwood (You can use `hostname` to get the hostname of your machine.)
```

However, if it is the first time that you are `ssh`'ing to localhost, you may
need to accept the trust on first use prompt.

```bash
The authenticity of host '<your hostname>' can't be established.
ECDSA key fingerprint is SHA256:<fingerprint here>.
Are you sure you want to continue connecting (yes/no)?
```
If the hostname does not resolve, try adding your machine name to `/etc/hosts`,
like so:
```bash
echo -e "127.0.0.1\t$HOSTNAME" | sudo tee -a /etc/hosts
```
## Step 1. Install needed dependencies

If you see `ssh: connect to host <> port 22: Connection refused`, enable remote
login in:
Run the following command to install the needed dependencies. You will be asked to enter the `sudo` password of your macOS system.

```bash
System Preferences -> Sharing -> Remote Login
source readmes/README.macOS.bash
```

If you see a password prompt, add your SSH key to the `authorized_keys` file,
like so:
> [!NOTE]
> This will install [Homebrew](https://brew.sh/) if missing.
## Step 2. Enable password-free SSH connection to localhost

1. Enable **Remote Login** on your macOS system by navigating to **System Preferences** \> **Sharing** \> **Remote Login**.
2. Run the following command to verify whether password-free SSH connection to localhost has been enabled on your operating system.

```bash
ssh $(hostname)
```

- If this command runs without error or requiring you to enter a password, run `exit` and go to [Step 3. Configure, compile, and install](#step-3-configure-compile-and-install).
- If you are required to enter a password, take the following steps to set up password-free SSH connection.

1. Run `ssh-keygen` and then `cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys`.
2. Run `ssh $(hostname)` again to check whether password-free connection is ready.
3. If ready, run `exit` and go to [Step 3. Configure, compile, and install](#step-3-configure-compile-and-install).

> [!NOTE]
>
> - If it is the first time you are using `ssh` connection to localhost, you might need to accept the trust on the first use prompt:
>
> ```bash
> The authenticity of host '<your hostname>' can't be established.
> ECDSA key fingerprint is SHA256:<fingerprint here>.
> Are you sure you want to continue connecting (yes/no)?
> ```
>
> - If your hostname does not resolve, try adding your machine name to `/etc/hosts`:
>
> ```bash
> echo -e "127.0.0.1\t$HOSTNAME" | sudo tee -a /etc/hosts
> ```
## Step 3. Configure, compile, and install
```bash
mkdir -p ~/.ssh
ssh-keygen -t rsa -b 4096 -C "your_email@example.com"
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
```
# Run the following commands under the `cloudberrydb/` dir.
# 1. Configure the build environment.
After troubleshooting, verify one more time that you can ssh without a
password and that `~/.ssh/environment` exists, by running:
BREWPREFIX=$(brew --prefix); export PATH="$BREWPREFIX/opt/gnu-sed/libexec/gnubin:$BREWPREFIX/opt/apr/bin:$PATH"; CXXFLAGS="-I $BREWPREFIX/include" CFLAGS="-ggdb -Og -g3 -fno-omit-frame-pointer -I $BREWPREFIX/include" LDFLAGS="-L $BREWPREFIX/lib" CC=$(which gcc-13) CXX=$(which g++-13) ./configure --enable-debug --prefix=$(cd ~; pwd)/install/cbdb;
```bash
ssh <hostname of your machine>
ls ~/.ssh/environment
```
# 2. Compile and install Cloudberry Database.
If the ssh succeeded with out asking for a password, and that file exists, you
are good to go. Remember to exit all of your ssh session before moving on.
make -j8
make -j8 install
## 4. Configure, compile, and install
# 3. Bring in Greenplum environment for Cloudberry Database into your running shell.
At this point head back the main [README.md](./README.md#build-the-database),
and continue from __Build the Database__.
source $(cd ~; pwd)/install/cbdb/greenplum_path.sh
## Troubleshooting macOS specific issues:
# 4. Install the Python dependencies.
### Workaround for libreadline / libxml2 on OSX 10.11 (El Capitan)
pip3 install --user -r readmes/python-dependencies.txt
Symptoms:
* After Running `./configure`,
* You see output
`configure: error: readline library not found`, and
* in `config.log` you see
`ld: file not found: /usr/lib/system/libsystem_symptoms.dylib for architecture x86_64`
# 5. Start a demo cluster.
There is an issue with Xcode 8.1 on El Capitan. Here's a workaround:
PORT_BASE=8000 make create-demo-cluster
```bash
brew install libxml2
brew link libxml2 --force
# gpdemo-env.sh contains PGPORT, COORDINATOR_DATA_DIRECTORY and MASTER_DATA_DIRECTORY values
source gpAux/gpdemo/gpdemo-env.sh
```
Other workarounds may include downgrading to Xcode 7.3, or installing the CLT
package from Xcode 7.3.

For more info, [this seems to be the best thread](https://github.com/Homebrew/brew/issues/972)

### Disable System Integrity Protection
## Step 4. Verify the cluster
1. You can verify whether the cluster has started successfully by running the following command. You will see many active `postgres` processes with ports ranging from `8000` to `8007`.
```bash
ps -ef | grep postgres
```
2. Connect to the Cloudberry Database and see the active segment information by querying the system table `gp_segement_configuration`. For detailed description of this table, see the Greenplum document [here](https://docs.vmware.com/en/VMware-Greenplum/6/greenplum-database/ref_guide-system_catalogs-gp_segment_configuration.html).
```sql
$ psql -p 8000 postgres
postgres=# select version();
postgres=# select * from gp_segment_configuration;
```
Example output:
```shell
$ psql -p 8000 postgres
psql (14.4, server 14.4)
Type "help" for help.
postgres=# select version();
version
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
PostgreSQL 14.4 (Cloudberry Database 1.0.0+1c0d6e2224 build dev) on x86_64-apple-darwin22.4.0, compiled by gcc-13 (Homebrew GCC 13.2.0) 13.2.0, 64-bit compiled on Sep 22 2023 10:56:01
(1 row)
postgres=# select * from gp_segment_configuration;
dbid | content | role | preferred_role | mode | status | port | hostname | address | datadir | warehouseid
------+---------+------+----------------+------+--------+------+-----------------------------+-----------------------------+----------------------------------------------------------------------------------------------------------+-------------
1 | -1 | p | p | n | u | 8000 | hashdatas-MacBook-Pro.local | hashdatas-MacBook-Pro.local | /Users/hashdata/Documents/GitHub/upstream/cloudberrydb/gpAux/gpdemo/datadirs/qddir/demoDataDir-1 | 0
8 | -1 | m | m | s | u | 8001 | hashdatas-MacBook-Pro.local | hashdatas-MacBook-Pro.local | /Users/hashdata/Documents/GitHub/upstream/cloudberrydb/gpAux/gpdemo/datadirs/standby | 0
3 | 1 | p | p | s | u | 8003 | hashdatas-MacBook-Pro.local | hashdatas-MacBook-Pro.local | /Users/hashdata/Documents/GitHub/upstream/cloudberrydb/gpAux/gpdemo/datadirs/dbfast2/demoDataDir1 | 0
6 | 1 | m | m | s | u | 8006 | hashdatas-MacBook-Pro.local | hashdatas-MacBook-Pro.local | /Users/hashdata/Documents/GitHub/upstream/cloudberrydb/gpAux/gpdemo/datadirs/dbfast_mirror2/demoDataDir1 | 0
2 | 0 | p | p | s | u | 8002 | hashdatas-MacBook-Pro.local | hashdatas-MacBook-Pro.local | /Users/hashdata/Documents/GitHub/upstream/cloudberrydb/gpAux/gpdemo/datadirs/dbfast1/demoDataDir0 | 0
5 | 0 | m | m | s | u | 8005 | hashdatas-MacBook-Pro.local | hashdatas-MacBook-Pro.local | /Users/hashdata/Documents/GitHub/upstream/cloudberrydb/gpAux/gpdemo/datadirs/dbfast_mirror1/demoDataDir0 | 0
4 | 2 | p | p | s | u | 8004 | hashdatas-MacBook-Pro.local | hashdatas-MacBook-Pro.local | /Users/hashdata/Documents/GitHub/upstream/cloudberrydb/gpAux/gpdemo/datadirs/dbfast3/demoDataDir2 | 0
7 | 2 | m | m | s | u | 8007 | hashdatas-MacBook-Pro.local | hashdatas-MacBook-Pro.local | /Users/hashdata/Documents/GitHub/upstream/cloudberrydb/gpAux/gpdemo/datadirs/dbfast_mirror3/demoDataDir2 | 0
(8 rows)
postgres=#
```
3. Now we can finally run `installcheck-world` to test everything works fine:
```bash
# In the folder where your cloned the source code
make installcheck-world
```
Congratulations 🎉! You've successfully installed and created a CloudberryDB cluster. Happy Hacking! 😉
Note that you may need to disable System Integrity Protection in order to bring
up the gpdemo cluster. Without doing this, psql commands run in child processes
spawned by gpinitsystem may have the DYLD_* environment variables removed from
their environments.

0 comments on commit 80b2191

Please sign in to comment.