Skip to content

Commit

Permalink
fix(core): Double prepare on lex unit for cts with union(stoneatom#226)
Browse files Browse the repository at this point in the history 
fix(core): MTR + Double prepare on lex unit for ctas with union(stoneatom#226)

rich doc to fix  env dependence "not found" (stoneatom#316)

fix(mtr): make the mtr passed (stoneatom#315) (stoneatom#317)

[summary]
How to run:
./mysql-test-run.pl --suite=stonedb --nowarnings --force --nocheck-testcases

1. test cases below are failed, we ignored them first:
issue410
issue415
issue433
issue446
issue515
load.test
2. Sys variables next are not set correctly, they should be added to `/mysql-test/include/default_mysqld.cnf`.
stonedb_insert_delayed = 0
stonedb_ini_allowmysqlquerypath=1

3.check-testcases: A new MTR option to enforce strict cleanup:
https://dev.mysql.com/blog-archive/fail-check-testcases-a-new-mtr-option-to-enforce-strict-cleanup

In `mtr` comment:
```
Some options that control enabling a feature for normal test runs,
can be turned off by prepending 'no' to the option, e.g. --notimer.
This applies to reorder, timer, check-testcases and warnings.
```
Currently we turn off it first to make mtr run success.

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

feat(stonedb): update the copyright date time (stoneatom#321) (stoneatom#324)

fix(core): fix crash when executing sql. (stoneatom#327) (stoneatom#328)

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

feat(index): Unified class member variable name with google style (stoneatom#311) (stoneatom#312)

[summary]
Class member name will be like:

class TableInfo {
  ...
 private:
  std::string table_name_;  // OK - underscore at end.
  static Pool<TableInfo>* pool_;  // OK.
};

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

fix(mtr):move character_set to test.character_set, fix main mtr_test and innodb mtr_test(stoneatom#277) (stoneatom#335)

* fix(mtr):fix innodb mtr and mv some test to test/innodb

* fix(mtr):move character_set to test/main/

fix(mtr):mv main mtr character_set test to test/main/cahracter_set

fix(mtr):mv myisam error mtr-test to test/main/myisam

fix(mtr):add optimizer_switch to include/default_mysqld.cnf

* fix(mtr):fix main mtr optimizer_switch amd mv some test to test/main

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

fix(mtr): Add more ctas queries for the same bug-fix from code-analysis perspectives(stoneatom#226)

fix bugs of stoneatom#309 and stoneatom#310 (stoneatom#313)

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

fix(core): Double prepare on lex unit for cts with union(stoneatom#226) (stoneatom#365)

fix(core): MTR + Double prepare on lex unit for ctas with union(stoneatom#226)

fix(mtr):mtr result stonedb change to tianmu(stoneatom#359) (stoneatom#368)

* fix(mtr):mysqlpump*.test sys_stoendb to sys_tianmu

fix(mtr):fix main result stonedb to tianmu

fix(mtr):mv innodb_fts.ngram_1 to test/innodb_fts

fix(mtr):fix mtr funcs_1 stonedb to tianmu

fix(mtr):fix innodb mtr result stonedb change to tianmu

fix(mtr):fix perfschema mtr result stonedb change to tianmu

* fix(mtr):fix mtr stoendb change to tianmu

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

fix(tianmu): fix group_concat_max_len warning (stoneatom#338) (stoneatom#363)

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

docs(stonedb): update docs and fix bugs (stoneatom#373)

* docs(Quick Deployment and Compiling Methods): update docs

Make the document more user-friendly for new users

* fix(docs): Let the official website pictures can be adaptive

* Update compile-using-ubuntu2004.md

In the install make step, add the go to directory：cd make-3.82

* Update compile-using-redhat7.md

In the install make step, add the go to directory：cd make-3.82

* Update compile-using-centos7.md

In the install make step, add the go to directory：cd make-3.82

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: haitaoguan <105625912+haitaoguan@users.noreply.github.com>

feat(doc):doc add innodb backup to stonedb demonstrate（stoneatom#353） (stoneatom#355)

* feat(doc):doc add innodb backup to stonedb demonstrate

* fix(doc):fix mydumper and mysqldump doc

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

feat(innodb): To support compiling innodb as C++ 17(stoneatom#375) (stoneatom#376)

To support innodb is compiled as C++ 17. For now, tianmu is compiling as C++17,
and C++17 has became mainstream of C++ standard. Therefore, now, we star to support
to compile Innodb as C++17 in stonedb.

This is backport from MySQL, which commit id is: 1a8a111d8f855a31d0aeffc8f02309b2b82dd410 and bug# is #32907274

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

fix stonedb for optimize exists subquery and deep memory copy stoneatom#369 (stoneatom#330, stoneatom#329, stoneatom#340, stoneatom#332) (stoneatom#369)

* fix stonedb for optimize exists subquery and deep memcopy copy

* Update parameterized_filter.cpp

fix ParameterizedFilter deep copy mind memory leak

* fix subquery when this is in sql

* fix code readability and defensive

* use shallow memory when paral hash join by class DimensionGroupMultiMaterialized

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

feat(stonedb): To support gcc 9 and adds `WITH_ROCKSDB` and `WITH_MARISA` to specify where (stoneatom#377)

rocksdb and marisa installed, respectively.

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

fix(mtr): Add test cases for NULL value select on UNIQUE constraint(stoneatom#281) (stoneatom#379)

* fix(core): Double prepare on lex unit for cts with union(stoneatom#226)

fix(core): MTR + Double prepare on lex unit for ctas with union(stoneatom#226)

* fix(mtr): Add test cases for NULL value select on UNIQUE constraint(stoneatom#281)

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

fix bug stoneatom#342 (stoneatom#346)

Co-authored-by: dfx <duanfuxiang>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

Doc(stonedb): Added extensions deccriptions in topics related to backup and recovery for O&M Guide(stoneatom#360) (stoneatom#361)

* updated some code snippets in Install RocksDB step

Signed-off-by: xuejiao-joy <821682573@qq.com>

* some description here

Signed-off-by: xuejiao-joy <821682573@qq.com>

Co-authored-by: Yuting <34496563+Nliver@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

fix(mtr): Add more ctas queries for the same bug-fix from code-analysis perspectives(stoneatom#226)
  • Loading branch information
@xiaoguangye
xiaoguangye committed Aug 11, 2022
1 parent 5a2e7df commit 444f014
Show file tree
Hide file tree
Showing 905 changed files with 8,084 additions and 6,998 deletions.
51 changes: 15 additions & 36 deletions .github/workflows/integration-compile-mtr.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ env:

jobs:
stonedb-build:
runs-on: ubuntu-18.04
runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v3
Expand All @@ -49,43 +49,23 @@ jobs:
- name: install libs
run: |
sudo git clone https://github.com/hustjieke/stonedb-libs.git
sudo git clone https://github.com/stoneatom/stonedb-libs.git
cd stonedb-libs
sudo tar -zxvf stonedb-boost.tar.gz
sudo tar -zxvf stonedb-gcc-rocksdb.tar.gz
sudo tar -zxvf stonedb-marisa.tar.gz
cd gcc7.3
sudo tar -zxvf bin.tar.gz
sudo tar -zxvf include.tar.gz
sudo tar -zxvf lib64.tar.gz
sudo tar -zxvf lib.tar.gz
sudo tar -zxvf share.tar.gz
cd libexec/gcc/x86_64-pc-linux-gnu/7.3.0
sudo tar -zxvf all.tar.gz
sudo tar -zxvf cc1plus.tar.gz
sudo tar -zxvf cc1.tar.gz
cd - && cd ..
sudo cp -r stonedb-boost /usr/local/stonedb-boost
sudo cp -r stonedb-boost/lib /usr/local/stonedb-boost/include
sudo cp -r stonedb-gcc-rocksdb /usr/local/stonedb-gcc-rocksdb
sudo cp -r stonedb-marisa /usr/local/stonedb-marisa
sudo cp -r gcc7.3 /usr/local/gcc7.3
sudo rm /usr/bin/gcc
sudo ln -s /usr/local/gcc7.3/bin/gcc /usr/bin/gcc
sudo rm /usr/bin/g++
sudo ln -s /usr/local/gcc7.3/bin/g++ /usr/bin/g++
sudo rm /usr/bin/c++
sudo ln -s /usr/local/gcc7.3/bin/c++ /usr/bin/c++
cd ..
- name: compile stonedb
run: |
sudo cmake ./ \
-DCMAKE_BUILD_TYPE=release \
-DCMAKE_INSTALL_PREFIX=/stonedb56/install \
-DMYSQL_DATADIR=/stonedb56/install/data \
-DSYSCONFDIR=/stonedb56/install \
-DMYSQL_UNIX_ADDR=/stonedb56/install/tmp/mysql.sock \
-DCMAKE_INSTALL_PREFIX=/stonedb57/install \
-DMYSQL_DATADIR=/stonedb57/install/data \
-DSYSCONFDIR=/stonedb57/install \
-DMYSQL_UNIX_ADDR=/stonedb57/install/tmp/mysql.sock \
-DWITH_EMBEDDED_SERVER=OFF \
-DWITH_STONEDB_STORAGE_ENGINE=1 \
-DWITH_MYISAM_STORAGE_ENGINE=1 \
Expand All @@ -96,16 +76,15 @@ jobs:
-DEXTRA_CHARSETS=all \
-DDEFAULT_CHARSET=utf8 \
-DDEFAULT_COLLATION=utf8_general_ci \
-DWITH_BOOST=/usr/local/stonedb-boost/include
-DWITH_BOOST=/usr/local/stonedb-boost \
-DWITH_ROCKSDB=/usr/local/stonedb-gcc-rocksdb \
-DWITH_MARISA=/usr/local/stonedb-marisa
sudo make VERBOSE=1 -j`nproc`
sudo make install -j`nproc`
#- name: mtr test
# run: |
# sudo mkdir -p /stonedb56/install/data/innodb
# sudo mkdir -p /stonedb56/install/binlog
# sudo mkdir -p /stonedb56/install/log
# sudo mkdir -p /stonedb56/install/tmp
# sudo chown -R mysql:mysql /stonedb56
# cd /stonedb56/install/mysql-test
# sudo ./mysql-test-run.pl --suite=stonedb --nowarnings --force
- name: mtr test
run: |
sudo mkdir -p /stonedb57/install/log
sudo chown -R mysql:mysql /stonedb57
cd /stonedb57/install/mysql-test
sudo ./mysql-test-run.pl --suite=tianmu --nowarnings --force --nocheck-testcases
2 changes: 2 additions & 0 deletions .gitignore
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ build/
cscope*
sql/build_info.h
.vscode
.idea
VERSION.dep
CMakeCache
CMakeFiles
Expand Down Expand Up @@ -112,3 +113,4 @@ CMakeFiles
CMakeFiles/*
CTestTestfile.cmake
COPYING
.vs/
2 changes: 1 addition & 1 deletion .mergify.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ pull_request_rules:
actions:
comment:
message: |
This pull request's title is not fulfill the requirements. @{{author}} please update it :point_down:.
This pull request's title should follow requirements next. @{{author}} please check it :point_down:.
Valid format:
Expand Down
35 changes: 32 additions & 3 deletions CMakeLists.txt
View file
Original file line number Diff line number Diff line change
Expand Up @@ -303,6 +303,29 @@ OPTION(WITH_AUTHENTICATION_LDAP
OPTION(FORCE_UNSUPPORTED_COMPILER "Disable compiler version checks" OFF)
MARK_AS_ADVANCED(WITHOUT_SERVER DISABLE_SHARED FORCE_UNSUPPORTED_COMPILER)

#for boost, to check boost params.
INCLUDE(cmake/boost.cmake)
#Link directories under boost, etc.
IF (WITH_BOOST)
INCLUDE_DIRECTORIES("${WITH_BOOST}/include")
LINK_DIRECTORIES("${WITH_BOOST}/lib/")
ENDIF ()

#for rocksdb, to check rocksdb
INCLUDE(cmake/rocksdb.cmake)
#Link directories under rocksdb, etc.
IF (WITH_ROCKSDB)
INCLUDE_DIRECTORIES("${WITH_ROCKSDB}" "${WITH_ROCKSDB}/include")
LINK_DIRECTORIES("${WITH_ROCKSDB}")
ENDIF ()

#for marisa, to check marisa
INCLUDE(cmake/marisa.cmake)
#Link directories under marisa, etc.
IF (WITH_MARISA)
INCLUDE_DIRECTORIES("${WITH_MARISA}" "${WITH_MARISA}/include")
LINK_DIRECTORIES("${WITH_MARISA}" "${WITH_MARISA}/lib")
ENDIF ()

include(CheckCSourceCompiles)
include(CheckCXXSourceCompiles)
Expand Down Expand Up @@ -481,6 +504,13 @@ IF (USE_CXX11 AND CMAKE_COMPILER_IS_GNUCC AND CMAKE_COMPILER_IS_GNUCXX)
"${CMAKE_EXE_LINKER_FLAGS_DEBUG} -static-libstdc++ -static-libgcc")
ENDIF()

IF(WITH_BOOST AND WITH_ROCKSDB AND WITH_MARISA)
SET(CMAKE_INSTALL_RPATH "${CMAKE_INSTALL_RPATH};${WITH_BOOST}/lib;${WITH_ROCKSDB};${WITH_MARISA}/lib")
ELSE()
SET(CMAKE_INSTALL_RPATH "${CMAKE_INSTALL_RPATH};${BOOST_INCLUDE_DIR};${ROCKSDB_INCLUDE_DIR};${MARISA_INCLUDE_DIR}")
ENDIF()
MESSAGE(STATUS "TO SET INSTALL RPATH FOR STONEDB: ${CMAKE_INSTALL_RPATH}")

OPTION(ENABLED_LOCAL_INFILE
"If we should enable LOAD DATA LOCAL by default" ${IF_WIN})
MARK_AS_ADVANCED(ENABLED_LOCAL_INFILE)
Expand Down Expand Up @@ -565,8 +595,6 @@ ELSE()
SET(DEFAULT_TMPDIR "\"${TMPDIR}\"")
ENDIF()

INCLUDE(cmake/boost.cmake)

IF (CMAKE_SYSTEM_NAME MATCHES "Linux")
OPTION(WITH_SYSTEMD "Enable installation of systemd support files" OFF)
IF (WITH_SYSTEMD)
Expand All @@ -583,7 +611,8 @@ INCLUDE(configure.cmake)

# Common defines and includes
ADD_DEFINITIONS(-DHAVE_CONFIG_H)
ADD_DEFINITIONS(-DSTONEDB)
ADD_DEFINITIONS(-DTIANMU)
ADD_DEFINITIONS(-D_GLIBCXX_USE_CXX11_ABI=1)
INCLUDE_DIRECTORIES(
${CMAKE_CURRENT_BINARY_DIR}/include
${CMAKE_SOURCE_DIR}/extra/rapidjson/include
Expand Down
23 changes: 13 additions & 10 deletions Docs/02-getting-started/quick-deployment.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,40 +5,43 @@ sidebar_position: 3.1

# Quick Deployment

## 1. Download the latest package
## **1. Download the latest package**
Click [here](https://static.stoneatom.com/stonedb-ce-5.6-v1.0.0.el7.x86_64.tar.gz) to download the latest installation package of StoneDB.
## 2. Upload and decompress the TAR package
## **2. Upload and decompress the TAR package**
```shell
tar -zxvf stonedb-ce-5.6-v1.0.0.el7.x86_64.tar.gz
```
Upload the installation package to the directory. The name of the folder extracted from the package is **stonedb56**.
## 3. Check dependencies
## **3. Check dependencies**
```bash
cd /stonedb56/install/bin
ldd mysqld
ldd mysql
```
If the command output contains keywords **not found**, some dependencies are missing and must be installed.
## 4. Modify the configuration file
If the command output contains keywords **not found**, some dependencies are missing and must be installed. <br />For example, `libsnappy.so.1 => not found` is returned:

- If your OS is Ubuntu, run the `sudo apt search libsnappy` command. The command output will inform you to install `libsnappy-dev`.
- If your OS is RHEL or CentOS, run the `yum search all snappy` command. The command output will inform you to install **snappy-devel** and **snappy**.
## **4. Modify the configuration file**
```bash
cd /stonedb56/install/
cp stonedb.cnf stonedb.cnf.bak
vi stonedb.cnf
```
Modify the path and parameters. If the installation folder is **stonedb**, you only need to modify the parameters.
## 5. Create an account
## **5. Create an account**
```bash
groupadd mysql
useradd -g mysql mysql
passwd mysql
```
## 6. Execute reinstall.sh
## **6. Execute reinstall.sh**
```bash
cd /stonedb56/install
./reinstall.sh
```
The process of executing the script is to initialize and start the StoneDB.
## 7. Log in to StoneDB
## **7. Log in to StoneDB**
```shell
/stonedb56/install/bin/mysql -uroot -p -S /stonedb56/install/tmp/mysql.sock
Enter password:
Expand All @@ -65,7 +68,7 @@ mysql> show databases;
+--------------------+
7 rows in set (0.00 sec)
```
## 8. Stop StoneDB
## **8. Stop StoneDB**
```shell
/stonedb56/install/bin/mysqladmin -uroot -p -S /stonedb56/install/tmp/mysql.sock shutdown
```
```
20 changes: 19 additions & 1 deletion Docs/03-O&M-Guide/01-backup-and-recovery/use-mydumper-full-backup.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -99,7 +99,7 @@ Application Options:
--load-data
--fields-terminated-by
--fields-enclosed-by
--fields-escaped-by Single character that is going to be used to escape characters in theLOAD DATA stament, default: '\'
--fields-escaped-by Single character that is going to be used to escape characters in the LOAD DATA stament, default: '\'
--lines-starting-by Adds the string at the begining of each row. When --load-data is usedit is added to the LOAD DATA statement. Its affects INSERT INTO statementsalso when it is used.
--lines-terminated-by Adds the string at the end of each row. When --load-data is used it isadded to the LOAD DATA statement. Its affects INSERT INTO statementsalso when it is used.
--statement-terminated-by This might never be used, unless you know what are you doing
Expand Down Expand Up @@ -249,6 +249,24 @@ Each table has two backup files:
- **database.table.00000.sql**: records table data.
- **database.table-metadata**: records table metadata.
***Extensions***
If you want to import data to StoneDB, you must replace **engine=innodb** with **engine=stonedb** in table schema file **database.table-schema.sql** and check whether the syntax of the table schema is compatible with StoneDB. For example, if the syntax contain keyword **unsigned**, it is incompatible. Following is a schema example after modification:
```
[root@dev-myos dumper]# cat zz.t_user-schema.sql
/*!40101 SET NAMES binary*/;
/*!40014 SET FOREIGN_KEY_CHECKS=0*/;
/*!40103 SET TIME_ZONE='+00:00' */;
CREATE TABLE `t_user` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`c_user_id` varchar(36) NOT NULL DEFAULT '',
`c_name` varchar(22) NOT NULL DEFAULT '',
`c_province_id` int(11) NOT NULL,
`c_city_id` int(11) NOT NULL,
`create_time` datetime NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=STONEDB AUTO_INCREMENT=10001 DEFAULT CHARSET=utf8;
```
### Backup principles
Expand Down
19 changes: 19 additions & 0 deletions Docs/03-O&M-Guide/01-backup-and-recovery/use-mysqldump-backup-and-restore.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -414,11 +414,30 @@ mysql> select * from dumpdb.dumptb;
/stonedb56/install/bin/mysqldump -uroot -p***** -P3306 --skip-opt --master-data=2 --single-transaction --set-gtid-purged=off --databases dumpdb > /tmp/dumpdb.sql
```
#### Back up the table schema of a specific database
```
/stonedb56/install/bin/mysqldump -uroot -p***** -P3306 -d --databases dumpdb > /tmp/dumpdb_table.sql
```
#### Back up data in a table of a specific database, excluding the schema
```
/stonedb56/install/bin/mysqldump -uroot -p***** -P3306 --skip-opt --master-data=2 --single-transaction --set-gtid-purged=off -t dumpdb > /tmp/dumpdb_table.sql
```
#### Use mysqldump to back up all databases, except system databases mysql, performation_schema, and information_schema
```bash
/stonedb56/install/bin/mysql -uroot -p****** -P3306 -e "show databases;" | grep -Ev "sys|performance_schema|information_schema|Database|test" | xargs /stonedb56/install/bin/mysqldump -uroot -p****** -P3306 --master-data=1 --skip-opt --databases > /tmp/ig_sysdb.sql
```
***Extensions***
The previous method is suitable for backing up small tables from InnoDB. If you want to back up a large table, we recommend that you back up the schema and the data file separately, and then run the `sed -i 's/<Original character string>/<New character string>/g' <File name>` to change the engine setting in the backup file. For example:
```
sed -i 's/ENGINE=InnoDB/ENGINE=STONEDB/g' <File name>
```
Then, perform recovery to import the table to StoneDB.
### Recovery
#### Import data to StoneDB
Expand Down
Loading

0 comments on commit 444f014

Please sign in to comment.