Skip to content

junjiemars/kit

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Kit more than code

Quick start

Now, a daily working environment can be build via few lines code on the fly. And effetive, robust, beautiful, consistent on multiple platforms.

There are two ways to use kit:

  • clone git repository to your working host
git clone --depth=1 https://github.com/junjiemars/kit.git
  • use single kit via bash
$ bash <(curl https://raw.githubusercontent.com/junjiemars/kit/master/<where-the-bash-script>)

Unix-like

Kits for Unix-like platforms.

Locate at ul directory.

Clone the current shell environment and the job list of crontab into a plain text file named crontab.list.

# make crontab.list file
$ crontab-env.sh

# load crontab.list
crontab crontab.list

A shortcut of exuberant Ctags.

$ ./ctags.sh --help
$ ./ctags.sh --language=c

Download and configure kit via web.

install-java-kits.sh

Install java programming environment.

$ HAS_ANT=1 HAS_ABCL=1 HAS_MAVEN=1 ./install-java-kit.sh

lr.sh

Log rotate.

nd.sh

A nginx configurator and shell maker.

sh.sh

Setup bash or zsh, aliases, paths and vars etc., on Windows, Darwin, Linux or Unix-like box, and just one line code you need to getting things done:

$ bash <(curl https://raw.githubusercontent.com/junjiemars/kit/master/ul/sh.sh)
# or zsh
$ $ zsh <(curl https://raw.githubusercontent.com/junjiemars/kit/master/ul/sh.sh)

For Windiows you can use Git Bash instead.

You can boot it from local storage two.

# git clone it from github to <kit-local-dir>
$ git clone --depth=1 https://github.com/junjiemars/kit.git <kit-local-dir>

# boot up from <kit-local-dir>
$ GITHUB_H=file://<kit-local-dir> <kit-local-dir>/ul/sh.sh

tc.sh

td.sh

unzip-cjk.sh

Unzip specify code page.

wget.sh

A shortcut of wget to mirror a website.

A tiny-handy Web server based on Netcat (often abbreviated to nc).

$ ./ws.sh --help
$ ./ws.sh --http-port=8080 --http-version=1.1

$ curl -v http://<host>:8080

Reference

Windows

e.sh

Get Windows default code page.

install-win-kits.sh

Setup Win32 programming environment.

Tools

nameurl
aria2chttps://github.com/aria2/aria2/releases

Issues

  • Disk slow on Windows 10: Disable write cache in Disk > Hardware > Policies

Darwin

Boot

Disk

UEFI

UEFI(Unified Extensible Firmware Interface) is known as a motherboard firmware interface for computers. With UEFI boot mode, your computer can initialize the hardware components and boot the OS stored on the hard disk.

GPT

GPT(GUID Partition Table) is a partitioning schemes that introduced by Intel by the 1990s.

MBR

sudo diskutil partitionDisk /dev/disk3 4 MBR \
  MS-DOS WIN 20G \
	MS-DOS LIN 20G \
	MS-DOS MAC 20G \
	MS-DOS DTA R
# enter command mode
sudo fdisk -e /dev/disk3

Recovery

References

Launchpad

remove-from-launchpad.sh

# replace APP_NAME_CASE_SENSITIVE with the app name
sqlite3 \
$(find /private/var/folders \( -name com.apple.dock.launchpad -a -user $USER \) \
2> /dev/null)/db/db \
"DELETE FROM apps WHERE title='APP_NAME_CASE_SENSITIVE';" \
&& killall Dock

Network

Routing

  • Non-Persistent Routes
# display the entire routing table
netstat -rn

# add non-persistent routes
sudo route -n add 192.168/16 10.0.1.1

# delete non-persistent routes
sudo route -n delete 192.168/16
  • Persistent Routes
# display all of the network names
networksetup -listallnetworkservices

# get IP address info for a specific network
networksetup -getinfo "Wi-Fi"

# display routes associated with a specific network
networksetup -getadditionalroutes "Wi-Fi"

# add a static route
networksetup -setadditionalroutes "Wi-Fi" 192.168.0.0 255.255.0.0 10.0.1.1

# delete all static routes
networksetup -setadditionalroutes "Wi-Fi"

DNS

  • /private/etc/hosts
# clean dns cache
sudo killall -HUP mDNSResponder

Port

  • Which Port is Listening
$ lsof -iTCP -sTCP:LISTEN -n -P
  • Who is Listening on the Port
$ lsof -i :<port> -n -P

Applications

  • /Applications
  • ~/Library
rm -r ~/Library/Saved Application State/${APP}
rm -r ~/Library/Containers/${APP}
rm -r ~/Library/Caches/${APP}
rm -r ~/Library/Application Support/${APP}
rm -r ~/Library/Application Scripts/${APP}
rm -r ~/Library/Preferences/${APP}
  • Remove Google software
# local uninstallation:
python ~/Library/Google/GoogleSoftwareUpdate/GoogleSoftwareUpdate.bundle/\
Contents/Resources/GoogleSoftwareUpdateAgent.app/Contents/Resources/\
install.py --uninstall
touch ~/Library/Google/GoogleSoftwareUpdate
# global uninstallation:
sudo python /Library/Google/GoogleSoftwareUpdate/GoogleSoftwareUpdate.bundle/\
Contents/Resources/GoogleSoftwareUpdateAgent.app/Contents/Resources/\
install.py --uninstall
sudo touch /Library/Google/GoogleSoftwareUpdate
  • Remove Oracle JDK
rm -r /Library/Java/JavaVirtualMachines/${JDK_VER}
rm -r /Library/PreferencePanes/JavaControlPanel.prefPane
rm -r /Library/Internet\ Plug-Ins/JavaAppletPlugin.plugin
rm -r /Library/Application Support/Oracle
  • Install OpenJDK
sudo port install openjdk${JDK_VER}
# JAVA_HOME=/Library/Java/JavaVirtualMachines/openjdk%{JDK_VER}/Contents/Home
  • Install Wireshark

#+BEING_SRC sh sudo port install wireshark3

wireshark-chmodbpf has the following notes: To fully complete your installation and use Wireshark to capture from devices (like network interfaces) please run:

sudo dseditgroup -q -o edit -a [USER] -t user access_bpf and change [USER] to the user that needs access to the devices. A reboot should not be required for this to take effect.

A startup item has been generated that will start wireshark-chmodbpf with launchd, and will be enabled automatically on activation. Execute the following command to manually disable it:

sudo port unload wireshark-chmodbpf

Launch Daemons and Agents

There are two types of services that launchd manages:

  • launch daemon can run without a user logged in. /System/Library/LaunchDaemons/ for Apple. /Library/LaunchDaemons/ for the rest.
  • launch agent need the user to be logged in. /System/Library/LaunchAgents/ for Apple. /Library/LaunchAgents/ for rest.

launchctl

launchctl interfaces with launchd to manage and inspect daemons, agents and XPC services.

References

APFS

References

Package Manager

MacPorts

Ubuntu

Using Ubuntu as example to explain Linux.

Hardware

Hardware is the basis.

lshw

lsmod

modprobe

dmesg

Intel CPU

	sudo apt install intel-microcode

AMD CPU

	sudo apt install amd64-microcode

Booting

References

init

service

service --status-all
ls -lh /etc/init.d/
service acpid status

systemctl

Control the systemd system and service manager.

systemctl show-environment
systemctl list-units
systemctl list-sockets
systemctl list-timers
systemctl list-jobs
systemctl status acpid

References

Performance

Limits

/etc/security/limits.conf, replace user1 with real user name.

* 	 soft     nproc          655350
* 	 hard     nproc          655350
* 	 soft     nofile         655350
* 	 hard     nofile         655350
user1 	 soft     nproc          200000
user1 	 hard     nproc          200000
user1 	 soft     nofile         200000
user1 	 hard     nofile         200000

Kernel Parameters

sysctl

Swap

# write
sysctl -w vm.swappiness=30
# load
sysctl -p
# check
sysctl -a |*grep swap.*
less /proc/sys/vm/swappiness

IO

IO request follows VFS -> Cache -> Scheduler -> Driver.

iostat

Scheduler

What scheduler?

cat /sys/block/sda/queue/scheduler

Change scheduler.

sudo modprobe kyber-iosched
echo "kyber" | sudo tee /sys/block/sda/queue/scheduler

Persistent scheduler.

  • change /etc/default/grub
GRUB_CMDLINE_LINUX="cgroup_enable=memory swapaccount=1 scsi_mod.use_blk_mq=1"
  • create /etc/udev/rules.d/60-scheduler.rules if it didn’t exist.
ACTION=="add|change", KERNEL=="sda", ATTR{queue/scheduler}="kyber"
  • apply udev
	sudo udevadm control --reload
	sudo udevadm trigger

Virtual Memory

vmstat

/proc/sys/vm

	(let ((write/s 1200.0) (memory 8000.0))
		(let* ((dirty-ratio (* (/ write/s memory) 100))
					 (background-ratio (ceiling (/ dirty-ratio 2.0))))
			(list dirty-ratio background-ratio)))
vm.dirty_background_ratio = 6
vm.dirty_ratio = 12
vm.dirty_writeback_centisecs = 500
vm.dirty_expire_centisecs = 3000

Memory

RAM is split into Zones

  • Direct Memory Access (DMA): This is the low 16 MB of memory. The zone gets its name because, a long time ago, there were computers that could only do direct memory access into this area of physical memory.
  • Direct Memory Access 32: Despite its name, Direct Memory Access 32 (DMA32) is a zone only found in 64-bit Linux. It’s the low 4 GB of memory. Linux running on 32-bit computers can only do DMA to this amount of RAM (unless they are using the physical address extension (PAE) kernel), which is how the zone got its name. Although, on 32-bit computers, it is called HighMem.
  • Normal: On 64-bit computers, normal memory is all of the RAM above 4GB (roughly). On 32-bit machines, it is RAM between 16 MB and 896 MB.
  • HighMem: This only exists on 32-bit Linux computers. It is all RAM above 896 MB, including RAM above 4 GB on sufficiently large machines.
getconf PAGESIZE
less /proc/zoneinfo
sudo cat /proc/pagetypeinfo
less /proc/buddyinfo
	(let ((mem-available 2996044))
		(list :write (* (/ mem-available 4 100) 10)
					:flood (* (/ mem-available 4 100) 30)))

CPU

cpufreqd

# show old
for i in 0 1 2 3; do cat /sys/devices/system/cpu/cpu$i/cpufreq/scaling_governor; done
# trans powersave to performance
# /etc/rc.local
for i in 0 1 2 3; do echo performance > /sys/devices/system/cpu/cpu$i/cpufreq/scaling_governor; done

Network

References

journal

journalctl

dmesg

snap

Location /var/lib/snapd/snaps/

Just keep the current ul/snap-rm.sh version.

Screen

  • kazam: screen recording and capturing program.

Docker

Docker on Linux

Run docker client via non root

Docker daemon run as root user in a group called docker by default.

$ sudo usermod -a -Gdocker <user>
$ sudo service docker[.io] restart
$ sudo reboot now

Port connection

Container linking

Basic CentOS Development Environment

Include basic building/networking tools, emacs/vim editors for c/c++/clang/python/lua development. You can use root or default sudoer: u/Hell0 to login and play.

*** Build from Dockerfile

$ docker build -t centos-dev https://raw.githubusercontent.com/junjiemars/kit/master/docker/dev/centos.dockerfile

Or you can download centos.dockefile then build from it.

*** Play with centos-dev Docker Container

  • once a time
$ docker run -w /home/u -h centos --privileged -u u -it --rm junjiemars/centos-dev /bin/bash
  • as daemon
# gdb or lldb needs privileged permission
$ docker run --name centos-dev -w /home/u -h centos --privileged -d junjiemars/centos-dev
$ docker exec -it -u u centos-dev /bin/bash
  • cannot change locale
$ localedef -i en_US -f UTF-8 en_US.UTF-8

** Basic Ubuntu Development Environment

Include basic building/networking tools, emacs/vim editors for c/c++/llvm/python/lua development. You can use root or default sudoer: u/Hell0 to login and play.

Build from Dockerfile

$ docker build -t ubuntu-dev https://raw.githubusercontent.com/junjiemars/kit/master/docker/dev/ubuntu.dockerfile

Or you can download ubuntu.dockefile then build from it.

Play with ubuntu-dev Docker Container

  • once a time
$ docker run -w /home/u -h ubuntu --privileged -u u -it --rm junjiemars/ubuntu-dev /bin/bash
  • as daemon
$ docker run --name ubuntu-dev -w /home/u -h ubuntu --privileged -d junjiemars/ubuntu-dev
$ docker exec -it -u u ubuntu-dev /bin/bash

Avoid slow apt-get update and install

  • Avoid IPv6 if you use a slow tunnel
$ sudo apt-get -o Acquire::ForceIPv4=true [update|install]
  • Use mirrors which is based on your geo location
# use mirror automatically
$ sudo cp /etc/apt/sources.list /etc/apt/sources.list.ori
$ sudo sed -i 's#http:\/\/archive.ubuntu.com\/ubuntu\/#mirror:\/\/mirrors.ubuntu.com\/mirrors.txt#' /etc/apt/sources.list

# check mirrors list that based on your geo
$ curl -sL mirrors.ubuntu.com/mirrors.txt
  • Aovid posioning mirrors: select another country

Docker on Windows 10

Now, the good news is Docker has native stable version for Windows 10 since 7/29/2016. If you need Docker Machine you can check DockerMachine on Windows 10.

Hyper-V Default Locations

  • Control Panel > Administrative Tools > Hyper-V Manager
  • Change Virtual Machines location
  • Change Virtual Hard Disks location

tty Issue

  • Mintty does not provide full TTY support;
  • Use cmd or PowerShell;

Internal Virtual Switch

Failed to Start

  • Hyper-V Manager: keep only one MobiLinuxVM and delete all the others

Basic Java Development Environment

Build from Dockerfile

$ docker build -t java-dev https://raw.githubusercontent.com/junjiemars/kit/master/docker/dev/java.dockerfile

Play with java-dev Docker Container

  • one time
$ docker run -w /home/u -h centos -u u -it --rm java-dev /bin/bash
  • as daemon
$ docker run --name java-dev -w /home/u -h centos --privileged -d java-dev
$ docker exec -it -u u java-dev /bin/bash

Install Java Programming Environment

Run into java-dev container and then run install-java-kits.sh

$ HAS_ALL=YES install-java-kits.sh

Docker for Database

PostgreSQL

Oracle

You can pull it from Docker Hub, or build it from oracle_xe.dockerfile. Beside, Oracle XE 11g2 could not be downloaded via curl, there needs some hack way to do it.

$ docker pull junjiemars/xe11g2:latest
$ docker run --name xe11g2 -p 1521:1521 -p 8080:9000 -d junjiemars/xe11g2:latest

Docker Machine on Windows 10

  • Install Docker Toolbox
  • Run Docker Quickstart Terminal
  • Play docker, it’s same on Linux box

Install Docker Toolbox

  • Kitematic is useless, don’t install it
  • Need VirtualBox and NIS6+

Configure Docker Quickstart Terminal

  • Font: On Windows, the Console’s font is ugly if the code page is 936 for Chinese locale. Change the Windows locale to English and change the font to Consolas or others thats good for English lauguage. Restart Windows then switch the locale back to your locale, then restart it again.
  • Mintty: Mintty is not based on Windows’ Console, it’s better than git-bash. To use Mintty via change Docker Quickstart Terminal’s the target in shortcut to
"C:\Program Files\Git\usr\bin\mintty.exe" -i "c:\Program Files\Docker Toolbox\docker-quickstart-terminal.ico" /usr/bin/bash --login -i  "c:\Program Files\Docker Toolbox\start.sh"
  • MACHINE_STORAGE_PATH: Environment variable points to docker’s image location.

Access Windows dir in Docker Host

  • Configure Shared folders on VirtualBox:
<vbox-folder-label-name> -> <windows-local-dir>
  • Mount the dir on Docker VM:
$ docker-machine ssh [machine-name]
$ mkdir -p /home/docker/<dir-name>
$ sudo mount -t vboxsf -o uid=1000,gid=50 <vbox-folder-label-name> /home/docker/<dir-name>
  • Run Docker Host with Volume:
$ docker run -d -v <vbox-folder-label-name>:<docker-host-mount-dir> <image>

tty mode

If you got cannot enable tty mode on non tty input, so

$ docker-machine ssh <default>

Sharing Files

  • machine -> host:
$ docker-machine scp <machine>:<machine-path> <host-path>
  • host -> machine:
$ docker-machine scp <host-path> <machine>:<machine-path>
  • container -> host
# copy from container to machine
$ docker cp <container-path> <machine-path>
# copy from machine to host
$ docker-machine scp <machine>:<machine-path> <host-path>
  • host -> container vice versa.

Docker Networking

Bridge

The default docker0 virtual bridge interface let communications:

  • container -> container
  • container -> host
  • host -> container

Overlay

SSH between Containers

  • Read from socket failed: Connection reset by peer
$ sudo ssh-keygen -t rsa -f /etc/ssh/ssh_hosts_rsa_key
$ sudo ssh-keygen -t dsa -f /etc/ssh/ssh_hosts_dsa_key

Tips

  • Container’s IP address
# on default bridge network
$ docker inspect --format "{{.NetworkSettings.IPAddress}}" <container-id|container-name>

# on specified network
docker inspect --format "{{.NetworkSettings.Networks.<your-network>.IPAddress}}" <container-id|container-name>
  • Link to Another Containers (/etc/hosts)
$ docker run --name n2 --link=n0 --link=n1 -d <docker-image>
"registry-mirrors": [
    "https://registry.docker-cn.com",
    "https://docker.mirrors.ustc.edu.cn"
]

Storage

# create mount the volume on /opt/vol
$ docker run --name n0 -w /home/u -h n0 -v /opt/vol -d <docker-iamge>

# mount a host volume on /opt/vol
$ docker run --name n0 -w /home/u -h n0 -v <host-path>:/opt/vol -d <docker-image>

# mount a host file
$ docker run --name n0 -w /home/u -h n0 -v ~/.bash_history:/home/u/.bash_history -d <docker-image>

Podman

	cd kit/podman
	make -C compose -f <xxx.Makefile> up

Podman on Ubuntu

	# install podman
	sudo apt install podman podman-gvproxy podman-plugins
	# install podman-compose
	sudo pip3 install podman-compose
	# install CNI

containers/podman#21024

Podman on Darwin

Raspberry

Ubuntu

Login with user ubuntu and default password ubuntu, then prompt to change the default password.

Install

On MacOS

sudo diskutil list
sudo diskutil unmountdisk /dev/xxx
xzcat ubuntu.img.xz | sudo dd of=/dev/xxx bs=4M

Network

Static IP Address

sudo vi /etc/netplan/50-cloud-init.yaml
network:
  version: 2
  ethernets:
    eth0:
      dhcp4: no
      addresses: [192.168.1.15/24,]
      gateway4: 192.168.1.1
      nameservers:
        addresses: [208.67.222.222,8.8.8.8]
      match:
        macaddress: xx:xx:xx:xx:xx:xx
      set-name: eth0
sudo netplan apply

Raspberry References

Database

Oracle

make -C docker/compose/ -f oraclexe.Makefile start

SQL*Plus

You can using raw sqlplus command program or an awsome tool sqlplus.sh, or Emacs.

Sqlite

Scheme

List tables

select name from sqlite_master where type='table';

Table structure #+BEING_SRC sql pragma table_info(<table>);

MySQL

make -C docker/compose/ -f mysql.Makefile start

System variables

mysql> select @@sql_mode;
mysql> SET GLOBAL sql_mode=(SELECT REPLACE(@@sql_mode,'ONLY_FULL_GROUP_BY',''));
mysql> SET GLOBAL sql_mode = sys.list_drop(@@GLOBAL.sql_mode, 'ONLY_FULL_GROUP_BY');
mysql> SET @@sql_mode = @@GLOBAL.sql_mode;

Functions

SELECT [Name],
       GROUP_CONCAT(DISTINCT [Name]  SEPARATOR ',')
       FROM tbl_cars

VCS

Git

Proxy

git config --global http.proxy 'socks5://127.0.0.1:<port>'
# or per-user: export ALL_PROXY='socks5://127.0.0.1:<port>'

# git something...

git config --global --unset http.proxy

Issue

cannot checkout branch - error: pathsepc

fix the .git/config file

[remote "origin"]
  url = https://github.com/junjiemars/kit.git
  fetch= +refs/heads/*:refs/remotes/origin/*

Subversion

On Windows, svn doesn’t support posix repository URL.

# create repository
svnadmin create </opt/lab/vcs/svn/repo>

# make hooks/pre-revprop-change file
chmod u+x hooks/pre-revprop-change

# checkout repository
svn checkout <file://E:/lab/vcs/svn/repo>

# update local
svn update

# commit
svn add .
svn commit -m"message"

# log
svn log --limit 8

hooks/pre-revprops-change

#!/bin/sh
exit 0;

Git and Subversion Bridge

Clone the SVN repository

git svn clone <file:///a/b/c>

Create a new branch to modify or work on a new feature

git checkout -b <new-branch>

Work on your newly created branch in the local repository as usual. Once complete, you can switch to the master branch and merge your changes to master.

git checkout master
git merge <new-branch>

Update your SVN repository

git svn rebase

Commit back to the SVN repository

git svn dcommit -m"commit back"

References

Language

Java

Install

# VER=openjdk-11, default version
sudo apt update
sudo install ${VER}-jdk ${VER}-dbg ${VER}-11-doc
sudo update-alternatives --install /usr/bin/java, java `echo $JAVA_HOME`/bin/java
sudo update-alternatives --install /usr/bin/javac, javac `echo $JAVA_HOME`/bin/javac
sudo update-alternatives --install /usr/bin/jhsdb, jhsdb `echo $JAVA_HOME`/bin/jhsdb

jps

jps -vl

jhsdb

jhsdb -h

jstack

jhsdb jstack -h

jmap

echo $PASSWORD | sudo -S jhsdb jmap --pid $PID

jd.sh

Allow jdb to attach process on Ubuntu.

OpenJFX

* --module-path /Users/<user>/Downloads/javafx-sdk-11/lib --add-modules=javafx.controls,javafx.fxml

Kotlin

Node.js

NODE_OPTIONS environment variable

export NODE_OPTIONS="--max-old-space-size=1024"

References

SSL

Certificate Signing Request

# generate a private key and a CSR(Certificate Signing Request)
openssl req -newkey rsa:2048 -nodes -keyout x.key -out x.csr
	openssl req \
					-newky rsa:2048 -nodes -keyout x.key \
					-x509 -days 365 -out z.crt
keytool -v -list -keystore <key-store>

References

Hardware

CPU

hwloc

numactl

All

IDE

IDEA Intellij

Performance

Find the configuration file: idea64.vmoptions at ~/.config/JetBrains.

-Xms128m
-Xmx1024m
-XX:+UseParallelGC
-XX:+UseAdaptiveSizePolicy
-XX:-UseConcMarkSweepGC
  • -Xms: specify the initial memory allocation pool.
  • -Xmx: specify the maximum memory allocation pool.
  • +UseParallelGC: faster responsibilies but may be unstable
  • -XX:+UseParallelGC: faster responsibilies but may be unstable
  • -XX:+UseAdaptiveSizePolicy: faster responsibilies but may be unstable
  • -XX:-UseConcMarkSweepGC: faster responsibilies but may be unstable

Unified Gradle

Default gradle home at ~/.gradle.

Open File=>=Settings=>=

Keymap

idea-jar-kit.sh import/export settings.

	# export to kit
	./idea-jar-kit.sh -x=.

	# import from kit
	./idea-jar-kit.sh -a=.

	# list
	./idea-jar-kit.sh -l=