Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP: Customizable transfer files #9

Open
wants to merge 8 commits into
base: master
Choose a base branch
from
Open

WIP: Customizable transfer files #9

Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
ef9a5ac
Basic push, still need to add the send all logs functionality and mak…
Aug 11, 2021
df365c6
Pushing the default yaml file for transfer, should be needed for test…
Aug 11, 2021
ca1206d
Adding code that will get the default logs if no additional log infor…
Aug 16, 2021
b5cff1b
Removing the code to download the dault log yaml, since we modified t…
Aug 16, 2021
28e0a3d
Adding requested changes from the code review, and changes that were …
Aug 16, 2021
79bea6b
Removing debugging commands from code
Aug 17, 2021
88743d3
Updating scripts for Pull Request, have removed zeek log array, copie…
Sep 9, 2021
344dbba
Update default logs transferred
ethack Oct 27, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Empty file modified connect_sensor.sh
100644 → 100755
View file
Open in desktop
Empty file.
72 changes: 72 additions & 0 deletions zeek-transport-default.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
Days: 7
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you change the default here to be 3 just to match the current behavior?

Logs:
#These headings are for user friendliness, they do not do anything
Default:
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since the headings don't do anything, I think it would be less confusing to make all the individual log files nested directly under Logs: (so de-indent one level) and make the Default:, Troubleshooting:, and All: comments instead of YAML keys.

conn
dns
http
ssl
x509
known_certs
#Troubleshooting:
#capture_loss
#notice
#stats
#All:
#dce_rpc
#dhcp
#dnp3
#ftp
#irc
#kerberos
#modbus
#modbus_register_change
#mysql
#ntlm
#ntp
#radius
#rdp
#rfb
#sip
#smb_cmd
#smb_files
#smb_mapping
#smtp
#snmp
#socks
#ssh
#syslog
#tunnel
#files
#ocsp
#pe
#netcontrol
#netcontrol_drop
#netcontrol_shunt
#netcontrol_catch_release
#openflow
#intel
#notice_alarm
#signatures
#traceroute
#known_hosts
#known_modbus
#known_services
#software
#barnyard2
#dpd
#unified2
#unknown_protocols
#weird
#weird_stats
#broker
#cluster
#config
#loaded_scripts
#packet_filter
#print
#prof
#reporter
#stderr
#stdout
#conn-summary
149 changes: 135 additions & 14 deletions zeek_log_transport.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,117 @@

export PATH="/sbin:/usr/sbin:$PATH" #Note that cron does _NOT_ include /sbin in the path, so attempts to locate the "ip" binary fail without this fix

default_user_on_aihunter='dataimport'
default_user_on_aihunter="dataimport"

get_send_logs() {
##These names were found at https://docs.zeek.org/en/master/script-reference/log-files.html
declare -a zeek_logs=( "barnyard2"
"broker"
"capture_loss"
"cluster"
"config"
"conn"
"conn-summary"
"dce_rpc"
"dhcp"
"dnp3"
"dns"
"dpd"
"files"
"ftp"
"http"
"intel"
"irc"
"kerberos"
"known_certs"
"known_hosts"
"known_modbus"
"known_services"
"loaded_scripts"
"modbus"
"modbus_register_change"
"mysql"
"netcontrol"
"netcontrol_catch_release"
"netcontrol_drop"
"netcontrol_shunt"
"notice"
"notice_alarm"
"ntlm"
"ntp"
"ocsp"
"openflow"
"packet_filter"
"pe"
"print"
"prof"
"radius"
"rdp"
"reporter"
"rfb"
"signatures"
"sip"
"smb_cmd"
"smb_files"
"smb_mapping"
"smtp"
"snmp"
"socks"
"software"
"ssh"
"ssl"
"stats"
"stderr"
"stdout"
"syslog"
"traceroute"
"tunnel"
"unified2"
"unknown_protocols"
"weird"
"weird_stats"
"x509"
)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not often this would happen, but adding a new log file would mean having to update this list and thus the installed script (which is what we're trying to avoid having to do). For instance, AC-Hunter recently added a new log file open_conn.log. If a user installs a plugin that generates a new log file or Zeek happens to start producing a new log we wouldn't be able to get that log without updating this script.

What if we just tested the string against a regex and only allowed ^[a-zA-Z0-9-_]+$? I'm not aware of anything that could be abused path traversal-wise with only those characters and I'm willing to take the tradeoff of allowing more character combinations in exchange for being more future-proof.

send_logs=()

destination="$1"
dest_log_trans="$2"
local_dest_dir="$3"

rsync $destination:$dest_log_trans $local_dest_dir

##Parse file for logs skip headerlines (indicated by :) and commented lines (indicated by #)
echo $local_dest_dir
request_logs=`cat $local_dest_dir | grep -v ":" | grep -v "#"`
echo ${requst_logs[0]}

##TODO check if the request logs is set to all if so we should set the send_logs to the zeek_logs variable
if [[ ${request_logs[*]} == *all* || ${request_logs[*]} == *All* || ${request_logs[*]} == *ALL* ]]; then
send_logs+=("${zeek_logs[@]}")
else
##Ensure we only send the zeek logs
for log in $request_logs
do
if [[ ${zeek_logs[*]} == *${log}* ]]; then
send_logs+=($log)
#else
# echo $log will not be transported
fi
done
fi

echo "${send_logs[*]}"
}


get_send_days() {
local_yaml_file=$1

days_back=`cat $local_yaml_file | grep "Days:" | cut -d ':' -f2`
rm $local_yaml_file

echo $days_back
}


can_ssh () {
Expand Down Expand Up @@ -81,14 +191,14 @@ usage () {

require_util () {
#Returns true if all binaries listed as parameters exist somewhere in the path, False if one or more missing.
while [ -n "$1" ]; do
if ! type -path "$1" >/dev/null 2>/dev/null ; then
echo Missing utility "$1". Please install it. >&2
return 1 #False, app is not available.
fi
shift
done
return 0 #True, app is there.
while [ -n "$1" ]; do
if ! type -path "$1" >/dev/null 2>/dev/null ; then
echo Missing utility "$1". Please install it. >&2
return 1 #False, app is not available.
fi
shift
done
return 0 #True, app is there.
} #End of requireutil


Expand Down Expand Up @@ -237,14 +347,25 @@ status "Sending logs to rita/aihunter server $aih_location , My name: $my_id , l
status "Preparing remote directories"
ssh $extra_ssh_params "$aih_location" "mkdir -p ${remote_top_dir}/$today/ ${remote_top_dir}/$yesterday/ ${remote_top_dir}/$twoda/ ${remote_top_dir}/$threeda/ ${remote_top_dir}/current/"

recv_loc="`pwd`/zeek-log-transport.yaml"
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of pwd we should copy the file to a place we know is writable by the current user. Here's some sample code that might be useful.

recv_dir="$(mktemp)"
recv_loc="$recv_dir/zeek-log-transport.yaml"
# clean up the temporary files on exit
trap "rm -rf '$recv_dir'" EXIT

request_logs=`get_send_logs $aih_location "/etc/AI-Hunter/zeek-log-transport.yaml" "$recv_loc"`
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are cases where we will want different retention and logs from different sensors. If the sensor is small enough we can transfer all the logs and keep them for longer, but another sensor's logs might be too big where we need to limit what is sent and only keep it for a limited time.

I can think of a couple ways to do this:

  • We could pull from a sensor-specific location first (e.g. $remote_top_dir/zeek-log-transport.yaml) and fall back to /etc/AI-Hunter/zeek-log-transport.yaml if that fails. This seems the most straightforward to me.
  • We could put sensor-specific config sections in the same /etc/AI-Hunter/zeek-log-transport.yaml file. This would be easier if we were actually parsing structured YAML somehow, but it might be too difficult with the current method of using grep.

request_days=`get_send_days $recv_loc`
logs_str=`echo "${request_logs[*]// /|}"`

cd "$local_tld" || fail "Unable to change to $local_tld"
send_candidates=`find . -type f -mtime -3 -iname '*.gz' | egrep '(conn|dns|http|ssl|x509|known_certs)' | sort -u`
if [ ${#send_candidates} -eq 0 ]; then
echo
printf "WARNING: No logs found, if your log directory is not $local_tld please use the flag: --localdir [bro_zeek_log_directory]"
echo

query="find . -type f -mtime -$request_days -iname '*.gz' | egrep '($logs_str)' | sort -u"
send_candidates="$(eval $query)"
if [ ${#send_candidates} -eq 0 ]; then
#if we don't have anything we assume that we need to check the last 3 days for logs needed for RITA
send_candidates=`find . -type f -mtime -3 -iname '*.gz' | egrep '(conn|dns|http|ssl|x509|known_certs)' | sort -u`
if [ ${#send_candidates} -eq 0 ]; then
echo
printf "WARNING: No logs found, if your log directory is not $local_tld please use the flag: --localdir [bro_zeek_log_directory]"
echo
fi
fi

status "Transferring files to $aih_location"
flock -xn "$HOME/rsync_log_transport.lck" timeout --kill-after=60 7080 $nice_me rsync $rsyncparams -avR -e "ssh $extra_ssh_params" $send_candidates "$aih_location:${remote_top_dir}/" --delay-updates --chmod=Do+rx,Fo+r
retval=$?
Expand Down