Merging to falco-master for 0.5.0 (#702)

* K8s fixes + max & windows build (#666)

* mac build (not tested)

* linux build and run

* Done
- add blocking connect/init mode to k8s
- sysdig connect and init are blocking now (faster startup)
- move k8s http to 1.1 to utilize keepalive
- fixed chunk purging bug
- reuse state socket for watch (no diconnect after state fetch)
Todo
- improve handler receive error handling
- test https
- blocking resolve

* watch redirection fix

* fix watch transition; detect http 1.1 watch emission end and reconnect promptly; fix jq filter order bug

* fix mac build

* fix linux compile error; add docker flag to handler

* windows build

* fix race condition when no data on first attempt; make k8s default http 1.1

* fix blocking read

* Add less to docker image

* Added s390x support to sysdig source (#667)

* Update ppm.h

Added support for s390x

* Update ppm.h

re committing changes related to s390x

* Revert "Added s390x support to sysdig source (#667)"

This reverts commit bf7ae5a.

* Added s390 support to sysdig source (#671)

* Update ppm.h

Added support for s390x

* Update ppm.h

re committing changes related to s390x

* build: Fix openssl build when not using the bundled library. (#672)

Otherwise, with
cmake -DCMAKE_BUILD_TYPE=Debug  -DUSE_BUNDLED_OPENSSL=OFF ..
one gets

```
[ 96%] Linking CXX executable csysdig
[ 97%] Linking CXX executable sysdig
/usr/bin/ld: ../libsinsp/libsinsp.a(k8s_handler.cpp.o): undefined reference to symbol 'SSL_CTX_use_PrivateKey_file'
/usr/lib/libssl.so.1.0.0: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status
make[2]: *** [userspace/sysdig/CMakeFiles/csysdig.dir/build.make:131: userspace/sysdig/csysdig] Error 1
make[1]: *** [CMakeFiles/Makefile2:275: userspace/sysdig/CMakeFiles/csysdig.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
/usr/bin/ld: ../libsinsp/libsinsp.a(k8s_handler.cpp.o): undefined reference to symbol 'SSL_CTX_use_PrivateKey_file'
/usr/lib/libssl.so.1.0.0: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status
make[2]: *** [userspace/sysdig/CMakeFiles/sysdig.dir/build.make:129: userspace/sysdig/sysdig] Error 1
make[1]: *** [CMakeFiles/Makefile2:323: userspace/sysdig/CMakeFiles/sysdig.dir/all] Error 2
make: *** [Makefile:139: all] Error 2
```

This is because linking to libssl and libcrypto is not done (after
find_package in CMakeLists.txt) when using
the system libraries.

Also, fix the curl ssl dependency.

sysdig-CLA-1.0-signed-off-by: Raghavendra Prabhu <me@rdprabhu.com>

* - keep single k8s socket opened throughout session
- fix transition from non-chunked to chunked k8s handler
- change active k8s handler filter from reference to pointer
- remove unnecessary socket handler docker flag
- early terminate k8s blocking request loop on JSON end detect
- increase k8s blocking loop sleep to 10ms
- fix some logs

* missing deployments fix

* fix, part II; Missing k8s metadata #251

* make iolen signed

* fix code formatting

* fix (un)signed warnings; lower k8s handler creation log severity to debug

* replace http parser

* Changes for s390 , removed unsupported syscalls (#676)

added checks for unused macros on s390x
sysdig-CLA-1.0-signed-off-by: Ketan Kunde ketan22584@gmail.com

* Parse Conf from Docker

* Detect/remove stale threadinfo in clone children

When parsing clone exit events, specifically for the child half of a
clone and when in a container, detect and potentially remove stale
threadinfo state for the child thread.

Generally the client have of a clone is responsible for creating the
thread state for the new thread, as long as the parent is in a
container. See the parent half of the "if(childtid == 0)" statement. We
simply need to verify in the child half that the parent really was in a
container.

You can find the parent thread id from the syscall return information,
which is moved up from below. Look up the parent thread and see if its
vtid/vpid differs from tid/pid. If so, any existing thread state must be
stale and remove it. Note that you can't use
evt->m_tinfo->get_parent_thread() directly, as that comes from the
existing potentially stale threadinfo.

This fixes #664.

* Remove cwd parsing from the driver because the function became sleepable in 4.8 (torvalds/linux@47be618).

When forking a new process, inherit the cwd from the parent.

* Use main_thread for set_cwd/get_cwd

* Mesos token auth (#673)

Support DC/OS token auth and HTTPS on Mesos

* Remove spurious code

* Probe builder with timeout (#683)

* add timeout to urlopen operations

* add timeout to download operations

* retry download max 10 times

* exlude 4.9 from ubuntu repos (#685)

* add msg end handler

* sysdig with https k8s-api failed #687

* windows compile errors

* return an exception when a filter only fiels is used for display

* bugfix: evtin.span.*.tags filter fields were not working properly

* evtin* fields can also be used as display fields now

* minor cleanup

* add stopwatch utility

* Fix compilation issues with kernel 4.9 (#684)

* Fix compilation issues with kernel 4.9

related commits:
torvalds/linux@4c737b4
torvalds/linux@b9d989c

* map io cgroup to blkio, fix for kernels >= 4.8

* Fix tracer code errors

* Fix ipv4 mapped ipv6 when used on sendto and receiver endpoint is 0

* Use https for all downloads.

Use https instead of http for all downloads within the install
script. In cases where the links refer to artifacts in our s3 bucket,
switch to https + s3.amazon.aws.com, which is already used by other urls
in the script.

This fixes falcosecurity/falco#152.

* Fix format memory leak (#694)

* Whitespace diffs.

Committing separate from other changes.

* Fix leak when fmt string ends with non-filtercheck

Make sure that any final rawstring_check added to the list of tokens is
also added to m_chks_to_free, so it is properly freed.

This fixes #693.

* Clean up utils header file to be self-contained (#696)

Currently, utils.h has a lot of implicit dependencies on other stl
header files as well as assuming the std namespace is available.

Clean it up so it can be included on its own (say, in
falcosecurity/falco#162).

* Fix typo in csysdig threads view

* a bit of work on the flame chisel

* support reading merged files

* wrong return value

* throttle k8s (#699)

* throttle max bytes per socket/cycle to 512k, max msgs for critical k8s entities to 100

* ifdef k8s caching

* adjust some commented (TBD) code

* fix the message limit logic

* osx build

* windows build, remove some warnings

* Revert "exlude 4.9 from ubuntu repos (#685)"

This reverts commit c183a57.

* Reset marathon group json together with marathon app one (#700)

* Reset marathon group json together with marathon app one

* Remove spurious app_it declaration

CMakeLists.txt

@@ -324,13 +324,13 @@ if(NOT WIN32 AND NOT APPLE)
 		set(CURL_LIBRARIES "${CURL_BUNDLE_DIR}/lib/.libs/libcurl.a")
 
 		if(NOT USE_BUNDLED_OPENSSL)
-			set(CURL_SSL_OPTION "")
+			set(CURL_SSL_OPTION "--with-ssl")
 		else()
 			set(CURL_SSL_OPTION "--with-ssl=${OPENSSL_INSTALL_DIR}")
+                        message(STATUS "Using bundled curl in '${CURL_BUNDLE_DIR}'")
+                        message(STATUS "Using SSL for curl in '${CURL_SSL_OPTION}'")
 		endif()
 
-		message(STATUS "Using bundled curl in '${CURL_BUNDLE_DIR}'")
-		message(STATUS "Using SSL for curl in '${CURL_SSL_OPTION}'")
 
 		ExternalProject_Add(curl
 			DEPENDS openssl

docker/dev/Dockerfile

@@ -24,7 +24,7 @@ RUN echo "deb http://httpredir.debian.org/debian jessie main" > /etc/apt/sources
 	gcc \
 	gcc-5 \
 	gcc-4.9 \
-	libelf1 && rm -rf /var/lib/apt/lists/*
+	libelf1 less && rm -rf /var/lib/apt/lists/*
 
 # Terribly terrible hacks: since our base Debian image ships with GCC 5.0 which breaks older kernels,
 # revert the default to gcc-4.9. Also, since some customers use some very old distributions whose kernel

docker/stable/Dockerfile

@@ -24,7 +24,7 @@ RUN echo "deb http://httpredir.debian.org/debian jessie main" > /etc/apt/sources
 	gcc \
 	gcc-5 \
 	gcc-4.9 \
-	libelf1 && rm -rf /var/lib/apt/lists/*
+	libelf1 less && rm -rf /var/lib/apt/lists/*
 
 # Terribly terrible hacks: since our base Debian image ships with GCC 5.0 which breaks older kernels,
 # revert the default to gcc-4.9. Also, since some customers use some very old distributions whose kernel

driver/main.c

@@ -1627,7 +1627,11 @@ TRACEPOINT_PROBE(syscall_enter_probe, struct pt_regs *regs, long id)
 	 * If this is a 32bit process running on a 64bit kernel (see the CONFIG_IA32_EMULATION
 	 * kernel flag), we switch to the ia32 syscall table.
 	 */
+#if LINUX_VERSION_CODE >= KERNEL_VERSION(4, 9, 0)
+	if(in_ia32_syscall()) {
+#else
 	if (unlikely(task_thread_info(current)->status & TS_COMPAT)) {
+#endif
 		cur_g_syscall_table = g_syscall_ia32_table;
 		cur_g_syscall_code_routing_table = g_syscall_ia32_code_routing_table;
 		socketcall_syscall = __NR_ia32_socketcall;
@@ -1689,7 +1693,11 @@ TRACEPOINT_PROBE(syscall_exit_probe, struct pt_regs *regs, long ret)
 	 * use 64bit syscall table. On 32bit __NR_execve is equal to __NR_ia32_oldolduname
 	 * which is a very old syscall, not used anymore by most applications
 	 */
+#if LINUX_VERSION_CODE >= KERNEL_VERSION(4, 9, 0)
+	if(in_ia32_syscall() && id != __NR_execve) {
+#else
 	if (unlikely((task_thread_info(current)->status & TS_COMPAT) && id != __NR_execve)) {
+#endif
 		cur_g_syscall_table = g_syscall_ia32_table;
 		cur_g_syscall_code_routing_table = g_syscall_ia32_code_routing_table;
 		socketcall_syscall = __NR_ia32_socketcall;

driver/ppm.h

@@ -122,6 +122,8 @@ long ppm_strncpy_from_user(char *to, const char __user *from, unsigned long n);
   #define SYSCALL_TABLE_ID0 0
 #elif defined CONFIG_PPC64
   #define SYSCALL_TABLE_ID0 0
+#elif defined CONFIG_S390
+  #define SYSCALL_TABLE_ID0 0
 #endif
 
 #define SYSCALL_TABLE_SIZE 512

driver/ppm_events.c

@@ -705,61 +705,6 @@ int val_to_ring(struct event_filler_arguments *args, uint64_t val, u16 val_len,
 	return PPM_SUCCESS;
 }
 
-/*
- * Get the current working directory for the current process.
- * Returns the pointer to the string, which is NOT going to be at the beginning
- * of buf.
- * Buf must be at least 1 page in size.
- */
-#if LINUX_VERSION_CODE > KERNEL_VERSION(2, 6, 20)
-char *npm_getcwd(char *buf, unsigned long bufsize)
-{
-	struct path pwd;
-	char *res;
-
-	ASSERT(bufsize >= PAGE_SIZE - 1);
-
-#if LINUX_VERSION_CODE >= KERNEL_VERSION(2, 6, 36) || defined CONFIG_VE
-	get_fs_pwd(current->fs, &pwd);
-#else
-	read_lock(&current->fs->lock);
-	pwd = current->fs->pwd;
-	path_get(&pwd);
-	read_unlock(&current->fs->lock);
-#endif
-
-	res = d_path(&pwd, buf, bufsize);
-
-	if (IS_ERR(res))
-		res = NULL;
-
-	path_put(&pwd);
-
-	return res;
-}
-#else /* LINUX_VERSION_CODE > KERNEL_VERSION(2, 6, 20) */
-char *npm_getcwd(char *buf, unsigned long bufsize)
-{
-	struct dentry *dentry;
-	struct vfsmount *mnt;
-	char *res;
-
-	ASSERT(bufsize >= PAGE_SIZE - 1);
-
-	read_lock(&current->fs->lock);
-	mnt = mntget(current->fs->pwdmnt);
-	dentry = dget(current->fs->pwd);
-	read_unlock(&current->fs->lock);
-
-	res = d_path(dentry, mnt, buf, bufsize);
-
-	if (IS_ERR(res))
-		res = NULL;
-
-	return res;
-}
-#endif
-
 static inline u8 socket_family_to_scap(u8 family)
 {
 	if (family == AF_INET)

driver/ppm_events.h

@@ -129,7 +129,6 @@ extern const struct ppm_event_entry g_ppm_events[];
 int32_t dpi_lookahead_init(void);
 int32_t f_sys_autofill(struct event_filler_arguments *args, const struct ppm_event_entry *evinfo);
 int32_t val_to_ring(struct event_filler_arguments *args, u64 val, u16 val_len, bool fromuser, u8 dyn_idx);
-char *npm_getcwd(char *buf, unsigned long bufsize);
 u16 pack_addr(struct sockaddr *usrsockaddr, int ulen, char *targetbuf, u16 targetbufsize);
 u16 fd_to_socktuple(int fd, struct sockaddr *usrsockaddr, int ulen, bool use_userdata, bool is_inbound, char *targetbuf, u16 targetbufsize);
 int addr_to_kernel(void __user *uaddr, int ulen, struct sockaddr *kaddr);

driver/ppm_fillers.c

@@ -812,7 +812,7 @@ static int append_cgroup(const char *subsys_name, int subsys_id, char *buf, int
 	int subsys_len;
 	char *path;
 
-#if LINUX_VERSION_CODE < KERNEL_VERSION(3, 15, 0)
+#if LINUX_VERSION_CODE < KERNEL_VERSION(3, 15, 0) || LINUX_VERSION_CODE >= KERNEL_VERSION(4, 9, 0)
 	int res;
 #endif
 
@@ -832,7 +832,17 @@ static int append_cgroup(const char *subsys_name, int subsys_id, char *buf, int
 		return 1;
 	}
 
-#if LINUX_VERSION_CODE >= KERNEL_VERSION(3, 15, 0)
+#if LINUX_VERSION_CODE >= KERNEL_VERSION(4, 9, 0)
+	// According to https://github.com/torvalds/linux/commit/4c737b41de7f4eef2a593803bad1b918dd718b10
+	// cgroup_path now returns an int again
+	res = cgroup_path(css->cgroup, buf, *available);
+	if (res < 0) {
+		ASSERT(false);
+		path = "NA";
+	} else {
+		path = buf;
+	}
+#elif LINUX_VERSION_CODE >= KERNEL_VERSION(3, 15, 0)
 	path = cgroup_path(css->cgroup, buf, *available);
 	if (!path) {
 		ASSERT(false);
@@ -1006,7 +1016,7 @@ static int f_proc_startupdate(struct event_filler_arguments *args)
 	struct mm_struct *mm = current->mm;
 	int64_t retval;
 	int ptid;
-	char *spwd;
+	char *spwd = "";
 	long total_vm = 0;
 	long total_rss = 0;
 	long swap = 0;
@@ -1149,14 +1159,9 @@ static int f_proc_startupdate(struct event_filler_arguments *args)
 		return res;
 
 	/*
-	 * cwd
+	 * cwd, pushed empty to avoid breaking compatibility
+	 * with the older event format
 	 */
-	spwd = npm_getcwd(args->str_storage, STR_STORAGE_SIZE - 1);
-	if (spwd == NULL)
-		spwd = "";
-
-	args->str_storage[STR_STORAGE_SIZE - 1] = '\0';
-
 	res = val_to_ring(args, (uint64_t)(long)spwd, 0, false, 0);
 	if (unlikely(res != PPM_SUCCESS))
 		return res;
@@ -3890,14 +3895,22 @@ static inline u16 ptrace_requests_to_scap(unsigned long req)
 	case PTRACE_GETFPXREGS:
 		return PPM_PTRACE_GETFPXREGS;
 #endif
+#ifdef PTRACE_SETFPREGS
 	case PTRACE_SETFPREGS:
 		return PPM_PTRACE_SETFPREGS;
+#endif
+#ifdef PTRACE_GETFPREGS
 	case PTRACE_GETFPREGS:
 		return PPM_PTRACE_GETFPREGS;
+#endif
+#ifdef PTRACE_SETREGS
 	case PTRACE_SETREGS:
 		return PPM_PTRACE_SETREGS;
+#endif
+#ifdef PTRACE_GETREGS
 	case PTRACE_GETREGS:
 		return PPM_PTRACE_GETREGS;
+#endif
 #ifdef PTRACE_SETSIGMASK
 	case PTRACE_SETSIGMASK:
 		return PPM_PTRACE_SETSIGMASK;

driver/syscall_table.c

@@ -288,7 +288,9 @@ const enum ppm_syscall_code g_syscall_code_routing_table[SYSCALL_TABLE_SIZE] = {
 	[__NR_link - SYSCALL_TABLE_ID0] = PPM_SC_LINK,
 	[__NR_unlink - SYSCALL_TABLE_ID0] = PPM_SC_UNLINK,
 	[__NR_chdir - SYSCALL_TABLE_ID0] = PPM_SC_CHDIR,
+#ifdef __NR_time
 	[__NR_time - SYSCALL_TABLE_ID0] = PPM_SC_TIME,
+#endif
 	[__NR_mknod - SYSCALL_TABLE_ID0] = PPM_SC_MKNOD,
 	[__NR_chmod - SYSCALL_TABLE_ID0] = PPM_SC_CHMOD,
 /* [__NR_lchown16 - SYSCALL_TABLE_ID0] = PPM_SC_NR_LCHOWN16, */

scripts/build-probe-binaries

@@ -17,6 +17,8 @@ PROBE_VERSION=$2
 REPOSITORY_NAME=$3
 BASEDIR=$(pwd)
 ARCH=$(uname -m)
+URL_TIMEOUT=300
+RETRY=10
 
 if [ ! -d $BASEDIR/output ]; then
 	mkdir $BASEDIR/output
@@ -140,7 +142,7 @@ function coreos_build_old {
 	cd $COREOS_DIR
 
 	if [ ! -f config_orig ]; then
-		wget ${VERSION_URL}coreos_developer_container.bin.bz2
+		wget --timeout=${URL_TIMEOUT} --tries=${RETRY} ${VERSION_URL}coreos_developer_container.bin.bz2
 		bunzip2 coreos_developer_container.bin.bz2
 		sudo kpartx -asv coreos_developer_container.bin
 		LOOPDEV=$(sudo kpartx -asv coreos_developer_container.bin | cut -d\  -f 3)
@@ -162,7 +164,7 @@ function coreos_build_old {
 	KERNEL_URL=https://www.kernel.org/pub/linux/kernel/v${MAJOR}.x/$TGZ_NAME
 
 	if [ ! -f $TGZ_NAME ]; then
-		wget $KERNEL_URL
+		wget --timeout=${URL_TIMEOUT} --tries=${RETRY} $KERNEL_URL
 	fi
 
 	if [ ! -d $DIR_NAME ]; then
@@ -197,7 +199,7 @@ function coreos_build_new {
 	cd $COREOS_DIR
 
 	if [ ! -f coreos_developer_container.bin ]; then
-		wget ${VERSION_URL}coreos_developer_container.bin.bz2
+		wget --timeout=${URL_TIMEOUT} --tries=${RETRY} ${VERSION_URL}coreos_developer_container.bin.bz2
 		bunzip2 coreos_developer_container.bin.bz2
 	fi
 	sudo kpartx -asv coreos_developer_container.bin
@@ -245,7 +247,7 @@ function boot2docker_build {
 
 	if [ ! -f $TGZ_NAME ]; then
 		echo Downloading $TGZ_NAME [Boot2Docker]
-		wget $KERNEL_URL
+		wget --timeout=${URL_TIMEOUT} --tries=${RETRY} $KERNEL_URL
 	fi
 
 	if [ ! -d $DIR_NAME ]; then
@@ -268,7 +270,7 @@ function boot2docker_build {
 		; do \
 	        patch -p1 < "$patch"; \
 		done
-		wget -O .config $KERNEL_CONFIG
+		wget --timeout=${URL_TIMEOUT} --tries=${RETRY} -O .config $KERNEL_CONFIG
 		cp .config ../config-orig
 		make olddefconfig
 		make modules_prepare
@@ -307,7 +309,7 @@ function ubuntu_build {
 
 	if [ ! -f $DEB ]; then
 		echo Downloading $DEB [Ubuntu]
-		wget $URL
+		wget --timeout=${URL_TIMEOUT} --tries=${RETRY} $URL
 		dpkg -x $DEB ./
 	fi
 
@@ -349,7 +351,7 @@ function rhel_build {
 
 	if [ ! -f $RPM ]; then
 		echo Downloading $RPM [RHEL and CentOS]
-		wget $URL
+		wget --timeout=${URL_TIMEOUT} --tries=${RETRY} $URL
 		rpm2cpio $RPM | cpio -idm
 	fi
 
@@ -387,7 +389,7 @@ function debian_build {
 		fi
 		if [ ! -f ${BASEDIR}/common-dependencies/debian/kbuild/${DEB} ]; then
 			echo Downloading ${DEB} [Ubuntu]
-			wget -P ${BASEDIR}/common-dependencies/debian/kbuild ${URL}
+			wget --timeout=${URL_TIMEOUT} --tries=${RETRY} -P ${BASEDIR}/common-dependencies/debian/kbuild ${URL}
 		fi
 		return
 	else
@@ -409,7 +411,7 @@ function debian_build {
 
 		if [ ! -f ${DEB} ]; then
 			echo Downloading ${DEB} [Ubuntu]
-			wget ${URL}
+			wget --timeout=${URL_TIMEOUT} --tries=${RETRY} ${URL}
 			dpkg -x ${DEB} ./
 		fi
 	fi

scripts/install-sysdig.in

@@ -27,18 +27,18 @@ function install_rpm {
 	if ! yum -q list dkms > /dev/null 2>&1; then
 		echo "* Installing EPEL repository (for DKMS)"
 		if [ $VERSION -eq 7 ] && [ $DISTRO = "centos" ]; then
-			rpm --quiet -i http://mirrors.kernel.org/centos/7/extras/x86_64/Packages/epel-release-7-5.noarch.rpm
+			rpm --quiet -i https://mirrors.kernel.org/centos/7/extras/x86_64/Packages/epel-release-7-5.noarch.rpm
 		elif [ $VERSION -eq 7 ]; then
-			rpm --quiet -i http://mirrors.kernel.org/fedora-epel/7/x86_64/e/epel-release-7-8.noarch.rpm
+			rpm --quiet -i https://mirrors.kernel.org/fedora-epel/7/x86_64/e/epel-release-7-8.noarch.rpm
 		else
-			rpm --quiet -i http://mirrors.kernel.org/fedora-epel/6/i386/epel-release-6-8.noarch.rpm
+			rpm --quiet -i https://mirrors.kernel.org/fedora-epel/6/i386/epel-release-6-8.noarch.rpm
 		fi
 	fi
 
 	echo "* Installing _COMPONENT_ public key"
 	rpm --quiet --import https://s3.amazonaws.com/download.draios.com/DRAIOS-GPG-KEY.public
 	echo "* Installing _COMPONENT_ repository"
-	curl -s -o /etc/yum.repos.d/draios.repo http://download.draios.com/_REPOSITORY_NAME_/rpm/draios.repo
+	curl -s -o /etc/yum.repos.d/draios.repo https://s3.amazonaws.com/download.draios.com/_REPOSITORY_NAME_/rpm/draios.repo
 	echo "* Installing kernel headers"
 	KERNEL_VERSION=$(uname -r)
 	if [[ $KERNEL_VERSION == *PAE* ]]; then
@@ -66,7 +66,7 @@ function install_deb {
 	echo "* Installing Sysdig public key"
 	curl -s https://s3.amazonaws.com/download.draios.com/DRAIOS-GPG-KEY.public | apt-key add -
 	echo "* Installing _COMPONENT_ repository"
-	curl -s -o /etc/apt/sources.list.d/draios.list http://download.draios.com/_REPOSITORY_NAME_/deb/draios.list
+	curl -s -o /etc/apt/sources.list.d/draios.list https://s3.amazonaws.com/download.draios.com/_REPOSITORY_NAME_/deb/draios.list
 	apt-get -qq update < /dev/null
 	echo "* Installing kernel headers"
 	apt-get -qq -y install linux-headers-$(uname -r) < /dev/null || kernel_warning

scripts/kernel-crawler.py

@@ -182,6 +182,7 @@
 # code.
 #
 urls = set()
+URL_TIMEOUT=30
 
 if len(sys.argv) < 2 or not sys.argv[1] in repos:
 	sys.stderr.write("Usage: " + sys.argv[0] + " <distro>\n")
@@ -192,28 +193,21 @@
 # patterns given. Save the result in `packages`.
 #
 for repo in repos[sys.argv[1]]:
-
-	root = urllib2.urlopen(repo["root"]).read()
-	versions = html.fromstring(root).xpath(repo["discovery_pattern"], namespaces = {"regex": "http://exslt.org/regular-expressions"})
-
-	for version in versions:
-		for subdir in repo["subdirs"]:
-
-			# The try - except block is used because 404 errors and similar
-			# might happen (and actually happen because not all repos have
-			# packages we need)
-			try:
-				source = repo["root"] + version + subdir
-				page = urllib2.urlopen(source).read()
-				rpms = html.fromstring(page).xpath(repo["page_pattern"], namespaces = {"regex": "http://exslt.org/regular-expressions"})
-
-				for rpm in rpms:
-					if "exclude_patterns" in repo and any(x in rpm for x in repo["exclude_patterns"]):
-						continue
-					else:
-						urls.add(source + str(urllib2.unquote(rpm)))
-			except:
-				continue
+    try:
+        root = urllib2.urlopen(repo["root"],timeout=URL_TIMEOUT).read()
+        versions = html.fromstring(root).xpath(repo["discovery_pattern"], namespaces = {"regex": "http://exslt.org/regular-expressions"})
+        for version in versions:
+            for subdir in repo["subdirs"]:
+                source = repo["root"] + version + subdir
+                page = urllib2.urlopen(source,timeout=URL_TIMEOUT).read()
+                rpms = html.fromstring(page).xpath(repo["page_pattern"], namespaces = {"regex": "http://exslt.org/regular-expressions"})
+                for rpm in rpms:
+                    if "exclude_patterns" in repo and any(x in rpm for x in repo["exclude_patterns"]):
+                        continue
+                    else:
+                        urls.add(source + str(urllib2.unquote(rpm)))
+    except:
+        continue
 
 #
 # Print URLs to stdout