tcp-tunnel can’t bind a connection #22

MCApollo · 2020-07-30T21:07:09Z

A few days ago, I wanted to try to make a buildserver out of this project (fork here), so I updated the source to qemu 5.0.0 and apply some PVE/Proxmox patches so it’ll work with my use case; Sadly there’s no serial output, but I’m trying to see if maybe iOS is continuing to run in the background.

I’ve been having some trouble SSH’ing into iOS with tcp-tunnel and see if someone maybe knows a answer; The distro is debian buster and my thoughts are a header conflict between apple-darwin and gnu-linux, help would be appreciated, Thanks!

sickcodes · 2020-08-01T20:45:48Z

Only works on MacOSX as the host because the networking system calls are different on Linux. I have also been trying to get around this.

Secondly, the no-frills way to SSH in is using ash shell.

I'll pull out a docker image so people who know how to fix it, can quickly get in started :)

sickcodes · 2020-08-01T21:36:55Z

I also see 97 which I think was bind error. It works on OSX thru KVM.

You'll also see 255 error if it's starting on boot. @aronsky said it's simply the hard coded system calls in getting a TCP socket in both the QEMU side and the tunnel in the xnu-qemu-arm64 tools folder.

If anyone knows how to change that to Unix ports that would be coolio.

OR

the easiest way to leap over it is possibly emulating the Darwin kernel via qemu which is what I have to do to ssh in, but it's heavy as I'm booting OSX each time. Maybe qiling framework or usercorn can remap those system calls automatically?

sickcodes · 2020-08-01T22:36:03Z

Here's his write up, https://alephsecurity.com/2020/03/29/xnu-qemu-tcp-tunnel/

Would be awesome to get the tunnel working on GNU/Linux!

aronsky · 2020-08-02T07:44:37Z

Hi, @MCApollo, the error in tcp-tunnel happens here: https://github.com/alephsecurity/xnu-qemu-arm64-tools/blob/master/tcp-tunnel/src/main.c#L295. This call to connect is handled by the host (QEMU) here: https://github.com/alephsecurity/xnu-qemu-arm64/blob/master/hw/arm/guest-socket.c#L128. Can you debug your execution of QEMU, set a breakpoint there, and verify that the passed addr variable is reasonable?

MCApollo · 2020-08-02T08:47:58Z

@aronsky https://gist.github.com/MCApollo/9b26e3986634515f6430bb040891b145

I don't really write C code that often, but that looks right; Let me know if you need any other information.

aronsky · 2020-08-02T08:57:28Z

It seems you set the breakpoint on qc_handle_bind, while the failure in your earlier screenshot happens inside qc_handle_connect. Can you run it again, with the breakpoint at qc_handle_connect? But even here, at qc_handle_bind, the value in addr->sin_family doesn't seem correct - 52 is a weird number, it's not AF_INET as it should be (which is 2 under both Linux and macOS)...

MCApollo · 2020-08-02T09:03:57Z

Sorry about that, beat me to the edit. I can't hit the breakpoint in qc_handle_connect, only bind hits.


bash-4.4# tunnel 2222:127.0.0.1:22

Thread 3 "qemu-system-aar" hit Breakpoint 4, qc_handle_bind (
    cpu=0x5555567f5310, sckt=0, g_addr=0x16fdffbe0, addrlen=16)
    at /home/mcapollo/iOS-builder/diff/xnu-qemu-arm64/hw/arm/guest-socket.c:88
88      {
(gdb) c
Continuing.
accept failed: 11

MCApollo · 2020-08-02T09:18:55Z

sys/errno.h:
#define EDEADLK 11 /* Resource deadlock avoided *

and there's no define for 52 in sys/socket.h, spooky.

Edit
I did plug a Ethernet cord into the box, that seemed to changed the behavior; The connection doesn't drop, but no data is being sent.

bash-4.4# /iosbinpack64/usr/local/bin/wget localhost       
--1970-01-01 00:21:36--  http://localhost/
Resolving localhost... ::1, 127.0.0.1
Connecting to localhost|::1|:80... failed: Connection refused.
Connecting to localhost|127.0.0.1|:80... connected.
HTTP request sent, awaiting response... No data received.
Retrying.

I'm starting to think that the mismatch of headers is causing a issue and is the root of the problem:
#define ENOLINK 97 /* Reserved */
#define ENETRESET 52 /* Network dropped connection on reset */

aronsky · 2020-08-02T10:03:49Z

Hm... It's interesting that the connect breakpoint is never hit, since in your previous screenshot it's where the problem happens... Are you executing tcp-tunnel with the same arguments? And there's definitely a good chance that the values mismatch between Linux and macOS is the issue, just trying to identify what exactly is happening.

MCApollo · 2020-08-02T10:11:02Z

Executed the example.com example and tried to connect to localhost, but for the life of me I can’t reproduce the issue in the picture.

Did mess with /etc/interfaces in the last couple days, if that’s some insight.

aronsky · 2020-08-02T10:15:42Z

What are the arguments to tcp-tunnel when you hit the breakpoint in qc_handle_bind? Do you get any errors from the iOS shell once you continue the execution?

MCApollo · 2020-08-02T10:34:40Z

There we go, the wget/example.com combo; connect happens on out, my bad.

--1970-01-01 00:12:43--  http://localhost/
Resolving localhost... ::1, 127.0.0.1
Connecting to localhost|::1|:80... failed: Connection refused.
Connecting to localhost|127.0.0.1|:80... 
Thread 3 "qemu-system-aar" hit Breakpoint 2, qc_handle_connect (cpu=0x5555567f5310, 
    sckt=0, g_addr=0x16fdffb58, addrlen=16)
    at /home/mcapollo/iOS-builder/diff/xnu-qemu-arm64/hw/arm/guest-socket.c:115
115     {
(gdb) s
118         VERIFY_FD(sckt);
(gdb) s
120         int retval = 0;
(gdb) s
122         if (addrlen > sizeof(addr)) {
(gdb) s
125             cpu_memory_rw_debug(cpu, (target_ulong) g_addr, (uint8_t*) &addr,
(gdb) p addr
$2 = {sin_family = 52, sin_port = 0, sin_addr = {s_addr = 0}, 
  sin_zero = "\000\360\337o\001\000\000"}
(gdb) s
cpu_memory_rw_debug (cpu=0x5555567f5310, addr=6171917144, buf=0x7fffafffe2c0 "4", 
    len=16, is_write=0) at /home/mcapollo/iOS-builder/diff/xnu-qemu-arm64/exec.c:3753
3753    {
(gdb) next
3757        cpu_synchronize_state(cpu);

Arguments for qc_handle_bind where the ssh example on xnu-tools page and qc_handle_connect is the example.com example.

No errors for the wget example, the connection now just doesn’t go through.

sickcodes · 2020-08-02T11:58:19Z

@MCApollo so excited! Can you push to your repo if you got the tunnel working on Linux? I pulled from your repo last night, I'll push my changes including Dockerfile.

Were you able to resize the disk on Linux?

I can mount and edit, just can't resize

aronsky · 2020-08-02T12:05:43Z

@MCApollo okay, looking good. However, the value of addr you're printing is actually before cpu_memory_rw_debug is called (and, therefore, is not initialized yet). Use n instead of s to step over function calls, and print (p addr) the value of addr after the call to cpu_memory_rw_debug, when the debugger is stopped at line 128.

sickcodes · 2020-08-02T12:12:48Z

@aronsky @MCApollo shouldn't internet be disabled until the device is unlocked or has a SIM card or fake WiFi network since we are booting into regular iOS?

aronsky · 2020-08-02T12:27:02Z

I don't think so - never ran into this issue before (and we never added a SIM card or a WiFi network to our machine).

MCApollo · 2020-08-02T18:09:12Z

@aronsky

122         if (addrlen > sizeof(addr)) {
(gdb) s
125             cpu_memory_rw_debug(cpu, (target_ulong) g_addr, (uint8_t*) &addr,
(gdb) n
128             if ((retval = connect(guest_svcs_fds[sckt], (struct sockaddr *) &addr,
(gdb) n
132                 cpu_memory_rw_debug(cpu, (target_ulong) g_addr, (uint8_t*) &addr,
(gdb) n
137         return retval;
(gdb) p addr
$5 = {sin_family = 2, sin_port = 20480, sin_addr = {s_addr = 584628317}, 
  sin_zero = "\002\b\000\000\000\000\000"}
(gdb) p retval
$6 = 0

_debug in the name and slipped my mind, now we see AF_INET.

sickcodes · 2020-08-02T20:49:46Z

@MCApollo @aronsky could be that we've disabled SLIRP?

Also regarding editing dyld cache so I can connect the debugger, is it normal that it takes a very long time to open?

Thirdly, I noticed that the 2 disks mount properly in MacOSX but don't mount both in Linux, I'll get a screenshot. That could be the issue regarding permissions.

aronsky · 2020-08-03T09:48:23Z

@MCApollo - interesting. So the call to connect succeeds. Yet in the shell, you get connect failed: 97?

MCApollo · 2020-08-03T12:37:53Z

@aronsky Let's recap then

Binding

tunnel 2222:127.0.0.1:22 &
[1] 95
bash-4.4# 
Thread 3 "qemu-system-aar" hit Breakpoint 2, qc_handle_bind (cpu=0x5555567f5310, 
    sckt=0, g_addr=0x16fdffbe0, addrlen=16)
    at /home/mcapollo/iOS-builder/diff/xnu-qemu-arm64/hw/arm/guest-socket.c:88
88      {
(gdb) s
91          VERIFY_FD(sckt);
(gdb) s
93          int retval = 0;
(gdb) s
95          if (addrlen > sizeof(addr)) {
(gdb) s
98              cpu_memory_rw_debug(cpu, (target_ulong) g_addr, (uint8_t*) &addr,
(gdb) n
101             if ((retval = bind(guest_svcs_fds[sckt], (struct sockaddr *) &addr,
(gdb) n
105                 cpu_memory_rw_debug(cpu, (target_ulong) g_addr, (uint8_t*) &addr,
(gdb) n
110         return retval;
(gdb) p addr
$1 = {sin_family = 2, sin_port = 44552, sin_addr = {s_addr = 0}, 
  sin_zero = "\000\000\000\000\000\004\000"}
(gdb) p retval
$2 = 0
(gdb) continue
Continuing.
accept failed: 11

Connecting

bash-4.4# tunnel out:80:93.184.216.34:80
/iosbinpack64/usr/local/bin/wget --tries=1 localhost &>/dev/null
[Switching to Thread 0x7fffaffff700 (LWP 15424)]

Thread 3 "qemu-system-aar" hit Breakpoint 1, qc_handle_connect (cpu=0x5555567f5310, 
    sckt=0, g_addr=0x16fdffb58, addrlen=16)
    at /home/mcapollo/iOS-builder/diff/xnu-qemu-arm64/hw/arm/guest-socket.c:115
115     {
(gdb) s
118         VERIFY_FD(sckt);
(gdb) s
120         int retval = 0;
(gdb) s
122         if (addrlen > sizeof(addr)) {
(gdb) s
125             cpu_memory_rw_debug(cpu, (target_ulong) g_addr, (uint8_t*) &addr,
(gdb) n
128             if ((retval = connect(guest_svcs_fds[sckt], (struct sockaddr *) &addr,
(gdb) n
132                 cpu_memory_rw_debug(cpu, (target_ulong) g_addr, (uint8_t*) &addr,
(gdb) n
137         return retval;
(gdb) p addr
$1 = {sin_family = 2, sin_port = 20480, sin_addr = {s_addr = 584628317}, 
  sin_zero = "\002\b\000\000\000\000\000"}
(gdb) p retval
$2 = 0
(gdb) c
Continuing.
bash-4.4#

No error, but no data throughput
I've tried a google server and 93.184.216.34 (example.com)- just in case that's a old IP for example.com, but:

iosbinpack64/usr/local/bin/wget localhost
--1970-01-01 00:09:19--  http://localhost/
Resolving localhost... ::1, 127.0.0.1
Connecting to localhost|::1|:80... failed: Connection refused.
Connecting to localhost|127.0.0.1|:80... failed: Connection refused.

Again, did plug a Ethernet cord into the box, was using a usb-wifi dongle before but the box could connect to the internet fine.
I'm going to assume the error in the picture was related to that, but still no connection to the outside world from iOS.

Bit a pain to test anything, launchd yells about issues and I haven't had the time to pull a different bootstrap off some recent jailbreak. Thanks for looking into this.

aronsky · 2020-08-03T13:39:30Z

Hm... So the ports seem wrong in both cases. Strange... I wonder what could cause that?

Edit: scratch that. It's network endianness. Not sure what could be wrong here...

MCApollo · 2020-08-03T18:43:05Z

Thread 3 "qemu-system-aar" hit Breakpoint 1, qc_handle_bind (
    cpu=0x5555567f5310, sckt=0, g_addr=0x16fdffbe0, addrlen=16)
    at /home/mcapollo/iOS-builder/diff/xnu-qemu-arm64/hw/arm/guest-socket.c:88
88������{
(gdb) s
91������    VERIFY_FD(sckt);
(gdb) n
93������    int retval = 0;
(gdb) n
95������    if (addrlen > sizeof(addr)) {
(gdb) n
98������        cpu_memory_rw_debug(cpu, (target_ulong) g_addr, (uint8_t*) &addr,
(gdb) n
101�����        addr.sin_port = ntohs(addr.sin_port);
(gdb) n
103�����        if ((retval = bind(guest_svcs_fds[sckt], (struct sockaddr *) &addr,
(gdb) p addr
$1 = {sin_family = 2, sin_port = 2222, sin_addr = {s_addr = 0}, 
  sin_zero = "\000\000\000\000\000\004\000"}
(gdb) c

ah, you’re right, congrats 🎉

MCApollo · 2020-08-04T06:49:02Z

MCApollo/xnu-qemu-arm64-tools@bb284ed

With that edit and botch-recreating addr, I can get a connection going, but the port number is wrong (big endian of input port) & qemu freezes waiting for a connection; Not sure what that’s about or related to the interrupt issue/feature.

I’ll continue to look into the wrong port number and hang when waiting for a connection.

sickcodes · 2020-08-04T12:17:24Z

I only see one mounted file system when running on Linux.

On OSX, the dirty RAM disk mounts as well. Only 1 mounted fs on Linux

I have to attach reattach hfs.main to be able to boot in, on OSX.

What about QEMU nesting the OSX qemu binary? That's essentially how I boot in now

sickcodes · 2020-08-04T12:38:20Z

Also @MCApollo Thanks for bringing the QEMU version up. I can't build it without the proxmox backup header file, how'd you get around that?

I've tried adding -disable-... proxmox backup,

I tried removing every reference in pbs.c and the makefile, how'd you do it?

MCApollo · 2020-08-05T05:47:23Z

@aronsky Figured it out, darwin still uses sin_len, which is where that 52 came from.

https://github.com/theos/sdks/blob/a2dd04acbc031f523c8b21d25f5e30494ed5735f/iPhoneOS11.2.sdk/usr/include/netinet/in.h#L376

struct sockaddr_in {
	__uint8_t	sin_len;
	sa_family_t	sin_family;
	in_port_t	sin_port;
	struct	in_addr sin_addr;
	char		sin_zero[8];
};

Using -isystem to CFLAGS, redefining stuff could work to match Darwin/iOS expected addr, even if it's silly.

qc_handle_bind:
$15 = {sin_len = 49 '1', sin_family = 2 '\002', sin_port = 44552, sin_addr = {
    s_addr = 0}, sin_zero = "\000\000\000\000\000\000\000"}
(gdb) p ((struct sockaddr *) &addr).sa_family 
$16 = 44552
(gdb) p addr
$17 = {sin_len = 49 '1', sin_family = 2 '\002', sin_port = 44552, sin_addr = {
    s_addr = 0}, sin_zero = "\000\000\000\000\000\000\000"}

sickcodes · 2020-08-05T20:11:57Z

I'm gonna try running the macOS qemu using https://github.com/darlinghq/darling

MCApollo · 2020-08-06T06:06:44Z

Some redefining and small edits, finally got SSH working.

I’ll work out a PR tomorrow, code’s a bit messy.

aronsky · 2020-08-06T06:11:42Z

Awesome!

sickcodes · 2020-08-06T10:41:35Z

So damn excited 😝😋

sickcodes · 2020-08-06T15:09:00Z

@MCApollo Feel free to post notes, I have time to help with the PR

MCApollo mentioned this issue Aug 6, 2020

Added Linux support to qc_handle_{accept,bind,connect} functions. #23

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

tcp-tunnel can’t bind a connection #22

tcp-tunnel can’t bind a connection #22

MCApollo commented Jul 30, 2020

sickcodes commented Aug 1, 2020

sickcodes commented Aug 1, 2020

sickcodes commented Aug 1, 2020

aronsky commented Aug 2, 2020

MCApollo commented Aug 2, 2020

aronsky commented Aug 2, 2020

MCApollo commented Aug 2, 2020 •

edited

Loading

MCApollo commented Aug 2, 2020 •

edited

Loading

aronsky commented Aug 2, 2020

MCApollo commented Aug 2, 2020

aronsky commented Aug 2, 2020

MCApollo commented Aug 2, 2020 •

edited

Loading

sickcodes commented Aug 2, 2020

aronsky commented Aug 2, 2020

sickcodes commented Aug 2, 2020

aronsky commented Aug 2, 2020

MCApollo commented Aug 2, 2020

sickcodes commented Aug 2, 2020

aronsky commented Aug 3, 2020

MCApollo commented Aug 3, 2020

aronsky commented Aug 3, 2020 •

edited

Loading

MCApollo commented Aug 3, 2020 •

edited

Loading

MCApollo commented Aug 4, 2020 •

edited

Loading

sickcodes commented Aug 4, 2020

sickcodes commented Aug 4, 2020

MCApollo commented Aug 5, 2020

sickcodes commented Aug 5, 2020

MCApollo commented Aug 6, 2020

aronsky commented Aug 6, 2020

sickcodes commented Aug 6, 2020

sickcodes commented Aug 6, 2020

tcp-tunnel can’t bind a connection #22

tcp-tunnel can’t bind a connection #22

Comments

MCApollo commented Jul 30, 2020

sickcodes commented Aug 1, 2020

sickcodes commented Aug 1, 2020

sickcodes commented Aug 1, 2020

aronsky commented Aug 2, 2020

MCApollo commented Aug 2, 2020

aronsky commented Aug 2, 2020

MCApollo commented Aug 2, 2020 • edited Loading

MCApollo commented Aug 2, 2020 • edited Loading

aronsky commented Aug 2, 2020

MCApollo commented Aug 2, 2020

aronsky commented Aug 2, 2020

MCApollo commented Aug 2, 2020 • edited Loading

sickcodes commented Aug 2, 2020

aronsky commented Aug 2, 2020

sickcodes commented Aug 2, 2020

aronsky commented Aug 2, 2020

MCApollo commented Aug 2, 2020

sickcodes commented Aug 2, 2020

aronsky commented Aug 3, 2020

MCApollo commented Aug 3, 2020

aronsky commented Aug 3, 2020 • edited Loading

MCApollo commented Aug 3, 2020 • edited Loading

MCApollo commented Aug 4, 2020 • edited Loading

sickcodes commented Aug 4, 2020

sickcodes commented Aug 4, 2020

MCApollo commented Aug 5, 2020

sickcodes commented Aug 5, 2020

MCApollo commented Aug 6, 2020

aronsky commented Aug 6, 2020

sickcodes commented Aug 6, 2020

sickcodes commented Aug 6, 2020

MCApollo commented Aug 2, 2020 •

edited

Loading

MCApollo commented Aug 2, 2020 •

edited

Loading

MCApollo commented Aug 2, 2020 •

edited

Loading

aronsky commented Aug 3, 2020 •

edited

Loading

MCApollo commented Aug 3, 2020 •

edited

Loading

MCApollo commented Aug 4, 2020 •

edited

Loading