Latest Experiences in Linux

While the Python web service running on Werkzeug has been performing fantastically for development purposes someone has managed to crash the server a few times since I've made it publicly available. I'm not entirely sure if this is an issue* with Werkzeug or my code. I use the term "issue" loosely since the approach I'm using simply isn't meant for a production environment and I should be using an actual web server like Apache. Still, since this kind of problem could arise regardless I decided to try to see what tools are available to diagnose the issue.

Remembering that one Command

One convenient trick I've learned in the midst of everything else is that Linux allows you to not only iterate through your previous commands but search them as well. By this time my command history has gotten fairly large so it's really helpful when I can't quite remember how I got a command to work the last time. To begin the search just press Ctrl + R

Transmission Logging

I decided I want to try to log all the packets coming and going from the service so I searched Google to see what people were recommending. At first glance netcat seemed to be capable of doing just that. An article over at hawk host gives an example of the command to set up the proxy as well as a great explanation of what it all means. I altered it to suit my needs.

mkfifo pipe
nc -l -p 80 < pipe | tee outgoing.log | nc server 8000 | tee pipe incoming.log

Before I could use this command I had to remove the redirect I when I created the lightweight application server. In order to remove this you first need to take a look at the table. You can use the following command.

iptables -t nat --line-numbers -n -L

And your output should look something like this.

Chain PREROUTING (policy ACCEPT)
num  target     prot opt source               destination         
1    REDIRECT   tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:80 redir ports 8000

Chain INPUT (policy ACCEPT)
num  target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
num  target     prot opt source               destination         

Chain POSTROUTING (policy ACCEPT)
num  target     prot opt source               destination       

In my case I only had one entry in the PREROUTING chain so I wanted to remove rule number 1.

iptables -t nat -D PREROUTING 1

Now that the redirect was out of the way I could try out my netcat proxy command.

Testing Services from the Command Line

At some point during all this testing I was getting pretty tired of switching back and forth between my PuTTy session and my browser. I had already managed to log one of the requests to the service into a file.

GET /Service/GetApplication?name=passport HTTP/1.1
Host: 192.168.123.101
Connection: keep-alive
Cache-Control: max-age=0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.57 Safari/537.36
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8

It occured to me that it was most likely possible to send this request straight from the command line. Surprisingly it was a simple cat command in combination with another nifty feature of Linux.

cat application.request > /dev/tcp/localhost/80

Piece of cake. The only drawback to this approach was that no output would be returned. In order to see that I'd have to use netcat in combination with the cat command.

cat application.request | netcat localhost 80

Again, fairly simple stuff. So how well did this work as a logging solution? The good news is that the requests were logged but the bad news was that netcat would stop listening after the first connection. While it turns out that there is a -k option which keeps netcat listening this only occurs after the current connection has closed. Still it felt like I was getting closer.

TCPServer

In the course of all my previous research I remembered seeing a link for an application called tcpserver which is a part of the ucspi-tcp package. I recommend installing the package as it has a number of other utilities that seem like they would be incredibly useful.

tcpserver waits for connections from TCP clients. For each connection, it runs prog, with descriptor 0 reading from the network and descriptor 1 writing to the network.

Great! Let's try a simple approach.

tcpserver 0 80 nc localhost 8000

This works but I immediately noticed a slight connection delay and occasionally the request would fail on a browser. I'm not entirely sure what the cause is. That said I still wanted to see if I could get the logging working.

tcpserver 0 80 tee logs/client.log | nc localhost 8000 | tee logs/server.log

Unfortunately this just seems to spin indefinitely and once the server is shut down the browser ends up showing it's own request so somewhere in this command wires are crossed. I was able to get mixed results by writing a bash script instead.

#!/bin/sh
tee logs/client.log | nc localhost 8000 | tee logs/server.log

Remember we need to make it executable

chmod 755 myscript

And then run it with tcpserver

tcpserver 0 80 ./myscript

I'm still working on understanding why the script functions but not the command but the bigger issue was that my log files were getting overwritten so I needed

#!/bin/sh
date=$(date +%Y%m%d)
client_count=$(ls -l logs/client.$date.* | grep ^- | wc -l)
count=$(ls -l logs/server.$date.* | grep ^- | wc -l)
if [ $client_count -gt $count ] ; then count=$client_count ; fi
tee logs/client.$date.$count.log | nc localhost 8000 | tee logs/server.$date.$count.log

When I run this and use cat to issue commands everything seems to work fine but I believe there are some concurrency issues when a browser hits the service. Ultimately I'm beginning to realize I've gone down a rabbit hole that's far off base from my original purpose so I believe I'll end things here and switch to another possibility.

Packet Sniffing

Rather than try to stand in between the communications an easier (and most likely safer) alternative would be to just run a packet sniffer. I'm still researching the various tools that Linux has available and I'll be writing more on my findings.

Conclusion

While it's unfortunate that the logging approach didn't work out I did gain a better understanding of Linux standard inputs and outputs as well as a few other neat tricks along the way.