CREATE YOUR OWN SMS GATEWAY

here i am gonna show you how to setup a simple sms gateway from a 3G USB dongle using python. the work-flow here is very simple. but before that let’s go through with some basics of 3G USB dongle.

in our gateway once a valid request from the internet is received to our gateway, it should communicate with the dongle and send the message included in the request to the relevant number in the request. so the most important part in this would be communicating with the dongle when the gateway receives some request.

basically your 3G dongle consists with a 3G modem and USB to serial converter. 3G modem is something like below image. to communicate with the modem, it needs a serial (RS-232) connection and this is where USB to serial converter comes to play.
as all modern laptops are consists of USBs this converter is also integrated inside the dongle so we can easily plug it and communicate. so the catch from here is that we can use RS-232 / serial to communicate with our dongle.

GSM / 3G modem module. a tiny compact module is used in a dongle
GSM / 3G modem module. a tiny compact module is used in a dongle

first let’s try to communicate with our dongle with the terminal and if it succeed we can automate that with python to use in our SMS gateway.

GSM / 3G module communication flow.
GSM / 3G module communication flow. source: http://www.engineersgarage.com/articles/gsm-gprs-modules

testing on terminal

first plug the dongle (assuming that you have inserted a sim card with credit in it 😛 ) and find out the port it opened. open terminal and issue dmesg | grep tty command to find out connected USB devices.
you will obtain an output like below.

[   14.124879] usb 1-1.4: GSM modem (1-port) converter now attached to ttyUSB0
[   14.126676] usb 1-1.4: GSM modem (1-port) converter now attached to ttyUSB1
[   14.127795] usb 1-1.4: GSM modem (1-port) converter now attached to ttyUSB2

now, let’s try to communicate with the dongle with the serial connection (RS 232). here we have to use AT commands to communicate. you can find the complete list of AT command set GSM/ 3G modems support from here.

we will be using picocom terminal program to communicate with the dongle via serial. install it as below.

sudo apt-get install picocom

now open the serial port. make sure the substitute relevant port of the dongle.

picocom /dev/ttyUSB0 -b 115200 -l

picocom v1.7

port is        : /dev/gsmmodem
flowcontrol    : none
baudrate is    : 115200
parity is      : none
databits are   : 8
escape is      : C-a
local echo is  : no
noinit is      : no
noreset is     : no
nolock is      : yes
send_cmd is    : sz -vv
receive_cmd is : rz -vv
imap is        : 
omap is        : 
emap is        : crcrlf,delbs,

Terminal ready

try issuing AT command first. you should get an output like following if everything so far alright. 🙂 issue following AT commands as well to make enable text mode for SMS.

AT
OK

AT+CMGF=1
OK

now let’s try to send a message. enter message in following format.
AT+CMGS="phonenumber"<CR>the message to send<ctrl+z>
after the number and the end of double quote hit enter, then type the message. hit ctrl + z.

it’ll give u a OK response and in a while you should receive a text message to your specified number. if all is good, its time to write our gateway piece. 😀

building the gateway

i’ll be using flask for the REST API and a python module called python-messaging for encoding text messages to PDU format.

first install those packages by:

pip install flask
pip install git+https://github.com/pmarti/python-messaging.git

now, lets get into coding. open a new file, say gateway.py. lets first write a function to open the serial port and issue relevant AT command to send the message.

from messaging.sms import SmsSubmit
import serial


def send_text(number, text, path='/dev/ttyUSB0'):
    sms = SmsSubmit(number, text)
    pdu = sms.to_pdu()[0]

    # print len(sms.to_pdu())

    # open the modem port (assumes Linux)
    ser = serial.Serial(path, timeout=1)
    # write the PDU length and wait 1 second till the
    # prompt appears (a more robust implementation
    # would wait till the prompt appeared)
    ser.write('AT+CMGS=%d\r' % pdu.length)
    print ser.readlines()
    # write the PDU and send a ctrl+z escape
    ser.write('%s\x1a' % pdu.pdu)
    ser.close()

now, let’s attach this as a response to a valid POST request. we’ll use HTTP basic auth for authentication.
1. first let’s write a simple function to check user name and password.
2. then basic auth handler to be used in POST request
3. finally the POST request

from flask import Flask, jsonify, request
from functools import wraps
app = Flask(__name__)

def check_auth(username, password):
    """check username / pass"""
    return username == 'user' and password == 'mypassword'


def requires_auth(f):
    @wraps(f)
    def decorated(*args, **kwargs):
        auth = request.authorization
        if not auth or not check_auth(auth.username, auth.password):
            return jsonify({'message': 'not logged in'}), 404
        return f(*args, **kwargs)
    return decorated


@app.route("/")
def hello():
    return "python sms gateway :)"

# POST request handler
@app.route('/send', methods=['POST'])
@requires_auth
def create_task():
    if not request.json or not 'number' in request.json:
        return jsonify({'message': 'no number provided'}), 400

    send_text(request.json['number'], request.json['text'])
    return jsonify({'message': 'message queued'}), 201

if __name__ == "__main__":
    app.run(host='0.0.0.0', port=4000)

so the full code for the gateway will look like below.

gateway.py

from messaging.sms import SmsSubmit
import serial
from flask import Flask, jsonify, request
from functools import wraps
app = Flask(__name__)


def check_auth(username, password):
    """check username / pass"""
    return username == 'user' and password == 'mypassword'


def requires_auth(f):
    @wraps(f)
    def decorated(*args, **kwargs):
        auth = request.authorization
        if not auth or not check_auth(auth.username, auth.password):
            return jsonify({'message': 'not logged in'}), 404
        return f(*args, **kwargs)
    return decorated


def send_text(number, text, path='/dev/ttyUSB0'):
    sms = SmsSubmit(number, text)
    pdu = sms.to_pdu()[0]

    # print len(sms.to_pdu())

    # open the modem port (assumes Linux)
    ser = serial.Serial(path, timeout=1)
    # write the PDU length and wait 1 second till the
    # prompt appears (a more robust implementation
    # would wait till the prompt appeared)
    ser.write('AT+CMGS=%d\r' % pdu.length)
    print ser.readlines()
    # write the PDU and send a ctrl+z escape
    ser.write('%s\x1a' % pdu.pdu)
    ser.close()




@app.route("/")
def hello():
    return "python sms gateway :)"


@app.route('/send', methods=['POST'])
@requires_auth
def create_task():
    if not request.json or not 'number' in request.json:
        return jsonify({'message': 'no number provided'}), 400

    send_text(request.json['number'], request.json['text'])
    return jsonify({'message': 'message queued'}), 201

if __name__ == "__main__":
    app.run(host='0.0.0.0', port=4000)

now run the gateway by issuing python gateway.py. our gateway server now would be running on http://localhost:4000 in the host machine. next let’s use postman to send a test request and ensure that our gateway is working. download postman from chrome web store if you don’t have it installed.

format the request as below. 😛

postman_1

postman_3

if all is good, you should now receive a text from the dongle.

you can find the complete source code of the project from https://github.com/wathmal/sms-gateway.

references:
* https://myraspberryandme.wordpress.com/2013/09/13/short-message-texting-sms-with-huawei-e220/
* https://github.com/pmarti/python-messaging/blob/master/doc/tutorial/sms.rst

RUN APACHE WITH NODE JS REVERSE PROXY – USING REDBIRD

if you are learning or using nodejs, most of you might encounter the problem of running nodejs with apache or nginx. there are several solutions for this issue, with several pros and cons for each solution. for this article i’ll only consider we have a web server with apache.

some of suggested solutions are:

  1. use a reverse proxy with apache (with mod_proxy) to serve nodejs apps
  2. disable apache and run nodejs on port 80 😀
  3. run nodejs on port 80 with a reverse proxy configured on nodejs

if you go with first method, you are really wasting the power of nodejs. but if you do a simple google search on this, lots of articles will suggest you or show you to go with this approach. i am not going to explain this in depth. following extracted text will explain what is the downside of this approach very well. 🙂

you just bought a brand new ferrari and you want to take it for a spin on the highway.. so you go there.. but there is a big fat truck in front of you that only goes 50mph and there is no way to get passed it.

that is basically what you are doing if you put apache (truck) in front of nodejs (ferarri) this defeats one of the main purposes of using nodejs having a FAST evented machine.. which does not allocate threads for requests.

in the second approach, the only downside is you will lost your apache server. so if you run some CMS es like wordpress / drupal you have to find other alternative options like our 3rd solution.

so the 3rd option, running a reverse proxy with nodejs. this idea is really crazy. 😀 following are the steps to do this.

  1. first we have to  make apache run on different port other than port 80.
  2. then configure nodejs to run on port 80.
  3. create the proxy script with redbird.
  4. configure PM2 to auto the start proxy server.

and we are done.

step 1: make apache run on different port

first open ports.conf in /etc/apache2/ directory.

sudo nano /etc/apache2/ports.conf

then change Listen 80 to Listen 8080
so the ports.conf will look like following.

Listen 8080

<IfModule ssl_module>
	Listen 8443
</IfModule>

<IfModule mod_gnutls.c>
	Listen 8443
</IfModule>

next open 000-default.conf in /etc/apache2/sites-available directory.
change <VirtualHost *:80> to the previously given port as <VirtualHost *:8080> and restart the apache server to reflect the changes.

sudo service apache2 restart

step 2: configure nodejs to run on port 80

to run nodejs on port 80, the script will need sudo permission. but there are several ways to do this without sudo permission, which is more safer. we can bind nodejs to listen on lower ports (1 – 1024) by following commands.

sudo apt-get install libcap2-bin
sudo setcap cap_net_bind_service=+ep /usr/local/bin/node

once you do this, we can listen to all lower ports including port 80 via our nodejs app.

step 3: create the proxy script

now we have to write the reverse proxy script. for an example let’s say we have a wordpress blog running with apache on http://mydomain.me:8080/blog/ (in the first step, me made apache run on port 80) and a nodejs app running on http://mydomain.me:3000/ and we want to make both of these apps available on port 80.

to write the proxy script we will be using a npm module called redbird. we can begin writing the script as follows.

 

var proxy = require('redbird')({port: 80, xfwd: false);

proxy.register("mydomain.me/blog", "http://mydomain.me:8080/blog");
proxy.register("mydomain.me", "http://mydomain.me:3000");

note than, this reverse proxy script is set to run on port 80. now if you run this script with node proxy.js we’ll be able to access our blog via http://mydomain.me/blog/ and our node app on port 3000 via http://mydomian.me/. 🙂

step 4: configure PM2 to auto start the proxy server

next step is to making this proxy script auto-start when the system boots. there is a great tool which will do this job for us called PM2. install and configure it if you haven’t already used it.

 

sudo npm install -g pm2
pm2 startup ubuntu

this last line will output a command for you to run on the terminal to make PM2 auto start on system boot. run it and you are almost done.

next step would be adding our proxy script to process list of PM2. then PM2 will take care of our script for us.

pm2 start proxy.js

also via PM2 we can start / stop / restart the process like we use services in linux.

references:
https://groups.google.com/d/msg/nodejs/HX_j8LqFAso/pma5pSQR08IJ
https://github.com/OptimalBits/redbird
https://www.digitalocean.com/community/tutorials/how-to-use-pm2-to-setup-a-node-js-production-environment-on-an-ubuntu-vps

SETTING APACHE REVERSE PROXY FOR TRANSMISSION WEB

howdy fellas! 🙂 today i had to setup apache reverse proxy with my transmission web interface. in my university wifi all ports are blocked other than port 80 and in order to access my transmission server i had to either setup an apache reverse proxy or make transmission daemon run on port 80.

for this, first you have to enable apache mod_proxy and mod_proxy_http modules.

sudo a2enmod proxy
sudo a2enmod proxy_http

next you need to edit the apache configuration file. so first open it with nano.

cd /etc/apache2/sites-available
sudo nano 000-default.conf

your default configuration file name might not same as 000-default.conf. provide the appropriate name.

next you need to add following configuration block to the conf file. here our goal is to access transmission web interface via http://mydomain.me/transmission instead of http://mydomain.me:9091

ProxyRequests Off
<Proxy *>
        AddDefaultCharset off
        Order Allow,Deny
        Allow from all
</Proxy>

ProxyPass /transmission http://mydomain.me:9091/transmission
ProxyPassReverse /transmission http://mydomain.me:9091/transmission

ProxyVia On

following isn’t the reason for HTTP 409 conflict error.
one thing to note is don’t add http://localhost:9091/transmission instead of http://mydomain.me:9091/transmission. in my configuration putting localhost instead of actual domain generated HTTP 409 conflict error.

save the file & exit. next restart the apache server and you are almost done.

sudo service apache2 restart

now you can access your transmission web interface via http://mydomain.me/transmission/web/. 🙂 so happy reverse proxying.

update *
note that you have to access transmission web interface via the exact url as above. if you loose or don’t add /web/ part you’ll receive HTTP 409 conflict error.

SERVER LOAD TESTING NGINX VS APACHE WITH DRUPAL

here is the million dollar questoin! 😀 what is the best web server ? nginx or apache ? if you dig some deep on this, you’ll come around that unless, you have a very big user base or limited hardware resources, it doesn’t matter.

so i wanted to test this with drupal, whether it actually makes a difference in a practical situation. (theoretically it should) thanks to AWS free tier i setup my test environment using AWS as below.

SETTING UP TEST ENVIRONMENT

following are the actions i performed in AWS console and in EC2,

  • launched an EC2 instance
  • launched RDS instance for mysql (for use as drupal’s DB)
  • installed both apache and nginx servers in EC2 instance
  • installed drupal and connected it to RBD server
  • added some dummy content in drupal site

next i created an image of my EC2 instance (AMI) and launched another instance with the same hardware features as previous EC2 instance in same region from the created AMI.

so the next steps followed are,

  • configured one instance to run apache only and another to run nginx only.
  • pointed drupal installation as root directory in both servers.

after this, i had two drupal installations with same content running on apache and nginx in same location.

CREATING A TEST PLAN

so the basic thing i wanted to test is,

server performance in limited hardware over a large traffic.

i used jmeter with performance monitor plugin for this. performance monitor plugin will allow you to measure server performance metrics in jmeter.(cool right ? 🙂 )

for the test plan, in jmeter

  • added thread group with 1000 threads to simulate 1000 users
  • inside thread group added HTTP GET request to root folder to simulate main page access in druapl site
  • added a performance monitor metric collector to measure server CPU & memory usage
  • added a summary report to get overall statistics
jmeter test plan
jmeter test plan

 

RESULTS

so i performed this same test on both of my EC2 instances. following are the results i obtained.

APACHE

requests failed throughput
1000 18.7% 4.437640063/hr
apache server performances for 1000 users
apache server performances

NGINX

requests failed throughput
1000 7.2% 4.4314258999/hr
nginx server performances for 1000 users
nginx server performances

CONCLUSION

so so so, here it is! answer for the million dollar question.

under a large traffic with same hardware resources, nginx performs way better than apache.

references:
https://www.digitalocean.com/community/tutorials/how-to-use-apache-jmeter-to-perform-load-testing-on-a-web-server