Entries from October 2007 ↓

I have been playing with Processing a lot lately and I have found the map() function to be extremely versatile. I got to thinking how useful it would be on the Arduino or some micro-controller platform. The map function is used as such:

   map(value, low1, high1, low2, high2)

where all of these are float values. So basically, you have a value (let’s say some kind of analog value) has a minimum of low1 and and maximum of high1. Now, let’s say you want to map this into a MIDI signal. You could just do something like:

   float midiValue = map(analogValue, low1, high1, 0, 127)

This will properly map your analog signal into a MIDI signal no matter what values it spits out. Anyway, I dug into the code and found the map function in the processing.core library (as suspected). Here it is (formatted for Arduino):

   float map(float val, float i1, float i2, float o1, float o2){
         return o1+(o2-o1) * ((val-i1) / (i2-i1));
   }

Use it but give credit to Casey Reas and Ben Fry, it is GPL.

**Update** I have been informed that this will be added to Arduino 11. **Update**

Last time I talked about a project I was working on and some python CGI research. Following up on that, that project also involves a little socket programming (which I chose to do in python again). Here is some wikipiedia info on internet sockets. We are going to be using streaming sockets with TCP (Transmission Control Protocol). Basically, a socket allows you to tunnel bytes (usually characters) from one computer to another (or many). Generally with TCP, you have a socket server that waits for socket clients to connect. Then it spawns off threads and serves that client socket resources. Usually, in the case of most sockets on the internet, those resources are html pages, pictures, etc. Sockets are very low level and in most cases considered a very complex programming subject. But, for simple projects, like the one I am working on, they are actually quite easy to implement. Here is the package for this post.

First take a look at threat_server.py, first we are importing everything from the socket module into the namespace, we are also importing urllib2, that is used to parse the Department of Homeland Security site and get the current threat level. More on that later. Next we establish some constants:

    HOST = ''
    PORT = 9055
    BUFSIZ = 1024
    ADDR = (HOST, PORT)

HOST is null b/c this is the server, this basically means localhost. PORT can be arbitrarily chosen so long as it is not being used by some other program. Don’t choose one under 100! I would just stick to numbers over 3000. BUFSIZ is naturally the size of the receive buffer. ADDR is a tuple containing HOST and PORT.
Now we have this chunk of code:

    tcpSerSock = socket(AF_INET, SOCK_STREAM)
    tcpSerSock.bind(ADDR)
    tcpSerSock.listen(5)

The first line establishes the variable tcpSerSock (our socket), and creates it as a tcp socket stream. AF_INET and SOCK_STREAM are predefined in the namespace when we import the socket module. This is the default configuration. Next we “bind” this socket to our address. Then the listen() function tells the socket the max amount of connections to listen for. This could just as easily be 1 for this example.
Next, we enter an infinite “while True” loop. and listen for a client socket:

     tcpCliSock, addr = tcpSerSock.accept()

After we get one, we enter another infinite loop where we wait for data from the client socket:

     data = tcpCliSock.recv(BUFSIZ)

We break the loop and listen for another client if this client sends “nothing” or the string “disconnect”. If the client sends “getTime”, we send the time. If they send “getThreat”, we send the current threat level. So on and so on. So, even though this isn’t a very powerful example, you can see the possibilities. Also, this code is not stable at all. There should be timeouts, multiple threads, etc., but that would be for a production quality piece of code and this is just an example.

Next open up the client, clienttest.py. This one is simple. You see some of the same setup as the server except HOST is foreign. If you are using this on your sever, change this to your domain. Also, make sure the port is chosen the same as your server program. The rest of this program is self-explanatory.
So, how to use this stuff. You can test this over your localhost or over a server. If you do the former, change the HOST in both programs to ‘localhost’ or maybe ‘127.0.0.1′. Then run the server, then run the client in a different shell. If you do the latter, FTP the server program to your server and run it there. Then run the client on your machine.
Well, this is all pretty simple but this is going somewhere. I will release more details later in the week when school dies down.

I recently got an idea for a project that I found will involve a little CGI programming so I did some research on python’s CGI library. CGI (Common Gateway Interface) is a way for programs running on your machine to access form and request information from your server. Here is a link you should read regarding the subject in python.

So, basically you can grab form data, process it, then print html to the requester. Python makes this so easy. Just use a line like:

form = cgi.FieldStorage()

Now you can access fields in the form variable by statements like:

value = form.getvalue("field_name")

So now you have user data in your python program! Just to show you exactly how easy this is. I have prepared a demonstration (Is your taste in beer similar to mine?). This link contains a simple form. Take a look at the source. You see the 3 radio types all have the name “fav_beer”. You also see that this form is being submitted to cgi-bin/cgi_form_test.py . That is the cgi script. Here is what it looks like (sorry for the shitty formatting, go copy and paste it into cgi_form.py or whatever):

    #!/usr/bin/python
    import cgi
    import cgitb
    cgitb.enable()

    # you need these 2 lines
    print "Content-type: text/html"
    print
    form = cgi.FieldStorage()
    beer = form.getvalue('fav_beer')

    print """
    <html><body>
    <p>
    """

    if (beer == "Abita"):
       print "My favorite beer is also Abita."
    else:
       print "Sorry, "+beer+" is not my favorite."

    print """
    </p></body></html>
    """

cgitb.enable() allows nice html formatted error messages but it is not really needed. As you can see, you directly print html to the standard output stream. The only place you will run into problems is with your host. First you need to make sure they support python and user cgi scripts. Also, you need to find out where you place the cgi scripts. Most hosts are in the domain root /cgi-bin/ folder. Dreamhost supports cgi scripts anywhere with any extensions. The next thing you need to take care of is permissions. In dreamhost you run ‘chmod 755 cgi_file.py’ . You usually need to give it permissions for every new file, or every-time it changes or updates, etc. Also, you should think a little about security. Never let user data directly effect system resources or system calls. For instance, never let one of the user input values be the name of a filename to modify or etc. Pretty simple stuff. Watch out for people trying to input huge values. Don’t use cgitb module in production (It shows people your code!). Other than all that, it’s easy. Enjoy.

Well, I had some time last night and I adapted my 6_pots experiment to work with Processing as promised. I did a little research, and other than the Processing Serial library page and this site, there isn’t a whole lot of information on the subject. Neither of these sites provide you with much technical information or technique allowing you to easily develop your own protocol so I figured I would write a little bit about it. Here is the package link.

First a little general theory about serial communication. Serial messages are sent bit by bit. For instance, when you send a string data type over a serial line, you are sending a series of bytes, each byte corresponding to a character. And you are sending each byte one bit at a time. There are many ways to represent a string on the bit level so how does the receiver know when the string starts, when it ends, how many bits per-byte, how many bits per second, etc? In this way, the information being sent is totally useless without some kind of protocol. A protocol is like an agreement or set of rules b/w a sender and a receiver. The first general rule of a protocol is establishing a conventional baud rate. The baud rate is the number of bits per second. This way the receiver knows at what rate to check the RX pin for being HIGH or LOW (0 or 1) and the sender knows at what rate to write each bit to it’s TX pin. Other important parts are parity and stop-bits. Arduino and processing make this invisible though and use (I believe, not sure) the generic asynchronous 8-N-1 configuration. This means 8 bits per byte, no parity, and 1 stop bit. That is the only thing you really need to know about the 8-N-1 configuration. If you want to go deeper, start here. So, with 8-bits per byte, you have a total of 2^8 = 256 values 0 to 255 || -127 to +127). Now, this doesn’t mean that you cannot send higher values, you just need to get creative. For instance, to send a value like 945, you could send characters ‘9′, ‘4′, ‘5′, and then some reserved value to tell the receiver that the value is complete and to “build” the value. The processing.serial library provides methods for this like readUntil(), bufferUntil(), etc. Check the Processing serial link above. You also need to establish a network relationship b/w the 2. Is it going to be ad-hoc where either can send any value at any time, or is it going to be a master-slave (handshaking) relationship? Both of these are for 2-way communication but we only have 1-way communication in this sketch.

Now, how to design this protocol. We want it to be as fast as possible so we choose a baud rate of 115200 bps. To set this up in Arduino, put this into setup():

   Serial.begin(115200);

and in Processing, put this into setup():

   port = new Serial(this, Serial.list()[0], 115200);

Oh, that brings up an important point. You need to know where your Arduino port is. Mine is always the first in the list but yours may differ. The best way to find out is to connect your Arduino and open the software. Open (Tools –> Serial port) and the one that you have selected should look something like (on os x, linux) /dev/tty.usbserial-XXXXXXX (not 100% sure on how it works in XP w/ COM ports and all). This is the FTDI driver. Mine is always the first in Serial.list() so I just choose the 0 index of the array. You can choose the proper index in this fashion or enter the entire string ex(“/dev/tty.usbserial-XXXXXXX” or whatever). Okay, so after that is setup, you need to decide the rest of the protocol. We are sending 2 bytes per message, one is the pot number (control number) and the next is the value. How does Processing distinguish which is which in the packet? Well, we send a “header byte” that we know will not be used in the packet (the control number byte or the value byte) b/c it is too big. So Processing waits for this header byte and when it gets it, it knows the next byte is the control number and the next byte after that is the value. Here is the Arduino code for that:

   void sendMessage(byte chan, byte val) {
       Serial.print(255, BYTE); // header byte
       Serial.print(chan, BYTE);
       Serial.print(val, BYTE);
   }

Now, the way Processing reads it is like this:

   if (port.available() > 0) {
       if (port.read() == 255) {
          try {
             grid[port.read()] = port.read();
          } catch (Exception e) {System.out.println(e);}
       }
    }

First it checks to see if there is anything in the buffer with port.available(). If this result is greater than 0, there is stuff in the buffer, else break. If that condition passes, it checks to see if the incoming byte is the header byte, if it isn’t, break. Now we enter a try loop. I’ll explain that in a second. Then there is the assignment. I used my knowledge of the compiler and how the read() function works here to save some lines of code. Because port.read() is a blocking function (the whole program waits for it to receive a byte) and this statement is evaluated from left to right (thanks to the compiler and JVM), we know that the first port.read() waits for the next byte (which will be the control number). We use this as the index of our grid. Then the next port.read() is evaluated and this assigns the result to the grid space. Some people may think this is dangerous but I don’t really care. So, now for the try loop explanation. The reason I did this is b/c at this speed, there are errors and sometimes the the first condition (port.available() > 0) passes on accident when the value is actually -1 (port not available). Maybe because the buffer has not cleared yet until after the condition (I don’t know). It would probably fix the situation to add a delay() at the end of the draw() loop or better yet, do port.clear() right-after this segment of code and make sure the buffer is flushed before it continues. It may also help to get rid of the print(grid[i]) statement in the for loop for rectangle drawing (take note that printing takes up a lot of time and shouldn’t be used as I did here unless you want your program to run slow). Anyway, the try-catch loop catches exceptions when the grid assignment throws an index array exception by trying to access an index of -1. Also, I used System.out.println() instead of the println() in the processing,core library b/c sometimes the compiler gets fussy about using anything outside of the java.lang library in a catch. This is like duct tape, I’ll let you know if I fix this later.

So, after that, the rest is pretty self-explanatory. The Arduino file processing_test.pde is implemented almost exactly the same as that midi_cc_test.pde file in the 6_pot –> MIDI experiment. The Processing sketch also draws some rectangles with the information and prints the value just for visual feedback:

   for (int i=0; i<6; i++) {
       rect((i*40)+20, 0, 40, grid[i]);
       println(grid[i]);
   }

Here is a screenshot:

****Here is an update. The same sketch but it has sound as well (need Ess library)****

I’ve been wanting to make some MIDI controllers for a while but I have never gotten past some crappy prototypes based on the PIC16F84A. So, after re-reading a really excellent book that a friend of mine at Berklee School of Music gave me, MIDI for the Professional, I decided to give it a try with the Arduino. I just wanted to make a simple 6 pot circuit that outputs values on 6 midi “Control Change (CC)” channels. You can download everything here (midi_test is a Max binary) and here are the details:

The first design rule I had was that it had to be a full midi implementation since I am running it through my Yamaha UX16 – USB MIDI Interface. So, I need:

• a baud rate of 31250 (MIDI standard)
• outputs 8 bit bytes
• only uses 7 bits as value so messages must be b/w 0 and 127
• needs to only send message when something changes

The first specification is simple. In Arduino’s setup() function, declare:

Serial.begin(31250);

To store the values in bytes, we need the appropriate data structure:

byte currentPot[6] = {0,0,0,0,0,0};

To make sure the bytes are only b/w 0 and 127 (and actually fit in a byte):

currentPot[i] = analogRead(i) / 8;

divide the result of analogRead() (0-1023) by 8 because 1023/8 = 127.875 . It just a simple way to map the values. Now, to make sure that it only sends when something changes, we have defined another data structure:

byte pot[6] = {0,0,0,0,0,0};

to store the past values of the pots. The program iterates through both byte arrays, if the value at i has changed by more than some delta, :

#define e 1

the message is sent and the current value replaces the old value. Make e = 0 if you want it to send a message every time there is a slight change but you may have some false sends due to the power supply not being completely clean causing some oscillations in the pot values. It may also be weird round off errors from the division by 8 (just a second guess, I don’t know for sure). Your experience may differ.

So, how are the messages built and sent? First build this schematic to the arduino (just the midi connector part):

This is provided by Tom Igoe. Check out this link for more. Now the software side:

void sendMidi(byte controlNum, byte val) {
   Serial.print(0xB0, BYTE);
   Serial.print(controlNum, BYTE);
   Serial.print(val, BYTE);
}

The first byte sent is 0xB0 (176, DEC) which is the HEX value to tell the MIDI device that the next bytes will make up a control change message. MIDI is best read in HEX format because there are 16 of everything so it all fits in neat labeled packages. For example, the different control change “control numbers” in HEX are B0 to BF. It basically maps out so that the first digit represents a type of message and the second represents the “channel” number. Genius protocol. If you still don’t understand but actually care, look here and/or here. The next byte is reserved for the control number. For us, this number will represent which pot the next value belongs to but in a general sense, it represents a “channel”. The next value is simply the value (0-127) being sent.

For the schematic of the pots, I got 6 10K pots and hooked them up in this arrangement. Pin 1 goes to GND, pin 2 goes to an analog pin, and pin 3 goes to 5 volts. This could use a drop resistor but that would take more time. Maybe for something more permanent. Connect these to pins 0-5. I did it like this:

The rest of this code is pretty self-explanatory. Let me know if you are confused. I also created a simple max patch (midi_test) that comes in the 6_pot package. Here is a screen-shot:

To use it open (File–>MIDI Setup) and change your midi input device to the symbol ‘a’. Then startup your arduino and tweak. Each channel is sent to different oscillators, some saw waves, some sine. The saw waves start at 400 Hz and so do the sine waves. Not too exciting but it is some kind of output. If you don’t have Max you can get the Runtime and run this. Also, if you don’t know how to use Max, turn up the volume by sliding up that vertical slider with your mouse (not all the way!) and hit the speaker button (audio on). Well, that is it for now. Maybe more on this later. I think next time I will talk about getting these values into Processing.

I wrote this piece of code to figure out the best ways to fit ripped movies onto DVDs. I can usually fit anywhere from 3 to 5 movies on a DVD and I realized after burning many discs, I was choosing very inefficient combinations. So, what this program does is goes through a directory of files and takes the ones with your chosen extensions [in my case usually (avi|mov|mpg)]. Then it generates every combination (not permutation) of those files inside a range of group sizes and picks the the best fit. Here is the heart of the logic:

def getBestFit(movies, target_size):
    free_target_space = target_size
    best_fit = []
    cont = True # True while there are still hits
    n = 2 # starts off at 2 and continues until there are no hits
    while cont:
        hit = False
        for uc in xuniqueCombinations(movies, n):
            total_bytes = 0L
            for mov in uc:
                total_bytes += mov.getSize()
            fit = target_size-total_bytes
            if ((fit > 0L) and (fit < free_target_space)):
                best_fit = uc
                free_target_space = fit
                hit = True
        cont = hit
        n += 1
    return (best_fit, free_target_space)

The parameter ‘movies’ is a list of ‘Movie’ objects (a generic class to store data of a movie file) and the parameter ‘target_size’ is a long representing the number of bytes on the target memory (DVD in my case). To try, simply run the file as __main__:

$ python MovieModule.py

Here is an example of filling out the output:

Enter absolute path > /Users/ben/Movies/films/

PATH = /Users/ben/Movies/films/
Enter target size in GB (float) > 4.38
TARGET_SIZE = 4.38 GB
Enter extensions separated by '|' > avi|mov|mpg
EXTENSIONS = ['avi', 'mov', 'mpg']

When entering the PATH, enter the absolute path to the directory that holds all the files (no subfolders) and be sure to end with a ‘/’. When entering the extensions to search for, chain as many as you want and separate by ‘|’. Only extensions with excactly 3 characters will work. These are not all flaws b/c I made this for myself only for this situation but some of you may be able to adjust it for more in your own code. Here is the 2nd output:

American_Beauty : 1.038962972 GB
Palindromes : 0.85508783 GB
V_for_Vendetta : 1.132145016 GB
Zodiac : 1.34692592 GB
free_target_space = 0.006878262 GB
total_bytes on disc = 4.373121738 GB

Here is the file (archived with gzip -9) : MovieModule.py