Hand to Hand Combat with Subversion and SmartSVN

In previous links I have extolled the virtue of both backups (obviously) and version control (less obviously) for what is in fact 'hobby' software development.  I had the 'bot under version control using Subversion on OSX with SmartSVN as the GUI client.  Somehow, however, I managed to screw things up to the point where it was unusable.  I spent a fair amount of time on the OSX box trying to figure out what I had done but finally decided that I wanted my repository to live on another server anyway.  Ubuntu was my most logical choice, as I have a couple of boxes running it already, so I got started.

I found some pretty straight forward instructions at this link and followed them closely.   When running on OSX I had relied on the SmartSVN GUI entirely but this time I wanted to have at least a basic understanding of how Subversion was working under the scenes.  This reflects my late realization that Subversion is actually doing all the heavy lifting and that this was being masked by the GUI.

In any case, following those instructions in detail gave me a pretty good handle on how Subversion works.   Everything went well locally but I did have some issues when I tried to access my repository remotely.   It turns out that in the process of getting my directory structures the way that I wanted them I had lost the changes that I had made to the 'svnserve.conf' file located in the 'conf' directory of the repository.   Without this file, and the passwd file also in the same directory, svnserve was denying me access.  My real problem was that I was convinced that I had done everything that I was supposed to have done...and did not look at the obvious (at least for a bit).

Once I had everything working at the command line it was simple to get SimpleSVN working again.

Here is a summary of the steps that I took based on (but with a few differences) the above mentioned link:

First create the directory structure for svn and your repository.   I put mine on the root thinking it might be easier to relocate it to it's own mount point later.   In this box I also created a group for subversion users and added myself to that group.

cd /
sudo mkdir svn
cd svn
sudo mkdir repos
cd repos
sudo groupadd svn
sudo chgrp svn /svn/repos
sudo chmod g+w /svn/repos
sudo chmod g+w /svn/repos
sudo usermod -a -G svn will

Note the two chmod commands.  The intent of these two commands is to make sure the svn group can write to the repository...and...that files created in the repository inherit that write-ability for the group.   I had to repeat these commands later so am not sure if they are needed here or not.

Now create the actual repository, or repositories, that you will use.   The umask is there so that group rights are protected...but...given my comment above I am not sure that you need these as the later chmods will accomplish the same thing?

umask 002
sudo svnadmin create /svn/repos/Arduino
umask 022

And finally the chmod's that I mentioned above.   Without these I was not able to commit to the repository as the group had, somehow, not gotten the right write rights all the way into the structures that subversion uses. 

sudo chmod -R g+w /svn/repos
sudo chmod -R g+w /svn/repos
sudo reboot

I did a reboot at this point but I think that all you really need is a logout / login to pickup the group rights for svn.

You should now be able to execute the following commands on your local machine to checkout the repository, create a file and add it to the repository, and then commit that change:

cd ~
svn checkout file:///svn/repos/Arduino
cd Arduino
cat 'This is a test file' > test.txt
svn add test.txt
svn commit -n 'Test file added'

You should have seen a version 0 checkout, that add of test.txt, and then a version 1 commit if all has gone well.

Next remote access...

First create a passwd file in the base directory that was created above.  In the case of the example this was /svn.   The format is below and should include the names for you (and your team if you have one!):

[users]
will = password

The go into each repository under repos and edit the svnserve.conf file (in the conf directory) to have the following minimum of entries:

[general]
anon-access = none
auth-access = write
password-db = /svn/passwd
realm = team

Note that I am pointing the password database to the file created above so you only need one of these.

Finally the command to start svnserve:

sudo svnserve -d -r /svn/repos

You should now be able to access your new repository from the network using the below command for a checkout:

svn checkout svn://workbench/Arduino --username will

Obviously with your IP address or host name substituted above!

Once all of the above was done I was also able to get back into SmartSVN and import this project.   Understanding for once what was going on in the background!  One last note...a very useful command while I was cleaning things up from my previous mess was 'rm -Rf .svn'.   This deletes the structures that mark a path as being in an svn repository.


The last thing that I did was to follow my own instructions, located here, to create a script to have svnserve start automagically on boot.   Here is the script that I created from my example:

#! /bin/sh
# /etc/init.d/noip

### BEGIN INIT INFO
# Provides:          svnserve
# Required-Start:    $remote_fs $syslog
# Required-Stop:     $remote_fs $syslog
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Starts svnserve for local code repositories
# Description:       A simple script based on one from www.stuffaboutcode.com.
### END INIT INFO

# If you want a command to always run, put it here

# Carry out specific functions when asked to by the system
case "$1" in
  start)
    echo "Starting svnserve"
    svnserve -d -r /home/svn/repos
    ;;
  *)
    echo "Usage: /etc/init.d/svnserve {start}"
    exit 1
    ;;
esac

exit 0

There are obviously a LOT of other configuration and commands that can be done from here but this is a bare minimum to get me back to where

Arduino Mega + LCD Touch Screen = Cool Beans!

The Arduino Mega, integrated with a Sainsmart LCD Touchscreen, is a pretty cool little instrument.  The configuration below consists of a Mega 2560, their SainSmart 3.2" TFT LCD Display (with an SD card adapter), and their adjustable shield to mount it to the 2560.

 

Hardware integration was pretty easy since it all just piles together.  Two minor challenges, the first being that my initial LCD display was defective on reciept from Amazon UK.   Easy replacement though.  The second issue concerns the model of display that I bought (as in cheap) and that it uses most of the pins on the Arduino and semi-blocks the rest of them!   I am pretty sure there is an I2C model that could help with the former though I am not sure about the latter.  

In any case, I only needed a couple pins and was able to access them by bending some patch wires to fit under the shield.   In the case of the reset line, I bent the pin from the shield away so I could connect a reset button (on the breadboard). Speaking of the breadboard...it supports the reset button, an LED that is used as a status indicator while data is being collected (so LCD refresh times don't impact the data collection), and a voltage divider with a variable resistor to simulate an input.

In the next post I will address software integration

Logging to an SD Card with Arduino - the Software

The sketch that I used for logging to an SD Card is available here for download.  It is pretty much based on the examples from the Arduino download but with a couple of interesting notes:

Nothing special here except that at one point I had put the initialization of the card where it could get executed twice.   Fail!   It is much happier in the setup section with one time things.

    // Make sure that the default SD chip select pin is set to output, even if
    // you don't use it (interferes with lcd???):
    pinMode(SS, OUTPUT);

    // See if the card is present and can be initialized:
    if (!SD.begin(chipSelect)) {
        utilLcd.lcdDisplay("SD Card is not ", "present/working", -1);
        while (1);
    }

Prior to starting to make observations we open an output file.

    // Open up the file we're going to log to and create a new file
    char filename[] = "READINGS.DAT";
    SD.remove(filename);
    dataFile = SD.open(filename, FILE_WRITE);
    // Did the file open work?
    if (! dataFile) {
        utilLcd.lcdDisplay("Could not create", "readings file", -1);
        while (1) ;
    }

Almost certainly overkill, but, I wanted to have the most dense data logging possible so am logging two longs in eight bytes.   Pretty much their native size if not format.  I have not done a timing between doing this and saving as ascii but I would hope for this to be more efficient!?!?

    // Convert the above from an unsigned long ints to a 8-byte array...
    buffer[0] = (int)((timestamp >> 24) & 0xFF);
    buffer[1] = (int)((timestamp >> 16) & 0xFF);
    buffer[2] = (int)((timestamp >> 8) & 0XFF);
    buffer[3] = (int)((timestamp & 0XFF));
    buffer[4] = (int)((voltage >> 24) & 0xFF);
    buffer[5] = (int)((voltage >> 16) & 0xFF);
    buffer[6] = (int)((voltage >> 8) & 0XFF);
    buffer[7] = (int)((voltage & 0XFF));
    // ... and save them in our data file
    dataFile.write(buffer, 8);

If we are logging to an SD card we may well NOT be doing anything further with it on the Arduino but I wanted to do so anyway.   Obviously we had closed and reopened the logging file and then I use this function to reach into that file and get the eight bytes that were logged.

/* ************************************************************************
Read a specific observations entry from our file into our global record buffer
************************************************************************ */
int observation(long entry) {
    if (dataFile.seek(entry * 8)) {
        for (int i = 0; i<8; i++) {
            buffer[i] = dataFile.read();
        }
        return(TRUE);
    }
    else {
        return(FALSE);
    }
}

Finally, an example of getting back the long value that had been saved.   In this case the voltage reading.

/* ************************************************************************
Return the voltage portion of our file buffer
************************************************************************ */
unsigned long voltageEntry() {
    unsigned long temp = ( ((unsigned long)buffer[4] << 24)
                   + ((unsigned long)buffer[5] << 16)
                   + ((unsigned long)buffer[6] << 8)
                   + ((unsigned long)buffer[7] ) );
    return(temp);
}

This code captured and logged at the rate of about 860 observations per second.

Logging to an SD Card with Arduino - the Hardware

The Arduino can make a lot of readings every second.  If you want to do some complicated processing with those observations you will be limited by the speed of the processor.  You can upload them in real time via the serial interface, or an add-on shield doing WLAN or Bluetooth.  If you want to run disconnected, logging to an SD Card could be the solution.  I don't have shields for either WLAN or Bluetooth but I suspect that logging to an SD Card would be the faster of the logging strategies.

In any case, I bought one for three quid from eBay now available even cheaper on Amazon!   The one that I bought is not integrated into a shield and so requires a little wiring...but it was cheap!  SD Cards use SPI, or Serial Peripheral Interface, to communicate.   This is provided on the Uno by pins 11, 12, and 13.   On the Mega it will be 50, 51, and 52.  The pin connections that are needed are shown below and by the included pictures:

SD Card Breakout Pin	Function	Arduino Pins - Uno / Mega	Comments
GND	Ground	GND	Only one ground is needed
+3.3	Power on 3.3v system	+3.3	Only one power source is needed
+5	Power on a 5v system	+5
CS	Chip Select	10 or 53	This pin may wander while the lower three are confined to where the board puts SPI.  Make sure your sketch reflects the right pin!
MOSI (sometimes called DO)	Master Output - Slave Input	12 or 50
SCK (sometimes call CLK)	System Clock	13 or 52
MISO (sometimes called I)	Master Input - Slave Output	11 or 51
GND	Ground	Ground

There is also a Card Detect pin on some cards (though not mine). It shorts to ground when a card is inserted. You should connect a pull up resistor (10K or so) and wire this to another pin if you want to detect when a card is inserted.

The testbed shown above consists of an Arduino Uno with a Keypad LCD Shield riding on a handy Screwshield Terminal Expansion Board for the Uno.  The SD Card is pretty power hungry, as is the LCD, so we are connected to a wall power dongle.   The breadboard in the foreground has a variable resistor as part of a voltage dividing circuit so I can vary my "readings" for test purposes.

In my next post I will talk about the software.

Arduino Pointers, Pointers, and more Pointers

As I work on my Arduinos I am constantly reminded that I always found pointer based languages hard to master.   I guess that is why I didnt make a career out of working in C or C++.   But the Arduino demands it if you are going to do anything complicated.   I have been struggling to remember my CS classes of so long ago but have gotten a lot of help from folks on the Arduino Forum.

With that help in hand I came up with a sketch (available here) that runs through most of the pointer combinations and permutations that I think I will need for the immediate future.   The output from this sketch is shown below:

Pointer to an integer
a is now = 200
Another way to use that pointer...
a is now = 300
...which allows you to do this
a is now = 301

Pointer to a char array
abcdefghijklmnopqrstuvwxyz0123456789 <- Printed using original assignment of 'myS'
a <- Printed using '*myS'
a <- Printed using pointer '*ptrmyS'
abcdefghijklmnopqrstuvwxyz0123456789 <- Printed using pointer 'ptrmyS'
abcdefghijklmnopqrstuvwxyz0123456789 <- Printed by walking pointer down char array
abcde#ghijklmnopqrstuvwxyz0123456789 <- after modifying using array notation
abcde#ghij#lmnopqrstuvwxyz0123456789 <- after modifying using pointer notation

Printed in a function using a variable...
abcde#ghij#lmnopqrstuvwxyz0123456789 ...passed using original assignment
abcde#ghij#lmnopqrstuvwxyz0123456789 ...passed using pointer '*ptr2myS
abcde#ghij#lmno#qrstuvwxyz0123456789 ...passed using pointer '*ptr2myS and modified

Pointers to an integer array...
...int *B = A + 5; = 5
...int C = *B + 10; = 15
...*B = 21; = 21

Pointers into a two dimensional array...
abcdefg <- Printed as myData[3]
d <- Printed as myData[3][3]
e <- Printed as *( myData[3] + 4 )
abcdefg <- Printed as myDataAsPtr[3]
defg <- Printed as myDataAsPtr[3] + 3
a <- Printed as *myDataAsPtr[3]
t <- Printed as *( myDataAsPtr[1] + 3 )
22222 <- Printed as *ptr after 'const char (*ptr)[10] = (myData) + 2;'

Pointers into a two dimensional array...
abcdefg <- Printed as myData[3]
d <- Printed as myData[3][3]
e <- Printed as *( myData[3] + 4 )
abcdefg <- Printed as myDataAsPtr[3]
defg <- Printed as myDataAsPtr[3] + 3
a <- Printed as *myDataAsPtr[3]
t <- Printed as *( myDataAsPtr[1] + 3 )
wertyu <- Printed as *ptr1 after 'const char (*ptr1)[10] = (myData) + 2;'

Pointers into an array of structures...
line2 <- Printed via function passed a pointer to a row
selGraph <- Printed via a pointer to a row
selStats
selGraph
line2
 <- Printed via function passed a pointer to a row

Pointers on Pointers in Arduino Land

My degree in CS dates back a long long ways...in fact so far back that "C" was only just starting to gain in popularity and Unix System V had just come out of Bell Labs into the university community.   Pointers made my head hurt then and they still do today!   The following code defines and then passes an array of structures to a function.  It took me forever, and a lot of help from the Arduino forum, to get this to work to this point...but...I have a lot of learning still to do.

struct F4Utft {
    String name;
    int x;
    int y;
    String formatName;
};

static struct F4Utft m01 = {"selStats", 15, 195, "BigFontS"};
static struct F4Utft m02 = {"selGraph", 215, 195, "BigFontS"};
static struct F4Utft m03 = {"line2", 15, 185, "None"};

struct F4Utft *Def_Menu[] = {&m01, &m02, &m03};

void setup() {
    // For debugging messages
    Serial.begin(57600);

    Serial.println("--->" + Def_Menu[1]->formatName);
    Serial.println("--->" + String(test(Def_Menu)));
}

void loop() {
    while (true) {

    }
}

String test(struct F4Utft** menu) {
    return(menu[1]->formatName);
}

The initializer gave me fits and there is another format (single statement with a structure declaration before each entry) that I liked better could not get to work...but this does so I am happy and can move on!

My next challenge is to try and understand why I can not seem to put this resultant structure into PROGMEM without it killing my script.......arghhhhh!

Arduino Convert Long to Char Array and Back

I wanted to log data from an Arduino to an SD Card in the most space and time efficient manner possible.   To me this meant storing a four byte long in four bytes.  Bit shifting has always made my head hurt so I turned to Google.   I found the below (in a  couple places) but it did not seem to work:

unsigned long int longInt = 1234567890;
    unsigned char byteArray[4];
              
    // convert from an unsigned long int to a 4-byte array
    byteArray[0] = (int)((longInt >> 24) & 0xFF) ;
    byteArray[1] = (int)((longInt >> 16) & 0xFF) ;
    byteArray[2] = (int)((longInt >> 8) & 0XFF);
    byteArray[3] = (int)((longInt & 0XFF));
 
    unsigned long int anotherLongInt;
 

    anotherLongInt = ( (byteArray[0] << 24) 
                   + (byteArray[1] << 16) 
                   + (byteArray[2] << 8) 
                   + (byteArray[3] ) );

At least until AWOL on the Arduino forum suggested the below:

anotherLongInt = ( ((unsigned long)byteArray[0] << 24) 
                   + ((unsigned long)byteArray[1] << 16) 
                   + ((unsigned long)byteArray[2] << 8) 
                   + ((unsigned long)byteArray[3] ) )

Eureka!  I can now store two longs in eight bytes with a couple of distinct advantages:

1. Result is fixed size which means I can seek into my file looking for specific entries
2. Size on disk would generally be less than storing a string representation (unless you are only doing tiny numbers) and particularly if you wanted to do one of fixed size.
3. Speedy string operations is not what you bought the Arduino for in the first place!  Saving 1,000 long to char arrays took a bit more than a second while the string version (even without padding to fixed size) took more than 10 seconds.

 I have no idea why AWOL's suggestion worked but I am happy to move on.

Simple Arduino Serial Communications Example

I had done a little serial work with the Arduino and my 'Bot but wanted to try another example.   In this case I have a sketch that  has the Arduino being the  receiver of a one way ASCII data stream.   The data stream uses two sync chars followed by two characters that will make up a two char integer. 

A real implementation would probably use character 127 but I am using "%" for illustrative purposes.  A real implementation would also recognize that the second of two chars of data could be a 127 and therefore you could see two sync characters where one of them is really data.

%%GF%%HG%34%%4%%%HH
   
First payload       - GF
Second payload      - HG
Garbage collected   - %34
Third payload       - 4%
Fourth payload      - HH

%%GF %%HG %34 %%4% %%HH

Here is a link to the sample code.

Simple Vertical Menu for Arduino LCD Shield

Decided that I needed a real simple menu using the buttons on the Arduino LCD Shield.   I only need a simple menu so am just using the up and down buttons with the select button for making a choice.

Here is the code.   The key working bits are shown by the below snippets.  First define the character arrays that will comprise your menu items.  They need to be no more than fifteen characters long.   Remember to terminate the list with an empty entry!

Note the use of the compile option PROGMEM which moves the menu items to program storage and saves short supply RAM.

// Enable storage of literals in program space rather than RAM
#include <avr/pgmspace.h>
 
    // The menu entries list needs to end with a blank entry and each item
    // can only be 15 characters long:
    //                      [               ]
    PROGMEM char* menu1[]={ "Menu 1, Item 0",
                            "Menu 1, Item 1",
                            "Menu 1, Item 2",
                            "Ascii Table   ",
                            ""};

Then invoke the menu as shown below:

    int selection = menuSelection(menu1);

If I find that I need more functionality I may add some to this routine later but at the moment all I need is a simple vertical menu.

Gertboard - What For?

With a collection of four RPi's already in hand (five if you count the fried one) it seemed only natural that I add a Gertboard.   In reading about it I am not sure why I would want a Gertboard as opposed to buying a less expensive Arduino and then integrating it with the RPi but I am interested enough to buy one to play with for a while.

So there it is.  I have downloaded my first sketch to the onboard Arduino and am not sure what to do next!   Getting it up and running was pretty easy thanks to Gordon Henderson and these posts re configuring the Gertboard for use of the Arduino.

The one thing that did cause some momentary angst was solved when I read some comments on Gordon's blog regarding a jumper on the Getboard.   Namely the one that bridges power from the RPi to the Gertboard!   The one in the Gertboard manual that is decribed as such: "There is one jumper that should be in place at all times on the board: the one connecting pins 1 and 2 in header J7".   Now, if I had only read the manual.

I have to ask though.   If it should be in place at all times...why is it a jumper?   Or why not have the board come with one already in place?

Arduino and Memory

To date I have not done very complicated work on the Arduino itself as my interface library allows me to offload any heavy duty processing to the host RPi.   I am now working on something that would run solely on the Arduino and am being forced to come to terms with the limitations of a micro controller environment.

• Program logic has to fit within the Arduino's flash memory which static (burned when you sketch is uploaded)
• There is EEProm memory available for sketches to use to store things that are needed across sessions.   This is in very short supply but in a microcontroller environment how much should you need anyway?
• Most importantly, however, is the random access memory, or RAM, needed for a program during it's execution.   This is in relatively short supply especially if you have any desire to use arrays.

Here is a table from the Arduino Playground that shows the constraints that must be observed:

	ATMega168	ATMega328P	ATmega1280	ATmega2560
Flash (1 Kbyte used for bootloader)	16 KBytes	32 KBytes	128 KBytes	256 KBytes
SRAM	1024 bytes	2048 bytes	8 KBytes	8 KBytes
EEPROM	512 bytes	1024 bytes	4 KBytes	4 KBytes

My development environment is an Uno (ATMega168) so it is pretty constrained.  In a test case a sketch with two integer arrays (a voltage reading and a matching time stamp) maxed out at 400 entries.  On my ATmega1280 that is driving the 'Bot the top limit was 1400 entries.   I have a Gert Board on order and it uses the one in between (ATMega328).

This was not a realistic case as any reasonably complicated program is going to use more RAM than my example.   There are ways of maximizing RAM.   One is simply to be cautious with everything and especially with buffers that can be reused and character strings that can be abbreviated.   Static information can also be stored in program memory using PROGMEM (here is an example).

One last comment on the Arduino and Memory.   How did I first realize that I was having problems with memory management???   Sketches either did not run at all or when they ran they were corrupted and doing REALLY weird things.  Drove me a little bit banana's.