A paper based on this project work was compiled and presented at the International Conference on Computer Vision and Robotics 2012 held at Bhubaneswar, India. The paper was selected as the best paper in the conference and Nandan Banerjee (presenter) was awarded with the IRNet Young Investigator Award. The paper was published in the “Proceedings of International Conference on Computational Vision and Robotics, 19th of August, 2012, pp. 20 – 25”.

Click here for the paper.

VIDEOS

Introduction

This project introduces Buttercup, a 5 DOF robotic manipulator system based on the Stanford model that is designed to autonomously play the game of chess in real time against human or robotic opponents. The Buttercup Chess Robot includes a Logitech HD webcam, a low cost custom made manipulator, algorithms for the efficient detection of the chess pieces on the chessboard and a robust control mechanism for the movement and control of the manipulator. Compared to prior work on robotic systems playing chess with specifically instrumented chess boards and/or pieces, the Buttercup chess robot is not confined to just playing chess but it can also be used as a standard manipulator and it can be controlled effectively using its own set of control commands.

I. Chess Move Detection

The chess pieces on the chessboard were detected using standard image processing techniques.

A. Prerequisites

For the image processing part in which the main objective was to find the current orientation of the board given the previous orientation, the following were used:

• OpenCV library (used along with qt)
• Logitech C270 webcam
• Intel Core2Duo processor running at 1200 MHz

B. Methodology

A webcam was positioned directly over the chessboard and the resolution was set at 640×480. The OpenCV image processing library was used to capture the screenshots and apply the image processing procedures. Following factors were identified as having a major effect on the performance on the image processing techniques:

• Texture/pattern of the chessboard
• Colour of the squares/pieces
• Lighting conditions

Therefore, the colours of the squares on the chessboard were switched to red and faint lemon yellow to offer a better contrast than the earlier black-and-white chessboard with black-and-white chess pieces on it. The most suitable light source was found to be a diffused light source on to the surface of the board. When a pointed light source from the side was found to have been used, shadows and glares thus created were sometimes misinterpreted as pieces.

C. Image processing algorithm

1. Set the region of interest – A screenshot is taken using the webcam and the region of interest is set manually. The region of interest is the part of the image where all the image processing procedures are to be applied.

2. Detect the corners and intersection points – Using the Shi-Tomasi corner detection algorithm, which is just a slight modification of the Harris corner detection algorithm, the corners of the chess squares are found out.

3. Find the top-leftmost, top-rightmost, bottom-leftmost, bottom-rightmost corners – The four endmost corners of the chessboard are then determined using normal pixel position calculations.

4. Compare the probably and the actual corners – The probable corners are found out by dividing the difference of the corner points by eight, thereby finding out the corner points of all the chess squares. Then the probable corners are compared with the actual corners and if the difference between their values suggests that they are neighbouring points, then the probable corner is replaced by the actual corner. In this way, the corners for all the 81 chess squares are thus found out. This method is used to remove the corners that are detected on the chess pieces. Also, because of bad ambient lighting conditions, sometimes a few of the corners are not detected. In that case, the probable corner acts as the corner of the chess square. Therefore, to avoid these two errors, this comparison and assertion procedure is performed.

5. Check for the square occupancy using the Canny edge detection algorithm – The Canny edge detection algorithm is then used to detect the edges of the chess pieces on the chessboard. From the already found values for the four coordinates of all the 64 chess squares, all the chess squares are chosen. The dilation operation is then applied on the edge detected binary image on each of the chess squares, and the number of white pixels is then compared with a designated threshold value. If it exceeds the threshold value, then the chess square is occupied otherwise empty.

6. Determination of the color of the chess piece – The image is thresholded. The threshold is chosen according to the ambient lighting conditions and the texture of the chessboard. After thresholding, the black pieces stay black whereas everything else becomes white. Then, the erosion operation is applied on the thresholded image on each of the chess squares, and the number of black pixels is then compared with a designated threshold value. If the value exceeds this threshold value, then the chess piece is black otherwise white if the square is not otherwise empty. Fig 3. (a) Showing a chessboard orientation with two black pieces and one white piece. (b) After thresholding, only the black pieces are deemed as objects and everything else is ignored.

D. Chess Move Detection

After the orientation of the chessboard has been successfully detected, the individual identities of every chess square is known, i.e. if the chess square is empty or not, and if not empty, then the color of the chess piece present in it. An eight by eight matrix is then constructed consisting of three values [0, 1 and 2]. ‘0’ signifies that the square is empty. ‘1’ signifies that the square has a white piece in it and a ‘2’ signifies that the square has a black piece in it. The image processing algorithm described above is applied on the entire chessboard and all the 64 values of the squares are hence determined and stored in the matrix. Then the matrix is compared to a previously recorded matrix of a previous image and the changes are noted. From these changes, it can be ascertained as to what move has been given by the opponent. After this move is realized, it is then fed as the input to the GNU Chess engine described in the next segment. The vision system could be thwarted simply by swapping chessmen so it is being assumed that there won’t be any foul play on the user’s part.

II. GNU Chess Interface

GNU Chess is a computer program which plays a full game of chess against a human or other computer program. The goal of GNU Chess is to serve as a basis for research. It has been used in numerous research contexts. It is free software, licensed under the terms of the GNU General Public License and is maintained by collaborating developers. It’s one of the oldest computer chess programs for Unix based computers and one of the earliest available with full source code.

On basic, current computer chess architecture, GNU Chess plays at senior master/weak international master strength (2500 ELO on simple hardware – Intel Core 2 Duo), without parallel processing, according to the IQ6 test suite.

In this project, the GNU Chess version 6.0.1 has been used to determine the next move by the Chess Robot. The GNU Chess engine has been interfaced in qt using the QProcess class. GNU Chess is started in the Xboard mode using the -x argument. The moves are sent to and received from the GNU Chess engine using standard coordinate algebraic notation through the standard input and output streams.

As mentioned in the previous section, the move ascertained after applying the image processing techniques is fed into the input stream of the GNU Chess engine which is interfaced using the QProcess class as already mentioned. The GNU Chess engine then figures out the appropriate response and it is then relayed to the robot control module which then automates the robot.

III. Mechanical Aspects of the serial manipulator

The various mechanical aspects of the serial manipulator can be categorized as follows:

• Mechanism development and actuation
• Stress and force analysis
• CAD Design
• Material selection
• Manufacturing of the serial manipulator

A. Mechanism development and actuation

The manipulator is intended to perform a pick and place task. A simple model for a serial manipulator for this purpose is a 2PR (two prismatic and one rotary joint) serial manipulator. The base link carries the first link and there exists a rotary joint between them. Between the first and second and between the second and third link there are prismatic joints. While the prismatic joints help in easily locating the end-effector of the serial manipulator in the desired point in the work space, the rotary joint increase the total work space and provides more maneuverability.

The rotary joint between the base and first link is established by fixing a high torque motor to the base and mounting the first link on the motor shaft. The actuation of the prismatic joints can be achieved by using linear motors. But due to its high cost, an efficient and comparatively cheaper screw joint mechanism actuated by a dc motor is used to achieve the linear motion.

B. Stress and force analysis

The arrangement of the last link makes it act like a cantilever beam. The forces acting on the last link are same as that on a cantilever beam. The length of the last link is decided by the farthest distance needed by the end effector to traverse. The links are linearly actuated by the screw joint motion for which they need to be cylindrical and shape and should carry screw threads. The pitch of the screw thread is decided by the velocity of the actuating motor and the required linear velocity. This cantilever beam type link is mounted on another link which too carries another prismatic joint which is actuated by a dc motor using the screw mechanism. The diameter of this cylindrical link is decide by calculated the cumulative effect of the bending moment and torsion of the cantilever beam on it and also the torsion due to its own inertia. This cylindrical link which supports the cantilever beam type link is mounted on a motor’s shaft fixed to the base.

The stress and force analysis helps to decide the actuation torque required for the motors to drive these joints. The cantilever beam type link required a motor with comparative larger shaft length. The base motor needed a high torque motor as it had to bear the weight of the links mounted on it. the problem of increased torque due to the inertia/weight of the links on the base is resolved by mounting the links via bearing on the base and then actuating the link from the top end instead of mounting the whole link arrangement on the shaft of the motor and revolving it on the shaft.

C. CAD Design

The serial manipulator consists of various joints and links. The actuating joints need to be placed properly for dynamic balancing. The shape of thread, its position on the cylindrical links, the position of the motors, the mounting clamps, the alignment of the motor shaft etc. are some of the various points that are to be kept in mind while designing the manipulator. Most of the time the stress and force analysis is just not enough for obtaining the correct design which is free from all types of faults. Many decisions regarding alignments and positioning of various parts need to be decided in a heuristic manner. Computer aided designing packages help one to develop such designs. CATIA is one such package and it has been used to develop the design of this serial manipulator. The CAD designing approach helps us to perceive the implemented mechanisms in a better way. The probable areas of fault can be figured out while studying a CAD model and it aids in making fine changes in the intricate details of the model.

D. Material Selection

Once the mechanism is decided and the resulting force and torques are determined, the materials are selected for individual links of the manipulator. The base has the support the whole weight the other links and the actuating joints. Cast iron suits the requirements for such a base. It is capable of absorbing shocks. The base made of cast iron reduces the vibration of the cantilever beam type link and thus helps in precise positioning of the end effector. The first link on the manipulator has to support the cantilever type link and thus have to act like a strong column. The hollow cylindrical shape is suitable for this purpose. The thickness is limited by the machining stresses induced while making the threads on the column. Mild steel is suitable for making the column. The cantilever type link too is a hollow cylinder with low weight. Aluminium is a suitable material for manufacturing it.

E. Manufacturing of the serial manipulator

While manufacturing the manipulator certain issues were kept in mind. The selection of proper material is just not enough to dampen the inherent vibration produced in the links. The proper positioning of the actuating mechanism, the alignment of the links and mounting of the motors also play a crucial role in dampening this vibration and thus helps to attain dynamic stability and consequently better performance. The link which acts as the supporting beam for the cantilever beam type end link is mounted on a bearing on the base. The bearing is welded to the base and a protrusion from the supporting link is press fit in to the bearing. The cantilever type link has a cuboidal end with a cylindrical hole carrying inner groves matching the threads on the supporting beam. The cuboidal end has a protrusion which runs in a straight grove on a bar mounted on the base. This helps to attain a prismatic joint’s to and fro motion with the help of the screw joint motion. Similarly the cantilever type link is fitted with a cuboidal block carrying a cylindrical hole with grooved inner threads that matches the threads on the cantilever type link and also possesses a protruded edge that runs on a grooved overhead bar. This cuboidal block that moves on the cantilever beam type link carries the end effector.

IV. Gripper, Structure and Movement of the manipulator

The manipulator has three degrees of freedom. The manipulator consists of 4 links and 3 joints. There are base linked can be termed as link 0 and the consecutive links can be numbered/named successively as link 1, 2 and 3. Joint 1 exists between link 0 and 1 and is a revolute joint. There exists two screw joints, one between link 1 and 2 and the other between link 2 and 3. The screw joints are good replacement for prismatic joints. The linear motors used for making a prismatic joint between to links is far more costly then the combined cost of machining threads (outer and inner) on links and that of a servo motor on which the links will be mounted after machining to form the screw joint.

The above diagram is the sketch of the serial manipulator and shows the joints and links. The linear motion achieved by the screw joint movement is also shown. The table/stand shows the base. Vertical column with the grooved mounting consists link one. The horizontal cantilever beam can be considered the link two. The end effector is the gripper which is in form of two spur gears with the gripping plates mounted on them. The revolute joint between the base and the first link is also shown.

Thus, the joints and links are as follows:

• Link 0 – Base
• Link 1 – Column
• Link 2 – Cantilever
• Link 3 – End effector/gripper
• Joint 1 – Revolute joint
• Joint 2 – Screw (vertical) joint
• Joint 3 – Screw (horizontal) joint

Joint 2 is responsible for vertical lift necessary for picking up the chess piece and go for the necessary movement once of has been grabbed properly by the gripper. Once the image processing of the grabbed images of the board and chess pieces on it are done, a move for the serial manipulator is decide and the commands for the necessary movements of the joints for execution of the decision are transferred to the microcontroller. It thus drives the servo motors in compliance with the commands generated. The initial task for the manipulator is always to reach the desired coordinates where it has to pick up the desired chess piece. The picking up of the chess piece is the next task. Then it needs to take the chess piece to the destined position and then place it back on the board at the new position.

Grabbing the chess piece, picking it up from the board and placing it back are similar kind of actions and can be grouped in a set. This set requires lowering of the gripper and holding the chess piece. Then it needs to be pulled up without disturbing the position of neighbouring chess pieces. This set of actions requires the movement of the grippers and joint 2.

The movement of joint 2 provides the lift necessary for picking and placing the pieces. The height of the lift should be such that while moving the grabbed pieces from one position to another, then it should be hit other pieces and thus should not disturb their position. This can be ensured by estimating the position from where the chess piece desired to be moved is grabbed by the gripper and the average height of the chess pieces above which this particular grabbed chess piece will pass during while traversing its path from its initial to the final position across the board.

For gripping the chess pieces spur gears are used on which gripping jaws are mounted. Actuation of the gears leads to the gripping action. The angle to which this gripping jaws can spread is decided by the angle by which this actuating spur gears (and in turn the shaft of the motor driving the spur gear arrangement) turn which is constrained by the distance r_L of the piece from the neighbouring chess pieces.

Moving the gripper to the initial position of the chess piece and taking it to the desired destination is another set. This set of task requires knowledge of the coordinates of the place where the chess piece is positioned on the chess board in reference to suitable axes.

If the manipulator starts with the end effector located at O (0,0) for reaching destination with coordinates P(r, θ), then the Joint 1 will move by angle θ and then joint 3 will rotate by a certain degree so that the end effector travels the radial distance ‘r’.

V. Robot Feedback and Control

As the name suggests, robot feedback refers to the sensory data from every joint of the manipulator which is used to effectively control the movement and orientation of the manipulator. The interface between the robot and the computer is via a control circuit with an AVR microcontroller at its centre. The microcontroller connects to the computer using the serial port via an USB to serial port converter. The manipulator motors are driven using high power motor drivers.

A. Sensory Data

Various sensors have been used to determine the instantaneous position of the manipulator. A 1 turn, 2 KO potentiometer was used to determine the angle of rotation of the manipulator about its base axis. It is referenced with a potential of 5 V and the value from the potentiometer is read in through the ADC with a resolution of 10 bits sampled at 172.8 kHz.

Two optical encoder wheels with IR sensors were used to determine the linear movement of the vertical and the horizontal screw drive mechanisms. The optical encoder wheel has ten protruding teeth with an IR sensor placed around it. So, for a count of 10 by the IR sensor, the screw has a complete rotation about its axis and hence moves the distance equal to the pitch of the screw. The minimum distance that can be determined by this movement is 1/10th of the pitch of the screw.

B. Control Circuit

The microcontroller, sensors, motor drivers and the serial port driver are the main elements of the control circuit. The microcontroller is an ATMEL ATMega32 microcontroller running at 11.0592 MHz. The output from the potentiometer (the rotation sensor) was input into the ADC2 pin and the output from the horizontal screw mechanism encoder was first compared using a comparator (LM324) and then input to the microcontroller. A 20 × 4 LCD (HD44780 driver) was connected to the microcontroller for debugging purposes. The motor drivers were built using complementary MOSFET transistors (IRF540 and IRF9540) and controlled using the microcontroller pins. The serial port driver was made using Hex Schmitt inverters (74HCT14) for Rx and Tx (UART) and an USB to serial port adapter. Everything was soldered on Vero boards. A 220 V AC to 12 V 10 A DC SMPS was used to effectively regulate the voltage output provided to the control circuit.

C. Serial Communication and Command Set

The computer connects to the robot through a serial connection to the microcontroller. The serial connection has a baud rate of 115.2 kbps. The manipulator is controlled through a set of commands which controls all the movements of the actuation motors precisely.

The command set includes the following commands:

• MTNR – Rotate the manipulator to the right
• MTNL – Rotate the manipulator to the left
• MTNVXXXX – Rotate the manipulator to the position XXXX
• MLNF – Move the Linear Screw forward
• MLNB – Move the Linear Screw backward
• MLNVXXXX – Move the Linear Screw to the position XXXX
• GPOP – Open the gripper
• GPCL – Close the gripper
• STOP – Stop all Motor movement
• BRAKE – Brake all motors
• RDERM – Read from the EEPROM
• WTERM – Write to the EEPROM
• TNSR – Read turn sensor value
• RESET – Reset the linear motion sensor count value
• CNT – Read the linear motion sensor count value

D. Chess piece movement using the manipulator

The locations of the various chess squares (in algebraic coordinate notation) on the chessboard are calculated, and then stored in a table in the program which communicates with the robot via the serial interface. When the output from the GNU Chess engine is directly fed to the program, it then sends a series of commands to the microcontroller specifying the angle of rotation and the position of the screws for the source and the destination position. The manipulator then picks up the desired piece from the source location and then places it to its respective destination position.

Implementation Details

The first module, i.e. the image processing module has been completed successfully. The program is able to determine the moves made by the human opponent properly. The only issues which hampered the detection of the corners of the chessboard effectively using image processing methods were bad lighting conditions and skewed orientation of the chessboard under the webcam. It was found that the best image processing results were attained when the chessboard was oriented in parallel to the viewing angle of the camera. Choosing a threshold value of 50 seemed to work in almost 99% of the cases where the ambient light came from an overhead white fluorescent light. After every move the opponent makes, the program is able to determine the movement of the piece between the old and new position.

The second module, i.e. the GNU Chess Interface module has also been successfully completed. The program is able to send and receive moves by running GNU Chess in background and using its standard input and output stream to effectively communicate with it and extract the moves given by the engine. It is running with no errors.

The third module, i.e. the control circuit and the microcontroller code was also completed in due time. The microcontroller circuit is being able to effectively control the movement of the robot motors and is being able to pick up, put down and move the pieces from one point to the other. The end effector is able to move to specified distances through the computer commands using the serial port properly. The turn motor is working properly and the angle of rotation is being ascertained correctly by the potentiometer reading. The specified angle movement for the turn motor is yet to be completed. The motor drivers are working without any problems and there has been no malfunction in the circuit till now. The optical encoders and the potentiometers have been decoupled properly and are not giving any erroneous readings. The automation of the chess move using a specified set of commands is yet to be coded.

The fourth module, i.e. the manipulator is almost completed. All the motors are working perfectly fine with the help of respective motor drivers. The screw mechanism used for the links are serving their purposes right by giving more precision and accuracy to pick and place chess pieces. Only problem that arises is with the vertical motion of manipulator used in case for picking the pieces as 10rpm seems very slow henceforth stalling this process. Also there is a little bend in the arm which creates a little difficulty while placing the pieces at far corners.

References

PAPERS

• G.D.Illeperuma, Unversity of Colombo. ”Using Image Processing Techniques to Automate Chess Game Recording” Proceedings, 27(2011) 76-83
• Cynthia Matuszek, Brain Mayton, Roberto Aimi, Marc Peter Deisenroth “Gambit : A Robust chess-Playing Robotics System”

BOOKS

• Learning Open CV Computer Vision with Open CV Library – Gary Bradski and Adrian Kaehler
• Digital Image Processing – Rafael C. Gonzalez and Richard E. Woods
• The book of QT4 , the art of building Qt applications – Daniel Molkentin
• Designing Embedded Hardware – John Catsoulis

WEB REFERENCES

• CircuitLab – The free Circuit simulator
• AVR Fuses Guide
• Linearizing Sensor readings guide
• AVR Freaks
• Programming AVR Fuse Bits at ScienceProg

DATASHEETS

• ATMEL ATMega32
• IRF540 n-channel MOSFET
• IRF9540 p-channel MOSFET
• HD44780 controller for the LCD

Project members

• Nandan Banerjee (Computer Science and Engg. Dept., NIT Durgapur)
  Electronics, Programming and Concept
• Debal Saha (Mechanical Engg. Dept., NIT Durgapur)
  Mechanical Design and Implementation
• Atikant Singh (Computer Science and Engg. Dept., NIT Durgapur)
  Documentation
• Professor Gautam Sanyal (Computer Science and Engg. Dept., NIT Durgapur)
  Guidance and Mentorship