Word Streak/Boggle With Friends Automation

This project combined the use of Python, OpenCV, and Keras (TensorFlow). After learning about Word Streak, of course the Software engineer in me knew that finding all the words in a grid could be solved with an algorithm. So that was the task I set out to do.

The first step was to create the algorithm of finding the words. I was able to find a plain text list of about 60,000 words. Some of the words in it aren't considered words in the game which results in it rejecting some of them, but that's ok. The majority of the words work.

I start at each letter in the grid and recursively check all the letters around the letters. I have some simple optimizations like the dictionary is split up in a map by the first letters. Another is if the combo of letters being checked is not found as the start of any of the words in the dictionary, it quits checking it.

As it finds words, it adds them to an all encompassing list. The score for the word is also calculated and it also takes into consideration the letter/word multipliers that are in the word.

Python libraries used are:

mss for the screen capture

pynput for the mouse automation

keras for the neural network

numpy to handle arrays and images

opencv to do the image processing

At first it was a manual process of typing in the letters so that it could find all the letters, but I wanted to keep going with some letter detection.

The first step was to extract out each letter from the boggle board. I use OpenCV to find all of the contours in the image. I then filter out anything that isn't 4 edges. Then I determine how square the contour is and if it is the correct size. This almost always leads to only the tiles being selected.

Sometimes a tile might be missed (especially in the case with DW/DL/TW/TL), since I know the basic size of each tiles and the spacing, I can interpolate where a square should be and still extract a tile anyway.

Once I have the letters extracted from the image and processed with thresholds and gaussians, I pass it in to the Neural Net.

The Neural Network part I used Keras and Tensorflow. I've been on kind of a deep learning/machine learning/neural net kick lately. So I thought I'd used that. I trained the net on a set of 1000 images per letter. Each image of the letter is of a different font style. This way when I passed in a picture of a letter extract from the boggle board, it would be able to know what letter it is without having seen the images of the boggle letters.

This is the network I used.

For the Qu case, I had to do the more simple technique (which in this case could probably work for all letters) where I subtract a canned Qu image from it. The sum of the pixel values for the Qu subtraction would be way less than letters that are not Qu.

Once the letters are all found, I then pass it into my original code of finding the words. It sorts it by the highest score first and then begins inputting the words into the game by controlling the mouse.

I have Android running in a Virtual Box VM. That is how the game is running on my computer.

Check out the video of it in action below!!