Semaine 1 (Week 1) lundi

~ lundi ~

Finished reading Ting-Fan's paper [1].
Noted down some plans for our project on the way. Still developing plans.

• First, need to clean data, so that the problem can become simpler. Data should have no clutter and have distinguishable plate color from food, ideally all white.
• In order to clean data, first need to run some samples so we can know how clean the data needs to be. So need to run the segmentation software.
• Lighting conditions: Ting-Fan had problems with illumination because the heat lamp and fluorescent light actually emitted different colors, that can be hard to adjust to, but for simple darker and lighter brightnesses, it might be doable. I wonder if there is a way to adjust the brightness of all input images to a threshold, so that they all have the same level of brightness, and thus similar color distributions on the histogram? That's something to Google and see
• Color spaces: Ting-Fan didn't use HSV at all. I wonder why? If brightness can be tuened, HSV may be useful, so the hue can be used, and the value can all be tuned to a similar level, maybe?
• Textons: Need to take a look at the textons paper from Ting-Fan's references and see what exactly it is and how it works. Seems like it works like a combination of color and texture, and it works better than color and gradient combined.

Got JSEG to compile and run in Linux [2].

• Download the source of jseg and jpeg-b6 from the URL. Unzip anywhere.
• To set up jpeg-b6 library, use dos2unix command to convert the jseg-b6/configure file to UNIX format, run ./configure to generate the jconfig.h file.
• Change the make file in JSEG directory to point to the jpeg-b6 file path for -L and -I CFLAGS. Then run make in JSEG directory. It should run and create "segdist" executable.
• Run segdist executable on images.
• First image tried with JSEG (original image came from the web):

• Image is ran with default parameters, so very over-segmented. Will need to tune later and find out if this software is suitable.

/* Never mind this doesn't make sense. I DO need to segment out some parts in order to figure out the ingredient!

In using JSEG, the difference between my goal and Ting-Fan's goal is that while he had multiple dishes on a plate and thus needed to segment each dish out from a single plate, I only want to segment out the entire food area from the plate. It could be difficult to segment out a continuous area for dishes with discrete objects, such as sushi. For others, such as spaghetti, it should be easier.

*/

As for the rationale of "segment then classify" or "classify then segment," my choice is clearly the former, because the nature of the project is to figure out the different ingredients on a plate first, and then use the combination of ingredients to classify the plate to a cuisine.

Thus I will be using JSEG to segment out the different ingredients, try to identify each one, and then from the ingredients, classify to a cuisine. So it's a two-step process:

1. Identify ingredients using some kind of features - color, texture, shape, etc.
2. Classify cuisine using a second set of features (, using the identified ingredients as input?).

Final outputs would be country / region names where the cuisine is from, e.g. Japanese, Italian, Mexican, Indian, etc. For each image, the output should be a set of probabilities (higher than a set threshold) of the cuisine, for example, an output could look like:

67.5 Japanese

23.8 Chinese

(7.2 Mexican, percentage lower than threshold of 15% and ommited from output)

Next step TODO:

• Tune JSEG and see how well it works to segment out the entire food area from the plate, instead of over-segmenting
• Further develop project plans and slim down things, using notes from reading Ting-Fan's paper as a reference
• Brain wrinkling: brightness tuning, textons
• Could look up if useful: LoG, DoG, whatever else in paper

[1] Wu, Ting-Fan. Cafeteria Vision: Identification and Amount Measurement of Foods in a Plate. http://cseweb.ucsd.edu/classes/fa06/cse252c/projects/twu.pdf

[2] Y. Deng and b. s. Manjunath. Unsupervised segmentation of color-texture regions in images and
video. IEEE Trans. Pattern Anal. Mach. Intell., 23(8):800–810, 2001.
 http://vision.ece.ucsb.edu/segmentation/jseg/software/

Semaine 0 (Week 0)

i. OpenCV installation tested in Linux.

Got really close to getting all the tests that came with installation working. First it couldn't find the extra test data, so grabbed opencv_extra from the latest repository, now all tests from the 3 shell scripts worked, except z-highgui (and beyond if there were more).

Really wanted to see all the tests pass, so installed GTK2+, reran cmake, reinstalled OpenCV, rebuilt with examples, recopied test files etc. Now the tests suddenly can't see opencv_extra, and it gets stuck at optflow-estimate-rigid for 10+ hours (no, I didn't sit there for 10 hours)... doesn't even get to z-highgui anymore. Should have stopped the first time...

Well since it worked the first round, I'll assume it works....


ii. Configured pkg-config and libraries, set up Makefile and a test C++ file doing simple matrices operations. Compiled and ran. Yay.

Did have to add opencv.conf to /etc/ld.so.conf.d/, and ran /sbin/ldconfig as root. Otherwise it couldn't find the library at run time.


iii. TODO next:

• Find and run the segmentation code Tingfan suggested and see what it does. Don't even remember, was it in Matlab?
• Downloaded JSEG, from UCSB, has makefile for Linux. Great.
• Take a look at Tingfan's code and paper and see how exactly he did things.

Bienvenue.

Tentative Project Title (tentative i.e. probably going to stay if can't come up with a better one):
Identifying the Cuisine of a Plate of Food

Hopefully we can actually get this working by the end of the universe.

Proposal can be accessed HERE.

Progress so far:

• Raw database more or less collected. Got Flickr query code working and downloaded several hundreds of food images in a few categories. Can download more once algorithm gets going and have more concrete idea of what is needed
• Got suggestions from Tingfan Wu for an off-the-shelf segmentation program. About to try out on some images once get OpenCV working.
• To do next: run on a few images, see what we need to do to clean the database.
• Still need to trial run Tingfan's project to see what it does and shed some light on mine
• Installed OpenCV in Linux, haven't figured out how to set up a project with it yet. Need to run their example files and set up a project and Makefile to link up all the libs etc. See how many tries we'll need to get it running (#1 thing they should teach us in undergrad - how to write shell scripts, #2 thing - how to write sophisticated Makefiles).