My brother Bijan recently asked me why I shoot RAW whenever possible. Sometime in the near future, I plan to explain that here in detail with accompanying example photos. For now, I'd like to explain two concerns about the availability of the RAW format in current digital cameras.

First of all, what is RAW? Most digital photographers shoot in JPEG, a lossy compression format. When we shoot JPEG, we choose aperture, ISO, shutter speed, focal length, composition, and perspective (ie, where we stand) prior to taking the shot. The camera takes the information recorded by the sensor (ie, the light gathered by the sensor) and, based on our pre-selected settings, applies adjustments including white balance, sharpening, noise reduction, contrast curves, and saturation; it then turns the image into a smaller file by eliminating much of the "extra" information that is less necessary for the photo to have the intended final look. This resulting compressed file format is JPEG. JPEG is a great format for people who don't want to do this processing by hand or don't know how to process, though I should note that many desktop applications can process for you as well as or better than your camera can. Being smaller, JPEG files keep storage costs and camera buffer needs lower. When we shoot RAW, we make the same decisions about aperture, ISO, shutter speed, focal length, composition, and perspective prior to taking the shot. After the shot, all information gathered from light striking the sensor while the shutter is open is stored for us to process later. In other words, we save the "raw" data. The advantage of this is that we get to make a lot of decisions after the shot. For an extreme and obvious example, let's say we shoot in JPEG mode and take a black and white photo with in camera settings for contrast and noise reduction set to "high." We get home and decide we wish the image were in color. Too bad. Wish the image had more detail? No problem, just decrease the noise reduction. Oops the camera's processing engine permanently through away that detail while it was trying to keep noise down. Blow some highlights? If only the contrast setting hadn't been quite so high... If only I had shot RAW. All of those decisions could have been made later, and I'd have my color, noisy but detailed, well-exposed shot. Many advantages of RAW are far more subtle. There are some other advantages of JPEG as well. Both of those issues will have to wait for another installment. Now back to my concerns about RAW.

RAW is being dropped from advanced compact cameras at an alarming rate. Canon and Nikon have abandoned it. Many of us think that this has been done in order to steer enthusiasts to DSLRs where we will spend thousands on lenses. Others feel that RAW files are being kept from us so we won't see how noisy these tiny, pixel-stuffed sensors are before in-camera noise reduction is applied. I also wonder whether companies are leaving out RAW in order to save money on hardware costs. RAW file capture and storage typically necessitates a larger memory buffer for reasonable continuous shooting rates. Whatever the reason, it is important to understand that RAW is not being kept from us because the cameras don't "make" a RAW file. When light hits the sensor, RAW data is recorded. They just need to fork it over. This point is illustrated by a Russian hack which makes RAW files available on Canon digital cameras with DIGIC II processors from which Canon had disabled end user access to RAW data. Many held out hope that Canon would keep RAW in their flagship prosumer G-series. No such luck. Canon's Chuck Westfall explained that this omission was a consequence of the pixel pitch being so small on the G7. By shoving 10 million pixels onto a 1/1.8" sensor, Canon made the G7 pixels so small and noisy that no RAW processor can eek more out of them than Canon's on-board DIGIC III processor can. I won't argue against the implication that little pixels are no good. Two of my cameras (Leica D2 and Canon 5D) have a large pixel pitch, and I think that when one compares them to cameras of the same technology generation, one can really tell the difference. Larger pixels tend to capture a greater dynamic range as well as to limit the amount of noise. I'm getting off topic though. Users of the tiny-pixelled G7 would still benefit from access to their RAW files. First of all, there is nothing magic about the DIGIC III processor. It is not more powerful than my Mac Pro, and it's processing algorithms will be dated as fast as Canon can produce the DIGIC IV. Therefore, to imply that the G7 can get as much or more out of the RAW files than we can is downright silly. Second, RAW is not only about how much detail or how little noise we can extract. It is about flexibility. When I shoot ISO 400 with the G7, the camera applies a certain amount of noise reduction (NR) to create a balance between detail preserved (NR destroys detail) and noise eliminated. Each camera's JPEG engine does this differently. For example, Panasonic's Venus III engine has been widely criticized for striking a balance that favors excessive noise reduction with resulting smearing of details. NR also creates the Fuji compact camera "watercolor effect" at very high ISO. Getting back to the G7, what if I want to preserve more detail at the expense of greater than usual noise for ISO 400? Often with black and white conversion, I choose such a compromise since color noise is not a factor in B&W. RAW would give me that option. Mr. Westfall's explanation is not good enough for me.

Now for the main reason I am posting this. It has recently come to my attention that, even in the cameras which still offer RAW, our RAW may not be as RAW as we think it is. I was reading about image quality comparisons between the Panasonic DMC-LX1 (branded by Leica as the D-LUX 2) and the DMC-LX2 (branded by Leica as the D-LUX 3). These two cameras are interesting to me because they are compact, offer RAW, offer a choice of 16:9, 3:2, and 4:3 aspect ratios, and have a wide angle lens (starting at 28mm when shooting in wide aspect format). The LX2 has more pixels, which as noted above is not always a good thing when sensor size limited. However, I was intrigued by the fact that LX2 RAW files, at typically between 19-20MB each, are considerably larger than my 5D RAW files. This prompted me to do a little digging. One reason for the LX2 RAW files to be so large is that they contain 16-bit color as opposed to 12-bit as in most RAW files. The other thing I found out about LX2 RAW files is more ominous. Panasonic appears to have applied a considerable amount of NR to the RAW files themselves! This reduces the amount of shadow noise considerably but also reduces the low contrast detail available in those shadows. Björn (viztyger) on DPReview was kind enough to make available his LX2 RAW analyses in those forums. Here is his post showing identically processed RAW files from the LX1 and LX2. I think the results speak for themselves. It is clear that the LX1 crop contains noise that the LX2 crop does not. Many would find this noise objectionable, and yet they could largely remedy the situation by applying NR software during or after RAW conversion on their computer. On the other hand, the shadow detail which is present in the LX1 crop is irrecoverable in the LX2 photo. No software could restore it.

This raises the question as to whether Panasonic is an evil company that is alone in violating the sanctity of RAW, whether everyone is doing it, or whether I have just been thinking about RAW the wrong way. One thing to consider is that the processing of recorded light data is not limited to a single processing engine such as DIGIC III or Venus III. There is so-called "on-chip" processing on the sensor itself. It has never been clear to me whether to think of this processing as distinct from what the main processing engine does or not. In other words, I wonder whether "on-chip" NR permanently eliminates detail the way other NR does, or alternatively whether it could somehow be improving signal to noise capture at the source. If anyone knows, please tell me. As to whether "everyone is doing it," it is becoming clear to be that the answer is probably "yes" to one extent or another. In a 2006 thread in the Rob Galbraith forums, Canon's Chuck Westfall had the following to say: "...This dates back to previous discussions on this board (circa 11/2003) where I stated that .CRW files from early EOS Digital SLRs (D30, D60, 10D, Digital Rebel/300D) were minimally sharpened in camera. Canon Inc. engineers confirmed this to be true. Conversely, no sharpening of any kind is applied to raw images captured as .TIF (as in EOS-1D, EOS-1Ds) or .CR2 (EOS 5D, 20D, Digital Rebel XT, 1D Mark II, 1D Mark II N, and 1Ds Mark II)." He went on to add, "To the best of my knowledge, there is no softening or smoothing per se on Canon RAW images. However, it's clear that on-chip noise reduction is performed prior to exporting the image data to the camera's image processing system." Although they are rascals for tampering with my RAW, I do respect Canon for having a prominent representative answer the tough questions posed in a public, third party forum. What about the number 2 producer of digital cameras? Jim Cockfield, who informed me of the aforementioned comments from Mr. Westfall, also gave me this link, which suggests that as of 2005 there was evidence that Nikon was modifying RAW files.

While every camera manufacturer's offered RAW is clearly more raw than their in-camera JPEGs, I now understand that not all RAW is equally raw. From now on when I shop for a new camera, I will not only have to ask "Does this camera have RAW?" but also "How RAW is it?"