My brother Bijan recently asked me why I shoot RAW whenever possible. Sometime in the near future, I plan to explain that here in detail with accompanying example photos. For now, I'd like to explain two concerns about the availability of the RAW format in current digital cameras.
First of all, what is RAW? Most digital photographers shoot in JPEG, a lossy compression format. When we shoot JPEG, we choose aperture, ISO, shutter speed, focal length, composition, and perspective (ie, where we stand) prior to taking the shot. The camera takes the information recorded by the sensor (ie, the light gathered by the sensor) and, based on our pre-selected settings, applies adjustments including white balance, sharpening, noise reduction, contrast curves, and saturation; it then turns the image into a smaller file by eliminating much of the "extra" information that is less necessary for the photo to have the intended final look. This resulting compressed file format is JPEG. JPEG is a great format for people who don't want to do this processing by hand or don't know how to process, though I should note that many desktop applications can process for you as well as or better than your camera can. Being smaller, JPEG files keep storage costs and camera buffer needs lower. When we shoot RAW, we make the same decisions about aperture, ISO, shutter speed, focal length, composition, and perspective prior to taking the shot. After the shot, all information gathered from light striking the sensor while the shutter is open is stored for us to process later. In other words, we save the "raw" data. The advantage of this is that we get to make a lot of decisions after the shot. For an extreme and obvious example, let's say we shoot in JPEG mode and take a black and white photo with in camera settings for contrast and noise reduction set to "high." We get home and decide we wish the image were in color. Too bad. Wish the image had more detail? No problem, just decrease the noise reduction. Oops the camera's processing engine permanently through away that detail while it was trying to keep noise down. Blow some highlights? If only the contrast setting hadn't been quite so high... If only I had shot RAW. All of those decisions could have been made later, and I'd have my color, noisy but detailed, well-exposed shot. Many advantages of RAW are far more subtle. There are some other advantages of JPEG as well. Both of those issues will have to wait for another installment. Now back to my concerns about RAW.
RAW is being dropped from advanced compact cameras at an alarming rate. Canon and Nikon have abandoned it. Many of us think that this has been done in order to steer enthusiasts to DSLRs where we will spend thousands on lenses. Others feel that RAW files are being kept from us so we won't see how noisy these tiny, pixel-stuffed sensors are before in-camera noise reduction is applied. I also wonder whether companies are leaving out RAW in order to save money on hardware costs. RAW file capture and storage typically necessitates a larger memory buffer for reasonable continuous shooting rates. Whatever the reason, it is important to understand that RAW is not being kept from us because the cameras don't "make" a RAW file. When light hits the sensor, RAW data is recorded. They just need to fork it over. This point is illustrated by a Russian hack which makes RAW files available on Canon digital cameras with DIGIC II processors from which Canon had disabled end user access to RAW data. Many held out hope that Canon would keep RAW in their flagship prosumer G-series. No such luck. Canon's Chuck Westfall explained that this omission was a consequence of the pixel pitch being so small on the G7. By shoving 10 million pixels onto a 1/1.8" sensor, Canon made the G7 pixels so small and noisy that no RAW processor can eek more out of them than Canon's on-board DIGIC III processor can. I won't argue against the implication that little pixels are no good. Two of my cameras (Leica D2 and Canon 5D) have a large pixel pitch, and I think that when one compares them to cameras of the same technology generation, one can really tell the difference. Larger pixels tend to capture a greater dynamic range as well as to limit the amount of noise. I'm getting off topic though. Users of the tiny-pixelled G7 would still benefit from access to their RAW files. First of all, there is nothing magic about the DIGIC III processor. It is not more powerful than my Mac Pro, and it's processing algorithms will be dated as fast as Canon can produce the DIGIC IV. Therefore, to imply that the G7 can get as much or more out of the RAW files than we can is downright silly. Second, RAW is not only about how much detail or how little noise we can extract. It is about flexibility. When I shoot ISO 400 with the G7, the camera applies a certain amount of noise reduction (NR) to create a balance between detail preserved (NR destroys detail) and noise eliminated. Each camera's JPEG engine does this differently. For example, Panasonic's Venus III engine has been widely criticized for striking a balance that favors excessive noise reduction with resulting smearing of details. NR also creates the Fuji compact camera "watercolor effect" at very high ISO. Getting back to the G7, what if I want to preserve more detail at the expense of greater than usual noise for ISO 400? Often with black and white conversion, I choose such a compromise since color noise is not a factor in B&W. RAW would give me that option. Mr. Westfall's explanation is not good enough for me.
Now for the main reason I am posting this. It has recently come to my attention that, even in the cameras which still offer RAW, our RAW may not be as RAW as we think it is. I was reading about image quality comparisons between the Panasonic DMC-LX1 (branded by Leica as the D-LUX 2) and the DMC-LX2 (branded by Leica as the D-LUX 3). These two cameras are interesting to me because they are compact, offer RAW, offer a choice of 16:9, 3:2, and 4:3 aspect ratios, and have a wide angle lens (starting at 28mm when shooting in wide aspect format). The LX2 has more pixels, which as noted above is not always a good thing when sensor size limited. However, I was intrigued by the fact that LX2 RAW files, at typically between 19-20MB each, are considerably larger than my 5D RAW files. This prompted me to do a little digging. One reason for the LX2 RAW files to be so large is that they contain 16-bit color as opposed to 12-bit as in most RAW files. The other thing I found out about LX2 RAW files is more ominous. Panasonic appears to have applied a considerable amount of NR to the RAW files themselves! This reduces the amount of shadow noise considerably but also reduces the low contrast detail available in those shadows. Björn (viztyger) on DPReview was kind enough to make available his LX2 RAW analyses in those forums. Here is his post showing identically processed RAW files from the LX1 and LX2. I think the results speak for themselves. It is clear that the LX1 crop contains noise that the LX2 crop does not. Many would find this noise objectionable, and yet they could largely remedy the situation by applying NR software during or after RAW conversion on their computer. On the other hand, the shadow detail which is present in the LX1 crop is irrecoverable in the LX2 photo. No software could restore it.
This raises the question as to whether Panasonic is an evil company that is alone in violating the sanctity of RAW, whether everyone is doing it, or whether I have just been thinking about RAW the wrong way. One thing to consider is that the processing of recorded light data is not limited to a single processing engine such as DIGIC III or Venus III. There is so-called "on-chip" processing on the sensor itself. It has never been clear to me whether to think of this processing as distinct from what the main processing engine does or not. In other words, I wonder whether "on-chip" NR permanently eliminates detail the way other NR does, or alternatively whether it could somehow be improving signal to noise capture at the source. If anyone knows, please tell me. As to whether "everyone is doing it," it is becoming clear to be that the answer is probably "yes" to one extent or another. In a 2006 thread in the Rob Galbraith forums, Canon's Chuck Westfall had the following to say: "...This dates back to previous discussions on this board (circa 11/2003) where I stated that .CRW files from early EOS Digital SLRs (D30, D60, 10D, Digital Rebel/300D) were minimally sharpened in camera. Canon Inc. engineers confirmed this to be true. Conversely, no sharpening of any kind is applied to raw images captured as .TIF (as in EOS-1D, EOS-1Ds) or .CR2 (EOS 5D, 20D, Digital Rebel XT, 1D Mark II, 1D Mark II N, and 1Ds Mark II)." He went on to add, "To the best of my knowledge, there is no softening or smoothing per se on Canon RAW images. However, it's clear that on-chip noise reduction is performed prior to exporting the image data to the camera's image processing system." Although they are rascals for tampering with my RAW, I do respect Canon for having a prominent representative answer the tough questions posed in a public, third party forum. What about the number 2 producer of digital cameras? Jim Cockfield, who informed me of the aforementioned comments from Mr. Westfall, also gave me this link, which suggests that as of 2005 there was evidence that Nikon was modifying RAW files.
While every camera manufacturer's offered RAW is clearly more raw than their in-camera JPEGs, I now understand that not all RAW is equally raw. From now on when I shop for a new camera, I will not only have to ask "Does this camera have RAW?" but also "How RAW is it?"
Keeping RAW Raw.
Sunday, May 20, 2007
Posted by Amin
Labels:
Canon,
D-LUX,
DIGIC,
G7,
LX1,
LX2,
Nikon,
noise,
RAW,
Venus,
Westfall
46 comments:
Subscribe to:
Post Comments (Atom)
I just discovered your excellent site and will spend some more time here. Although I feel that the LX2 is an inspiring camera, the crude noise reduction in RAW files bothers me. Like you point out, especially for black and white I would like to be able to determine the balance between low contrast detail and grain myself. The Canon G7 seems to use somewhat more sophisticated noise reduction, but its lens is not wide enough for my style of photography. So I am quite interested in the Ricoh GX100. I look forward to finding out just how raw its RAW files are, and how good an image quality one can squeeze out of that enticing photographic tool.
Kind Regards, Björn (viztyger)
Björn, thank you for sharing your thoughts about the LX2 and for your kind words about my site. When I compare the G7 and GX100, I also plan to borrow my dad's LX1 for additional comparisons of processed RAW files. Hopefully the results will be interesting. Regards, Amin
The ".RW2" files produced by the DMC-LX3 are smaller still at around 12 MB than the DMC-LX2 files (around 16 MB). Several folks claim that this is a result of 12-bit data stored on the LX3 (as opposed to the 16-bit data reported to be stored on the LX2).
The other day I took a test RW2 on a Leica D-Lux 4 (the ".RWL" files alleged to be the same as the Lumix ".RW2" files in all ways but the file extension name).
The subject, ISO, focal length, and F Number was the same as a previous shot (around the same time of day with similar lighting) taken with my FZ50 as a 20 MB ".RAW" file, and both the ".RAW" and the ".RW2" were post-processed with the same SilkyPix DS 3.x software application. This is unrelated to the known "barrel distortion correction" issue(s).
Although the FZ50 has a larger (and quite) likely higher quality Leica lens system, and the subject was a brick building at several hundred feet, I definitely see markedly reduced and inferior detail in the ".RW2" image taken with the Leica D-Lux 4 (presumed to be essentially the same as my soon to arrive DMC-LX3).
I would say that the (12 MB) ".RW2" files produced by the LX3 (while still potentially better than the in-camera JPGs) very likely (in a manner similar to the LX2 that you report) throw out low-level contrast detail.
The upshot - when I really need RAW format for detail at a distance, I guess that I will need to stick with my DMC-FZ50.
It appears that the JPEG and/or "baby-RAW" files of the DMC-LX3 will be limited to the more up-close and less-detailed work.
Following the lovely FZ30 (Venus II) - the JPG meat-job that Venus III produces, and the somewhat abated damage caused by the Venus IV, now followed by this ".RW2" format "baby-RAW" performance, truly qualifies Panasonic for the status of "consumer-driven banality" (if not that of evil itself ... :)
Four Lumix cameras in my collection, low-level detail continues to take a drubbing as time goes by.
-Sisyphus, the Detail Man
A "SMOKING GUN" AS TO PANASONIC APPLYING NOISE REDUCTION TO ".RW2" FORMAT FILES PRODUCED BY THE DMC-LX3?
At this Panasonic web-site:
http://panasonic.com.ph/web/cid/MainCont/1823
one finds a DMC-LX3 review (obviously deemed as favorable by Panasonic).
Note the following statement about reducing "the noise reduction and sharpness level to -2" from the review (in the "Image Quality" section):
"Now onto the RAW image. The .RW2 files can be converted to more common formats like TIFF or JPEG with the supplied SilkyPix RAW image-processing software, but the result wasn't as good as expected. This can probably be attributed to the software, and not the lens. We experimented several ways to get around this problem and found a better method of capturing superior-quality pictures. In the Film Mode settings, we reduced the noise reduction and sharpness level to -2, and then processed the RAW image in SilkyPix to TIFF format. After which, we opened the processed file in another image-editing software and tweaked the image from there. The result was significantly better than what we would have achieved if we had relied solely on SilkyPix."
One would reasonably infer from the above statements that the DMC-LX3 Noise Reduction and Sharpness settings (thought by many to relate to the on-camera JPG generation only) DO, in fact, also affect the content of the ".RW2" files generated by the camera ...
A (later inserted) "editor's note" states:
"Editor's Note:
Subsequent to a second review unit being extensively tested, we have verified that the image quality issue mentioned in our review previously was found to be isolated to the earlier evaluation unit. We have since updated our review with new comments in the Image Quality section and awarded an Editors' Choice to the LX3."
Something very fishy going on here, it seems ... If one LSI chip-set behaved in that way, then so do the rest. Note that the "editor's note" alludes to a single "defective unit", but does NOT deny or revise their original statements (about the NR and Sharpness controls affecting the image quality of the ".RW2" file)!
Given that the 10 Mpixel DMC-FZ50 RAW files are 20 MB in size, one can reason that Panasonic must be throwing *some* information out in the case of their compressed (12 MB in size) ".RW2" files.
And (as my previous post describes), low-level contrast detail (and detail in general) definitely appears to suffer from the effects of such file-size compression.
Extensive internet searching has not revealed any other published information about this subject. yet, it seems to here be admitted by Panasonic themselves ... !!!
-Sisyphus, the Detail Man
I've had my LX3 for about a month. It obviously uses light well (fast lens), but I also feel it REPRESENTS light very well. I particularly like the light in landscape photos from it. That being said, I have wondered about low contrast detail, and fine detail myself. The info provided here about RW2 files is very interesting, and much appreciated. I wonder if the smaller RW2 file size is the problem ? Could that smaller size be the result of a lossless conversion ? Maybe the NR in the RW2 files, if it is there , alone is the culprit? I wish Panasonic (and Leica) could be more open about what's going on. I also wish that a future LX3 firmware update might allow one to reduce NR even further, and maybe, if possible even allow the camera to use another type of RAW file. Regards, Jack
Just some further thoughts from someone who admittedly isn't so technically knowledgeable. Amin's original post talked about the LX2's RAW file size as larger than the LX1's the implication being that a greater amount of in camera NR applied to the LX2 RAW files might account for some of the file size difference. So in this case greater RAW file size seems to have gone along with greater in camera NR.
In addition to my LX3 I also have a Canon S70. I checked the file size of some of the CRW RAW files from it and found them to vary from 7.9MB to 9.4MB roughly. The S70 is a 7MP camera so the amount of "headroom" there seems to be in the same range as for the LX3 and its RW2 files. Again, I'm wondering if the size per se of the RW2 files as compared to other types of RAW files is a limiting factor for IQ ? I notice the Panasonic 4/3 cameras also seem to use RW2. I'd appreciate any info on the subject - Jack P.S. I've noticed converting the mode from 8bits to 32bits, even without any other post processing, after opening the RW2 files in Photoshop CS4 seems to produce a strong improvement in the IQ.
Jack 19 -
I wish that there was a way to query the original poster Amin in order to benefit from his extensive knowledge in this matter as it applies to the LX3 ".RW2" files. I've also thought of trying to contact Dave Coffin (the DCdraw programmer).
Though my aforementioned test shot (a fairly tough one of a brick building at a couple hundred feet) seems to indicate less detail in the 12 MB ".RW2" from the LX3 (compared to the 20 MB ".RAW" from my FZ30), it is also true that the Leica lens system on the FZ50 is larger in size and higher in quality.
Another possible factor is that the LX3 clearly underexposes by about 1/2 stop (which I was *not* aware of when I took the test shot mentioned). Thus, merely maxing the live Histogram will result in about 1/2 stop lower exposure. This takes a maximal toll on the minimal number of digital bits available to resolve details in the lower "shadow" ranges of the image.
Even if it is true (as I have read, but not verified) that the LX3 ".RW2" files are storing only 12-bits (as opposed to the full 16-bits reputed to have been stored in the LX2 ".RW2" files), 12 bits (equaling 12 stops) would appear to far exceed the actual usable dynamic range of either of these cameras. Thus, (it would seemm that) the bit-depth (at such levels) may not be a factor.
Despite what Amin writes about the differences between the LX1 and LX2 file sizes and apparent NR (which I do not doubt), it would (intuitively) seem that *larger* file sizes would be associated with *more* NR (if it is being performed at all on any of these "RAW" format files). If anything, it would seem that *smaller* file sizes might be evidence of *more* NR! Go figure.
The statement in the CNET review (that Panasonic re-posts on their very own site) reporting that setting the (what would be assumed to be JPG-only) NR and Sharpness controls down to their minimum levels (of -2) resulted in better results in Silkypix RAW post-processing has prompted me to do the same at all times. Fortunately, these settings strike me as appropriate and desirable settings for LX3 JPG generation, anyway.
Indeed, the size of "RAW" format files in various camera brands seems to vary (and be even smaller still than the LX3) in some cases. It is unclear what is going on, and to what extent these differences correspond to the number of Mpixels of the various camera's image sensors.
Don't hold your breath for a LX3 firmware upgrade that would modify the ".RW2" file structure (or the on-camera NR settings, even if they *do* affect the ".RW2" files). The many pleadings that Panasonic offer a firmware upgrade allowing the NR to be turned down on the wretched Venus III Engine JPGs of the FZ50 got absolutely nowhere ...
I will be re-shooting my aforementioned test shots (at +1/2 stop Exposure Compensation with the LX3, and at exactly 60mm focal length with my FZ30) in order to (again) try to determine whether we have a real problem that goes beyond the differences in FZ50 and LX3 optics.
In less demanding shooting situations, I am finding the LX3 ".RW2" files to be fairly workable so far - but I need to get more mileage with the LX3 under my belt before I form a more evolved opinion.
Please post any test results or impressions (that you or others may have) concerning the difference between the (20 MB) ".RAW" files (of the FZ50) and the (12 MB) ".RW2" files of the LX3 in particular. The differences between the LX1, LX2, and LX3 files seem less relevant (to me) - unless that data clearly establishes a "smoking gun" regarding pre-RAW Noise Reduction being applied on the part of the Panasonic Lumix designers.
Sisyphus, The Detail Man
CORRECTION OF DECEMBER 17, 2009, 8:58 AM POST (in [CAPS] below):
"Despite what Amin writes about the differences between the LX1 and LX2 file sizes and apparent NR (which I do not doubt), it would [NOT] (intuitively) seem that *larger* file sizes would be associated with *more* NR (if it is being performed at all on any of these "RAW" format files). If anything, it would seem that *smaller* file sizes might be evidence of *more* NR! Go figure.
-Sisyphus, The Detail Man
nonymous, thanks a lot for your very informative, and thought provoking post.
Unfortunately, I don't have an FZ50, and have no experience with Panasonic's 20MB".RAW" files.
I take your point about the RW2 smaller file size not intuitively suggesting less NR. I guess you are meaning more NR could equal throwing out more data which would equal less file size ( ?) I realize, though, that you aren't saying that the smaller RW2 file size is proof of NR, or greater NR.
Thanks for the info about the LX3 underexposing. I'll be interested to hear if your overexposing 1/2 stop to correct for the camera's predisposition helps the low contrast detail situation. I may try over exposing a bit myself, and seeing how that effects what I can recover in post processing with CS4. I should say that I almost always should raw. I am currently using the -2 NR setting with everything else at 0. I might try -2 sharpness as well to see how that goes.
I am a painter who could be termed a "photo-realist" ( I work from photo references) so I have a lot of 'pixel peeping' experience, and value sharpness. On the other hand I am also very responsive to the global, or overall impressions of images - their ambience, etc.. I most say, while there may be some loss of low contrast detail due to NR integral due perhaps to the LX3's internal hardware ( sensor architecture?), and /or raw (RW2) files that it's images have something else very positive going for them. They ring very true in a " beauty is truth, truth beauty" sense. The LX3 portrays light beautifully, and accurately to my eyes. There is a beautiful refinement, and subtlety in it's output. Not very scientific, and just one guy's opinion, but I see what i see, and i see it very consistently in images captured by the LX3. What accounts for this, and to what extent it is the result of purposeful design, or happy accident I don't know. There is certainly a lot more opportunity for camera/ software designers to shape, or effect output these days as compared to the days of film cameras. They seem to be operating in a philosophic realm making philosophic decisions.
I came across this link the other day - http://www.dxomark.com/index.php/eng/Image-Quality-Database/Compare-cameras/ It gives comparisons of the output from various camera sensor arrays ( prior to the writing of raw files I think) . I compared the Canon G11, Canon S90, and the LX3. The apparent greater tonal, color, etc. sensitivity of the Canon cameras sensor arrays as opposed to that of the LX3 runs counter to what i see in the images from each, but it is interesting.
Thank you once again Anonymous for your thoughts and your information . Please post any further thoughts or observations you have in the future. I will try and do the same.
Jack19 -
Jack: "I guess you are meaning more NR could equal throwing out more data which would equal less file size (?)"
Sisyphus: Correct. If anything, NR (if it is being applied) would reduce detail, which would (in turn) tend to reduce (rather than enlarge) file-size.
The other factor is how many bits are stored in the creation of a RAW file. The original article by Amin (who I have *tried* to alert to our current questions regarding the LX3 ".RW2" files) states:
"One reason for the LX2 RAW files to be so large is that they contain 16-bit color as opposed to 12-bit as in most RAW files."
His original article was inspired by the apparent noise reduction effects seen by observation of the LX2 RAW format images. But, he is (above) addressing the effects of the number of bits of color-depth increasing the file-sizes (and does not imply that the suspected NR is a suspected culprit for increased file-size).
Jack: "I'll be interested to hear if your overexposing 1/2 stop to correct for the camera's predisposition helps the low contrast detail situation."
Sisyphus: As I was not aware of the following when I took the Leica D-Lux 4 (very similar to my now acquired LX3) RW2 test shot and compared it to my previous FZ50 RAW test shot, I am suspecting (but have yet to re-test) that this exposure-level issue may well be a contributing factor to the impression of decreased low-level detail reported by a few folks on the internet, as well as the widely reported impression that the "shadow" areas seem unusually dark on LX3 images. A number of reviews have opined that the LX3 underexposes by 1/3 to 2/3 stops.
I have scoured the internet for discussions of the low-level detail image quality of RW2 type files in general. Other than two or three folks briefly (and vaguely) snarling on various forum posts that the ".RW2" image quality appears inferior to the larger Panasonic ".RAW" files (from cameras such as the FZ50), I cannot find *any* specific and informative data (from Panasonic, or anyone else, for that matter) ...
I am aware of the Dxo Labs data for "ISO Sensitivity" for the LX3 at:
http://www.dxomark.com/index.php/eng/Image-Quality-Database/Panasonic/Lumix-DMC-LX3
From the results for the (claimed) ISO 80, 100, 200, and 400, I calculate that the average discrepancy between claimed and (DxO) measured ISO equals -0.6362 stops (-34.75% in linear units). Thus, it is (approximately) -2/3 stops, requiring +2/3 EV in manual Exposure Compensation to correct.
This correlates well with my continuing experience with my LX3 using the multi-area (full frame) metering mode. Some shots require only +1/3 EV, but most require +2/3.
Note: Although *absolute* EV is referenced to the ISO value [ EV = LOG2 ( 100 / ISO * (F)^2 * S ) ], Exposure Compensation relates to *relative* changes, and is thus independent of the specific ISO sensitivity that the camera exhibits).
It very much appears that the LX3 internal calculations of exposure level (and the Live Histogram) are "tuned" to the *claimed* ISO sensitivities (and not to DxO Labs' measurements). The fact that +2/3 EV Exposure Compensation is (usually) required) appears to be fairly strong evidence that the DxO Labs measurements are accurate.
TO BE CONTINUED
- Sisyphus
CONTINUED FROM PREVIOUS POST:
This leads to the (irritating) fact that I have to go past maximum on the Live Histogram (which does begin to "clip" the highlights of the in-camera JPGs) and "guess" in order to properly expose. One shot I processed showed the additional "headroom" before "clipping" in the (same image's) RW2 file was (about) +1/2 stop. My *hope* is that this headroom (may) extend a bit farther (to +2/3 stops) - so that (at least) the RW2 does not have blown highlights as a result of properly setting the exposure (even though the corresponding JPG will have clipped highlights).
Setting the Exposure Compensation to +2/3 EV has worked out OK for some JPGs - but has blown-out highlights on others beyond repair in (JPG) post-processing.
I believe that Panasonic is well aware of this (claimed/actual) ISO discrepancy, and that this is a deliberate (though misleading) marketing move on their part. When one thinks about it, this (for the uninformed user) makes the image sensor (seem to be) less noisy, and makes the user *feel* good about their results at (for instance, a claimed ISO sensitivity of 400) - when, in fact (according to DxO Labs) they are in actuality shooting at ISO = 259. "Never mind" that the result is an underexposed shot (which I have tended to do with all my Lumix cameras myself, in order to avoid blowing out highlight-detail). Thus, users note that the "shadows" appear unusually dim, etc.
It would be nice if the Panasonic marketing department would be more truthful about the (actual) ISO - but I would not hold my breath for that ... It's all about emotions (and less about facts) in the market-consumer-state.
- Sisyphus
Jack19 -
Jack: "I am currently using the -2 NR setting with everything else at 0. I might try -2 sharpness as well to see how that goes."
If you are a person who post-processes your shots, my experience (and personal preferences) would recommend setting the Sharpness to -2 as well as setting the Noise Reduction to -2. The less NR used, the less that the image needs to be Sharpened ...
You may be interested in reading my ("Detail Man") posts (and a RAW-TIF-JPG derived) image at the thread:
http://forums.steves-digicams.com/panasonic-leica/158891-they-call-noise-free-fz35-38-a-2.html
where I report that *more* than default Sharpening (in Silkypix DS 3.x) combined (as well) with some Unsharp Masking applied at the final down-sized TIF pixel-size resulted in *less* sharpening than the LX3 in-camera JPG when Sharpness was set at -2!
(Also), be sure to use either the "MY FILM" or "STANDARD" Film Modes (as all the others have different (and unknown) parameters settings (even when a parameter is set at ZERO, etc.)
- Sisyphus
CORRECTION TO DEC @@ 2009 12:19 AM POST:
-0.63262 stops (LOG Base 2) is equal to a linear factor of -35.66% (and not the -34.75%) stated in the the above post.
- Sisyphus ("The Detail Man", indeed)
Jack19 -
Jack (referring to LX3 image quality):
" ... it's images have something else very positive going for them. They ring very true in a " beauty is truth, truth beauty" sense. The LX3 portrays light beautifully, and accurately to my eyes. There is a beautiful refinement, and subtlety in it's output."
Sisyphus: I am liking what I have (so far) seen, as well. While we wrestle with machines, the final "product" is very much an aesthetic (and inherently subjective) one (much like music) that passes and resonates (or does not resonate) in out "mind's eye", indeed.
(From my personal tastes), the in-camera JPG Sharpening is a bit excessive (even with Sharpness at -2), and the under-exposure issues previously mentioned tend to give it a specific quality (albeit perhaps more shadows than might be desired - though that can be compensated for using Exposure Compensation).
I like what I see much better than my LZ5, better than my FZ30, much better than FZ50 JPGs (which use the horrible Venus III JPG Engine), and better than my TZ4 (which also uses the Venus IV Engine, but really obscures detail in order to obviate noise concerns).
- Sisyphus
Jack19 -
Jack: "I came across this link the other day - http://www.dxomark.com/index.php/eng/Image-Quality-Database/Compare-cameras/ It gives comparisons of the output from various camera sensor arrays ( prior to the writing of raw files I think)"
Sisyphus: Your statement raises an interesting question. I have read several folks state on the internet that DxO Labs bypasses the lens systems in performing their test (illuminating the image sensor directly). However, I would speculate that they are not able to access sensor data any more directly than by using any particular camera's RAW file output (if/when available).
They primarily test DSLRs, etc. So it may well be that they are looking at RAW file output exclusively in their published tests.
- Sisyphus
MESSAGE TO ARTICLE AUTHOR "AMIN":
I hope that my a Jack's conversations about the LX3 in general (as well as it's ".RW2" file image quality) has not strayed (too much) "off topic" of this thread ... :)
We (and perhaps other) are very interested in any knowledge and expertise that Amin might be able to lend to the questions surrounding (possible) Noise Reduction being applied to the LX3 ".RW2" files (in a manner similar to that which he suspects is the case with the LX2 files).
I tried to leave a message at the "Contact Us" web-page for the Serious Compacts site, but there seems to be no way (with my Firefox 3.56 browser, anyway) to submit the message. So, I posted a comment at that web-page (partially copied below) in the hopes that it might lead to Amin seeing my inquiry:
"I recently discovered the 2007 thread about "Keeping RAW raw" at:
http://www.seriouscompacts.com/2007/05/keeping-ra...
Having added a few comments and questions (as "Detail Man") to the thread -
I would like to somehow alert the original posts author ("Amin"), in the hopes that his present knowledge might well be able to contribute at this time to information regarding if/how Panasonic may be processing sensor data with noise reduction algorithms *prior* to generation of the RW2 files in the LX3 (which uses the newer 1/1.63" image sensor)."
- Sisyphus, The Detail Man
CORRECTION TO DEC 22 2009 12:47 AM POST:
My apologies. "Detail Man" in his old age has become a bit foggy and sloppy (as well as, perhaps, characteristically obsessed with minor details that most could hardly care less about). In the interest of accuracy (and correctly stated without errors or typos on this 3rd attempt):
-0.6362 stops (LOG Base 2) is equal to a linear factor of -35.66%.
- Sisyphus ("The Detail Man", indeed)
CONTRAST SETTINGS ON THE LX3
At risk of further straying "off-topic" in my ramblings, and for the benefit of folks who post-process their LX3 JPG images, I wanted to report regarding the Contrast settings (in the LX3 "MY FILM" or "STANDARD" Film Modes):
I have conducted some (moderately) rigorous tests at the LX3's (claimed) ISO 80, using a "dynamic range" test pattern containing a total of 8 stops range of gray-levels that can be found at:
http://www.simpelfilter.de/analyse/images/digicam-test2.png
DxO labs specs the LX3 "Tonal Range" at (claimed) ISO 80 as 7.42 bits. A binary "bit" being equivalent to a "stop", the contrast-range of the test-pattern (slightly) exceeds the contrast-range of the LX3 (this is a good thing, making the test performed reasonably meaningful).
I then observed the luminance histogram of the uploaded images in PaintShopPro 9.01, noting the effects of the different LX3 Contrast settings.
Contrast = 0 setting over-emphasized the highlights and shadows, attenuating the mid-tones considerably.
Contrast = -2 setting over-emphasized the mid-tones, and clearly vanquished the highlights and shadows.
Contrast = -1 setting yielded a histogram which (nearly perfectly) covered the bottom of the shadows to the top of the highlights.
Further enamoring me to a Contrast setting of -1, the histogram (in PSP9) of my very first (and properly exposed) real-world test shot spanned the entire range from 0% to 100% beautifully - just like in my test shot.
Thus, for folks who post-process (and can always increase Contrast later in the process if necessary), a Contrast setting of -1 takes in the the entire tonal-range *without* over-emphasizing Highlights and Shadows in the JPGs generated.
- Sisyphus
Anonymous, Thanks for your additional thoughts and info. So it's your feeling that even the in camera contast setting, along with the sharpness and NR settings affect it's RW2 files ? As I mentioned earlier, I shoot mainly raw, in fact basically only raw. By the way I have been using 'Standard' film setting. Thanks for your advice there. I suppose I might try 'Film' sometime . So would you say these settings also affect raw shooting/files?
Assuming the in camera settings mentioned above are affecting my raw files, I'm doing some test shots now , and will try and report the results, later, after I've had time to examine them thoroughly. Thanks, by the way, for doing the test with the grey test pattern. Your results there are very interesting. For my tests I'm basically trying to shoot the same scenes or subjects using normal exposure and -2NR, 0 sharpness, 0 contrast, then a shot using normal exposure -2NR, -2 sharpness, 0 contrast, and a shot with normal exposure again and-2NR, -2 contrast, and -1 contrast. I'll then try and shoot the same three setting combos with a +1/3.or +2/3 exposure.
It's strange that Panasonic seems to have pegged their default or 0 exposure reading to their purported rather than their actual ISO settings ( maintain the myth - or what?). In any case if their ISO 80 is actually 53, or some such, that's fine by me. I actually prefer the idea that it's lower - a bit noisy for it's ISO reading or not. I have noticed the LX3 output shadow areas sometimes seeming a bit dark or compressed in their darkness, but have been using levels adjustments in CS4, when necessary, with good success. I have been more concerned, though, with texture, and low contrast detail in more brightly lit areas like the texture of a stucco wall in full sunlight, or in cloudy day diffuse light, etc.. I'll be interested to see how boosting the exposure a bit and/or changing the contrast setting affects what I can recover from the RW2 files in Photoshop. I am, like you, concerned with blown highlights though. I'll have to see how boosting the exposure goes, and what I can recover in post processing of the raw images.
Beyond the ISO readings I was interested in the other measurements of the LX3's raw output ( as per what you said it seems not readings directly from sensor arrays as I was thinking ) as compared to that of the Canon G11 and S90 cameras. The measurements of color and tonal responsiveness seem to favor the Canon cameras, but I find the LX3's output or images to be more subtle and sensitive - go figure!
Regarding the 12 bit data stored by the LX3 versus the 16 bit data stored by the LX2, I'll readily admit my technical ignorance , and risk making a stupid remark. I don't know if this is relevant to the above, but I really do see a sort of subtle, but clear, improvement when I change the image mode from 8bit to 32bit in CS4.
That's about it for now. I haven't had a chance to read your Detail Man posts yet, but look forward to doing so. I also look forward to hearing about your further exploration of the the FZ50 - LX3 distant brick wall relative image detail issue, and whether an exposure setting, or something else might narrow the gap there. In the end I'd like to squeeze a bit more detail out of the LX3, but am quite happy with it's raw ( haven't shot JPEG yet so can't comment about that) output as it is. I'll be hoping for improvement of course, but also looking for downsides, and weighing trade-offs in trying various camera settings. Thanks again. I hope we can keep this discussion going , and it would be great if Amin or others would weigh in. Regards, Jack
One further thought about the exposure setting/ image detail issue. I can imagine how boosting the exposure can help with the too dark, and undifferentiated shadows, but I wonder how it might effect the detail in more brightly lit areas like the stucco wall texture I talked about in my last post. I have 4"x5" film camera photos made of my paintings. I then have them scanned in a photo lab so I can have a digital record, and be able to develop an e-mailable image. They create a large 100MB, or so, Tiff. At the suggestion of the lab workers I've had the scans made somewhat darker than normal according to them so as to preserve detail, or so I'll be able to recover more detail when I do further processing of the image . Those darker Tiffs have worked well , and I've been able to create 'true to the paintings' - bright, detailed, correctly colored images from them. Just occurs to me that maybe I'll be better off with the LX3 normal or zero exposure setting as per this experience, and as the shadows I'm most concerned with don't fall so near the left end of the histogram - or at least I don't think they do(?) Anywho... it's all thought/experimentation provoking. I'll be trying things out.
Jack19 -
Jack: "So it's your feeling that even the in camera contast setting, along with the sharpness and NR settings affect it's RW2 files?"
Sisyphus: No. The only settings that (may, it possibly seems) affect RW2 output (other than ISO Exposure Compensation, focus, metering etc., of course) are the NR and Sharpness settings in the "Film Modes". This is (entirely based upon the satements in the CNET review that Panasonic themselves choose to (re-post) at their own web-page (hence, my "smoking gun?" statement.
I have not experimented with my LX3 in order to try to verify this - as I (from the get-go) want as little NR as possible, and the sharpening of the LX3 seems excessive to me (even at a -2 setting). I post-process all my JPGs, and perform sharpening at the end of the process (using Unsharp Masking after the final processed TIF (or BMP) has been re-sized downwards to it's final pixel-size (before finally converting the end-product to JPG, if appropriate).
Jack: "By the way I have been using 'Standard' film setting. Thanks for your advice there. I suppose I might try 'Film' sometime . So would you say these settings also affect raw shooting/files?"
Sisyphus: No, I am not saying that. As far as I know (or I should say, suspect), *only* the NR and Sharpness settings within the "Film Modes" (may) affect RW2 output (see above).
Jack: "I have noticed the LX3 output shadow areas sometimes seeming a bit dark or compressed in their darkness, but have been using levels adjustments in CS4, when necessary, with good success. I have been more concerned, though, with texture, and low contrast detail in more brightly lit areas like the texture of a stucco wall in full sunlight, or in cloudy day diffuse light, etc." ...
... "I'll have to see how boosting the exposure goes, and what I can recover in post processing of the raw images."
Sisyphus: With my FZ30 and FZ50 to some extent, and more so for my LZ5, and to a large degree with my TZ4 - I tended to *not* push the Live Histogram all the way to right (max) - because (even though the output JPGs did not show any clipping on the PSP9 histogram), highlight detail was getting lost to an objectionable degree.
The reason that I am trying to push the "gain" on the LX3 as far as I can (pushing the Live Histogram all the way to the right, and even farther) is because of the seemingly evident error in manufacturing of the LX3 of ignoring the (in actuality) lower ISO when setting their "zero" exposure point (and, as well, the Live Histogram).
This is for the following generalized reason (affecting all digital image sensors). As far as information to resolve and record tonal (luminance), a full 1/2 of the capacity is committed to the information of the highest-intensity "step" (factor of two). 1/4 is committed to next-lower "step", and so on ...
As one can see, there are precious few discrete information levels left by the time on gets down to the "shadows".
Thus, in the interest of maximizing the resolution of "shadow-detail", it becomes quite important to maximize the signal/noise ratio - by setting the exposure as high as practically possible (without "clipping" or in some discernible way degrading the "highlight-detail".
As a result the "window" for the exposure setting is rather narrow (in order to do justice to the low-level as well as the high-level information in the image).
You may well already know this from experience. I just wanted to fully state my own reasoning.
Jack: "Regarding the 12 bit data stored by the LX3 versus the 16 bit data stored by the LX2, I'll readily admit my technical ignorance , and risk making a stupid remark. I don't know if this is relevant to the above, but I really do see a sort of subtle, but clear, improvement when I change the image mode from 8bit to 32bit in CS4."
Sisyphus: Adding extra bits *cannot* add extra information - and (it seems) may cause *additional* quantization noise in to the processing. However, you are not the only person that I have seen state this impression.
My guess is that it is due to to on (or both) of the following effects:
(1) Using digital signal processing that proceeds at higher bit levels in the post-processing application may help to reduce unwanted artifacts of that post-processing; and/or
(2) An emotional (placebo) effect.
Whatever the case (even if extra color channel bits are somehow "created" by numerical interpolation techniques in a (lower-to-higher bit number) conversion - I don't think that any *new* information can be produced by such interpolations.
This is much like when (some, mostly older) digital cameras would (using interpolation) create in-camera images (the pixel-count of which) exceeded the actual number of photo-sensors.
I don't think that such "alchemy" can really add new information - though post-processors or printers may (for their own subtle reasons) create better results from such "information-padded" formats.
Jack: "In the end I'd like to squeeze a bit more detail out of the LX3, but am quite happy with it's raw (haven't shot JPEG yet so can't comment about that) output as it is."
Sisyphus: (Pretty much) everything that I have been talking about is (similarly) applicable to RAW/RW2 - except for camera settings that we understand to (definitely) only affect JPG (and not RW2) output.
In terms of exposure-level issues - the only difference is how much extra "headroom" exists (above the maximum on the Live Histogram on the LX3). The JPG highlights "clip" at that precisely that point, whereas the RW2 appears to allow about 1/2 of a "step" (factor of two) higher exposure before similarly "clipping" the highlight information.
Note: I have started using the term "step" to indicate a factor of TWO in my posting - as the term "stop" (formally, but, it seems, not commonly in usual practice) refers (specifically) to an "F-stop".
One "step" increase equals one "Exposure Value (EV)" increase, which equals a factor of TWO in the value of incoming light to the sensor (or a factor of TWO increase in ISO Sensitivity).
Jack: "I have 4"x5" film camera photos made of my paintings. I then have them scanned in a photo lab so I can have a digital record, and be able to develop an e-mailable image. They create a large 100MB, or so, Tiff. At the suggestion of the lab workers I've had the scans made somewhat darker than normal according to them so as to preserve detail, or so I'll be able to recover more detail when I do further processing of the image . Those darker Tiffs have worked well , and I've been able to create 'true to the paintings' - bright, detailed, correctly colored images from them. Just occurs to me that maybe I'll be better off with the LX3 normal or zero exposure setting as per this experience, and as the shadows I'm most concerned with don't fall so near the left end of the histogram - or at least I don't think they do(?)"
Sisyphus: In the case of your paintings, you have film camera and scanner characteristics entering into the "equation" ...
Both of the above items may well have a wider (tonal, as in luminance) dynamic range than does the LX3 (only 7.42 bits, equaling a linear factor of 171.255).
Thus, the LX3 (even with RW2 format) likely afford you a narrower "window" of tonal-range within which to resolve both your highlight as well as your shadow details to your satisfaction.
Accomplishing the above (may) require you to (more carefully) set initial exposure levels - in order to place your LX3 image sensor information *over* the lower-limit of shadow resolution, and (at the same time) *under* the upper-limit of preserving highlight resolution.
Due to such sensitivities and limitations of small image sensors, I usually narrow my shots (as I shoot them) down to two separate guesses of the "right" exposure level and take (twice as many) shots (separately, and individually) at each of the two exposure levels. Then I later choose from the shots in manual selection processes prior to post-processing (or try to process both, choosing from the final results).
Nothing about final image quality comes easy (for me, anyway ... :) That's why I call myself "Sisyphus"!
CLARIFICATION OD STATEMENT IN MY DEC 22, 2009. 10:33 PM POST
Jack: "By the way I have been using 'Standard' film setting. Thanks for your advice there. I suppose I might try 'Film' sometime . So would you say these settings also affect raw shooting/files?"
A MORE CLEAR ANSWER THAN ORIGINALLY POSTED IS PROVIDED HERE:
Sisyphus: "Standard" and "My Film" are both (as you know) individual "Film Modes" of the LX3.
Within any/all of the "Film Modes" (based upon the statements made in the CNET review re-posted by Panasonic themselves, and *not* based upon my own personal testing), there seems to be a possibility that (for *any/all* Noise Reduction and Sharpness settings within *any/all* of the LX3 "Film Modes") the content of the RW2 format camera output (may) be affected.
Note that the (smoking gun?) language within the review speaks generally of the "Film Modes" (and not about any one particular "Fil Mode"):
"In the Film Mode settings, we reduced the noise reduction and sharpness level to -2, and then processed the RAW image in SilkyPix to TIFF format. After which, we opened the processed file in another image-editing software and tweaked the image from there. The result was significantly better than what we would have achieved if we had relied solely on SilkyPix."
The Panasonic site reposting the revue can be found at:
http://panasonic.com.ph/web/cid/MainCont/1823
and relevant statement in the original revue itself can be found at:
http://asia.cnet.com/reviews/digitalcameras/0,39001468,62045250-4,00.htm
Mr. Leonard Goh:
I have recently purchased a LX3 (Firmware Version 2.1). The (12 MB size) ".RW2" files produced by the LX3 appear (under certain circumstances) to be inferior in terms of image detail relative to the (20 MB size) ".RAW" files produced by my FZ50 under similar shooting parameters and light upon the same subjects at the same distance (using Silkypix DS SE Version 3.0.2.1).
There exists language within your review dated September 2008 at:
http://www.cnet.com.au/panasonic-lumix-dmc-lx3-339290909.htm
that appears to imply that you (and/or associates) found that adjusting the "Noise Reduction" and "Sharpness" controls (which are commonly thought to only affect in-camera JPG encoding) affect the content of the ".RW2" image files produced by the LX3:
"The .RW2 files can be converted to more common formats like TIFF or JPEG with the supplied SilkyPix RAW image-processing software, but the result wasn't as good as expected. This can probably be attributed to the software, and not the lens. We experimented several ways to get around this problem and found a better method of capturing superior-quality pictures. In the Film Mode settings, we reduced the noise reduction and sharpness level to -2, and then processed the RAW image in SilkyPix to TIFF format. After which, we opened the processed file in another image-editing software and tweaked the image from there. The result was significantly better than what we would have achieved if we had relied solely on SilkyPix."
The "Editor's Note" appearing near the beginning of the presently posted version of the LX3 Review states:
"Subsequent to a second review unit being extensively tested, we have verified that the image quality issue mentioned in our review previously was found to be isolated to the earlier evaluation unit. We have since updated our review with new comments in the Image Quality section and awarded an Editors' Choice to the LX3."
This language implies that nothing about the "second review unit" has led the review author(s) to amend the language in the article quoted above.
This language seem to clearly imply that the particular "Film Mode" adjustments "Noise Reduction" as well as "Sharpening" do (by your observations) directly affect the content (and thus the characteristics) of the ".RW2" image files recorded.
If the first evaluation unit behaved in this manner, there seems no reason why the second evaluation unit would not (also) behave in this manner - and your (revised) review does not indicate that the second evaluation unit behaved otherwise ...
I notice that the Panasonic company themselves (on their own web-page) proudly re-posts your positive review of the LX3 at:
http://panasonic.com.ph/web/cid/MainCont/1823
Well informed and competent folks (such as Amin, at Serious Compacts) have in the past (May, 2007) published articles that imply that it seems to appear that the (previous) LX2 appears to apply noise-reduction prior to user-accessible raw file output, whereas the LX1 did not do so. See his article at:
http://www.seriouscompacts.com/2007/05/keeping-raw-raw.html
Recently, current LX3 owners are beginning to wonder about the nature of the ".RW2" image files generated by the LX3 ... and are becoming curious as to whether Noise Reduction (and, it would seem from your findings, Sharpening) is being applied to the ".RW2" image files generated. Thorough searches for information relative to these questions yield nothing (from Panasonic, or in general on the internet).
Would you (or your associates) be so kind as to clarify the above stated issue of whether NR/Sharpening is applied prior to ".RW2" image file output on the DMC-LX3 with Panasonic? In the absence of such clarifications, it would appear from your LX3 review (re-posted by Panasonic themselves) that such is in fact the case.
Thanks,
Milo
CURRENT SENIOR DIGITAL IMAGING EDITORS AT CNET CONTACTED
All efforts to *directly* email Leonard Goh have bounced ... ;-)
Thus, the current relevant editors at CNET (Josh Goldman and Lori Grunin) have been contacted with a (CNET web-based, and character-size limited) email referring them to the above post of inquiry to Leonard Goh (and/or his current associates).
Stay tuned ... and we will see whether CNET will follow-up on this issue with Panasonic (that their LX3 review by Leonard Goh appears to state is, in fact, the case).
Sisyphus, The Detail Man
AWAITING FOLLOW-UP BY LORI GRUNIN
I notice that senior CNET Digital Imaging Editor Lori Grunin authored a (later) Feb 9, 2009 CNET review of the DMC-LX3 at the (main) CNET web-site at:
http://reviews.cnet.com/digital-cameras/panasonic-lumix-dmc-lx3/4505-6501_7-33175903.html?tag=txt%3bpage
that does not (for whatever reasons) mention this issue, whatsoever ...
Leonard Goh's earlier review (re-posted on the Panasonic web-page) is presently posted at CNET's "Australia" and "Asia" sites - and he is not contactable via email (despite all efforts).
The other CNET Senior digital Editor (Josh Goldman) is "out of the office" through Jan 4, 2010.
Thus, it appears that Lori Grunin (having herself written a later review of the DMC-LX3 for CNET) would/should be the appropriate person to contact Panasonic directly for some answers as to the inner-workings of the LX3 in it's generation and recording of ".RW2" image files.
We anxiously await Lori Grunin's report to us - following a definitive statement on the part of Panasonic as to the facts surrounding these important matters!
The Detail Man
Sisyphus, Thanks for the further info, and for your considerable efforts to get to the bottom of things regarding the RW2 files written by the LX3. I really hope that someone from CNET or Panasonic responds so we LX3 owners can have some clarity on the noise reduction, and sharpening in those files. I also hope to learn more about the exposure setting questions you've raised.
Sorry, but I mistakenly remembered a "Film" as opposed to the customize-able "My Film" setting within film modes. There is no such "Film" option. I probably was thinking of "Smooth". Anyhow, even though it probably doesn't effect my raw shooting I am staying with the "Standard" setting within film modes. I will experiment over time with 0,-1, and -2 NR, and sharpness settings within the "Standard" film mode. I'll forget about the contrast setting , or leave it at 0 to be safe, unless I shoot JPEGs in which case I'll take your tip and try out a -1 contrast setting.
Regarding the 32bit mode conversion in CS4, I don't know? Maybe it's interpolation or placebo effect. I doubt the later though. Maybe you can try it yourself and see. In any case I notice an improvement from doing it even absent any other adjustments. I just open the image from Adobe's raw converter and click Image>Mode>32bit, and nada mas. Try it out if you get a chance.
I'm starting to wonder now if pretty much all digital camera makers have been using raw files that involve some in camera noise reduction and sharpening. In Panasonic's case they've had some objection to their NR in past camera model so they apparently have improved that now at least in the JPEGs produced by the LX3. They apparently have also allowed us users to reach back into the raw (RW2) files at least a short distance via the max. -2 NR, and sharpness settings and partially undo those things. It's interesting that this wasn't openly discussed by them in their manual, or elsewhere, but I am grateful that it was at least quietly (very quietly) offered by them. Personally, I think I see an improvement in my RW2 files when selecting -2NR, and I've just started fooling around with the minus-ing the sharpness setting. I'm still forming my opinions about the effects of these settings when shooting raw.
Sisyphus, I'm glad to know about the exposure setting issues you've uncovered. It's interesting that the amount of apparent underexposure corresponds with the difference between published and measured ISOs. In spite of that correspondence, I have trouble accepting that the underexposure at 0 exposure setting is a mistake by Panasonic. I have to believe they knew quite well about the camera's exposure settings, and made conscious decisions regarding them.
Anyhow, whatever they've done, and allowed regarding the raw files , and whatever they've done regarding exposure settings, they have produced a camera capable of taking remarkable photos. Maybe given the limitations of the design, etc. , they've been sort of benignly covert, and allowed only limited access to the raw files to keep us 'deconstructionists' from messing things up or from destroying the whole. P.S. any further results regarding your FZ50/LX3 test?
LEONARD GOH CONTACTED BY CNET (DEC 23 2009)
Senior Digital Imaging Editor Lori Grunin at CNET has read all of the relevant information, and has referred the matter to CNET Reviewer Leonard Goh (for his further research, clarification with Panasonic, and public response).
Stayed tuned to this message thread (and/or CNET itself) for an elucidation of the TRUTH surrounding this matter of what the Panasonic LX3 in actuality does to it's RW2 image files prior to recording them.
- Milo (aka "Sisyphus, The Detail Man")
LG: I refer to your mail to CNET's Lori Grunin who had forwarded me your mail. I understand your concerns over Panasonic's RAW files and if there was any in-camera processing applied to it.
As such, may I assert that this review was done last year using the initial firmware, which has since been upgraded several times.
During my tests then, I did discover that by applying sharpening and noise reduction in-camera did somewhat return better RAW files, but I am not sure if the latest firmware still exhibited such traits.
DM: Your CNET review of the LX3 is dated Aug 26, 2008. The LX3 firmware Version 1.1 was not available until Dec 15th, 2008. You must have been evaluating unit(s) with the original 1.0 firmware.
This web-page does a nice job of listing all of the issues publicly stated by Panasonic to have been addressed by all versions of the LX3 firmware updates since the time of your review:
http://www.lx3-photography.com/search/label/LX3%20FIRMWARE
Note that there exists no mention of any changes to the LX3 firmware at any time that would either modify or affect the behavior that you have reported.
Thus, it is reasonable to assume from your statements that you have in your LX3 review uncovered the fact that the LX3 (from Version 1.0 to the current Version 2.1 released in Oct, 2009) does, in fact, perform (both) Noise Reduction as well as Sharpening on the ".RW2" files that it records.
Further, the user-controllable settings of "NR" and "Sharpness" that exist in the LX3 "Film Modes" - which have by the consumer been widely thought to affect the nature of the in-camera JGP files (ONLY) - exhibit some amount of control over the degree to which (pre-RW2) Noise Reduction and Sharpening are applied in the recorded ".RW2" image files.
Panasonic (in the case of the DMC-LX3) has is the past (and, it appears, continues to date) to apply Noise Reduction as well as Sharpening to ".RW2" the image files that it records! This is important NEWS!
This is a very important issue for all consumers who presently do (or will contemplate) recording ".RW2" image files on the LX3 (as well as any/all of the other models of Panasonic cameras with user controllable NR and Sharpness that utilize the ".RW2" file format)!
LG: Also, I processed the RAW files with the supplied SilkyPix software which performance was mediocre. If you use Adobe Camera RAW, I think that can deliver much better results.
DM: That possibility does not in any way belie the importance of your informative findings!
LG: Due to the flood of new cameras and imaging technologies which were announced this year, I didn't have much time to try the new firmware nor the latest version of SilkyPix. But rest assure that when situation permits, I would give them a go again.
DM: I recognize your cautions about drawing assumptions about current LX3 firmware versions without personally verifying with an evaluation unit. However, (I personally) would be grabbing a (firmware Version 2.1) DMC-LX3 (not to mention every other relevant Panasonic camera) unit off the shelf as soon as I could for evaluation. This is genuine and important NEWS!
LG: Thank you for reading CNET Asia and your feedback.
Regards,
Leonard
DM: Thanks for your timely reply. We will be anxiously awaiting the results of your upcoming further scrutiny into these very important matters.
Please communicate directly with Panasonic in order to determine their official response to your findings. CNET (as well as consumers) deserve straight answers from Panasonic on these matters! The fact that Panasonic themselves re-posts your Aug 26, 2008 revue and findings on their own corporate web-page at:
http://panasonic.com.ph/web/cid/MainCont/1823
has relegated any claims on their part that such information is "proprietary" to the level of "absurd".
Demand some answers! The venerable CNET (and the consuming public who CNET serves) deserve no less.
In the course of enumerating my "plaintiff wail" (regarding LX3 RW2 image files AND Silkypix) here and there, The very sharp and informative "Odklizec" (a moderator for the the "Ricoh" and "Shortcut" (Silkypix)forums has made an informed guess about Silkypix-specific operations that is not inconsistent with CNET's Leonard Goh's experiences and statements, as well.
I would (certainly) rather that "Odklizec's" statement be true - as opposed to the possibility that there exists a Panasonic pre-RW2 NR/Sharpening operation going on when the LX3 generates a RW2 image file! He states:
"I personally found LX3 RAW files very good and detailed especially in dcraw based RAW editors (Rawtherapee)." ...
... "I would not be surprised if Silkypix would do also an automatic noise reduction based of the in-camera noise reduction settings." ...
... "All in all, I think the LX3 RW files are not (visibly) altered by in-camera NR ..."
http://www.ricohforum.com/viewtopic.php?f=22&t=1556&sid=11961080da9cee0a97fb1b65868902a9
I agree (from personal observation) that RAWtherapee 2.41 *does* provide superior RW2 image file details (in it's default settings).
Speaking of other possibilities in lieu of using Silkypix, I will also be purchasing DxO Optics Pro 6.x shortly, as they state that they are soon to release a LX3 "module" that will make the LX3 compatible with their esteemed "raw" processing application.
Based on the (I hope, likely) assumption that DxO Optics Pro does not exhibit what (may) be uniquely "Silkypix-specific" behavior (of tracking and responding to user NR and SHARPNESS controls that one would assume, or at least might perhaps hope, would be restricted to the in-camera JPGs created only), it will be interesting to see how the competition stacks up.
I am looking forward to hearing more from Leonard Goh regarding his experiences, as well as to performing further analysis of the low-level spatial-frequency resolution of the RW2 data in image files created by the LX3.
Any users of (both) Silkypix as well as the RAWtherapee (and/or DxO Optics Pro) software applications who have processed LX3 RW2 image files with these programs out there? What do you find to appear to be the case?
Detail Man
Anonymous,
Thanks to you and Odklizec for the further info. I, of course, hope that it does turn out that the LX3's RW2 files are fee of NR. That would be great news!
The suggestion has been made that the LX3's RW2 files may be free of NR, but contain instructions on performing NR that certain softwares like Silkypix then act on according to the camera NR setting. I don't have RAWtherapee. I have PS CS4 with Raw Editor 5.6 , and Silkypix, as it came with my camera. After reading your post today I decided to open some of my RW2 files in PS and Silkypix and compare them closely, side by side at 100% or 'actual pixel' zoom level. I have been doing experiments taking raw photos using the LX3's 'Standard' Film Mode with -2NR, and 0 NR for the past few weeks. I used 8 or 10 of those photos today in making my comparisons. They were all landscapes/cityscapes with a lot of fine detail, including a lot of distant detail, taken in a variety of weather, and lighting conditions. I found the Silkypix images to be consistently less detailed/more smeared than the PS images. The degree of difference seemed to be about the same whether I was using images from shots taken with -2NR, or 0 NR camera settings. Regarding that sameness of difference, a couple of possibilities occur. One is that both PS CS4 raw editor 5.6, and Silkypix respond to NR camera settings, but PS does it's NR better. A second possibility is that PS isn't doing any NR based on embedded info, and that Silkypix is , but not based on camera settings - just the same amount for each and every RW2 file (?).... Just some guesses
Finally, I'm pretty happy with what I'm getting from my LX3 RW2 files in CS4 , but I'd appreciate hearing from you or Odklizec as to how CS4 raw editor 5.6 stacks up compared to other softwares regarding detail resolution. Thanks! and Happy New Year !, Jack
Jack,
Greetings! I'm glad that you have been "on the case" about this ...
Jack: "I don't have RAWtherapee"
DM: RAWtherapee Version 2.41 is compatible with LX3 RW2 image files (and free) at:
http://www.rawtherapee.com/?mitem=3&artid=45
Jack: "I have PS CS4 with Raw Editor 5.6 , and Silkypix, as it came with my camera."
DM: Your recently purchased LX3 likely came with a recent version of Silkpix DS SE. Could you verify that the version that you have installed is Version 3.x (and not Version 2.x)? See "Help" - "Version Information".
Silkypix DS (Version 3.021 for Windows, Sep 18, 2009) SE is at:
http://www.isl.co.jp/SILKYPIX/english/p/support/download/
Did you you use particular "default" settings in and Silkypix when doing your comparisons (or particular favorite settings relating to NR, Sharpening, False Color Correction, and Demosaic-Sharp)?
An interesting factoid about the Silkypix "Noise Level" setting:
"Noise level on the NR tab has a dual effect. It sets a threshold below which no sharpening is applied and therefore has some effect even if Noise cancel is zero."
http://www.shortcutinc.com/forums/showthread.php?t=334
Jack: "I found the Silkypix images to be consistently less detailed/more smeared than the PS images."
DM: At (default) settings, or over any other (favorite or otherwise) ranges of adjustment (of NR, Sharpening, etc.) in the two different programs?
Jack: "The degree of difference seemed to be about the same whether I was using images from shots taken with -2NR, or 0 NR camera settings."
DM: Interesting. Did you try to compare the different shots (at the two different camera NR settings) in Silkypix *alone*, as well?
Jack: "I'd appreciate hearing from you or Odklizec as to how CS4 raw editor 5.6"
DM: I've never used and do not own PS. Have used PaintShop Pro 9 and Silkypix DS SE, played with RAWtherapee 2.41, and have recently been using the trial version of DxO Optics Pro 6.11 (which will soon support the LX3 RW2 image files). Their Panasonic FZ30 (JPG only) module, and processing yields superior results on my FZ30 JPGs to anything that I have been able to accomplish with (loss-less) JPG post-processing using PSP 9.
I've just purchased Dxo Optics Pro 6.11 Standard Edition (being offered at 30% discount = $109 through Dec 31, 2009 (Greenwich Mean Time in France).
That's quite a bargain compared to Silkypix DS Pro or Version 4 or (to a considerable degree) the Adobe offerings, yet the capabilities of their RAW/JPG processing appears formidable and widely respected. See:
http://www.dxo.com/intl/photo/dxo_optics_pro
(Not knowing whether or not "Odklizec" has/will read *this* comment-thread), you might try posting this question to him on the Ricoh Forum (where he is a moderator).
One interesting web-page that I have found in my meanderings is:
http://alphacorner.eu/index.php/porady/wywolywanie-arw-rawow-a-szumy-porownanie
Right-clicking and selecting "Play" in the Flash-menu will bring up the "Wybierz Program" display section of the web-page.
One is able to compare the PS RAW processing plugins at different settings: Lightroom 2.5 (and 3 Beta); RAWtherapee 2.4; DxO Optics Pro 6; Capture One 5 (and a couple of other applications).
Unfortunately Silkypix is not included in the "gamut" there.
While I will continue to use Silkypix for my Panasonic FZ50 RAW image file processing (which is not compromised as in the case of the LX3 RW2 image files), the formidable auto-corrections (for *all* optical distortions of the supported cameras) and processing power of DxO Optics Pro is looking like it may well become my "tool of choice" for LX3 RW2 image files. The DxO LX3 "module" is slated to be released by the end of Jan, 2010.
Best Regards, Detail Man
A REQUEST FOR DISCLOSURE BY SHORTCUT SOFTWARE
A request that the Shortcut Software Forum Administrator Roberto Ross determine and disclose exactly what is (or is not) going on with Silkypix DS SE (the Panasonic-camera-only versions) where it comes to processing LX3 RW2 image files has been made on the Shortcut Software Forum.
See the following post entitled:
"What (exactly) does Silkypix SE do (or not do) with LX3 RW2 Image Files?" (posted on Dec 30, 2009, at 9:55 PM PST) at:
http://www.shortcutinc.com/forums/showthread.php?t=709
Greetings Detail Man,
It's late here, but I thought I'd respond at least in brief for now.
My Silkypix version is 3.0 SE . I clicked on the files in my computer to 'open' them in Silkypix. I did nothing further to the images in Silkypix except zoom to 100%. Likewise in PS I merely 'opened' them in camera raw. I examined the images in camera raw, zoomed them to 100%, and compared them to the Silkpix image from the same RW2 file split screen style, moving around the images looking at and comparing different areas.
I do have a set of raw shots of the same scenes taken in pairs, one shot being -2 NR, and the other 0 NR. I haven't compared shots from those pairings with each other in Silkpix yet, but I'll try and do that in the next few days, and report what about what I see.
Good luck with the DXO Optics software. Thanks for the link to the Wybierz Program. I'll check it out when I get a chance. Thanks for pursuing this NR matter in such an intelligent, and comprehensive way. It feels great to be getting some clarity ! Cheers, Jack
Oh, I forgot to add, I have a Mac. As far as I know RAWtherapee isn't available in a Mac version? Do I have that right?
Jack,
Jack: "I do have a set of raw shots of the same scenes taken in pairs, one shot being -2 NR, and the other 0 NR. I haven't compared shots from those pairings with each other in Silkpix yet, but I'll try and do that in the next few days, and report what about what I see."
DM: That would be great if you have the chance. The more information the better in our empirical attempts to discern what is (or is not) going on with the LX3 and Silkypix DS 3.x SE!
Jack: "As far as I know RAWtherapee isn't available in a Mac version?"
DM: I believe that you are correct.
Jack: "Good luck with the DXO Optics software."
DM: Thank you! I am really impressed with DxO Optics Pro performance (on only JPGs so far), and very much looking forward to the soon upcoming release of their LX3 module!
BTW - DxO Optics Pro 6 is going to be released in early 2010 for Mac OS X4 - X6. Note that it can presently be pre-purchased at the 30% discount (for $109 USD) *if* you purchase it from DxO by midnight GMT (in France) on Dec 31, 2009 (which would be 7:00 PM EST in the US).
The interface is friendly, the compensation (with the upcoming LX3 module) for *all* LX3 optical distortions/aberrations is comprehensive and automatic. The quality and sophistication of the Noise Reduction, as well as the "Dxo Lighting" compensation (for highlight and shadow levels) is renowned and quite possibly unparalleled in the industry.
Also, they include a Plugin that works in/with Adobe PS, to boot!
It sure is a *fine* value at $109!
And no, I don't get a commission ... ;-) I'm just enthusiastic!
Detail Man
Detail Man,
Thanks for the DXO Optics special offer info. It's intriguing, but I don't have time to check it all out today before midnight. I'm open to it in the future though, and I'll be interested in hearing how you find it.
PS CS4 camera raw 5.6 seems to fully, and ably do the geometric lens distortion corrections ( barrel, pincushion ) of the images from LX3 RW2 files. Other possible automatic LX3 RW2 'corrections', like NR in PS, are still a bit of an open question, as we have been discussing. On another topic, I'm finding I can do a lot with "levels" in PS, and then there is "curves" ,if required, but the DXO lighting feature sounds good. - Jack
Detail Man, I can't say it's the least impulsive thing I've ever done... but, I bit the bullet, and went for the DXO Optics Pro 5 -Standard Edition ( with their promise of a free upgrade to Pro 6 for Mac, along with its LX3 module when it becomes available sometime in the first quarter of 2010). I say it feels a bit impulsive, because I'd never really looked at, or researched DXO until this morning, I already have PS CS4, etc.. I value your opinion though, and was very impressed with the DXO website, and their description of their corrections. Anywho... I got the pre midnight 12/31 $109.- U.S. price thanks to you! I have to leave for work, and won't be activating the DXO software for a few days perhaps, and until the LX3 module comes out I won't be able to try it fully, but it does look promising . So once again thanks, and Happy New Year! - Jack
Jack,
Good show about your timely acquisition of DxO Optics Pro! You will find that it (with the LX3 module) will be able to correct not just the barrel/pincushion distortions of the LX3 lens, but also: chromatic aberrations, including purple fringing (which has to be performed manually in Silkypix);
light falloff (darkened corners);
and lens softness around the edges and corners of the image.
A couple well written reviews (with sample images) can be found at:
http://www.kenrockwell.com/tech/dxo/optics-pro.htm
and
http://www.photographyblog.com/reviews/dxo_optics_pro_6_review/
As owners of the LX3, we are fortunate that they are characterizing, and releasing a module for, our camera (as they almost exclusively concentrate on much fancier and higher priced DSLRs).
The "road-map" for upcoming module releases is at:
http://www.dxo.com/intl/photo/dxo_optics_pro/product_editions/roadmap_dop_modules
I am finding the Technical Support personnel to be excellent - intelligent and responsive.
Further investigation (on your behalf) seems to indicate that the Adobe Plugin that comes with DxO Optics Pro works with Lightroom only (which you did not say that you presently own).
However, not only is DxO Optics Pro (at your discounted purchase price) less than half the cost of Lightroom 2, but I have read several accounts that using Optics Pro itself (to make all adjustments) yields far superior results to that of Lightroom.
So far, it's all quiet on the Shortcut Software (Silkypix) Forum front regarding "What (exactly) does Silkypix SE do (or not do) with LX3 RW2 Image Files?" at:
http://www.shortcutinc.com/forums/showthread.php?t=709
I would not be surprised if the forum moderators do not necessarily know for sure, and the forum administrator and/or Shortcut Software will decline to comment (under the rubric of "proprietary information") ...
Panasonic (despite CNET's possible inquiries) may likely be unresponsive to inquiries, and direct inquiries to Shortcut Software (who may well stay mum).
So, until such a time as reliable information relating the primary issue at hand (LX3 RW2 spatial-frequency detail when "raw" processing) emerges, I won't be further filling up Serious Compacts' hard-disks with comments.
I guess that folks may well just have to figure things out for themselves (so do stay on the case, Jack, and I will be investigating, as well).
All the "focus-pocus" surrounding these mattershas sent me over to DxO Optics Pro for my LX3 RW2 processing (instead of using the present Silkypix DS SE version for Panasonic cameras).
There are only two features that I find Silkypix has that DxO Optics Pro 6.x presently does not:
(1) A visual "color wheel" display depicting the status of the fine control-settings (which I find to be a useful visual mnemonic); and
(2) The ability to generate smooth, gradually-slope-changing Tone Curves with a minimum number of active points (such as only two points) - as opposed to the maddening process of trying to construct smooth step-wise approximations using many, many active points. I (for one) do not want to have to purchase the DxO "Film Pack" just to be able to (easily) construct such smoothly-sloping tone curves ...
I am presently in the process of "lobbying" DxO to add these useful features to the Optics Pro 6.x user interface, as well. Please feel free to join me in contacting DxO Technical Support if the above features would, indeed, be desirable to you as a DxO Optics Pro user!
Thanks for the links to the DXO Optics Pro reviews, and for the info about the Adobe plug in. Unfortunately, I don't have Lightroom, but I can manage without using DXO Optics Pro as a plug-in. I am interested to see what the DXO software can do, particularly when the LX3 module is released. It will be interesting to compare those 'tailored to the camera/lens' results with those I get from CS4. The LX3 seems to do so many things right. I'm hoping to get a bit more sharpness and fine detail in it's images without compromising their strengths.
I've been sort of 95% satisfied with what I've been getting from the LX3 RW2 files in CS4. I know it's a quest for the another 2 or 3 % , but that 2 or 3% can in make a "+50% "difference in terms of things clicking in, or being right, or convincing ! ( like audiophiles who spend 1000% more on their sound systems for an additional 3% of sound quality , but its money well spent for them because that additional 3% makes all the difference)
I haven't noticed purple fringing much at all in the LX3 RW2 images in CS4. Not sure why ? - lens maybe? ...or other? All the lens geometric - barrel/pincushion distortions seem well corrected automatically in CS4, per info embedded in the LX3 RW2 files. I'm wondering now about possible automatic NR and/or sharpening done in CS4 per instructions embedded in the raw (RW2) files. Maybe PS does those things automatically as well ?, and just does them better than Silkypix ? I've never heard that they do, but at this point I wonder.
Anyhow, it's all hard to sort out. I may do a few modest test of my own in the future, as I mentioned in earlier posts, but my main concern is just to get the best , or most personally satisfying, images I can from my LX3 RW2 files. DXO's approach of applying tailor-made corrections sounds promising. I'm glad about my purchase. I just had to see what it will do once the LX3 module is released.
My workflow with the LX3 RW2 files in CS4 has been to only sometimes, when needed, use the eyedropper in the raw editor 5.6 to change WB. I then open the image , and again, only sometimes, when needed, apply what are usually slight levels adjustments. That's basically it. I've experimented with sharpening, unsharp mask, but I haven't been satisfied with the results. I hoping I'll be happier with the way DXO 'retrieves' (to use an optimistic word) detail.
One last word for now Detail Man. It is the holiday season, both in the West, and in Asia, and in Japan in particular surrounding the new year. Hopefully , there will be a response from CNET. Panasonic themselves, or others to your questions in the coming weeks.
With Panasonic (with or without diligent inquires from CNET)likely to remain a silent corporate monolith(and, at, best, likely to point to Shortcut Software for answers regarding their ".RW2" image file processing in Silkypix), ...
and little clarity seeming likely to be forthcoming from Shortcut Software via inquiries made in posts on their Support Forum,
I am continuing the LX3 saga (and the DxO Optics Pro 6.x sizzle) on the Ricoh Forum DMC-LX3 discussion thread starting at:
http://ricohforum.com/phpbb/viewtopic.php?f=22&t=1556
.
With Panasonic (with or without diligent inquires from CNET)likely to remain a silent corporate monolith(and, at, best, likely to point to Shortcut Software for answers regarding their ".RW2" image file processing in Silkypix), ...
and little clarity seeming likely to be forthcoming from Shortcut Software via inquiries made in posts on their Support Forum,
I am continuing the LX3 saga (and the DxO Optics Pro 6.x sizzle) on the Ricoh Forum DMC-LX3 discussion thread starting at:
http://ricohforum.com/phpbb/viewtopic.php?f=22&t=1556
.