Raw histogram and why does it matter

OK, last try at explaining RAW Histograms (in post processing) and why they matter, as simply as possible..

This time with pictures and as little techno-speak as possible. Hopefully this will help make some sense of the previous discussion. I walked outside and shot this not so exciting flower for this purpose - no great art here, just an example that will have some relevance to the discussion.

I exposed this with the most important highlight detail being recorded all the way "to the right" (ETTR), just below clipping. How do we know it's just below clipping/ Read on.

Here are the initially poor results...

The RAW file with default Lightroom import processing is on the left, the SOOC jpeg is on the right, with their respective Lightroom histograms below them. Hmm, the Lightroom histograms do not show any highlight clipping for either image yet, though the camera did better, there clearly appears to be a significant loss of highlight detail with both.

What's going on here? Well, after all the preceding discussion it should have been made clear by now that the histograms we see in our editors (most, but not all editors) are only representative of the images we are looking at on the screen after processing, and does not show what information is present in the RAW file....Hey, wait a minute, I didn't apply any processing, that's just how it opened.. while it’s true that no sliders were moved in the development of that image, if you're looking at an image from a RAW file in your editor, significant processing has already been applied to the RAW file "under the hood" before you ever get to see it and, in this case, it messed some stuff up.

Let's look at the RAW Histogram for this file (via RawDigger, thanks Iliah) that is representative of the information present in the RAW file...

As can be gleaned from this, all 3 three channels are intact. the Green channel is maxed out and extends all the way up to, but not significantly into clipping. It's generally considered wise to leave yourself something of a safety buffer, but here we have pretty well maxed-out the exposure for this image. If there was significant highlight clipping present, you would see a spike at the clipping threshold at right side of the histogram (just below 16000). Also, note where the EV 0 line is here (Donald B), for clarity, I moved it from the usual -3 EV to where the clipping threshold actually is (it's really just a handy scale to use as you see fit.)

So, the if RAW highlight detail wasn't clipped and the exposure was OK, why does the flower look like crap? Because both the in-camera processing and Lighroom's default import processing both got it wrong - usually Lightroom's (and other editors') default import processing is easy enough to work with, but sometimes it really makes a mess of things and you have to work around it (a subject for another thread). A RAW histogram won't fix any of this for you, but it can be a great diagnostic tool for troubleshooting potential exposure problems (as we did here), and can (with some trial and error) be invaluable for dialing in a reliable methodology for making precisely controlled RAW exposures (this example took me exactly one try).

My exposures are almost always as expected now, so RawDigger isn't something I use regularly, but what a great tool for helping make that previous sentence a reality and, of course, it's especially useful for making sense of any sort of exposure issues.

Here's a before and after with the same RAW file, again with the Lightroom histograms below each one. Note that despite the very significant difference in highlight detail between the two images, the two Lightroom histograms don't look all that dissimilar...

On the left is the same inexplicably blown RAW default import processing result. On the right is the same file but with different Lightroom development settings ...and how about that, the RAW Histogram was correct, there was viable highlight detail after all (but it's still not an especially good photo).

I hope some of that made sense.

The raw image file doesn't "look" like anything (except strings of code) until it's been processed into a viewable image. The image you're showing as "raw" is actually a processed image which only needs some minor adjustments to look like a "normal" image.

There is no such thing as a "raw" image which is viewable, only raw data which can be processed into an actual viewable image.

For anyone interested in some of the various stages of processing and what the image looks like at different stages, the link below gives some examples...

Processing RAW Images in MATLAB, Rob Sumner Department of Electrical Engineering, UC Santa Cruz (There are images from various stages of processing in the last few pages of this PDF.)

Uh...

The image data in a raw file is that of a monochromatic one. You can view the image if you extract it from the raw file.

It looks like this:

Cool thanks. will give them a miss and just use compressed as they are so close to uncompressed.

Don

Losslessly compressed files aren't much bigger than craw files, and can be smaller.

Thank you for your post. i was getting very tired of a certain member constantly saying my images were 3 stops under exposed when they were clearly not, just like you have demonstrated.

I haven't seen anyone in this thread say your image was 3 stops underexposed. Can you post a link to the post or copy/paste the sentence along with its author?

the problem was is that when the L lossy compressed files were processed via ACR they had less fine detail ,texture and tonal information than the compressed file, size was very similar though.

That sentence makes no sense to me. What's a L lossy compressed file? What's the other compressed file you're talking about?

Could you post a histogram for the whole frame, with a logarithmic Y-axis ?

I'll post something similar I've seen on Panasonic shortly.

Switching the y-axis to logarithmic certainly makes the data clearer visually. Yes, the clipping is clearer now. So really, I had only about 1/2 a stop of highlight headroom in the raw data reds and blues and no headroom in the greens in the original shot (_MG_1468) where the camera's histogram was beginning to show clipping.

Of course, extracting the encoded raw data and then processing that into a monochrome mosaic pattern is one early stage example of processing raw data into a viewable image. I could "extract" the encoded raw image data in other ways, with a text editor for example, but I wouldn't have a viewable image. A program which is capable of: extracting the encoded raw data, understanding what it represents, and processing (convert, transform, choose your word preference) the raw data into some type of recognizable image form is needed, if I want it to be viewable.

There are a number of steps contained in the raw processing pipeline to arrive at a full color (or monochrome) reconstruction of the recorded raw image data. The image can be viewed at various stages of processing. Thanks for sharing one example, which is a PNG file labeled as "raw" — but close enough for demonstration purposes. There are other examples at different stages which can be found elsewhere, such as in the PDF link I provided.

The posted "raw" image that I was referring to was essentially completely processed except for a few adjustments to white balance and tone to present a more normal appearance. It does open the door to the question of: just how raw is the raw image you see displayed when you're processing it in your favorite raw processing app?

sony has added 3 more raw compression formats LL LM LS for my a74. just did another test with 3 of them .uncompressed, L lossy compressed, compressed. and even at 3000 % with raw ,psd, jeg from raw i couldnt tell the images apart.

Neither does JPEG, or even TIFF.

Lossy compressed files are not even close to raw. It's either lossy, or raw.

Interesting that many were able to see the problems with lossy and forced Sony to add lossless compression.

tell that to sony

I don't talk to marketing.

Your program opens them so they must be raw and so does ACR