Friday, October 24, 2014


Thursday, May 1, 2014

Astrophotography DSLR RAW Data Calibration: preprocessing

This thread has its origins in Craig Stark’s article about the non-linearity of DLSR RAW data, as well as discussions on various forums about preprocessing of DSLR RAW frames.

As a Linux user Pixinsight was the first and for a long time the only astro preprocessing application I have used, until very recently - cutting my teeth painfully. Having said that, Pixinsight is a brilliant program and it was my application that was at fault, assuming that the PI method, essentially 16 bit, was appropriate to DSLR RAW data. Exhaustive practical assessments led to the procedure described below, with much better and more consistent results.

Astroart, is however, a much easier application to use for DSLR preprocessing because the presets, strangely enough, conform to the following. So this is not new by any means. It is more likely that DSLR preprocessing has somehow been transformed by well meaning 16 bit conventional wisdom improperly advised.

The non-linearity of DSLR RAW data (compared to 16bit CCD linear data) is a product of camera firmware corrections designed to create a RAW image that is easily processed out of the box! Non-linearity is a technical term for data that has been manipulated in some way from the RAW linear state captured by the CMOS sensor. This is the case with DSLR RAW images (data) to produce a nice picture.

Because DSLR RAW data is manipulated in camera, it has already been processed to a point and requires unique handling during preprocessing; that is, during the calibration process. To avoid further data loss, typically experienced by applying 16 bit processing methods to DSLR RAW data; the following is recommended

1. Do NOT bias subtract the dark frames or the light frames, as you would if scaling the dark frames.
2. DO bias subtract the flat frames only.
3. DO subtract the master dark frame from the light frames - bias in the dark method.
4. Do NOT apply pixel rejection algorithms to bias, dark or flat frames - avoid data loss.
5. DO take flats and bias at the lowest ISO.
6. DO apply pixel rejection algorithms appropriately to calibrated light frames.

The issues discussed here were more evident when preprocessing cooled, temperature regulated, DSLR images, and does not rule out the same application to uncooled DSLR RAW data. However, if you have one of those JTW supercooled DSLRs, perhaps dark frames are a thing of the past. A carefully constructed master bias and flat may be all that is required.


“The point of this discussion is, that problems arise with DSLR RAW data when dark frames are bias subtracted and then the master dark frame is subtracted from the light frames, as well as subtracting a master bias - as one would do if scaling darks. This is a process normally applied to linear data and is not suited to DSLR RAW data calibration. Linear processes applied to DSLR data may produce less than optimal and sometimes devastating results due to data truncation.”

Because DSLR RAW data is processed in the camera, it is “clean” to begin with, and applying pixel rejection algorithms to calibration frames is unnecessary. However light frames are exposed to unfriendly light sources and pixel rejection is necessary.

A note on bias frames - CMOS sensors do not have a bias, and instead, produce a pattern of fixed noise, which, to all tense and purpose, may be considered bias and applied in the same way.

The practice of not calibrating dark frames is common place, and has been for years; “the-bias-is-in-the-dark.” However, the advantages and reasons, applied to DSLR RAW data, may not be clearly understood - “it’s always been done that way.” Some software defaults to dark scaling and is not suitable for DSLR RAW data.

“A typical DSLR RAW calibration data set, comprises uncalibrated darks and bias subtracted flats. Flats and bias frames can be taken at the cameras lowest ISO setting.” Denoising the master flat frame is recommended by some imagers.

AstroArt preprocessing is a one time operation, and uncomplicated. In PixInsight, master bias, master dark and master flat frames are prepared separately to avoid bias subtraction of dark and light frames. Either way, pixel rejection is applied to light frames only.

Thursday, October 31, 2013

Welcome to FlatPress!

This is a sample entry, posted to show you some of the features of FlatPress.

The more tag allows you to create a “jump” between an excerpt and the complete article.

[Read More…]

Saturday, December 29, 2012

The Black Generic: a Trout fly

For something completely different. “The Black Generic” (say that with a thick Scottish accent). Presented here for posterity. See further down, for a small snail pattern that was very effective in small dams, too.

“The Generic” (which never had a name, until yesterday) was an attempt to accentuate the form of nymphs - and it worked. The hook shank bent upwards, giving the body a curved posture, the long flared tail, tight skinny abdomen and bulbous bushy thorax were intentional. I noticed that variations were less successful. A thinly tied or short fiber thorax being the main flaws. It needs to look, curvy, leggy, cheeky and thoroughly provocative - hey! look at me, I’m here, come and get me!

My rough sketch doesn’t do it justice and I no longer have my fly tying gear.


Fished, in shallow runs, deep, sinking and on the rise, this fly caught plenty of fish in streams North of Melbourne, some 25 years ago, when I had the time to fish - Deep Creek and Jacksons Creek in particular, when they were healthy, before the drought devastated them. I understand that recent rains have regenerated the area and that fish are more plentiful.

The appearance is a long bulging thorax with a very bushy hackle to 2/3 of the body length. The abdomen is thinly wound single layer of silk with tightly wound silver wire segments and a long flared tail.


Hook size: 14-16 longshank - slightly bent upward ~2/3 from the eye.
Body : Black silk.
Tail: Soft black hen 6-9 curved fibres
Segment: Silver wire - anterior only.
Carapace: Black crow wing.
Hackle: 3 dark brown - black Ostrich herl with long fibres.
Head: Tie off crow wing with 3 half hitches and clip to cover eyelet - a flared finish.


Starting near the hook bend (not the 2/3 bend), tie in the tail, flaring with turns of silk. Tie in the wire and wind the body tightly (a single layer) to the bend (2/3 bend). Wind in narrow segments with the wire and tie off and clip at the 2/3 bend.

At and continuing from the 2/3 bend, tie in a good bunch of crow wing and several Ostrich herl. Wind silk to eye and half hitch. Wind Ostrich herl to within ~1/2mm of the eye and secure with half hitches. Bring crow wing over and tie off with half hitches immediately behind the eye and trim to cover the eye.


It may be necessary to tease out the hackle under the crow wing with a dubbing needle.

You may wax the thread, but I don’t. I prefer the fly waterlogged and sinking fast in moving or deep water.

Tying the crow wing shiny side up is an alternative presentation.

Small Snail pattern:

A second and very effective little fly, was a very small snail pattern. Hook size 16. Olive thread. A few winds of olive chenille with an olive carapace and a few turns of yellow chenille. About half and half.

Fished above weed beds in particular. Very easy fly to make.

Monday, September 24, 2012

Dithering: a DSLR astrophotographers best friend

There has been a lot of interest in this thread in recent times so I’ve provided a short summary, mainly for the benefit of newcomers to DSLR astrophotography - I hope it’s helpful.

Dithering, is in my view, one of the most useful and effective techniques applied to image acquisition for increasing SNR, applied to DSLR cameras, saving hours of processing time and frustratingly poor results.

There has been a bit of discussion on various forums about substituting dithering for calibration. Sounds attractive, but;

***dithering is not intended to be a substitute for calibration - I recommend reading this, as well***

Dithering and image reduction serve different purposes with the same aim - increased SNR. Temperature must be considered when using DSLRs, requiring dark subtraction. Dithering however, will hide the majority of temperature related calibration errors/inaccuracies as well as several other types of artifacts… read on.

You can also read about dithering in Berry and Burnell’s, “Handbook of Astronomical Image Processing,” where they recommend displacement of images by at least 12 pixels. There are several informative academic papers on-line, as well.

Backyard EOS has dithering capability, however, I have never used the program. My setup is Arduino based, controlling the RA or DEC axis between images, simply by slewing to present the camera to the target, displaced by 10 - 15 pixels or more between images - it’s that basic.

The comparison image, is intended to accentuate the underlying issues with the image on the right. No attempt has been made to minimize the effect with post processing. The image was stacked and stretched - please note that the red streaks were not evident in individual subs and only appeared after integration. In-fact, I naively spent hours trying to salvage that image - a complete waste of time. The image on the left was taken with the same camera, dithered.

Rather than spending time eradicating/covering up unsightly problems, time was spent lifting out detail, which in the image on the right was partly obliterated by poor acquisition - no dithering.


Here is the pattern I follow… it keeps the image within the sensor boundary. I use a look up table in the Arduino program to schedule the correct hand controller button activation.


More info

Struggling to produce a decent image, it became apparent that the physical and electronic characteristics of DSLR CMOS sensors demand more attention and an understanding of the limitations of the camera’s sensor was essential.

I went to the trouble of replacing the factory IR filter with a special purpose astronomical filter to increase the transmission of Ha wavelengths and built a sensor cooling system to reduce dark current/thermal noise - this all worked quite well. But, there is more to sensor technology and manufacture than meets the eye.

For instance, the anti-aliasing filter, which forms part of the dust reduction system in Canon DSLR cameras is designed to reduce Moire, an artifact produced by the Bayer (RGB) colour matrix. However, the reduction of moire tends to soften the image. Even with a focus mask at ‘perfect focus’ - consequently, the AA filter should be removed.

But that’s not the end of the story. Among the millions of pixels that make up the sensor light gathering matrix, a small percentage are dead (don’t work). Some are on all the time (hot) and overall, pixels differ in their ability to convert photons to electrons. For daylight photography this isn’t a problem, as a rule.

There’s more. The pixel matrix and associated electronics produce fixed pattern noise. Heating of the sensor during extended operation also increases noise. Random noise is a function of arriving photons and is different for every frame. The optical properties and cleanliness of the sensor, filters and lens also produce artifacts.

Combined, this all conspires to ruin the image to which you have dedicated copious amounts of time at the wrong end of the day, when perhaps, you should be sleeping.

So what’s the solution? In principle, reducing the effects of the deleterious electronic and physical influences inherent in the optical system is quite simple, in practice however, it involves another layer of complexity; that is, DITHERING!.

Dithering is an authentic solution because it addresses noise suppression, optical and sensor artifacts - dithering does not replace proper image calibration techniques. It will however, greatly improve results and avoid problems that no amount of calibration or sensible image processing can resolve.

So, what is dithering?

Dithering is the practice of shifting the sensor (camera) between images, so that each new image is slightly offset from the previous image. The image is sampled by different pixels; that is, the image moves, ideally, by 10 - 15 pixels, from the position of the preceding exposure. With careful management the target image remains well within the sensor boundary.

Dithering can be random using a hand controller and estimating the offset by timing the button push in DEC or RA. Better still, an automated system that dithers in a box shaped spiral, or maze pattern. The goal is to avoid a succession of images occupying the same or neighbouring pixels (which produces poor results) and to prevent the target object moving out of the FOV.


Dithering, particularly with DSLR cameras improves signal to noise ratio for very little effort. It can be win win for the astrophtographer.

Executed properly, dithering deals effectively with random noise, hides hot and cold pixels, improves flat fielding and sub-pixel sampling; that is, capturing the image over a range of pixels means that we are not sampling the same and possibly less efficient pixels repeatedly for the same object location.

Calibration is not always as effective as we would like with DSLR images. And even if the images weren’t calibrated, a dithered stack would produce pretty good results.