VR Tests, Pass 3 Part 2: Not Impressed

I’m looking for expert opinion, but my current conclusion is that my lens is broken. I’m supposed to get a 3-stop increase in handholdability from VR on this lens, and I don’t get anything like that. I still want comments on whether I’m being unreasonable about what I categorize as “critically sharp”. I think I’m being somewhat lax, myself; if it looks good at 1:1 I’ll call it critically sharp, and not inquire further.

So, here’s a set of carefully-shot tests, at increasing shutter speeds. Each setting I shot 10 times; then I rested for at least 5 minutes before resuming (I don’t think my arms and hands are really that shaky, but I was trying to give the lens every chance). I was using aperture priority, and adjusting the ISO, so I didn’t really get 10 shots the same each time; apparently I was too close to a boundary, so I got some at one shutter speed and some at another. They’re named and categorized based on the shutter speed recorded in the EXIF data. All test shots are at 200mm and f/5.6. The camera is the Nikon D700. The lens is the Nikkor 70-200/2.8 VR AF-S G. Click through to see the 100% crop of the center of the frame for each test. See the tripod results (showing what really sharp results look like, and what the full frame looks like before cropping down to just the center) here.

At 1/15 VR off: Good 0, marginal 1 (#1), bad 0.

1/15 VR on: Good 1 (#2), marginal 2 (#3,4), bad 1 (#1).

1/20 VR off: Good 2 (#2,7), marginal 0, bad 6 (#1,3,4,5,6,8).

1/20 VR on: Good 1 (#6), marginal 3(#1,3,5), bad 2(#2,4).

1/30 VR on: Good 0, marginal 3(#1,2,3), bad 0.

1/40 VR off: Good 1 (#4), marginal 0, bad 9 (#1,2,3,5,6,7,8,9,10).

1/40 VR on: Good 0, marginal 5(#1,2,4,6,7), bad 2(#3,5).

1/60 VR off: Good 1(#2), marginal 3(#6,7,9), bad 5(#1,3,4,5,8).

1/60 VR on: Good 0, marginal 0, bad 3(#1,2,3).

1/80 VR off: Good 0, marginal 0, bad 1(#1).

1/80 VR on: Good 2(#2,5), marginal 1(#3), bad 3(#1,4,6,7).

1/125 VR off: Good 1(#1), marginal 0, bad 0.

1/160 VR off: Good 1(#6), marginal 4(#1,7,8,9), bad 4(#2,3,4,5).

1/160 VR on: Good 8(#2,4,5,6,7,8,9,10), marginal 1(#3), bad 1(#1).

Once again, I get a surprising proportion (i.e. greater than 0) of usable shots at 1/20 and 1/40 seconds without VR. I wasn’t leaning on anything, sitting, resting my arms on anything; I was standing up all by myself, and holding the camera in my own two hands. And I get very little improvement in sharp shots, few critically sharp shots, when VR is turned on.

Anybody else with this lens, do you get better VR results than this? Is mine broken?

Notes for future testing: Target is good.  Procedure (with rest) is good. Need to use manual exposure. Need to have more light, so I can try some higher shutter speeds.

Continue reading VR Tests, Pass 3 Part 2: Not Impressed

VR Tests, Pass 3, Part 1: The Manual is Right

Yes, this topic has been annoying me quite a bit; why do you ask?

I was doing a set of tripod-based tests tonight, under the conditions I’ve established for the planned hand-held series, to produce example of just how sharp this target should look through this lens. (One of the questions last time was whether I was expecting too much sharpness.)

I was shooting very carefully, doing nearly everything I could to eliminate sources of vibration: tripod, LiveView mode (so the mirror was up), remote release (so my touching the body didn’t introduce any vibrations), contrast-detection focus (so any alignment issues in the other AF system won’t cause focus errors).

I shot 5 copies of each test case; each group of 5 looks substantially the same, indicating that my procedures were at least consistent from shot to shot.

Here’s one of the full frames, reduced to a sensible jpeg size, to give you an idea of what the setup was. All the images after this will be 100% crops from the center when you click through the thumbnail.

Full frame of 1/13sec VR off exposure
Full frame of 1/13sec VR off exposure

First, here are 100% crops from a few shots without VR at various ISOs and shutter speeds:

VR off, 1.30sec
VR off, 1.30sec
VR off, 1/25sec
VR off, 1/25sec
VR off, 1/13sec.
VR off, 1/13sec.

Now, here are 100% crops from a few shots with VR at the one ISO (Lo.1) and shutter speed (1.30sec) I tested:

VR On, 1.30sec
VR On, 1.30sec
VR on, 1.30sec
VR on, 1.30sec
VR on, 1.30sec
VR on, 1.30sec

That, as the saying goes, is just not right.

The manual says to NOT use VR on a tripod. Based on this step of the experiment, I have to say that the manual is definitely right on this one! (The manuals for various generations of Nikon VR give different answers to whether it should be used on a tripod. I checked the manual for this specific lens, and it definitely says not to use VR on a tripod. They’re right!)

So, some day soon, the rest of the test, actual hand-held shots with VR on and off, using this same target.

And, Finally, Tokina 28-70 ATX-Pro f/2.6-2.8

I got this to use with my Nikon N90 in 1994, when I first went into autofocus. It had a very high reputation then and since, but some of my professional contacts on mailing lists reported not-so-good performance from newer and supposedly better Tokina zooms.

I was considering sticking with this as my main lens for a while after I got the D700 (which brought me back into full-frame photography), but this required testing.

Results seemed pretty conclusive to me.

28mm f/2.8

Tokina ATX-Pro 28-70mm f/2.8 at 28mm, f/2.8, corner
Tokina ATX-Pro 28-70mm f/2.8 at 28mm, f/2.8, corner
Tokina ATX-Pro 28-70mm f/2.8 at 28mm, f/2.8, center
Tokina ATX-Pro 28-70mm f/2.8 at 28mm, f/2.8, center
Tokina ATX-Pro 28-70mm f/2.8
Tokina ATX-Pro 28-70mm f/2.8 at 28mm, f/2.8

Those corners are really pretty appalling. This was a reshoot, because I was worried I’d somehow done it wrong in the initial shoot.

 

 

 

 

28mm f/4

Tokina ATX-Pro 28-70mm f/2.8 at 28mm, f4, corner
Tokina ATX-Pro 28-70mm f/2.8 at 28mm, f4, corner
Tokina ATX-Pro 28-70mm f/2.8 at 28mm, f/4, center
Tokina ATX-Pro 28-70mm f/2.8 at 28mm, f/4, center
Tokina ATX-Pro 28-70mm f/2.8 at 28mm, f/4
Tokina ATX-Pro 28-70mm f/2.8 at 28mm, f/4

Some improvement; but more, some reason to believe that the f/2.8 result wasn’t a mistake somehow.

 

 

 

 

50mm f/2.8

Tokina ATX-Pro 28-70mm f/2.8 at 50mm, f/2.8, corner
Tokina ATX-Pro 28-70mm f/2.8 at 50mm, f/2.8, corner
Tokina ATX-Pro 28-70mm f/2.8 at 50mm, f/2.8, center
Tokina ATX-Pro 28-70mm f/2.8 at 50mm, f/2.8, center
Tokina ATX-Pro 28-70mm f/2.8 at 50mm, f/2.8
Tokina ATX-Pro 28-70mm f/2.8 at 50mm, f/2.8

Considerable improvement in the corners; not so much in the center, I almost wonder if that was me? This is why sane people use tripods for lens testing, I believe. But that one was at 1/1000 sec., it really doesn’t seem likely it was camera shake.

 

 

 

 

50mm f/5.6

Tokina ATX-Pro 28-70mm f/2.8 at 50mm, f/5.6, corner
Tokina ATX-Pro 28-70mm f/2.8 at 50mm, f/5.6, corner
Tokina ATX-Pro 28-70mm f/2.8 at 50mm, f/5.6, center
Tokina ATX-Pro 28-70mm f/2.8 at 50mm, f/5.6, center
Tokina ATX-Pro 28-70mm f/2.8 at 50mm, f/5.6
Tokina ATX-Pro 28-70mm f/2.8 at 50mm, f/5.6

There we go, that’s about where it’s really supposed to be.

 

 

 

 

70mm f/2.8

Tokina ATX-Pro 28-70mm f/2.8 at 70mm, f/2.8, corner
Tokina ATX-Pro 28-70mm f/2.8 at 70mm, f/2.8, corner
Tokina ATX-Pro 28-70mm f/2.8 at 70mm, f/2.8, center
Tokina ATX-Pro 28-70mm f/2.8 at 70mm, f/2.8, center
Tokina ATX-Pro 28-70mm f/2.8 at 70mm, f/2.8
Tokina ATX-Pro 28-70mm f/2.8 at 70mm, f/2.8

Funny-looking at the long end, too; more in the middle really. Coma maybe? I haven’t done serious pixel-peeping at these yet.

 

 

 

 

70mm f/5.6

Tokina ATX-Pro 28-70mm f/2.8 at 70mm, f/5.6 corner
Tokina ATX-Pro 28-70mm f/2.8 at 70mm, f/5.6 corner
Tokina ATX-Pro 28-70mm f/2.8 at 70mm, f/5.6, center
Tokina ATX-Pro 28-70mm f/2.8 at 70mm, f/5.6, center
Tokina ATX-Pro 28-70mm f/2.8 at 70mm, f/5.6
Tokina ATX-Pro 28-70mm f/2.8 at 70mm, f/5.6

Nice in the center, okay at the corner now.

 

 

 

 

Conclusion

So, as you may have guessed by now, I’ve ordered the Nikkor AF-S 24-70 f/2.8.

Nikkor 17-55mm f/2.8

My first-ever “major brand” zoom lens (though in fact Tokina, Tamron, and these days Sigma are “major” by any reasonable standards). So my first zoom lens made by a company that makes camera bodies. No, wait, Sigma makes camera bodies (and Vivitar did, back when they were “major”, may still). Anyway, my first zoom lens carrying the same brand as my camera does.

I got this because the Tokina 28-70/2.8 and the Tamron 28-105/2.8 were too long at the wide end to be comfortable as my main lens on a DX (1.5x crop factor) body. I had the Nikkor 18-70 kit lens for a while (bought used), but I got too much flare in situations that shouldn’t really have given me flare, and I fairly quickly bought this and sold the 18-70 on. This was a fine lens for me, and I regret selling it. I’ll regret it even more if I end up with another DX body after all; but I can’t afford to hold it unused while spending the money for the 24-70/2.8 that I’ve ordered.

Testing this is aimed at providing good examples.

I shot these tests on the D700 in DX mode, so the originals are only 5.1 megapixel images.

17mm f/2.8

Nikkor AF-S 17-55mm f/2.8 DX at 17mm, f/2.8, center
Nikkor AF-S 17-55mm f/2.8 DX at 17mm, f/2.8, center
Nikkor AF-S 17-55mm f/2.8 DX at 17mm, f/2.8, corner
Nikkor AF-S 17-55mm f/2.8 DX at 17mm, f/2.8, corner
Nikkor AF-S 17-55mm f/2.8 DX
Nikkor AF-S 17-55mm f/2.8 DX at 17mm, f/2.8

This ought to be the worst case, everything at its widest. Not bad, eh?

17mm f/8

Nikkor AF-S 17-55mm f/2.8 DX at 17mm, f/8, corner
Nikkor AF-S 17-55mm f/2.8 DX at 17mm, f/8, corner
Nikkor AF-S 17-55mm f/2.8 DX at 17mm, f/8, center
Nikkor AF-S 17-55mm f/2.8 DX at 17mm, f/8, center
Nikkor AF-S 17-55mm f/2.8 DX at 17mm, f/8
Nikkor AF-S 17-55mm f/2.8 DX at 17mm, f/8

Still not truly wonderful in the corners; in fact not much improved from f/2.8. But I think that’s more to the credit of f/2.8 in this case.

55mm f/2.8

Nikkor AF-S 17-55mm f/2.8 DX at 55mm, f/2.8, corner
Nikkor AF-S 17-55mm f/2.8 DX at 55mm, f/2.8, corner
Nikkor AF-S 17-55mm f/2.8 DX at 55mm, f/2.8, center
Nikkor AF-S 17-55mm f/2.8 DX at 55mm, f/2.8, center
Nikkor AF-S 17-55mm f/2.8 DX at 55mm, f/2.8
Nikkor AF-S 17-55mm f/2.8 DX at 55mm, f/2.8

55mm f/8

Nikkor AF-S 17-55mm f/2.8 DX at 55mm, f/8, corner
Nikkor AF-S 17-55mm f/2.8 DX at 55mm, f/8, corner
Nikkor AF-S 17-55mm f/2.8 DX at 55mm, f/8, center
Nikkor AF-S 17-55mm f/2.8 DX at 55mm, f/8, center
Nikkor AF-S 17-55mm f/2.8 DX at 55mm, f/8
Nikkor AF-S 17-55mm f/2.8 DX at 55mm, f/8

Mmmm, sharp. Well, still far from perfect in the corner.

Tokina 12-24mm f/4

Officially it’s “AT-X 124 AF PRO DX”. Their first shot at a crop-factor DSLR lens, and quite a big success; lots of people on the net like it for example.

A previous round of test shots when I first got this lens is here.

And why am I testing this when I’m leaving DX and about to sell it? Well, I needed to test my old 28-70, and I decided I should include other lenses as sanity-checks, so I’m looking at more than one set of results.

 

 

 

 

12mm f/4

Tokina AT-X124 Pro DX 12-24mm F4 at 12mm, f/4, center
Tokina AT-X124 Pro DX 12-24mm F4 at 12mm, f/4, center
Tokina AT-X124 Pro DX 12-24mm F4 at 12mm, f/4
Tokina AT-X124 Pro DX 12-24mm F4 at 12mm, f/4
Tokina AT-X124 Pro DX 12-24mm F4 at 12mm, f/4, corner
Tokina AT-X124 Pro DX 12-24mm F4 at 12mm, f/4, corner

So not so shabby really, I don’t think.

 

 

 

 

12mm f/8

Tokina AT-X124 Pro DX 12-24mm F4 at 12mm, f/8, corner
Tokina AT-X124 Pro DX 12-24mm F4 at 12mm, f/8, corner
Tokina AT-X124 Pro DX 12-24mm F4 at 12mm, f/8, center
Tokina AT-X124 Pro DX 12-24mm F4 at 12mm, f/8, center
Tokina AT-X124 Pro DX 12-24mm F4 at 12mm, f/8
Tokina AT-X124 Pro DX 12-24mm F4 at 12mm, f/8

There really isn’t anything much wrong with this by f/8.

 

 

 

 

24mm f/4

Tokina AT-X 124 Pro DX 12-24mm F4 at 24mm, f/4, corner
Tokina AT-X 124 Pro DX 12-24mm F4 at 24mm, f/4, corner
Tokina AT-X 124 Pro DX 12-24mm F4 at 24mm, f/4, center
Tokina AT-X 124 Pro DX 12-24mm F4 at 24mm, f/4, center
Tokina AT-X 124 Pro DX 12-24mm F4 at 24mm, f/4
Tokina AT-X 124 Pro DX 12-24mm F4 at 24mm, f/4

The 24mm end is pretty fuzzy in the corner too, at f/4.

24mm f/8

Tokina AT-X 124 Pro DX 12-24mm F4 at 24mm, f/4, corner
Tokina AT-X 124 Pro DX 12-24mm F4 at 24mm, f/8, corner
Tokina AT-X 124 Pro DX 12-24mm F4 at 24mm, f/4, center
Tokina AT-X 124 Pro DX 12-24mm F4 at 24mm, f/8, center
Tokina AT-X 124 Pro DX 12-24mm F4 at 24mm, f/4
Tokina AT-X 124 Pro DX 12-24mm F4 at 24mm, f/8

Pretty nice, especially in the center. Possibly better than the Nikkor at f/5.6, even.