To clarify, I only rent the house, so capital improvements like that are beyond my concern/authority, though when I do buy it's not unlikely there will be similar things to consider.

If I did own it and had $10,000 to put into it, I bet I'd get a lot more bang for the buck out of insulation than changing the heating system. I'm not sure if the most cost-effective first step would be replacing the old windows (more than a 10,000 job in itself), doing the walls, etc., but getting rid of some of those drafts is a must. The windows are bad enough that even if "shut", without piling all sorts of foam and tape on them even in a 15mph breeze enough air gets through to make the mini blinds bang around. Those things would help no matter what your energy source.

I'm astonished that none of the previous owners has done even the most rudimentary work like that, beyond putting up the most cheap and flimsy storm windows a few decades ago. There have been quite a few mods over the years, including putting in modern baseboard radiators instead of the old clunky tall ones, and the zone controls and associated plumbing and pumps, which looks newish (within 20 years) -- a fair amount of cash all told. Why post-energy crisis in an oil-heated home in New England you'd prioritize like that, I have no idea.

As for solar, you never see anyone here using it. I can think of a few reasons. Cloud cover is one, though that's not as big of an issue as in say, Portland, Oregon, where it's entirely possible to have 29 out of 31 days in January be 100% overcast. Here, you tend to get clear but bitterly cold days after a snowstorm (i.e., sitting under an artic air mass after the cold front passes).

Higher on the list would be the fact that solar doesn't work so well when you have a foot of snow on top of your roof, as we often do in the winter. Keeping it almost always snow-free would require a much steeper pitch than it has already; the current roof prevents it from getting dangerously deep but not nearly as snow-free as you'd want for solar collectors. That's not a cheap mod and would make the house look awfully wierd. Or you could have the collectors not lying flat on the roof maybe, but that increases their wind load significantly (important not only in winter storms but summer hurricanes, which we are also at risk for). Even then you'd get wind drifts around them, and the heat from the panels would end up with them caked with ice from the snow they melt. There's also just less roof area per square foot of house to heat here than what you're used to in CA, as the houses have much smaller footprints but are taller -- and much more tightly packed, which creates more shading problems at low sun angles. I'm sure there are solutions, but really solving it may call for having shaped the house or even the neighborhood with solar in mind when built.

And of course you gotta keep plenty of antifreeze in your solar loop. It's not so bad here, but up in Northern Vermont when doing a winter weekend at a friend's family's house up there we got a cold spell where it hit -47 at night. A cup of straight antifreeze I set out overnight was slushy; a bottle of tequilla was frozen solid. I'm quite aware of how that felt, since crazy me went out at 2am to see Hale-Bopp.