One Backbone, Three Heads: Multitask Learning for Scene and Emission Estimation
How a single ResNet-50 trunk can jointly predict Places365 scene categories, binary scene attributes, and a five-class carbon emission estimate — and what actually breaks when you fine-tune one head in isolation.
Why one model instead of three
The Sustainable Vision project needed three predictions from a single photograph: which of the 365 Places categories the scene belongs to, which binary attributes describe it, and roughly how carbon-intensive the depicted environment is. The naive approach is three independently trained networks.
That is wasteful and, more importantly, it throws away the strongest available signal. Carbon intensity is not a property you can read off pixels directly — it is inferred almost entirely from what the scene is. A highway, a coal plant, and a forest trail have wildly different emission profiles precisely because they are different scenes. Sharing a trunk lets the emission head borrow representations the scene head has already learned.
The architecture: shared trunk, three heads
The backbone is a ResNet-50 pretrained on Places365. Above the pooled feature vector sit three independent heads: a 365-way softmax for scene category, a sigmoid layer for binary attribute prediction, and a five-way softmax for emission level running from very low to very high.
Each head has its own loss, and the training objective is a weighted sum. Weighting matters more than the architecture. Left unweighted, the 365-way scene loss dominates the gradient and the emission head learns almost nothing.
class MultitaskResNet(nn.Module):
def __init__(self, n_scenes=365, n_attrs=102, n_emission=5):
super().__init__()
backbone = resnet50(weights=Places365_Weights.DEFAULT)
self.trunk = nn.Sequential(*list(backbone.children())[:-1])
feat = backbone.fc.in_features
self.scene_head = nn.Linear(feat, n_scenes)
self.attr_head = nn.Linear(feat, n_attrs)
self.emission_head = nn.Sequential(
nn.Dropout(0.3), nn.Linear(feat, n_emission)
)
def forward(self, x):
z = self.trunk(x).flatten(1)
return self.scene_head(z), self.attr_head(z), self.emission_head(z)Three heads over one trunk. The emission head gets dropout because its label set is the smallest and overfits first.
The label scarcity problem
Places365 gives you abundant scene labels. Nobody gives you abundant carbon emission labels for arbitrary photographs. This asymmetry is the central difficulty of the project, and it is a very common shape of problem in applied machine learning: the task you care about has the least data.
The workable answer is to treat the emission head as a small supervised layer over a representation learned from the abundant task, and to accept that its confidence intervals are wider than the scene head. Pretending otherwise produces a model that looks precise and is not.
When one head has orders of magnitude fewer labels than another, do not report their accuracies side by side without saying so. A 91% emission confidence and a 91% scene confidence are not the same claim.
Fine-tuning without catastrophic forgetting
To adapt emission estimation to a narrower domain, the model was fine-tuned on the Intel Image Classification dataset. The critical constraint: fine-tuning touches only the emission head. The trunk and the scene head stay frozen.
Unfreezing the trunk during a short fine-tune on a small, narrow dataset is the fastest way to destroy the Places365 representation that makes the whole architecture work. You gain a point of emission accuracy and lose the scene classifier entirely. Freezing is not a shortcut here; it is the correct decision.
Reading the output honestly
A representative inference returns a top-5 scene distribution — street at 47.9%, downtown at 6.5%, and so on — alongside attribute probabilities and an emission estimate of medium at 91.6%.
That 91.6% deserves scrutiny. A five-class softmax trained on limited labels is systematically overconfident, and the number reflects the model committing to a bucket rather than a calibrated probability that the true label is medium. Temperature scaling on a held-out split brings reported confidence much closer to observed accuracy, and costs nothing at inference time.
Sample scene top-2 vs emission confidence
Public sample output — not a held-out accuracy claim. Scene probability mass is spread; emission confidence is concentrated.
Source — Sustainable Vision README inference example
Sample inference output (public README example)
| Head | Top prediction | Reported confidence |
|---|---|---|
| Scene (Places365) | street | 47.91% |
| Scene (2nd) | downtown | 6.50% |
| Emission level | medium | 91.55% |
Which checkpoint ships
Two checkpoints came out of training: the Places365 base model and the Intel fine-tuned variant. The fine-tuned checkpoint is the deployment default because emission estimation is the product-facing task, and it performs materially better there.
The base checkpoint is kept and documented rather than deleted. If a future use case needs scene classification without Intel-specific emission adaptation, retraining from scratch to recover it would be an expensive way to undo a decision that a stored artefact already handles.
What generalises from this
Multitask learning is usually presented as an efficiency trick — fewer parameters, one deployment. In practice its real value is the transfer of representation from a data-rich task to a data-poor one that you could not train well in isolation.
The design questions that actually determine success are unglamorous: how you weight the losses, which parameters you freeze during adaptation, and whether you report calibrated confidence. Get those right and a standard ResNet-50 is more than enough backbone.