From failed starch works to a wartime asset, this brick ruin has seemingly always been an unwanted castle of a forgotten island.