Controlling alarms from any device

I have a hub and audio in the same "room" and speaker group, and both are even located several feet from each other. Yet it when an alarm is going off on one device, and the other device picks up my "Hey google", I get:

"There aren't any alarms going off right now"

Even as the chime is blaring right over it.