It appears as though this has been resolved by a reboot of each node.
NetApp support informed that some network cards get locked after a reboot and the solution is to reboot the nodes again.
There is a KB out on this but we don't have the specific CNA's that the article mentions. This may be an extension of that issue though. NetApp pulled some logging from my system while we were troubleshooting so maybe they'll find something in there that points to the locking issue.