Availability Zones and Resource Isolation¶
Overview¶
While Ironic does not implement traditional OpenStack Availability Zones like Nova and Neutron, it provides a three-tier approach for resource partitioning and isolation that achieves comprehensive availability zone functionality:
- Multiple Ironic Deployments: Completely separate Ironic services
targeted by different Nova compute nodes
- Conductor Groups: Physical/geographical resource partitioning within
a deployment
Shards: Logical grouping for operational scaling within a deployment
This document explains how these mechanisms work together and how to achieve
sophisticated availability zone functionality across your infrastructure.
This document does not cover similar effect which can be achieved
through the use of API level Role Based Access Control through the
owner
and lessee
fields.
Comparison with Other OpenStack Services¶
Service |
Mechanism |
Purpose |
---|---|---|
Nova |
Availability Zones (host aggregates) |
Instance placement across fault domains |
Neutron |
Agent AZs |
Network service HA |
Ironic |
Multiple Deployments + Conductor Groups + Shards |
Complete service isolation + physical partitioning + operational scaling |
Ironic’s Three-Tier Approach¶
Tier 1: Multiple Ironic Deployments¶
The highest level of isolation involves running completely separate Ironic services that Nova and other API users can target independently.
Use Cases:
Complete geographic separation (different regions/countries)
Regulatory compliance requiring full data isolation
Independent upgrade cycles and operational teams
Implementation: Configure separate Nova compute services to target different Ironic deployments using Nova’s Ironic driver configuration.
Benefits:
Complete fault isolation - failure of one deployment doesn’t affect others
Independent scaling, upgrades, and maintenance
Different operational policies per deployment
Complete API endpoint separation
Tier 2: Conductor Groups (Physical Partitioning)¶
Within a single Ironic deployment, conductor groups provides physical resource partitioning.
Use Cases:
Separate nodes by datacenter/availability zone within a region
Separate nodes by conductor groups for conductor resource management
Isolate hardware types or vendors
Create fault domains for high availability
Manage nodes with different network connectivity
Conductor groups control which conductor manages which nodes. Each conductor can be assigned to a specific group, and will only manage nodes that belong to the same group.
A classical challenge of Ironic is that it is able to manage far more
Bare Metal nodes than a single nova-compute
service is designed to
support. The primary answer for this issue is to leverage Shards first,
and then continue to evolve based upon operational needs.
See: Conductor Groups for detailed configuration.
Tier 3: Shards (Logical Partitioning)¶
The finest level of granularity for operational and client-side grouping.
Use Cases:
Horizontal scaling of operations
Parallelize maintenance tasks
Create logical groupings for different teams
Shards can be used by clients, including Nova, to limit the scope of their
requests to a logical and declared subset of nodes which prevents multiple
nova-compute
services from being able to see and work with the same
node on multiple nova-compute
services.
Note
Shards are client-side constructs - Ironic itself does not use shard values internally.
Added in version 1.82: Shard support was added in API version 1.82.
Warning
Once set, a shard should not be changed. Nova’s model of leveraging the Ironic API does not permit this value to be changed after the fact.
Common Deployment Patterns¶
Pattern 1: Multi-Region with Complete Isolation¶
Use Case: Global deployment with regulatory compliance
Implementation:
Multiple Deployments:
ironic-us-east
,ironic-eu-west
,ironic-apac
Nova Configuration: Separate compute services per region
Conductor Groups: Optional within each deployment
Shards: Operational grouping within regions
Example Nova Configuration:
# nova-compute for US East region
[ironic]
auth_url = https://keystone-us-east.example.com/v3
endpoint_override = https://ironic-us-east.example.com
# nova-compute for EU West region
[ironic]
auth_url = https://keystone-eu-west.example.com/v3
endpoint_override = https://ironic-eu-west.example.com
Note
The above indicated endpoint_override
configuration is provided
for illustrative purposes to stress endpoints would be distinctly
different.
Pattern 2: Single Region with Datacenter Separation¶
Use Case: Metro deployment across multiple datacenters
Implementation:
Single Deployment: One Ironic service
Conductor Groups:
datacenter-1
,datacenter-2
,datacenter-3
Nova Configuration: Target specific conductor groups
Shards: Optional operational grouping
In this case, we don’t expect BMC management network access to occur between datacenters. Thus each datacenter is configured with it’s own group of conductors.
Example Configuration:
# Configure Nova compute to target specific conductor group
[ironic]
conductor_group = datacenter-1
# Configure conductors (ironic.conf)
[conductor]
conductor_group = datacenter-1
# Assign nodes
baremetal node set --conductor-group datacenter-1 <node-uuid>
Note
Some larger operators who leverage conductor groups have suggested
that it is sometimes logical to have a conductor set without a
conductor_group
set. This helps prevent orphaning nodes because
Ironic routes all changes to the conductor which presently manages
the node.
Pattern 3: Operational Scaling Within Datacenters¶
Use Case: Large deployment requiring parallel operations
Implementation:
Single Deployment: One Ironic service
Conductor Groups: By datacenter or hardware type
Shards: Operational batches for maintenance/upgrades
Nova Configuration: May target specific conductor groups
Example:
# Set up conductor groups by hardware
baremetal node set --conductor-group dell-servers <node-uuid-1>
baremetal node set --conductor-group hpe-servers <node-uuid-2>
# Create operational shards for maintenance
baremetal node set --shard maintenance-batch-1 <node-uuid-1>
baremetal node set --shard maintenance-batch-2 <node-uuid-2>
Pattern 4: Hybrid Multi-Tier Approach¶
Use Case: Complex enterprise deployment
Implementation: All three tiers working together
Example Architecture:
# Deployment 1: Production East Coast
# Nova compute service targets ironic-prod-east
[ironic]
endpoint_override = https://ironic-prod-east.example.com
conductor_group = datacenter-east
# Within this deployment:
baremetal node set --conductor-group datacenter-east --shard prod-batch-a <node-uuid>
# Deployment 2: Production West Coast
# Nova compute service targets ironic-prod-west
[ironic]
endpoint_override = https://ironic-prod-west.example.com
conductor_group = datacenter-west
Nova Integration and Configuration¶
Targeting Multiple Ironic Deployments¶
Nova’s Ironic driver can be configured to target different Ironic services:
Per-Compute Service Configuration:
# /etc/nova/nova.conf on compute-service-1
[ironic]
auth_url = https://keystone-region1.example.com/v3
endpoint_override = https://ironic-region1.example.com
conductor_group = region1-zone1
# /etc/nova/nova.conf on compute-service-2
[ironic]
auth_url = https://keystone-region2.example.com/v3
endpoint_override = https://ironic-region2.example.com
conductor_group = region2-zone1
Advanced Options:
[ironic]
# Target specific conductor group within deployment
conductor_group = datacenter-east
# Target specific shard within deployment
shard = production-nodes
# Connection retry configuration
api_max_retries = 60
api_retry_interval = 2
See also
Nova Ironic Hypervisor Configuration for complete Nova configuration details.
Scaling Considerations¶
Nova Compute Service Scaling:
Single nova-compute can handle several hundred Ironic nodes efficiently.
Consider multiple compute services for >1000 nodes per deployment. Nova-compute is modeled on keeping a relatively small number of “instances” per nova-compute process. For example, 250 baremetal nodes.
One nova-compute process per conductor group or shard is expected.
A
conductor_group
which is independent of a nova-compute service configuration can be changed at any time. A shard should never be changed once it has been introduced to a nova-compute process.
Multi-Deployment Benefits:
Independent scaling per deployment
Isolated failure domains
Different operational schedules
Integration Considerations¶
Network Considerations¶
Ironic’s partitioning works alongside physical network configuration:
Physical networks can span multiple conductor groups
Consider network topology when designing conductor group boundaries
Ensure network connectivity between conductors and their assigned nodes
See also
Networking with the Bare Metal service for detailed network configuration guidance
Nova Placement and Scheduling¶
When using Ironic with Nova:
Nova’s availability zones operate independently of Ironic’s partitioning
Use resource classes and traits for capability-based scheduling
See also
Create flavors for use with the Bare Metal service for flavor and scheduling configuration
API Client Usage¶
Working Across Multiple Deployments¶
When managing multiple Ironic deployments, use separate client configurations:
# Configure client for deployment 1
export OS_AUTH_URL=https://keystone-east.example.com/v3
export OS_ENDPOINT_OVERRIDE=https://ironic-east.example.com
baremetal node list
# Configure client for deployment 2
export OS_AUTH_URL=https://keystone-west.example.com/v3
export OS_ENDPOINT_OVERRIDE=https://ironic-west.example.com
baremetal node list
Filtering by Conductor Group¶
# List nodes by conductor group
baremetal node list --conductor-group datacenter-east
# List ports by node conductor group
baremetal port list --conductor-group datacenter-east
Filtering by Shard¶
# List nodes by shard
baremetal node list --shard batch-a
# Get shard distribution
baremetal shard list
# Find nodes without a shard assignment
baremetal node list --unsharded
Combined Filtering Within Deployments¶
# Within a single deployment, filter by conductor group and shard
baremetal node list --conductor-group datacenter-1 --shard maintenance-batch-a
# Set both conductor group and shard on a node
baremetal node set --conductor-group datacenter-east --shard batch-a <node-uuid>
# Get overview of resource distribution
baremetal shard list
baremetal conductor list
Best Practices¶
Deployment Strategy Planning¶
Assess isolation requirements: Determine if you need complete service separation
Plan geographic distribution: Use multiple deployments for true regional separation
Design conductor groups: Align with physical/network boundaries
Implement shard strategy: Plan for operational efficiency
Configure Nova appropriately: Match Nova compute services to your architecture
Operational Considerations¶
Multiple Deployments:
Maintain consistent tooling across deployments
Plan for cross-deployment migrations if needed
Monitor each deployment independently
Coordinate upgrade schedules
Within Deployments:
Monitor conductor distribution:
baremetal shard list
Ensure conductor redundancy per group
Align network topology with conductor groups
Automate shard management for balance
Nova Integration:
Plan compute service distribution across deployments
Monitor nova-compute to Ironic node ratios
Test failover scenarios between compute services
Naming Conventions¶
Naming patterns can be defined by the infrastructure operator and below are some basic suggestions which may be relevant based upon operational requirements.
Conductor Groups:
Geographic:
datacenter-east
,region-us-west
,rack-01
Hardware:
dell-servers
,hpe-gen10
,gpu-nodes
Network:
vlan-100
,isolated-network
Shards:
Operational:
maintenance-batch-1
,upgrade-group-a
Size-based:
small-nodes
,large-memory
Temporal:
weekend-maintenance
,business-hours
Decision Matrix¶
Choose your approach based on requirements:
Requirement |
Multiple Deployments |
Conductor **Groups |
Shards |
---|---|---|---|
Complete isolation |
✓ Best |
✓ Good |
✗ No |
Independent upgrades |
✓ Complete |
✓ Partial |
✗ No |
Geographic separation |
✓ Best |
✓ Good |
✗ No |
Operational scaling |
✗ Overhead |
✓ Good |
✓ Best |
Resource efficiency |
✗ Lower |
✓ Good |
✓ Best |
Troubleshooting¶
Multiple Deployment Issues¶
Connectivity Problems:
# Test connectivity to each deployment
baremetal --os-endpoint-override https://ironic-east.example.com node list
baremetal --os-endpoint-override https://ironic-west.example.com node list
Nova Configuration Issues:
# Check Nova compute service registration
openstack compute service list --service nova-compute
# Verify Nova can reach Ironic
grep -i ironic /var/log/nova/nova-compute.log
Cross-Deployment Node Migration:
# Export node data from source deployment
baremetal node show --fields all <node-uuid>
# Import to destination deployment (manual process)
# Note: Requires careful planning and may need custom tooling
Common Issues Within Deployments¶
Orphaned nodes: Nodes without matching conductor groups cannot be managed
# Find nodes without conductor groups
baremetal node list --conductor-group ""
# List available conductor groups
baremetal conductor list
Unbalanced shards: Monitor node distribution across shards
# Check shard distribution
baremetal shard list
# Find heavily loaded shards
baremetal node list --shard <shard-name> | wc -l
Missing conductor groups: Ensure all groups have active conductors
# Check conductor status
baremetal conductor list
# Verify conductor group configuration
# Check ironic.conf [conductor] conductor_group setting
Migration Scenarios¶
Moving nodes between conductor groups:
# Move node to different conductor group
baremetal node set --conductor-group new-group <node-uuid>
Reassigning shards:
# Change node shard assignment
baremetal node set --shard new-shard <node-uuid>
# Remove shard assignment
baremetal node unset --shard <node-uuid>
Warning
Shards should never be changed once a nova-compute service has identified a node in Ironic. Changing a shard at this point is an unsupported action. As such, Ironic’s API RBAC policy restricts these actions to a “System-Scoped Admin” user. Normal Admin users are denied this capability due the restriction and requirement on the nova-compute side of the consumption of shards.
See Also¶
Conductor Groups - Detailed conductor group configuration
Networking with the Bare Metal service - Physical network considerations
Reference Deploy Architectures - Reference architectures
<no title> - Multi-tenant deployments
Tuning Ironic - Performance tuning considerations