GoogleCloudPlatform/terraformer
CLI tool to generate terraform files from existing infrastructure (reverse Terraform). Infrastructure to Code
| repo name | GoogleCloudPlatform/terraformer |
| repo link | https://github.com/GoogleCloudPlatform/terraformer |
| homepage | |
| language | Go |
| size (curr.) | 63703 kB |
| stars (curr.) | 2572 |
| created | 2019-05-01 |
| license | Apache License 2.0 |
Terraformer
A CLI tool that generates tf/json and tfstate files based on existing infrastructure
(reverse Terraform).
- Disclaimer: This is not an official Google product
- Created by: Waze SRE
Table of Contents
- Capabilities
- Installation
- Supported Providers
- Major Cloud
- Cloud
- Infrastructure Software
- Network
- VCS
- Monitoring & System Management
- Community
- Contributing
- Developing
- Infrastructure
Capabilities
- Generate
tf/json+tfstatefiles from existing infrastructure for all supported objects by resource. - Remote state can be uploaded to a GCS bucket.
- Connect between resources with
terraform_remote_state(local and bucket). - Save
tf/jsonfiles using a custom folder tree pattern. - Import by resource name and type.
- Support terraform 0.12 (for terraform 0.11 use v0.7.9).
Terraformer uses Terraform providers and is designed to easily support newly added resources. To upgrade resources with new fields, all you need to do is upgrade the relevant Terraform providers.
Import current state to Terraform configuration from a provider
Usage:
import [provider] [flags]
import [provider] [command]
Available Commands:
list List supported resources for a provider
Flags:
-b, --bucket string gs://terraform-state
-c, --connect (default true)
-С, --compact (default false)
-f, --filter strings google_compute_firewall=id1:id2:id4
-h, --help help for google
-O, --output string output format hcl or json (default "hcl")
-o, --path-output string (default "generated")
-p, --path-pattern string {output}/{provider}/ (default "{output}/{provider}/{service}/")
--projects strings
-z, --regions strings europe-west1, (default [global])
-r, --resources strings firewalls,networks or * for all services
-s, --state string local or bucket (default "local")
-v, --verbose verbose mode
Use " import [provider] [command] --help" for more information about a command.
Permissions
Read-only permissions
Filtering
Filters are a way to choose which resources terraformer imports. It’s possible to filter resources by its identifiers or attributes. Multiple filtering values are separated by :. If an identifier contains this symbol, value should be wrapped in ' e.g. --filter=resource=id1:'project:dataset_id'. Identifier based filters will be executed before Terraformer will try to refresh remote state.
Resource ID
Filtering is based on Terraform resource ID patterns. To find valid ID patterns for your resource, check the import part of the Terraform documentation.
Example usage:
terraformer import aws --resources=vpc,subnet --filter=aws_vpc=myvpcid --regions=eu-west-1
Will only import the vpc with id myvpcid. This form of filters can help when it’s necessary to select resources by its identifiers.
Planning
The plan command generates a planfile that contains all the resources set to be imported. By modifying the planfile before running the import command, you can rename or filter the resources you’d like to import.
The rest of subcommands and parameters are identical to the import command.
$ terraformer plan google --resources=networks,firewalls --projects=my-project --regions=europe-west1-d
(snip)
Saving planfile to generated/google/my-project/terraformer/plan.json
After reviewing/customizing the planfile, begin the import by running import plan.
$ terraformer import plan generated/google/my-project/terraformer/plan.json
Resource structure
Terraformer by default separates each resource into a file, which is put into a given service directory.
The default path for resource files is {output}/{provider}/{service}/{resource}.tf and can vary for each provider.
It’s possible to adjust the generated structure by:
- Using
--compactparameter to group resource files within a single service into oneresources.tffile - Adjusting the
--path-patternparameter and passing e.g.--path-pattern {output}/{provider}/to generate resources for all services in one directory
It’s possible to combine --compact --path-pattern parameters together.
Installation
From source:
- Run
git clone <terraformer repo> - Run
GO111MODULE=on go mod vendor - Run
go build -vfor all providers OR build with one providergo run build/main.go {google,aws,azure,kubernetes and etc} - Run
terraform initagainst aninit.tffile to install the plugins required for your platform. For example, if you need plugins for the google provider,init.tfshould contain:
provider "google" {}
Or alternatively
- Copy your Terraform provider’s plugin(s) to folder
~/.terraform.d/plugins/{darwin,linux}_amd64/, as appropriate.
From Releases:
- Linux
export PROVIDER={all,google,aws,kubernetes}
curl -LO https://github.com/GoogleCloudPlatform/terraformer/releases/download/$(curl -s https://api.github.com/repos/GoogleCloudPlatform/terraformer/releases/latest | grep tag_name | cut -d '"' -f 4)/terraformer-${PROVIDER}-linux-amd64
chmod +x terraformer-${PROVIDER}-linux-amd64
sudo mv terraformer-${PROVIDER}-linux-amd64 /usr/local/bin/terraformer
- MacOS
export PROVIDER={all,google,aws,kubernetes}
curl -LO https://github.com/GoogleCloudPlatform/terraformer/releases/download/$(curl -s https://api.github.com/repos/GoogleCloudPlatform/terraformer/releases/latest | grep tag_name | cut -d '"' -f 4)/terraformer-${PROVIDER}-darwin-amd64
chmod +x terraformer-${PROVIDER}-darwin-amd64
sudo mv terraformer-${PROVIDER}-darwin-amd64 /usr/local/bin/terraformer
Using a package manager
If you want to use a package manager:
- Homebrew users can use
brew install terraformer. - Chocolatey users can use
choco install terraformer.
Links to download Terraform Providers:
- Major Cloud
- Cloud
- Infrastructure Software
- Network
- Cloudflare provider >1.16 - here
- VCS
- GitHub provider >=2.2.1 - here
- Monitoring & System Management
- Community
Information on provider plugins: https://www.terraform.io/docs/configuration/providers.html
Use with GCP
Example:
terraformer import google --resources=gcs,forwardingRules,httpHealthChecks --connect=true --regions=europe-west1,europe-west4 --projects=aaa,fff
terraformer import google --resources=gcs,forwardingRules,httpHealthChecks --filter=google_compute_firewall=rule1:rule2:rule3 --regions=europe-west1 --projects=aaa,fff
List of supported GCP services:
addressesgoogle_compute_address
autoscalersgoogle_compute_autoscaler
backendBucketsgoogle_compute_backend_bucket
backendServicesgoogle_compute_backend_service
bigQuerygoogle_bigquery_datasetgoogle_bigquery_table
cloudFunctionsgoogle_cloudfunctions_function
cloudsqlgoogle_sql_database_instancegoogle_sql_database
dataProcgoogle_dataproc_cluster
disksgoogle_compute_disk
dnsgoogle_dns_managed_zonegoogle_dns_record_set
firewallsgoogle_compute_firewall
forwardingRulesgoogle_compute_forwarding_rule
gcsgoogle_storage_bucketgoogle_storage_bucket_aclgoogle_storage_default_object_aclgoogle_storage_bucket_iam_bindinggoogle_storage_bucket_iam_membergoogle_storage_bucket_iam_policygoogle_storage_notification
gkegoogle_container_clustergoogle_container_node_pool
globalAddressesgoogle_compute_global_address
globalForwardingRulesgoogle_compute_global_forwarding_rule
healthChecksgoogle_compute_health_check
httpHealthChecksgoogle_compute_http_health_check
httpsHealthChecksgoogle_compute_https_health_check
imagesgoogle_compute_image
instanceGroupManagersgoogle_compute_instance_group_manager
instanceGroupsgoogle_compute_instance_group
instanceTemplatesgoogle_compute_instance_template
instancesgoogle_compute_instance
interconnectAttachmentsgoogle_compute_interconnect_attachment
kmsgoogle_kms_key_ringgoogle_kms_crypto_key
logginggoogle_logging_metric
memoryStoregoogle_redis_instance
monitoringgoogle_monitoring_alert_policygoogle_monitoring_groupgoogle_monitoring_notification_channelgoogle_monitoring_uptime_check_config
networksgoogle_compute_network
nodeGroupsgoogle_compute_node_group
nodeTemplatesgoogle_compute_node_template
projectgoogle_project
pubsubgoogle_pubsub_subscriptiongoogle_pubsub_topic
regionAutoscalersgoogle_compute_region_autoscaler
regionBackendServicesgoogle_compute_region_backend_service
regionDisksgoogle_compute_region_disk
regionInstanceGroupManagersgoogle_compute_region_instance_group_manager
routersgoogle_compute_router
routesgoogle_compute_route
schedulerJobsgoogle_cloud_scheduler_job
securityPoliciesgoogle_compute_security_policy
sslPoliciesgoogle_compute_ssl_policy
subnetworksgoogle_compute_subnetwork
targetHttpProxiesgoogle_compute_target_http_proxy
targetHttpsProxiesgoogle_compute_target_https_proxy
targetInstancesgoogle_compute_target_instance
targetPoolsgoogle_compute_target_pool
targetSslProxiesgoogle_compute_target_ssl_proxy
targetTcpProxiesgoogle_compute_target_tcp_proxy
targetVpnGatewaysgoogle_compute_vpn_gateway
urlMapsgoogle_compute_url_map
vpnTunnelsgoogle_compute_vpn_tunnel
Your tf and tfstate files are written by default to
generated/gcp/zone/service.
Use with AWS
Example:
terraformer import aws --resources=vpc,subnet --connect=true --regions=eu-west-1 --profile=prod
terraformer import aws --resources=vpc,subnet --filter=aws_vpc=vpc_id1:vpc_id2:vpc_id3 --regions=eu-west-1
Profiles support
AWS configuration including environmental variables, shared credentials file (~/.aws/credentials), and shared config file (~/.aws/config) will be loaded by the tool by default. To use a specific profile, you can use the following command:
terraformer import aws --resources=vpc,subnet --regions=eu-west-1 --profile=prod
You can also provide no regions when importing resources:
terraformer import aws --resources=cloudfront --profile=prod
In that case terraformer will not know with which region resources are associated with and will not assume any region. That scenario is useful in case of global resources (e.g. CloudFront distributions or Route 53 records) and when region is passed implicitly through environmental variables or metadata service.
Supported services
accessanalyzeraws_accessanalyzer_analyzer
acmaws_acm_certificate
alb(supports ALB and NLB)aws_lbaws_lb_listeneraws_lb_listener_ruleaws_lb_listener_certificateaws_lb_target_groupaws_lb_target_group_attachment
api_gatewayaws_api_gateway_authorizeraws_api_gateway_documentation_partaws_api_gateway_gateway_responseaws_api_gateway_integrationaws_api_gateway_integration_responseaws_api_gateway_methodaws_api_gateway_method_responseaws_api_gateway_modelaws_api_gateway_resourceaws_api_gateway_rest_apiaws_api_gateway_stageaws_api_gateway_usage_planaws_api_gateway_vpc_link
auto_scalingaws_autoscaling_groupaws_launch_configurationaws_launch_template
budgetsaws_budgets_budget
cloud9aws_cloud9_environment_ec2
cloudfrontaws_cloudfront_distribution
cloudformationaws_cloudformation_stackaws_cloudformation_stack_setaws_cloudformation_stack_set_instance
cloudtrailaws_cloudtrail
cloudwatchaws_cloudwatch_dashboardaws_cloudwatch_event_ruleaws_cloudwatch_event_targetaws_cloudwatch_metric_alarm
codebuildaws_codebuild_project
codecommitaws_codecommit_repository
codedeployaws_codedeploy_app
codepipelineaws_codepipelineaws_codepipeline_webhook
customer_gatewayaws_customer_gateway
datapipelineaws_datapipeline_pipeline
devicefarmaws_devicefarm_project
dynamodbaws_dynamodb_table
ec2_instanceaws_instance
eipaws_eip
elasticacheaws_elasticache_clusteraws_elasticache_parameter_groupaws_elasticache_subnet_groupaws_elasticache_replication_group
ebsaws_ebs_volumeaws_volume_attachment
elastic_beanstalkaws_elastic_beanstalk_applicationaws_elastic_beanstalk_environment
ecsaws_ecs_clusteraws_ecs_serviceaws_ecs_task_definition
ecraws_ecr_lifecycle_policyaws_ecr_repositoryaws_ecr_repository_policy
eksaws_eks_cluster
elbaws_elb
emraws_emr_clusteraws_emr_security_configuration
eniaws_network_interface
esaws_elasticsearch_domain
firehoseaws_kinesis_firehose_delivery_stream
glueglue_crawleraws_glue_catalog_databaseaws_glue_catalog_table
iamaws_iam_groupaws_iam_group_policyaws_iam_group_policy_attachmentaws_iam_instance_profileaws_iam_policyaws_iam_policy_attachmentaws_iam_roleaws_iam_role_policyaws_iam_useraws_iam_user_group_membershipaws_iam_user_policy
igwaws_internet_gateway
iotaws_iot_thingaws_iot_thing_typeaws_iot_topic_ruleaws_iot_role_alias
kinesisaws_kinesis_stream
kmsaws_kms_keyaws_kms_alias
lambdaaws_lambda_event_source_mappingaws_lambda_functionaws_lambda_function_event_invoke_configaws_lambda_layer_version
media_packageaws_media_package_channel
media_storeaws_media_store_container
mskaws_msk_cluster
nataws_nat_gateway
naclaws_network_acl
organizationaws_organizations_accountaws_organizations_organizationaws_organizations_organizational_unitaws_organizations_policyaws_organizations_policy_attachment
rdsaws_db_instanceaws_db_parameter_groupaws_db_subnet_groupaws_db_option_groupaws_db_event_subscription
route53aws_route53_zoneaws_route53_record
route_tableaws_route_tableaws_main_route_table_associationaws_route_table_association
s3aws_s3_bucketaws_s3_bucket_policy
sesaws_ses_configuration_setaws_ses_domain_identityaws_ses_email_identityaws_ses_receipt_ruleaws_ses_receipt_rule_setaws_ses_template
sgaws_security_groupaws_security_group_rule(if a rule cannot be inlined)
snsaws_sns_topicaws_sns_topic_subscription
sqsaws_sqs_queue
subnetaws_subnet
transit_gatewayaws_ec2_transit_gateway_route_tableaws_ec2_transit_gateway_vpc_attachment
wafaws_waf_byte_match_setaws_waf_geo_match_setaws_waf_ipsetaws_waf_rate_based_ruleaws_waf_regex_match_setaws_waf_regex_pattern_setaws_waf_ruleaws_waf_rule_groupaws_waf_size_constraint_setaws_waf_sql_injection_match_setaws_waf_web_aclaws_waf_xss_match_set
waf_regionalaws_wafregional_byte_match_setaws_wafregional_geo_match_setaws_wafregional_ipsetaws_wafregional_rate_based_ruleaws_wafregional_regex_match_setaws_wafregional_regex_pattern_setaws_wafregional_ruleaws_wafregional_rule_groupaws_wafregional_size_constraint_setaws_wafregional_sql_injection_match_setaws_wafregional_web_aclaws_wafregional_xss_match_set
vpcaws_vpc
vpc_peeringaws_vpc_peering_connection
vpn_connectionaws_vpn_connection
vpn_gatewayaws_vpn_gateway
Global services
AWS services that are global will be imported without specified region even if several regions will be passed. It is to ensure only one representation of an AWS resource is imported.
List of global AWS services:
budgetscloudfrontiamorganizationroute53waf
Attribute filters
Attribute filters allow filtering across different resource types by its attributes.
terraformer import aws --resources=ec2_instance,ebs --filter="Name=tags.costCenter;Value=20000:'20001:1'" --regions=eu-west-1
Will only import AWS EC2 instances along with EBS volumes annotated with tag costCenter with values 20000 or 20001:1. Attribute filters are by default applicable to all resource types although it’s possible to specify to what resource type a given filter should be applicable to by providing Type=<type> parameter. For example:
terraformer import aws --resources=ec2_instance,ebs --filter=Type=ec2_instance;Name=tags.costCenter;Value=20000:'20001:1' --regions=eu-west-1
Will work as same as example above with a change the filter will be applicable only to ec2_instance resources.
Due to fact API Gateway generates a lot of resources, it’s possible to issue a filtering query to retrieve resources related to a given REST API by tags. To fetch resources related to a REST API resource with a tag STAGE and value dev, add parameter --filter="Type=aws_api_gateway_rest_api;Name=tags.STAGE;Value=dev".
SQS queues retrieval
Terraformer uses AWS ListQueues API call to fetch available queues. The API is able to return only up to 1000 queues and an additional name prefix should be passed to filter the list results. It’s possible to pass QueueNamePrefix parameter by environmental variable SQS_PREFIX.
Use with Azure
Support Azure CLI, Service Principal with Client Certificate & Service Principal with Client Secret
Example:
# Using Azure CLI (az login)
export ARM_SUBSCRIPTION_ID=[SUBSCRIPTION_ID]
# Using Service Principal with Client Certificate
export ARM_SUBSCRIPTION_ID=[SUBSCRIPTION_ID]
export ARM_CLIENT_ID=[CLIENT_ID]
export ARM_CLIENT_CERTIFICATE_PATH="/path/to/my/client/certificate.pfx"
export ARM_CLIENT_CERTIFICATE_PASSWORD=[CLIENT_CERTIFICATE_PASSWORD]
export ARM_TENANT_ID=[TENANT_ID]
# Service Principal with Client Secret
export ARM_SUBSCRIPTION_ID=[SUBSCRIPTION_ID]
export ARM_CLIENT_ID=[CLIENT_ID]
export ARM_CLIENT_SECRET=[CLIENT_SECRET]
export ARM_TENANT_ID=[TENANT_ID]
./terraformer import azure -r resource_group
List of supported Azure resources:
analysisazurerm_analysis_services_server
databaseazurerm_mariadb_configurationazurerm_mariadb_databaseazurerm_mariadb_firewall_ruleazurerm_mariadb_serverazurerm_mariadb_virtual_network_ruleazurerm_mysql_configurationazurerm_mysql_databaseazurerm_mysql_firewall_ruleazurerm_mysql_serverazurerm_mysql_virtual_network_ruleazurerm_postgresql_configurationazurerm_postgresql_databaseazurerm_postgresql_firewall_ruleazurerm_postgresql_serverazurerm_postgresql_virtual_network_ruleazurerm_sql_databaseazurerm_sql_active_directory_administratorazurerm_sql_elasticpoolazurerm_sql_failover_groupazurerm_sql_firewall_ruleazurerm_sql_serverazurerm_sql_virtual_network_rule
diskazurerm_managed_disk
network_interfaceazurerm_network_interface
network_security_groupazurerm_network_security_group
resource_groupazurerm_resource_group
storage_accountazurerm_storage_account
virtual_machineazurerm_virtual_machine
virtual_networkazurerm_virtual_network
Use with AliCloud
You can either edit your alicloud config directly, (usually it is ~/.aliyun/config.json)
or run aliyun configure and enter the credentials when prompted.
Terraformer will pick up the profile name specified in the --profile parameter.
It defaults to the first config in the config array.
terraformer import alicloud --resources=ecs --regions=ap-southeast-3 --profile=default
For all supported resources, you can do
# https://unix.stackexchange.com/a/114948/203870
export ALL_SUPPORTED_ALICLOUD_RESOURCES=$(terraformer import alicloud list | sed -e 'H;1h;$!d;x;y/\n/,/')
terraformer import alicloud --resources=$ALL_SUPPORTED_ALICLOUD_RESOURCES --regions=ap-southeast-3
List of supported AliCloud resources:
dnsalicloud_dnsalicloud_dns_record
ecsalicloud_instance
keypairalicloud_key_pair
natalicloud_nat_gateway
pvtzalicloud_pvtz_zonealicloud_pvtz_zone_attachmentalicloud_pvtz_zone_record
ramalicloud_ram_rolealicloud_ram_role_policy_attachment
rdsalicloud_db_instance
sgalicloud_security_groupalicloud_security_group_rule
slbalicloud_slbalicloud_slb_server_groupalicloud_slb_listener
vpcalicloud_vpc
vswitchalicloud_vswitch
Use with DigitalOcean
Example:
export DIGITALOCEAN_TOKEN=[DIGITALOCEAN_TOKEN]
./terraformer import digitalocean -r project,droplet
List of supported DigitalOcean resources:
cdndigitalocean_cdn
certificatedigitalocean_certificate
database_clusterdigitalocean_database_clusterdigitalocean_database_connection_pooldigitalocean_database_dbdigitalocean_database_replicadigitalocean_database_user
domaindigitalocean_domaindigitalocean_record
dropletdigitalocean_droplet
droplet_snapshotdigitalocean_droplet_snapshot
firewalldigitalocean_firewall
floating_ipdigitalocean_floating_ip
kubernetes_clusterdigitalocean_kubernetes_clusterdigitalocean_kubernetes_node_pool
loadbalancerdigitalocean_loadbalancer
projectdigitalocean_project
ssh_keydigitalocean_ssh_key
tagdigitalocean_tag
volumedigitalocean_volume
volume_snapshotdigitalocean_volume_snapshot
Use with Fastly
Example:
export FASTLY_API_KEY=[FASTLY_API_KEY]
./terraformer import fastly -r service_v1
List of supported Fastly resources:
service_v1fastly_service_acl_entries_v1fastly_service_dictionary_items_v1fastly_service_dynamic_snippet_content_v1fastly_service_v1
Use with Heroku
Example:
export HEROKU_EMAIL=[HEROKU_EMAIL]
export HEROKU_API_KEY=[HEROKU_API_KEY]
./terraformer import heroku -r app,addon
List of supported Heroku resources:
account_featureheroku_account_feature
addonheroku_addon
addon_attachmentheroku_addon_attachment
appheroku_app
app_config_associationheroku_app_config_association
app_featureheroku_app_feature
app_webhookheroku_app_webhook
buildheroku_build
certheroku_cert
domainheroku_domain
drainheroku_drain
formationheroku_formation
pipelineheroku_pipeline
pipeline_couplingheroku_pipeline_coupling
team_collaboratorheroku_team_collaborator
team_memberheroku_team_member
Use with Linode
Example:
export LINODE_TOKEN=[LINODE_TOKEN]
./terraformer import linode -r instance
List of supported Linode resources:
domainlinode_domainlinode_domain_record
imagelinode_image
instancelinode_instance
nodebalancerlinode_nodebalancerlinode_nodebalancer_configlinode_nodebalancer_node
rdnslinode_rdns
sshkeylinode_sshkey
stackscriptlinode_stackscript
tokenlinode_token
volumelinode_volume
Use with OpenStack
Example:
terraformer import openstack --resources=compute,networking --regions=RegionOne
List of supported OpenStack services:
blockstorageopenstack_blockstorage_volume_v1openstack_blockstorage_volume_v2openstack_blockstorage_volume_v3
computeopenstack_compute_instance_v2
networkingopenstack_networking_secgroup_v2openstack_networking_secgroup_rule_v2
Use with Vultr
Example:
export VULTR_API_KEY=[VULTR_API_KEY]
./terraformer import vultr -r server
List of supported Vultr resources:
bare_metal_servervultr_bare_metal_server
block_storagevultr_block_storage
dns_domainvultr_dns_domainvultr_dns_record
firewall_groupvultr_firewall_groupvultr_firewall_rule
networkvultr_network
reserved_ipvultr_reserved_ip
servervultr_server
snapshotvultr_snapshot
ssh_keyvultr_ssh_key
startup_scriptvultr_startup_script
uservultr_user
Use with Kubernetes
Example:
terraformer import kubernetes --resources=deployments,services,storageclasses
terraformer import kubernetes --resources=deployments,services,storageclasses --filter=kubernetes_deployment=name1:name2:name3
All Kubernetes resources that are currently supported by the Kubernetes provider, are also supported by this module. Here is the list of resources which are currently supported by Kubernetes provider v.1.4:
clusterrolebindingkubernetes_cluster_role_binding
configmapskubernetes_config_map
deploymentskubernetes_deployment
horizontalpodautoscalerskubernetes_horizontal_pod_autoscaler
limitrangeskubernetes_limit_range
namespaceskubernetes_namespace
persistentvolumeskubernetes_persistent_volume
persistentvolumeclaimskubernetes_persistent_volume_claim
podskubernetes_pod
replicationcontrollerskubernetes_replication_controller
resourcequotaskubernetes_resource_quota
secretskubernetes_secret
serviceskubernetes_service
serviceaccountskubernetes_service_account
statefulsetskubernetes_stateful_set
storageclasseskubernetes_storage_class
Known issues
- Terraform Kubernetes provider is rejecting resources with “:” characters in their names (as they don’t meet DNS-1123), while it’s allowed for certain types in Kubernetes, e.g. ClusterRoleBinding.
- Because Terraform flatmap uses “.” to detect the keys for unflattening the maps, some keys with “.” in their names are being considered as the maps.
- Since the library assumes empty strings to be empty values (not “0”), there are some issues with optional integer keys that are restricted to be positive.
Use with OctopusDeploy
Example:
export OCTOPUS_CLI_SERVER=http://localhost:8081/
export OCTOPUS_CLI_API_KEY=API-CK7DQ8BMJCUUBSHAJCDIATXUO
terraformer import octopusdeploy --resources=tagsets
accountsoctopusdeploy_account
certificatesoctopusdeploy_certificate
environmentsoctopusdeploy_environment
feedsoctopusdeploy_feed
libraryvariablesetsoctopusdeploy_library_variable_set
lifecycleoctopusdeploy_lifecycle
projectoctopusdeploy_project
projectgroupsoctopusdeploy_project_group
projecttriggersoctopusdeploy_project_deployment_target_trigger
tagsetsoctopusdeploy_tag_set
Use with RabbitMQ
Example:
export RABBITMQ_SERVER_URL=http://foo.bar.localdomain:15672
export RABBITMQ_USERNAME=[RABBITMQ_USERNAME]
export RABBITMQ_PASSWORD=[RABBITMQ_PASSWORD]
terraformer import rabbitmq --resources=vhosts,queues,exchanges
terraformer import rabbitmq --resources=vhosts,queues,exchanges --filter=rabbitmq_vhost=name1:name2:name3
All RabbitMQ resources that are currently supported by the RabbitMQ provider, are also supported by this module. Here is the list of resources which are currently supported by RabbitMQ provider v.1.1.0:
bindingsrabbitmq_binding
exchangesrabbitmq_exchange
permissionsrabbitmq_permissions
policiesrabbitmq_policy
queuesrabbitmq_queue
usersrabbitmq_user
vhostsrabbitmq_vhost
Use with Cloudflare
Example using a Cloudflare API Key and corresponding email:
export CLOUDFLARE_API_KEY=[CLOUDFLARE_API_KEY]
export CLOUDFLARE_EMAIL=[CLOUDFLARE_EMAIL]
./terraformer import cloudflare --resources=firewall,dns
or using a Cloudflare API Token:
export CLOUDFLARE_API_TOKEN=[CLOUDFLARE_API_TOKEN]
./terraformer import cloudflare --resources=firewall,dns
List of supported Cloudflare services:
accesscloudflare_access_application
dnscloudflare_zonecloudflare_record
firewallcloudflare_access_rulecloudflare_filtercloudflare_firewall_rulecloudflare_zone_lockdown
Use with GitHub
Example:
./terraformer import github --organizations=YOUR_ORGANIZATION --resources=repositories --token=YOUR_TOKEN // or GITHUB_TOKEN in env
./terraformer import github --organizations=YOUR_ORGANIZATION --resources=repositories --filter=github_repository=id1:id2:id4 --token=YOUR_TOKEN // or GITHUB_TOKEN in env
Supports only organizational resources. List of supported resources:
membersgithub_membership
organization_blocksgithub_organization_block
organization_projectsgithub_organization_project
organization_webhooksgithub_organization_webhook
repositoriesgithub_repositorygithub_repository_webhookgithub_branch_protectiongithub_repository_collaboratorgithub_repository_deploy_key
teamsgithub_teamgithub_team_membershipgithub_team_repository
user_ssh_keysgithub_user_ssh_key
Notes:
- Terraformer can’t get webhook secrets from the GitHub API. If you use a secret token in any of your webhooks, running
terraform planwill result in a change being detected: =>configuration.#: "1" => "0"in tfstate only.
Use with Datadog
Example:
./terraformer import datadog --resources=monitor --api-key=YOUR_DATADOG_API_KEY // or DATADOG_API_KEY in env --app-key=YOUR_DATADOG_APP_KEY // or DATADOG_APP_KEY in env
./terraformer import datadog --resources=monitor --filter=datadog_monitor=id1:id2:id4 --api-key=YOUR_DATADOG_API_KEY // or DATADOG_API_KEY in env --app-key=YOUR_DATADOG_APP_KEY // or DATADOG_APP_KEY in env
List of supported Datadog services:
dashboarddatadog_dashboard
downtimedatadog_downtime
monitordatadog_monitor
screenboarddatadog_screenboard
syntheticsdatadog_synthetics_test
timeboarddatadog_timeboard
userdatadog_user
Use with New Relic
Example:
NEWRELIC_API_KEY=[API-KEY]
./terraformer import newrelic -r alert,dashboard,infra,synthetics
List of supported New Relic resources:
alertnewrelic_alert_channelnewrelic_alert_conditionnewrelic_alert_policy
dashboardnewrelic_dashboard
infranewrelic_infra_alert_condition
syntheticsnewrelic_synthetics_monitornewrelic_synthetics_alert_condition
Use with Keycloak
Example:
export KEYCLOAK_URL=https://foo.bar.localdomain
export KEYCLOAK_CLIENT_ID=[KEYCLOAK_CLIENT_ID]
export KEYCLOAK_CLIENT_SECRET=[KEYCLOAK_CLIENT_SECRET]
terraformer import keycloak --resources=realms,openid_clients
terraformer import keycloak --resources=realms --filter=keycloak_realm=name1:name2:name3
Here is the list of resources which are currently supported by Keycloak provider v.1.12.0:
groupskeycloak_groupkeycloak_group_membershipskeycloak_group_roleskeycloak_default_groups
openid_clientskeycloak_openid_clientkeycloak_openid_client_service_account_rolekeycloak_openid_user_attribute_protocol_mapperkeycloak_openid_user_property_protocol_mapperkeycloak_openid_full_name_protocol_mapperkeycloak_openid_audience_protocol_mapperkeycloak_openid_group_membership_protocol_mapperkeycloak_openid_hardcoded_claim_protocol_mapperkeycloak_openid_hardcoded_role_protocol_mapper
realmskeycloak_realmkeycloak_ldap_user_federationkeycloak_ldap_full_name_mapperkeycloak_ldap_group_mapperkeycloak_ldap_msad_user_account_control_mapperkeycloak_ldap_user_attribute_mapperkeycloak_required_action
roleskeycloak_role
scopeskeycloak_openid_client_scopekeycloak_openid_client_default_scopeskeycloak_openid_client_optional_scopes
userskeycloak_user
Use with Logz.io
Example:
LOGZIO_API_TOKEN=foobar LOGZIO_BASE_URL=https://api-eu.logz.io ./terraformer import logzio -r=alerts,alert_notification_endpoints // Import Logz.io alerts and alert notification endpoints
List of supported Logz.io resources:
alertslogzio_alert
alert_notification_endpointslogzio_endpoint
Use with Commercetools
This provider use the terraform-provider-commercetools. The terraformer provider was build by Dustin Deus.
Example:
CTP_CLIENT_ID=foo CTP_CLIENT_SCOPE=scope CTP_CLIENT_SECRET=bar CTP_PROJECT_KEY=key ./terraformer plan commercetools -r=types // Only planning
CTP_CLIENT_ID=foo CTP_CLIENT_SCOPE=scope CTP_CLIENT_SECRET=bar CTP_PROJECT_KEY=key ./terraformer import commercetools -r=types // Import commercetools types
List of supported commercetools resources:
api_extensioncommercetools_api_extension
channelcommercetools_channel
product_typecommercetools_product_type
shipping_zonecommercetools_shipping_zone
statecommercetools_state
storecommercetools_store
subscriptioncommercetools_subscription
tax_categorycommercetools_tax_category
typescommercetools_type
Contributing
If you have improvements or fixes, we would love to have your contributions. Please read CONTRIBUTING.md for more information on the process we would like contributors to follow.
Developing
Terraformer was built so you can easily add new providers of any kind.
Process for generating tf/json + tfstate files:
- Call GCP/AWS/other api and get list of resources.
- Iterate over resources and take only the ID (we don’t need mapping fields!).
- Call to provider for readonly fields.
- Call to infrastructure and take tf + tfstate.
Infrastructure
- Call to provider using the refresh method and get all data.
- Convert refresh data to go struct.
- Generate HCL file -
tf/jsonfiles. - Generate
tfstatefiles.
All mapping of resource is made by providers and Terraform. Upgrades are needed only for providers.
GCP compute resources
For GCP compute resources, use generated code from
providers/gcp/gcp_compute_code_generator.
To regenerate code:
go run providers/gcp/gcp_compute_code_generator/*.go
Similar projects
terraforming
Terraformer Benefits
- Simpler to add new providers and resources - already supports AWS, GCP, GitHub, Kubernetes, and Openstack. Terraforming supports only AWS.
- Better support for HCL + tfstate, including updates for Terraform 0.12.
- If a provider adds new attributes to a resource, there is no need change Terraformer code - just update the Terraform provider on your laptop.
- Automatically supports connections between resources in HCL files.
Comparison
Terraforming gets all attributes from cloud APIs and creates HCL and tfstate files with templating. Each attribute in the API needs to map to attribute in Terraform. Generated files from templating can be broken with illegal syntax. When a provider adds new attributes the terraforming code needs to be updated.
Terraformer instead uses Terraform provider files for mapping attributes, HCL library from Hashicorp, and Terraform code.
Look for S3 support in terraforming here and official S3 support Terraforming lacks full coverage for resources - as an example you can see that 70% of S3 options are not supported:
- terraforming - https://github.com/dtan4/terraforming/blob/master/lib/terraforming/template/tf/s3.erb
- official S3 support - https://www.terraform.io/docs/providers/aws/r/s3_bucket.html
