Target’s use of Hadoop expanded quickly beyond the initial projects into a large, multi-tenant cluster performing over 50,000 jobs per day by hundreds of users across dozens of projects. Often cluster users would desire to quickly test future versions, new packages and different configurations but were limited to using the stage level cluster. The platform management team would have to make pipeline changes in CI/CD which would delay quick proof of concept style work typically suited for virtual environments. The need for private, project, or team level development environments was quickly realized by the influx of requests to the Hadoop platform engineering team. Custom software and configuration needed to be tested in a development environment that mocked production to minimize the risk of breaking efforts by other tenants.
The following requirements were taken into account when working on the project: provide a secured environment with the ability to leverage preconfigured LDAP and Kerberos servers, out-of-the-box usability, allowing you to log in with preconfigured user accounts, and custom user management utilities to administer the cluster. We also wanted to provide a fully customizable experience, where cluster size, cluster configurations, and secure assets such as users and passwords would be configurable beforehand.
To accomplish this goal, we built the product around two major components: a cookbook, to install and configure the OS and to lay down cluster configuration, and an orchestration component, to orchestrate virtualization of cloud-based clusters and run the chef deployment across the cluster. We also created a Docker image to isolate dependencies and created the cookbook in such a way that it could be used independently of the orchestration component.
As a result of releasing REDstack at Target, we are able to provide on-demand cloud clusters to development and product teams in under an hour. We are able to let users leverage existing internal and external cloud platforms to spin up sandbox environments as well as larger cloud-based managed clusters, and provide full root access to the people who own them. Since its release, we have seen over 500 deployments and rebuilds across the enterprise with over 125 clusters active currently.
By enabling users with on-demand clusters, the ability to work on an environment with full control and no effect on other users is available to users. It unlocks greater potential for users as they are able to iterate rapidly and not worry about failure. We continue to see development, discovery, and innovation take place by users in these environments, and our platform team continues to leverage the system for software testing and upgrades. Since creating and releasing the product, we have also been able to open source the product and allow the system to be used with public clouds as well as private cloud within an enterprise.