Shopkeeper is a python package that implements Data Platform Resources -- data producers, data consumers and datasets -- as custom infrastructure components. This project is in its early phases, exploring whether Pulumi can provide a rich self service interface to a data platform.
See honestgrowth.no for strategy and design.
Coming soon! For now, you can peek at the test cases in pulumi-shopkeeper/tests.
Prerequisites:
- Docker
- (recommended) Visual Studio Code, with the Dev Containers extension
- A test AWS subscription (
aws sso login)
This project can be tested and developed in a (rather opinionated) devcontainer based on the (rather large) Pulumi/Pulumi image. This image gives us Java, Go, dotnet and python, which we need to build and test pulumi packages across all supported languages.
To get started, build the container yourself, providing your own github handle as an arg. This will be used to create a non-root container user, and is later by the devcontainer configuration.
export GITHUB_USERNAME="jaketbouma"
cd .devcontainer
docker build -f dev.Dockerfile --build-arg GITHUB_USERNAME=$GITHUB_USERNAME$ -t shopkeeper-dev:latest ./Review the .devcontainer/devcontainer.json file. I use chezmoi to sync dotfile configuration and secrets into the development container using a postStartCommand script. This is mostly for terminal cosmetics and to persist terminal history. You can create your own .devcontainer/${GITHUB_USERNAME}.postStartCommand.sh and adjust as needed.
Poetry Poe is used as the task runner, and has tasks for tests, cleanups, etc.
cd pulumi-shopkeeper
poetry install
poetry run poeHappy hacking!