Fulgens: a `build & local deploy & run` script generator
Motivation
Over the last 15 years I have built a couple of projects (all the stuff on www.oglimmer.de) and while it is very simple to build a Java project via a brain-friendly ‘mvn package’ it is always a bit cumbersome to start-up a project you haven’t worked on for quite a while.
Starting a project usually needs a build, a local deployment of the webserver and the database, an initial set up of the database and sometimes a couple of config changes to connect everything together — how this all is done heavily depends on the project and the used technologies.
I wanted to have a system which builds, locally deploys and runs all of my projects with a consistent syntax.
It should be super easy to locally start a project and to play with different versions of Java, Node or database backends. And it should also support Docker and local deployments.
Now this is where Fulgens comes into play.
Possible solutions with existing technologies
We have maven, gradle, npm and many other standardized build tools to install dependencies and easily build a piece of software. And in my opinion the most important features of these systems are, that you don’t need any particular knowledge on how to build the software. A build is as easy as ‘mvn package’ or ‘npm install’. So great, the problem how to build an unknown piece of software is already solved.
But how to start the software locally? ‘npm start’? Oh wait, it needs a database…. And when it comes to Java you are lost even more. For sure, you could write a maven config to start a database and initialize it, but that gets really ugly, it isn’t the purpose of maven and thus it is far away from ‘easily starting the software locally’.
But we have provisioning tools like Ansible/Chef/Puppet!
These system tend to have a high complexity, as they have been built to solve a much bigger problem: installing infrastructure — not providing a local setup! Maybe Ansible is easy enough — at least it’s just an SSH-based remote shell command executor. Still Ansible needs a whole bunch of configuration files and the execution mechanism was made for SSH connections, it’s again too complex for what we actually want: just a simple local deployment.
Can’t Docker spin up environments?
A docker-compose.yml file is well suited to spin up all software components of your project but it doesn’t change your config files, it doesn’t setup your database and most important it doesn’t give you the flexibility to run your components outside of Docker. So even leaving the last aspect aside, you still need some bash code to cover the missing pieces to (initially) start your project.
Defining our goals for a new solution
1.) We want a single simple description file as the input configuration, a bit like Dockerfile, pom.xml or package.json.
2.) We want a (generated) shell script with minimal dependencies that builds, deploys, configures and runs our project.
3.) We want support for local builds as well as docker-based build.
4.) Different runtime environments should be supported:
- Downloaded temporary local software
- Docker
- Vagrant (VirtualBox)
- (re-)usage of already installed local software
5.) Initial and/or continuous setups in combination with temporary or permanent components should be supported.
6.) Last but not least, the generated script should be self-describing, almost zero-knowledge should be needed to run it. Spinning up a working local environment should be as easy as ‘mvn package’.
Introducing Fulgens and the Fulgensfile.js
Let’s assume we have a Java, web-based project, using a Mysql database.
To start this project, you would probably need to build the java project, start the Mysql database daemon, create a schema, set up some tables and data there and finally deploy the generated war file into a Tomcat server while adding a configuration file.
Let’s write a Fulgensfile.js to describe the project:
module.exports = { config: {
SchemaVersion: "1.0.0",
Name: "JavaWebProjectExample"
}, software: {
javaCode: {
Source: "mvn",
Artifact: "target/JavaWebProjectExample.war"
}, tomcat: {
Source: "tomcat",
Deploy: "javaCode"
}
}
}
This is a minimal length Fulgensfile.js for a Java, web-based project as it describes how the Java code should be build and where the generated artifact can be found on the filesystem.
It also defines a Tomcat web server and connects the result of the first step into the Servlet container.
Let’s add a database:
...
software: {
javaCode: {
Source: "mvn",
Artifact: "target/JavaWebProjectExample.war"
}, mysql: {
Source: "mysql",
Mysql: {
Schema: "java_code",
Create: [ "./src/db/mysql.dump" ]
}
}, tomcat: {
Source: "tomcat",
Deploy: "javaCode"
}
}
...
The new object “mysql” in the Fulgensfile.js will start a Mysql instance inside Docker, it will create a schema ‘java_code’ and import the sql file ./src/db/mysql.dump.
There is still one problem, the JavaCode.war doesn’t know how to find the Mysql host if it is not ‘localhost’.
...
software: {
javaCode: {
Source: "mvn",
Artifact: "target/JavaWebProjectExample.war",
configFile: {
Name: "java.properties",
Content: [{
Source:"mysql",
Regexp: ".*db.host.*",
Line: "\"db.host\": \"$$VALUE$$\""
}],
LoadDefaultContent: "src/main/resources/default.properties",
AttachAsEnvVar: [
"JAVA_OPTS", "-Dconfig.properties=$$SELF_NAME$$"
]
}
}, mysql: {
Source: "mysql",
Mysql: {
Schema: "java_code",
Create: [ "./src/db/mysql.dump" ]
}
}, tomcat: {
Source: "tomcat",
Deploy: "javaCode"
}
}
...
The final piece defines a configuration file for our WAR. It specifies the db.host variable and assigns the mysql host name to it. Finally the file name will be assigned to a -D parameter called config.properties via the tomcat environment variable JAVA_OPTS.
To support Vagrant the Fulgensfile.js needs a section ‘Vagrant’ on the initial config object:
module.exports = { config: {
SchemaVersion: "1.0.0",
Name: "JavaWebProjectExample",
Vagrant: {
Box: 'ubuntu/xenial64',
Install: 'maven openjdk-8-jdk-headless mysql-client-5.7 docker.io'
}
},
...
This defines the packages needed on a fresh installation of Ubuntu 16.04.
The complete source code can be found here.
Generating the bash script
After installing Fulgens from the npm repository via npm -g install fulgens`, a bash script can be generated using the command:fulgens Fulgensfile.js >run_local.sh
(You need to give the generated bash script executable rights via chmod 755 run_local.sh)
The script can be started with -h to get the help information:
Executing the script to build, deploy and run the project
One of the most simple things one can do with the script is to start it via ./run_local.sh -f. This will build the WAR file, start the Mysql database, set up the schema, table and initial data and finally start a Tomcat with the deployed WAR file. As we have given -f the script will finally tail Tomcat’s log file.
If we want to start the Tomcat inside a Docker container, we use ./run_local.sh -t tomcat:docker. This still builds the WAR file, starts the Mysql database, sets up the schema, table and initial data and finally starts a Tomcat within Docker with the deployed WAR file.
The build can also be done inside a Docker container. ./run_local.sh -t javacode:docker will again build, deploy and start the project. But this time the maven build, called ‘javacode’, will be executed inside Docker.
To build, deploy and start the project inside a Vagrant (VirtualBox) environment, you can use./run_local.sh -V.
Documenting and limiting versions
Fulgens can also be used to document and limit software versions.
Let’s assume our project must be build with Java 1.8, the Mysql must be a Version 5.x and the Tomcat a 7.0.92 using JRE-8.
In this case a versions object can be added to the Fulgensfile.js.
...
versions: {
javaCode: {
Docker: "3-jdk-8",
JavaLocal: "1.8",
KnownMax: "Java 1.8"
},
mysql: {
Docker: "5",
KnownMax: "Mysql 5.x"
},
tomcat: {
Docker: "7.0.92-jre8-slim",
TestedWith: "7 on Java 8"
}
}
...The attributes `KnownMax` and `TestedWith` are for documentation only and can hold any string. The attributes Dockerand JavaLocal are actually limiting the Docker or Java versions used by Fulgens.
Real world references
Here are some examples where I used Fulgens for my own projects.
My project ‘Code Your Restaurant (cyc)’ defines a Fulgensfile.js that first builds a Java project, then starts a Couchdb where it adds 3 views. The project consists of 2 parts, a backend server which is a plain Java program and a WAR file hosted on a Tomcat. Both of them need config files.
Another project is called ‘Lunchy’, a Java web applicaton using a Mysql database. The Fulgensfile.js builds the Java project, starts the Mysql, deploys the WAR file to the Tomcat and starts the Tomcat. To use utf-8, a config file is applied to the Mysql, after the database started the schema, tables and initial data is created via ‘mvn -DcreateTables=true process-resources’.
The next project ‘Told You So’, uses a Fulgensfile.js which builds a project in need of a different pom.xml for Java >= 9. It also startes a CouchDB and Tomcat.
The project ‘Linky’ defines a Fulgensfile.js that starts with cloning the git repository of Lucene, building it and starting Lucene as a standalone Java process. Then a CouchDB is started and 2 schemas with initial views are created. Finally a Node program is started. The CouchDB and Node have their own config files and Node is provided several environment variables.
A Nodejs based project called ‘Citybuilder’ defines a CouchDB while a Node executable is started in this Fulgensfile.js.
The last project ‘Grid Game One (ggo)’ has a very simple Fulgensfile.js. It builds a project and starts it on a Tomcat.
Limitations
This project is in an early development stage and all features correlate strongly with what I needed for my projects. Further enhancements and extensions will depend on feedback from other users.
Currently Fulgens supports:
- maven (to build)
- java (to start)
- node (to start)
- shell script (to start)
- tomcat (to host war files)
- mysql (as a database backend)
- couchdb (as a database backend)
- redis (as a database backend)
