Write Your First Package
Introduction
This guide takes ~15 minutes and will walk you through writing your first Kurtosis package. You'll set up a Postgres database and an API server which connects to the database.
- Start a containerized Postgres database
- Seed your database with test data
- Connect an API server to your database
- Configure your package to take in parameters, so users can easily change the data in their database
- Publish your package
TL;DR Version
This quickstart is in a "code along" format. You can also dive straight into running the end results and exploring the code too.
Open the Playground: Start
Click on the "New Workspace" button! You don't have to worry about the Context URL, Editor or Class. It's all pre-configured for you.
Setup
Requirements
Before you proceed, please make sure you have:
- Installed and started the Docker engine
- Installed the Kurtosis CLI (or upgraded to latest if you already have it)
This guide will have you writing Kurtosis Starlark. You can optionally install the VSCode plugin to get syntax highlighting, autocomplete, and documentation.
Hello, World
First, create and cd into a directory to hold the project you'll be working on:
mkdir kurtosis-postgres && cd kurtosis-postgres
Next, turn the directory into a Kurtosis package using the package init command:
kurtosis package init
You'll see a Starlark file called main.star inside your new directory with the following contents (more on Starlark in the "Review" section coming up soon):
def run(plan):
# TODO
plan.print("hello world!")
Syntax Highlighting Tips for IDEs
If you're using Visual Studio Code, you may find our Kurtosis VS Code Extension helpful when writing Starlark.
If you're using Vim, you can add the following to your .vimrc to get Starlark syntax highlighting:
" Add syntax highlighting for Starlark files
autocmd FileType *.star setlocal filetype=python
or if you use Neovim:
autocmd BufNewFile,BufRead *.star set filetype=python
Finally, run the script:
kurtosis run --enclave kurtosis-postgres main.star
You should get the following output:
INFO[2023-03-15T04:27:01-03:00] Creating a new enclave for Starlark to run inside...
INFO[2023-03-15T04:27:05-03:00] Enclave 'kurtosis-postgres' created successfully
> print msg="Hello, world"
Hello, world
Starlark code successfully run. No output was returned.
INFO[2023-03-15T04:27:05-03:00] ===================================================
INFO[2023-03-15T04:27:05-03:00] || Created enclave: kurtosis-postgres ||
INFO[2023-03-15T04:27:05-03:00] ===================================================
Name: kurtosis-postgres
UUID: a78f2ce1ca68
Status: RUNNING
Creation Time: Wed, 15 Mar 2023 04:27:01 -03
========================================= Files Artifacts =========================================
UUID Name
========================================== User Services ==========================================
UUID Name Ports Status
Congratulations - you've written your first Kurtosis code!
Review: Hello World
In this section, you created a Kurtosis package containing a main.star file that simply told Kurtosis to print Hello, world. The .star extension corresponds to Starlark, a Python dialect also used by Google and Meta for configuring build systems.
Your code created a simple plan, shown by the line of output > print msg="Hello, world". The plan represents the set of instructions that need to run to set up your application.
Run Postgres
First, remove the "Hello World" enclave you created in the previous step:
kurtosis clean -a
Replace the contents of your main.star file with the following:
POSTGRES_PORT_ID = "postgres"
POSTGRES_DB = "app_db"
POSTGRES_USER = "app_user"
POSTGRES_PASSWORD = "password"
def run(plan, args):
# Add a Postgres server
postgres = plan.add_service(
name = "postgres",
config = ServiceConfig(
image = "postgres:15.2-alpine",
ports = {
POSTGRES_PORT_ID: PortSpec(5432, application_protocol = "postgresql"),
},
env_vars = {
"POSTGRES_DB": POSTGRES_DB,
"POSTGRES_USER": POSTGRES_USER,
"POSTGRES_PASSWORD": POSTGRES_PASSWORD,
},
),
)
Now, run your script with:
kurtosis run --enclave kurtosis-postgres main.star
You'll see in the result that the kurtosis-postgres enclave now contains a Postgres instance:
Name: kurtosis-postgres
UUID: a30106a0bb87
Status: RUNNING
Creation Time: Tue, 14 Mar 2023 20:23:54 -03
========================================= Files Artifacts =========================================
UUID Name
========================================== User Services ==========================================
UUID Name Ports Status
b6fc024deefe postgres postgres: 5432/tcp -> postgresql://127.0.0.1:59299 RUNNING
Review: Run Postgres
So what happened? Three things actually:
- Interpretation: Kurtosis first ran your Starlark to build a plan for what you wanted done (in this case, starting a Postgres instance)
- Validation: Kurtosis then ran several validations against your plan, including validating that the Postgres image exists and you have enough memory & CPU to run it
- Execution: Kurtosis finally executed the validated plan inside the enclave to start a Postgres container
Note that Kurtosis did not execute anything until after Interpretation and Validation completed. You can think of Interpretation and Validation like Kurtosis' "compilation" step for your distributed application: you can catch many errors before any containers run, which shortens the dev loop and reduces the resource burden on your machine.
We call this approach multi-phase runs. While this approach has powerful benefits over traditional scripting, it also means you cannot reference Execution values like IP address in Starlark because they simply don't exist at Interpretation time. We'll explore how Kurtosis gracefully handles values generated during the Execution phase at the Interpretation phase later on in the quickstart.
This section introduced Kurtosis' ability to validate that definitions work as intended, before they are run - helping developers catch errors sooner & save resources when configuring multi-container environments.
Add some data
A database without data is a fancy heater, so let's add some.
Your two options for seeding a Postgres database are:
- Making a sequence of PSQL commands via the
psqlbinary - Using
pg_restoreto load a package of data
Both are possible in Kurtosis, but for this tutorial we'll use pg_restore to seed your database with a TAR of DVD rental information, courtesy of postgresqltutorial.com.
Without Kurtosis
Normally going this route (using pg_restore) requires downloading the seed data to your local machine, starting Postgres, writing a pile of Bash to copy the seed data to the Postgres server, and then finally running the pg_restore command. If you forget to check if the database is available, you may get flakes when you try to use the seeding logic in a test.
Alternatively, you could use Docker Compose to volume-mount the data TAR into the Postgres server, but you'd still need to handle Postgres availability and sequencing the pg_restore afterwards.
With Kurtosis
By contrast, Kurtosis Starlark scripts can use data as a first-class primitive and sequence tasks such as pg_restore into the plan.
Let's see it in action, and we'll explain what's happening afterwards.
Update your main.star with the following:
data_package_module = import_module("github.com/kurtosis-tech/awesome-kurtosis/data-package/main.star")
POSTGRES_PORT_ID = "postgres"
POSTGRES_DB = "app_db"
POSTGRES_USER = "app_user"
POSTGRES_PASSWORD = "password"
SEED_DATA_DIRPATH = "/seed-data"
def run(plan, args):
# Make data available for use in Kurtosis
data_package_module_result = data_package_module.run(plan, {})
# Add a Postgres server
postgres = plan.add_service(
name = "postgres",
config = ServiceConfig(
image = "postgres:15.2-alpine",
ports = {
POSTGRES_PORT_ID: PortSpec(5432, application_protocol = "postgresql"),
},
env_vars = {
"POSTGRES_DB": POSTGRES_DB,
"POSTGRES_USER": POSTGRES_USER,
"POSTGRES_PASSWORD": POSTGRES_PASSWORD,
},
files = {
SEED_DATA_DIRPATH: data_package_module_result.files_artifact,
}
),
)
# Load the data into Postgres
postgres_flags = ["-U", POSTGRES_USER,"-d", POSTGRES_DB]
plan.exec(
service_name = "postgres",
recipe = ExecRecipe(command = ["pg_restore"] + postgres_flags + [
"--no-owner",
"--role=" + POSTGRES_USER,
SEED_DATA_DIRPATH + "/" + data_package_module_result.tar_filename,
]),
)
Now, run the following to see what happens:
kurtosis clean -a && kurtosis run --enclave kurtosis-postgres .
(Notice you are using . instead of main.star)
The output should also look more interesting as your plan has grown bigger:
INFO[2023-03-15T04:34:06-03:00] Cleaning enclaves...
INFO[2023-03-15T04:34:06-03:00] Successfully removed the following enclaves:
60601dd9906e40d6af5f16b233a56ae7 kurtosis-postgres
INFO[2023-03-15T04:34:06-03:00] Successfully cleaned enclaves
INFO[2023-03-15T04:34:06-03:00] Cleaning old Kurtosis engine containers...
INFO[2023-03-15T04:34:06-03:00] Successfully cleaned old Kurtosis engine containers
INFO[2023-03-15T04:34:06-03:00] Creating a new enclave for Starlark to run inside...
INFO[2023-03-15T04:34:10-03:00] Enclave 'kurtosis-postgres' created successfully
INFO[2023-03-15T04:34:10-03:00] Executing Starlark package at '/tmp/kurtosis-postgres' as the passed argument '.' looks like a directory
INFO[2023-03-15T04:34:10-03:00] Compressing package 'github.com/john-snow/kurtosis-postgres' at '.' for upload
INFO[2023-03-15T04:34:10-03:00] Uploading and executing package 'github.com/john-snow/kurtosis-postgres'
> upload_files src="github.com/kurtosis-tech/awesome-kurtosis/data-package/dvd-rental-data.tar"
Files with artifact name 'howling-thunder' uploaded with artifact UUID '32810fc8c131414882c52b044318b2fd'
> add_service name="postgres" config=ServiceConfig(image="postgres:15.2-alpine", ports={"postgres": PortSpec(number=5432, application_protocol="postgresql")}, files={"/seed-data": "howling-thunder"}, env_vars={"POSTGRES_DB": "app_db", "POSTGRES_PASSWORD": "password", "POSTGRES_USER": "app_user"})
Service 'postgres' added with service UUID 'f1d9cab2ca344d1fbb0fc00b2423f45f'
> exec recipe=ExecRecipe(command=["pg_restore", "-U", "app_user", "-d", "app_db", "--no-owner", "--role=app_user", "/seed-data/dvd-rental-data.tar"])
Command returned with exit code '0' with no output
Starlark code successfully run. No output was returned.
INFO[2023-03-15T04:34:21-03:00] ===================================================
INFO[2023-03-15T04:34:21-03:00] || Created enclave: kurtosis-postgres ||
INFO[2023-03-15T04:34:21-03:00] ===================================================
Name: kurtosis-postgres
UUID: 995fe0ca69fe
Status: RUNNING
Creation Time: Wed, 15 Mar 2023 04:34:06 -03
========================================= Files Artifacts =========================================
UUID Name
32810fc8c131 howling-thunder
========================================== User Services ==========================================
UUID Name Ports Status
f1d9cab2ca34 postgres postgres: 5432/tcp -> postgresql://127.0.0.1:62914 RUNNING
Does your Postgres have data now? Let's find out by opening a shell on the Postgres container and logging into the database:
kurtosis service shell kurtosis-postgres postgres
From there, listing the tables in the Postgres can be done with:
psql -U app_user -d app_db -c '\dt'
...which will reveal that many new tables now exist:
List of relations
Schema | Name | Type | Owner
--------+---------------+-------+----------
public | actor | table | app_user
public | address | table | app_user
public | category | table | app_user
public | city | table | app_user
public | country | table | app_user
public | customer | table | app_user
public | film | table | app_user
public | film_actor | table | app_user
public | film_category | table | app_user
public | inventory | table | app_user
public | language | table | app_user
public | payment | table | app_user
public | rental | table | app_user
public | staff | table | app_user
public | store | table | app_user
(15 rows)
Feel free to explore the Postgres container. When you're done, run either exit or press Ctrl-D.
Review: Add some data
So what just happened?
We ran the directory as a package
By using the kurtosis run . syntax rather than kurtosis run main.star, you told Kurtosis to run your directory as a Kurtosis package (specifically, a runnable package). Because your directory had a kurtosis.yml file (created for you during kurtosis package init), your directory met the requirements for a Kurtosis package which allowed you to declare dependencies on external packages using Kurtosis’ built-in packaging/dependency system.
To see this in action, the first line in your local main.star file was used to import, and therefore declare a dependency on, an external package called data-package using a locator:
data_package_module = import_module("github.com/kurtosis-tech/awesome-kurtosis/data-package/main.star")
... which we then ran locally:
data_package_module_result = data_package_module.run(plan, {})
This external Kurtosis package, named "data-package" contains the seed data for your Postgres instance that we referenced earlier as a .tar file.
Special note here that we used a locator to import an external package from the awesome-kurtosis repository and not a regular URL. Learn more about how they differ here.
You imported seed data into your Kurtosis package
The main.star file in that external "data-package" contained Starlark instructions to store the .tar data as a files artifact using the upload_files Starlark instruction:
TAR_FILENAME = "dvd-rental-data.tar"
def run(plan, args):
dvd_rental_data = plan.upload_files("github.com/kurtosis-tech/awesome-kurtosis/data-package/" + TAR_FILENAME)
result = struct(
files_artifact = dvd_rental_data, # Needed to mount the data on a service
tar_filename = TAR_FILENAME, # Useful to reference the data TAR contained in the files artifact
)
return result
A files artifact is Kurtosis' first-class data primitive and is a TGZ of arbitrary files living inside an enclave. So long as a files artifact exists, Kurtosis knows how to mount its contents on a service.
You mounted and seeded the data into your Postgres instance
Next, you mounted the seed data, stored in your enclave now as a files artifact, into your Postgres instance using the ServiceConfig.files option:
postgres = plan.add_service(
name = "postgres",
config = ServiceConfig(
# ...omitted...
files = {
SEED_DATA_DIRPATH: data_package_module_result.files_artifact,
}
),
)
Then to seed the data, you used the exec Starlark instruction:
plan.exec(
service_name = "postgres",
recipe = ExecRecipe(command = ["pg_restore"] + postgres_flags + [
"--no-owner",
"--role=" + POSTGRES_USER,
SEED_DATA_DIRPATH + "/" + data_package_module_result.tar_filename,
]
),
Here, you saw one of Kurtosis' most loved features: the ability to modularize and share your distributed application logic using only a Github repository. We won't dive into all the usecases now, but the examples here can serve as a good source of inspiration.
Add an API
In this section we'll add a PostgREST API in front of the database and see how Kurtosis handles inter-service dependencies.
Replace the contents of your main.star with this:
data_package_module = import_module("github.com/kurtosis-tech/awesome-kurtosis/data-package/main.star")
POSTGRES_PORT_ID = "postgres"
POSTGRES_DB = "app_db"
POSTGRES_USER = "app_user"
POSTGRES_PASSWORD = "password"
SEED_DATA_DIRPATH = "/seed-data"
POSTGREST_PORT_ID = "http"
def run(plan, args):
# Make data available for use in Kurtosis
data_package_module_result = data_package_module.run(plan, {})
# Add a Postgres server
postgres = plan.add_service(
name = "postgres",
config = ServiceConfig(
image = "postgres:15.2-alpine",
ports = {
POSTGRES_PORT_ID: PortSpec(5432, application_protocol = "postgresql"),
},
env_vars = {
"POSTGRES_DB": POSTGRES_DB,
"POSTGRES_USER": POSTGRES_USER,
"POSTGRES_PASSWORD": POSTGRES_PASSWORD,
},
files = {
SEED_DATA_DIRPATH: data_package_module_result.files_artifact,
}
),
)
# Load the data into Postgres
postgres_flags = ["-U", POSTGRES_USER,"-d", POSTGRES_DB]
plan.exec(
service_name = "postgres",
recipe = ExecRecipe(command = ["pg_restore"] + postgres_flags + [
"--no-owner",
"--role=" + POSTGRES_USER,
SEED_DATA_DIRPATH + "/" + data_package_module_result.tar_filename,
]),
)
# Add PostgREST
postgres_url = "postgresql://{}:{}@{}:{}/{}".format(
"postgres",
POSTGRES_PASSWORD,
postgres.ip_address,
postgres.ports[POSTGRES_PORT_ID].number,
POSTGRES_DB,
)
api = plan.add_service(
name = "api", # Naming our PostgREST service "api"
config = ServiceConfig(
image = "postgrest/postgrest:v10.2.0",
env_vars = {
"PGRST_DB_URI": postgres_url,
"PGRST_DB_ANON_ROLE": POSTGRES_USER,
},
ports = {POSTGREST_PORT_ID: PortSpec(3000, application_protocol = "http")},
)
)
Now, run the same dev loop command as before (and don't worry about the result, we'll explain that later):
kurtosis clean -a && kurtosis run --enclave kurtosis-postgres .
We got a failure, just like we might when building a real system!
> add_service name="api" config=ServiceConfig(image="postgrest/postgrest:v10.2.0", ports={"http": PortSpec(number=3000, application_protocol="http")}, env_vars={"PGRST_DB_ANON_ROLE": "app_user", "PGRST_DB_URI": "postgresql://postgres:password@{{kurtosis:4d65eca66b5749df8988419ae31dda21:ip_address.runtime_value}}:5432/app_db"})
There was an error executing Starlark code
An error occurred executing instruction (number 4) at DEFAULT_PACKAGE_ID_FOR_SCRIPT[54:27]:
add_service(name="api", config=ServiceConfig(image="postgrest/postgrest:v10.2.0", ports={"http": PortSpec(number=3000, application_protocol="http")}, env_vars={"PGRST_DB_ANON_ROLE": "app_user", "PGRST_DB_URI": "postgresql://postgres:password@{{kurtosis:4d65eca66b5749df8988419ae31dda21:ip_address.runtime_value}}:5432/app_db"}))
Caused by: Unexpected error occurred starting service 'api'
Caused by: An error occurred waiting for all TCP and UDP ports being open for service 'api' with private IP '10.1.0.4'; as the most common error is a wrong service configuration, here you can find the service logs:
== SERVICE 'api' LOGS ===================================
09/May/2023:19:18:41 +0000: Attempting to connect to the database...
09/May/2023:19:18:41 +0000: {"code":"PGRST000","details":"connection to server at \"10.1.0.3\", port 5432 failed: FATAL: password authentication failed for user \"postgres\"\n","hint":null,"message":"Database connection error. Retrying the connection."}
09/May/2023:19:18:41 +0000: connection to server at "10.1.0.3", port 5432 failed: FATAL: password authentication failed for user "postgres"
postgrest: thread killed
== FINISHED SERVICE 'api' LOGS ===================================
Caused by: An error occurred while waiting for all TCP and UDP ports to be open
Caused by: Unsuccessful ports check for IP '10.1.0.4' and port spec '{number:3000 transportProtocol:0 applicationProtocol:0xc006662e10 wait:0xc00662d510}', even after '2' retries with '500' milliseconds in between retries. Timeout '15s' has been reached
Caused by: An error occurred while calling network address '10.1.0.4:3000' with port protocol 'TCP' and using time out '14.499139733s'
Caused by: dial tcp 10.1.0.4:3000: i/o timeout
Error encountered running Starlark code.
Here, Kurtosis is telling us that the add_service instruction on line 54 of your main.star (the one for ensuring PostgREST is up) is timing out when checking for ports being open.
Investigating the issue
If you check the service's logs, printed in the error message right after this header == SERVICE 'api' LOGS ===================================, you will see that there is an authentication error.
The enclave state is usually a good place to find more clues. If you look at the bottom of your output, you'll see the following state of the enclave:
Name: kurtosis-postgres
UUID: 5b360f940bcc
Status: RUNNING
Creation Time: Tue, 14 Mar 2023 22:15:19 -03
========================================= Files Artifacts =========================================
UUID Name
323c9a71ebbf crimson-haze
========================================== User Services ==========================================
UUID Name Ports Status
45b355fc810b postgres postgres: 5432/tcp -> postgresql://127.0.0.1:59821 RUNNING
From the above, we can see that the PostgREST service (named: api) is not in the 'User Services' list, so we can infer that it crashed when it was starting.
You can also grab the PostgREST logs...
kurtosis service logs kurtosis-postgres api
...and we can see that the PostgREST is dying:
15/Mar/2023:01:15:30 +0000: Attempting to connect to the database...
15/Mar/2023:01:15:30 +0000: {"code":"PGRST000","details":"FATAL: password authentication failed for user \"postgres\"\n","hint":null,"message":"Database connection error. Retrying the connection."}
15/Mar/2023:01:15:30 +0000: FATAL: password authentication failed for user "postgres"
postgrest: thread killed
Looking back to your Starlark code, you can identify the problem: it's creating the Postgres database with a user called app_user, but it's telling PostgREST to try and connect through a user called postgres:
POSTGRES_USER = "app_user"
# ...
def run(plan, args):
# ...
# Add a Postgres server
postgres = plan.add_service(
name = "postgres",
config = ServiceConfig(
# ...
env_vars = {
# ...
"POSTGRES_USER": POSTGRES_USER,
# ...
},
# ...
),
)
# ...
postgres_url = "postgresql://{}:{}@{}:{}/{}".format(
"postgres", # <---------- THE PROBLEM IS HERE
POSTGRES_PASSWORD,
postgres.ip_address,
postgres.ports[POSTGRES_PORT_ID].number,
POSTGRES_DB,
)
In the line declaring the postgres_url variable in your main.star file, replace the "postgres", string with POSTGRES_USER, to use the correct username we specified at the beginning of our file. Then re-run your dev loop:
kurtosis clean -a && kurtosis run --enclave kurtosis-postgres .
Now at the bottom of the output we can see that the PostgREST service is RUNNING correctly:
Name: kurtosis-postgres
UUID: 11c0ac047299
Status: RUNNING
Creation Time: Tue, 14 Mar 2023 22:30:02 -03
========================================= Files Artifacts =========================================
UUID Name
323c9a71ebbf crimson-haze
========================================== User Services ==========================================
UUID Name Ports Status
ce90b471a982 postgres postgres: 5432/tcp -> postgresql://127.0.0.1:59883 RUNNING
98094b33cd9a api http: 3000/tcp -> http://127.0.0.1:59887 RUNNING
Review: Add an API
In this section, you spun up a new PostgREST service (that we named api for readability) with a dependency on the Postgres service. Normally, PostgREST needs to know the IP address or hostname of the Postgres service, and we said earlier that Starlark (the Interpretation phase) can never know Execution values.
So how did the services get connected?
Answer: Execution-time values are represented at Interpretation time as future references that Kurtosis will replace at Execution time with the actual value. In this case, the postgres_url variable here...
postgres_url = "postgresql://{}:{}@{}:{}/{}".format(
POSTGRES_USER,
POSTGRES_PASSWORD,
postgres.ip_address,
postgres.ports[POSTGRES_PORT_ID].number,
POSTGRES_DB,
)
...used the postgres.ip_address and postgres.ports[POSTGRES_PORT_ID].number future references returned by adding the Postgres service, so that when postgres_url was used as an environment variable during PostgREST startup...
api = plan.add_service(
name = "api", # Naming our PostgREST service "api"
config = ServiceConfig(
# ...
env_vars = {
"PGRST_DB_URI": postgres_url, # <-------- HERE
"PGRST_DB_ANON_ROLE": POSTGRES_USER,
},
# ...
)
)
...Kurtosis simply swapped in the correct Postgres container execution-time values. While future references take some getting used to, we've found the feedback loop speedup to be very worth it.
What you've just seen is Kurtosis' powerful ability to gracefully handle data generated at runtime to set up service dependencies in multi-container environments. You also saw how seamless it was to run on-box CLI commands on a container.
Modifying data
Now that you have an API, you should be able to interact with the data.
Inspect your enclave:
kurtosis enclave inspect kurtosis-postgres
Notice how Kurtosis automatically exposed the PostgREST container's http port to your machine:
28a923400e50 api http: 3000/tcp -> http://127.0.0.1:59992 RUNNING
In this output the http port is exposed as URL http://127.0.0.1:59992, but your port number will be different.
You can paste the URL from your output into your browser (or Cmd+click if you're using iTerm) to verify that you are indeed talking to the PostgREST inside your kurtosis-postgres enclave:
{"swagger":"2.0","info":{"description":"","title":"standard public schema","version":"10.2.0.20230209 (pre-release) (a1e2fe3)"},"host":"0.0.0.0:3000","basePath":"/","schemes":["http"],"consumes":["application/json","application/vnd.pgrst.object+json","text/csv"],"produces":["application/json","application/vnd.pgrst.object+json","text/csv"],"paths":{"/":{"get":{"tags":["Introspection"],"summary":"OpenAPI description (this document)","produces":["application/openapi+json","application/json"],"responses":{"200":{"description":"OK"}}}},"/actor":{"get":{"tags":["actor"],"parameters":[{"$ref":"#/parameters/rowFilter.actor.actor_id"},{"$ref":"#/parameters/rowFilter.actor.first_name"},{"$ref":"#/parameters/rowFilter.actor.last_name"},{"$ref":"#/parameters/rowFilter.actor.last_update"},{"$ref":"#/parameters/select"},{"$ref":"#/parameters/order"},{"$ref":"#/parameters/range"},{"$ref":"#/parameters/rangeUnit"},{"$ref":"#/parameters/offset"},{"$ref":"#/parameters/limit"},{"$ref":"#/parameters/preferCount"}], ...
Now make a request to insert a row into the database (replacing $YOUR_PORT with the http port from your enclave inspect output for the PostgREST service that we named api)...
curl -X POST -H "content-type: application/json" http://127.0.0.1:$YOUR_PORT/actor --data '{"first_name": "Kevin", "last_name": "Bacon"}'
...and then query for it (again replacing $YOUR_PORT with your port)...
curl -X GET "http://127.0.0.1:$YOUR_PORT/actor?first_name=eq.Kevin&last_name=eq.Bacon"
...to get it back:
[{"actor_id":201,"first_name":"Kevin","last_name":"Bacon","last_update":"2023-03-15T02:08:14.315732"}]
Of course, it'd be much nicer to formalize this in Kurtosis. Replace your main.star with the following:
data_package_module = import_module("github.com/kurtosis-tech/awesome-kurtosis/data-package/main.star")
POSTGRES_PORT_ID = "postgres"
POSTGRES_DB = "app_db"
POSTGRES_USER = "app_user"
POSTGRES_PASSWORD = "password"
SEED_DATA_DIRPATH = "/seed-data"
POSTGREST_PORT_ID = "http"
def run(plan, args):
# Make data available for use in Kurtosis
data_package_module_result = data_package_module.run(plan, {})
# Add a Postgres server
postgres = plan.add_service(
name = "postgres",
config = ServiceConfig(
image = "postgres:15.2-alpine",
ports = {
POSTGRES_PORT_ID: PortSpec(5432, application_protocol = "postgresql"),
},
env_vars = {
"POSTGRES_DB": POSTGRES_DB,
"POSTGRES_USER": POSTGRES_USER,
"POSTGRES_PASSWORD": POSTGRES_PASSWORD,
},
files = {
SEED_DATA_DIRPATH: data_package_module_result.files_artifact,
}
),
)
# Load the data into Postgres
postgres_flags = ["-U", POSTGRES_USER,"-d", POSTGRES_DB]
plan.exec(
service_name = "postgres",
recipe = ExecRecipe(command = ["pg_restore"] + postgres_flags + [
"--no-owner",
"--role=" + POSTGRES_USER,
SEED_DATA_DIRPATH + "/" + data_package_module_result.tar_filename,
]),
)
# Add PostgREST
postgres_url = "postgresql://{}:{}@{}:{}/{}".format(
POSTGRES_USER,
POSTGRES_PASSWORD,
postgres.hostname,
postgres.ports[POSTGRES_PORT_ID].number,
POSTGRES_DB,
)
api = plan.add_service(
name = "api",
config = ServiceConfig(
image = "postgrest/postgrest:v10.2.0",
env_vars = {
"PGRST_DB_URI": postgres_url,
"PGRST_DB_ANON_ROLE": POSTGRES_USER,
},
ports = {POSTGREST_PORT_ID: PortSpec(3000, application_protocol = "http")},
)
)
# Insert data
if "actors" in args:
insert_data(plan, args["actors"])
def insert_data(plan, data):
plan.request(
service_name = "api",
recipe = PostHttpRequestRecipe(
port_id = POSTGREST_PORT_ID,
endpoint = "/actor",
content_type = "application/json",
body = json.encode(data),
)
)
Now clean and run, only this time with extra args to kurtosis run:
kurtosis clean -a && kurtosis run --enclave quickstart . '{"actors": [{"first_name":"Kevin", "last_name": "Bacon"}, {"first_name":"Steve", "last_name":"Buscemi"}]}'
Using the new http URL on the api service in the output, query for the rows you just added (replacing $YOUR_PORT with your correct PostgREST http port number)...
curl -XGET "http://127.0.0.1:$YOUR_PORT/actor?or=(last_name.eq.Buscemi,last_name.eq.Bacon)"
...to yield:
[{"actor_id":201,"first_name":"Kevin","last_name":"Bacon","last_update":"2023-03-15T02:29:53.454697"},
{"actor_id":202,"first_name":"Steve","last_name":"Buscemi","last_update":"2023-03-15T02:29:53.454697"}]
Review: Modifying data
Mechanically, we first created a JSON string of data using Starlark's json.encode builtin. Then we used the request Starlark instruction to shove the string at PostgREST, which writes it to the database:
plan.request(
service_name = "api",
recipe = PostHttpRequestRecipe(
port_id = POSTGREST_PORT_ID,
endpoint = "/actor",
content_type = "application/json",
body = json.encode(data),
)
)
At a higher level, Kurtosis automatically deserialized the {"actors": [{"first_name":"Kevin", "last_name": "Bacon"}, {"first_name":"Steve", "last_name":"Buscemi"}]} string passed as a parameter to kurtosis run, and put the deserialized object in the args parameter to the run function in main.star:
def run(plan, args):
This section showed how to interact with your environment, and also how to parametrize it for others to easily modify and re-use.
Publishing your Kurtosis package for others to use
At this point you should have 2 files:
main.starkurtosis.yml
These 2 files in a Github repository, with the kurtosis.yml file's name key matching github.com/YOUR_GITHUB_USERNAME/REPOSITORY_NAME, are all that's needed for anyone with Kurtosis and an internet connection to instantiate the environment you described in your main.star file.
Once you've uploaded it to a Github repository, anyone can spin up the same system to their liking (using your parameters), anywhere they want (e.g. Docker, Kubernetes, local, remote, etc).
As an example, a consumer of the Kurtosis package need only run the following command to reproduce the system and add another row to the database of actors:
kurtosis run github.com/YOUR_GITHUB_USERNAME/REPOSITORY_NAME '{"actors": [{"first_name":"Harrison", "last_name": "Ford"}
where YOUR_GITHUB_USERNAME is your Github username, and REPOSITORY_NAME is the name of your Github repository that houses your kurtosis.yml and main.star files.
Conclusion
And that's it - you've written your very first package in Kurtosis! You experienced a number of workflows that Kurtosis optimizes to make setting up an environment easy for your package users. Namely, you:
- Started a Postgres database in an ephemeral, isolated environment.
- Seeded your database by importing an external Starlark package from the internet.
- Set up an API server for your database and gracefully handled dynamically generated dependency data.
- Inserted & queried data via the API.
- Parameterized data insertion for future use by other users.
- Made your Kurtosis package consumable by any user by publishing a Kurtosis package to Github.
To dig deeper, visit other sections of our docs where you can read about what Kurtosis is, understand the architecture, and hear our inspiration for starting Kurtosis.
Second, we'd love to hear from you! Please don't hesitate to:
- Use
kurtosis feedbackto file an issue on our Github - Ask questions in our Github Discussions forum
- Join our community on Discord
- Reach out to us on Twitter
Finally, if you found Kurtosis interesting or useful, we'd appreciate a star on Github!