Pulumi: Infrastructure-as-a-Code in Java, Kotlin and Scala - JVM Weekly vol. 58

I've been waiting for this for a long time! Because today we can start again with a set of JEPs, which we will probably see around JDK 22. In addition, Pulumi, and interesting releases.

1. First set of new JEPs following the release of JDK 21.

JEP 455: Primitive types in Patterns, instanceof, and switch (Preview)

This JEP is unexpectedly complex - though, I should have anticipated this, considering it deals with primitive types, which invariably makes everything more intricate. Hence, I will illustrate with an example.

In reality, there are certain limitations associated with the use of primitive type patterns in Java. For instance, while we have the ability to model JSON documents using records (as demonstrated in the following code), there is a constraint on nested primitive type patterns.

if (json instanceof JsonObject(var map)
    && map.get("age") instanceof JsonNumber(double age))
{
    return new Customer(name, (int)age); 
}

In the provided code snippet, we anticipate the "age" value to consistently be an integer. However, we are only able to retrieve the value as a double and must depend on manual conversion (casting) to an integer. This issue arises from the way JSON depicts numbers and how numerous Java JSON libraries subsequently interpret them.

JSON doesn't differentiate among various number types. According to the JSON standard, all numbers are kept as floating point numbers. This implies that if we have the value 'age: 25' in a JSON document, it's interpreted as a floating-point number, even if it might represent age as an integer value.

When Java interprets a value from JSON, numerous libraries such as javax.json, org.json, and so on, perceive all numbers as double. This is because double can symbolize both integers and floating point numbers.

In the given example, when you attempt to match the "age" value with a JsonNumber pattern, the value is presented as a double - this is the most universal method of representing floating-point numbers in Java. If you desire the value as an int, you are required to explicitly cast it. However, be aware that this may involve a risk of precision loss for numbers that don't perfectly fit within the int range.

The suggested approach is to permit the use of primary type patterns more universally, ensuring their compatibility with instanceof and allowing their use in a broader set of scenarios. For instance, we aim to directly incorporate int into the JsonNumber pattern, so that the pattern only corresponds if the double value within the JsonNumber object can be transformed to int without any data loss.

if (json instanceof JsonObject(var map)
    && map.get("name") instanceof JsonString(String name)
    && map.get("age") instanceof JsonNumber(int age))
{
    return new Customer(name, age);         // bez konwersji!
}

This method can remove the requirement for comprehensive and possibly hazardous transformations later in the code..

This is just one example, you will find the meaning of more in the JEP.

JEP 458: Launch Multi-File Source-Code Programs

The upcoming JEP follows the one that simplified the creation of basic applications. It's uncertain if commercial developers recall, but the Java application launcher java from the JDK 11 permits uncompiled .java files to be executed. Why hasn't this approach been utilized more? The reason is that, until now, its use was restricted to a single file. After all, any experienced Java programmer understands that a single class can at most instantiate a structural pattern, but it's not sufficient for writing anything realistically useful.

The functionality will now be expanded to enable the launching of programs delivered as multiple Java source files, with automatic compilation of dependencies - provided they maintain the standard package structure. If a class in one source file refers to a class in another file, the launcher will automatically locate and compile the latter file. It's also feasible to utilize previously compiled classes. Transitioning from a single .java file to multiple files necessitates more complex steps for Java developers and learners, such as using javac. So we have another nod to newcomers, as seen with JEP 445: Unnamed Classes and Instance Main Methods (Preview).

JEP 457: Class-File API (Preview)

To clarify the purpose of this JEP, it's necessary to first discuss ASM. ASM is a Java library intended for the dynamic manipulation and generation of bytecode. It offers low-level analysis, creation, and alteration of existing classes. Many developers utilize ASM to enhance classes with features like logging or monitoring, as well as to generate new code following bytecode specifications. Libraries such as AspectJ, Kryo, FindBugs or JaCoCo, which use ASM, serve as examples. Additionally, it forms the foundation for the widely used (mainly among tool developers) ByteBuddy.

Interestingly, the JDK also utilizes ASM within its internals. ASM has emerged as a crucial library in executing certain JDK functions, such as generating proxies for lambdas. This is due to the fact that, in practice, despite the JDK possessing its own tools for bytecode manipulation, it is ASM that has consistently been one of the primary tools employed for such tasks over the years due to its performance and adaptability. JEP 457 aimed to offer a standard JDK for reading, writing, and modifying Java class files, thereby enabling the elimination of the internal use of ASM in the JDK. The initiative also strives to substitute existing uses of the ASM library in the JDK, which could eventually result in the removal of ASM from the JDK.

The subject has gained significant importance in the recent years. Initially, the development of JVM and the class file format was relatively slow, but the speed of this evolution has notably increased in the recent years. For instance, entities like Valhalla frequently introduce new (although often ephemeral) need for new bytecodes and field descriptors. The necessity to update external libraries such as ASM, which hinders JVM development, becomes increasingly impractical. The establishment of a dedicated standard library in the JDK for class modification, which evolves alongside the JVM, will enhance the flexibility of the entire process. Moreover, it will lead to the unification of all the custom solutions that currently exist in the JDK.

You can find additional information about the Class File API in the video:

---

Additionally, JEP 456: Unnamed Variables and Patterns has also been released, which basically unchanged stabilises JEP 443, known from JDK 21.

---

It's also quite fascinating to delve into Draft Valhalla, which discusses Null-Restricted Value Class Types (!). However, I plan to take a measured approach to this - I believe we'll dedicate an entire separate section to it once it becomes more stable.

2. Pulumi - Infrastructure-as-a-Code in Java, Kotlin and Scala

And now things are about to get untypical as we delve into the topic of Pulumi. But don't fret, I have a solid reason for this. Unfamiliar with Pulumi? I understand that you're not currently reading "Infrastructure as a Code Weekly", so let's begin with a quick overview. In my opinion, the simplest method to explain Pulumi is by comparing it with its main rival, Terraform.

For individuals without experience in DevOps, infrastructure, and other cloud-related aspects (and a small piece of advice - if you don't have a very precise career path, you should gain one), Terraform is a tool that manages infrastructure similarly to other code. However, it's not exactly the same "code", as Terraform employs a specialized HCL language to declaratively describe resources in the cloud. The idea is that users specify their desired infrastructure, and Terraform ensures this state is achieved with providers for different cloud platforms - all in a fully declarative manner.

However, as programmers (right?), it's time to consider the competition. Pulumi, the main hero of this section, is similar to Terraform in that it's used to manage infrastructure as code. However, it has a significant difference: it permits the use of standard programming languages (like Python, TypeScript) instead of a dedicated language. This enables Pulumi users to utilize loops, conditions, and other language features when defining infrastructure, providing additional flexibility over Terraform. This becomes incredibly beneficial when attempting to perform slightly more 'intelligent' tasks in our infrastructure than merely piecing together a few yamls from the internet.

The Girl He Told You Not to Worry About

Last week, Pulumi secured a significant amount of funding to further develop its tools. This happened during a period when Venture Capital is not as generous as before, indicating the potential of the tool. This event coincided with Terraforma's licensing problems, which you can find more details about here. This news prompted me to discuss the JVM versions of the Pulumi SDK, particularly since engineers from my main company, VirtusLab, played a significant role. As a business, we are deeply embedded in the open-source and JVM tooling universe, accountable for the Scala compiler among other things. Believing in Pulumi's vision, we collaborated with them to create an official Java SDK (created mainly by Anton Tayanovskyy and Paweł Prażak), and variants in Kotlin (being wrapper over Java SDK in practice, made by Michał Fudała, Dariusz Dzikon and Julia Plewa) and Scala (written from scratch by Łukasz Biały and Michał Pałka).

This latest release holds a special place for me. I firmly believe that the future will see the realization of the Kotlin Multiplatform vision, where a single language can be used to create all project 'artifacts' - the whole package. It's no surprise that application code can be written in Kotlin, but JetBrains is also deeply involved in Compose - the UI layer - and Gradle, which will be discussed later today, is also deeply committed to the Kotlin DSL. Infrastructure has been neglected, and Terraform with its HCL is causing significant disruption. This is why we ensured the creation of Pulumi for Kotlin. Julia Plewa shared the project's vision and the "first steps" in the article Pulumi Kotlin - The missing piece in Kotlin multi-platform, which perfectly encapsulates the entire project's vision.

Besom, the Scala SDK for Pulumi, shares a similar philosophy. The developers' goal is to enable developers to realistically create 'full-stack' applications in their preferred language, without the hassle of dealing with yams. It also offers robust type safety and pure, lazy, functional evaluation semantics, which are greatly cherished by Scala enthusiasts. The project can be found on Github. Additionally, here's a link to a tweet by Łukasz Biały announcing the project, which effectively highlights the philosophies behind it.

Do you agree with this method? If yes, give Pulumi a shot - it's been quite a while since I, for one, enjoyed writing "infra" this much.

And, and Pulumi didn't pay me a penny for advertising. I just like the tool 💎🙌.

3. Release Radar

JVector

Vector databases are an expanding area in the database realm and the broader IT community, due to their application in LLMs. In fact, vector search is crucial in contemporary applications that utilize generative AI, simplifying the process for developers to augment the knowledge base of models with extra data. This enables sophisticated language models to deliver accurate responses, preventing mistakes or 'hallucinations'.

It's not unexpected to see comparable technologies being developed in Java. I believe many readers already know that a number of database solutions rely on the JVM. These encompass Apache Cassandra, HBase - a columnar storage for the Hadoop file system, and Elasticsearch, a search and analysis engine built on Lucene.

JVector is a vector search engine that can be embedded, written entirely in Java. It powers DataStax Astra and integrates with Apache Cassandra, as previously mentioned. JVector's closest counterpart is the vector search of Apache Lucene. Lucene does implement the HNSW vector search algorithm, which is speedy but tends to be memory-hungry. JVector, on the other hand, is based on the more advanced DiskANN algorithm, making it over 10 times faster than Lucene when dealing with large datasets. JVector is quick, memory-efficient, disk-aware, parallel, easy to embed, and incremental. The JVector project aims for easy integration while still maintaining high performance. For instance, it utilizes the Vector API and SIMD instructions from Panama (which, for clarity, is still in incubation).

Gradle 8.4

The Gradle team has introduced version 8.4. The most significant update is the compilation support for JDK 21, and the embedded Kotlin has been upgraded to version 1.9.10. Interestingly, Kotlin 1.9.10 does not yet support JDK 21, implying that Gradle requires JDK 20 to function.

In this release, Windows also got the Java compiler daemons-based acceleration that other systems received in version 8.3. The update includes minor enhancements like optimized memory settings for code quality control tools (such as Checkstyle, CodeNarc, and PMD) in larger codebases, a better Checkstyle HTML report format, and compatibility with the JVM distributed by JetBrains.

Also a significant modification is the easier creation of role-focused configurations, with anticipated support for basic Gradle plug-ins for use as soon as the forthcoming version 9.0. The Kotlin DSL has also been improved, with stable support for simple property assignment using the = operator.

Bonus: Small invitation

And finally, if you reside in Prague or the Stuttgart vicinity, I would like to extend an invitation for you to meet me in person. Considering my upcoming week is somewhat hectic (which might lead to the absence of an edition next week, but I'll strive!).

First, I'm heading to Ludwigsburg, Germany for EclipseCon, where I'll be presenting GraalVM, CRaC, Leyden and friends - in search of TRULY cloud-native Java on the 19th of October.

Then a swift teleportation (since I'm unsure of what else to label it) to Prague 🇨🇿 to attend GeeCON on 20 October.

This is how we roll 😎

Even if you're not attending the conferences, if anyone is interested in meeting up for a coffee, I believe we can arrange it. If needed, you have my email.