• Bootstrapping Kotlin from scratch

    From Emmanuel Bourg@21:1/5 to All on Fri Apr 2 23:00:02 2021
    Hi all,

    I've attempted a different approach for bootstrapping Kotlin and I'd
    like to share with you the result. For the reminder, Kotlin is extremely difficult to build from sources because it is written itself in Kotlin
    and its build system is based on Gradle which also depends on Kotlin,
    that's the perfect scenario for a serious headache.

    Instead of starting with a relatively recent release of Kotlin and a
    binary bootstrap compiler and standard library that are subsequently
    rebuilt, I tried to start further back in time with a release that
    doesn't depend on Kotlin and then build up a path toward the recent
    releases.

    TL;DR it didn't work, but I'd like to document the process if someone
    plans to follow this path in the future.

    The journey starts with locating the latest version of the Kotlin
    compiler that doesn't have .kt source files, and that's the version
    0.6.786 (tagged as build-0.6.786). It goes back to October 2013. At this
    time Ant was the build system (the Ant build was dropped in the version build-1.2.30-dev-715 tagged in January 2018, so Gradle isn't an
    immediate concern). Java 6 was the version targeted by the built but I
    assume Java 7 (released in 2011) was used. For this bootstrapping
    process I'm using Java 8, which is the lowest version available in
    Debian unstable and the baseline for the current ant package.

    Kotlin 0.6.786 requires the IntelliJ SDK 132.27, the issue is the
    ancient SDK builds are not tagged in the Git repository. There are only branches for the stable releases (131, 133, 135...). Fortunately, the
    code on the branch 131 is good enough to build Kotlin 0.6.786. The
    branch 133 is fine too with a minor tweak (the class StubBuildingVisitor requires an extra constructor to preserve the backward compatibility
    with Kotlin < 0.6.1203). The SDK 133 can be used to build Kotlin up to
    the version 0.6.2122.

    A side note on the SDK: its dependencies are checked into the lib
    directory of the repository as binary jar files, and some of them were modified. We can replace them with Debian libraries, except one, ASM4,
    which was relocated to the org.jetbrains.asm4 package and got a few undocumented changes. Theoretically the lib/src directory contains the
    source archives for these modifications, but for ASM the sources don't
    compile and some modifications are missing (the jar was modified several
    times but the source archive was never updated). Without these
    modifications the Kotlin build fails with Java 8. After some
    investigation, it appears that the required modification is a change to
    the ClassReader class that ensures that parsing unsupported class file
    formats doesn't trigger an exception. This is a modification that's is
    still carried today by the Jetbrains variant of ASM (https://github.com/JetBrains/intellij-deps-asm/commit/db1b691).

    So Kotlin 0.6.786 + SDK 133 is able to build Kotlin up to the version
    0.6.895. It fails on the version 0.6.900 due to a new annotation (jet.KotlinTraitImpl) added to Kotlin 0.6.855 and now used by the
    compiler. I opted to add the annotation to Kotlin 0.6.786 and continue.
    It works up to Kotlin 0.6.1364.

    With Kotlin 0.6.1379 the build fails due to the "externalannotations"
    attribute added to the <withKotlin> Ant task in Kotlin 0.6.1130 and
    first used in this version. So Kotlin 0.6.1364 now replaces 0.6.786 as
    the bootstrap compiler.

    Kotlin 0.6.1364 + SDK 133 builds Kotlin up to the version 0.6.1475 (note
    that 0.6.1379 fails due to the use of closure-compiler for the Kotlin to JavaScript compiler, I've simply disabled this part of the build). The
    version 0.6.1485 failed with an error I couldn't explain:

    [javac2] ERROR: /home/ebourg/kotlin/compiler/frontend/src/org/jetbrains/jet/lang/psi/psiUtil/jetPsiUtil.kt:
    (84, 58) Type mismatch: inferred type is (???) -> jet.Boolean but com.google.common.base.Predicate<org.jetbrains.jet.lang.psi.JetElement>
    was expected
    [javac2] ERROR: /home/ebourg/kotlin/compiler/frontend/src/org/jetbrains/jet/lang/psi/psiUtil/jetPsiUtil.kt:
    (84, 60) Cannot infer a type for this parameter. To specify it
    explicitly use the {(p : Type) => ...} notation

    It looks like the problematic function JetPsiUtil.getOutermostLastBlockElement() is neither used by the
    compiler nor the standard library, so I've removed it.

    The next failure is with Kotlin 0.6.1507 due to the new org.jetbrains.annotations.ReadOnly annotation which was moved to the
    runtime but used by the compiler before the runtime is built. Copying
    the annotation into the compiler source solves the issue.

    The next stop is Kotlin 0.6.1799 where the compilation breaks due to a
    bytecode verification error (java.lang.VerifyError: Bad type on operand
    stack). I suspect the bytecode verification got stricter in Java 8 and
    the bytecode generated by Kotlin back then is now considered invalid. I
    worked around the issue by adding the -noverify JVM argument (a trick
    shared by the Kotlin CLI script, so I assume it's ok).

    In Kotlin 0.6.1862 the Java to Kotlin converter (j2kConverter) fails to
    build, I'm not sure why but since it is optional I disabled it too.

    In Kotlin 0.6.1910 the following error appears:

    [javac2] ERROR: /home/ebourg/kotlin-from-scratch/kotlin.upstream/compiler/frontend/src/org/jetbrains/jet/lang/evaluate/OperationsMapGenerated.kt:
    (104, 64) Unresolved reference: compareTo

    I worked around this by commenting the offending line.

    The process continues up to Kotlin 0.6.2052 and this is the end of the
    road for Kotlin 0.6.1364 as the bootstrap compiler. The next version
    0.6.2063 has unresolved references in
    compiler/builtins-serializer/src/run.kt. Kotlin 0.6.2052 becomes the new bootstrap compiler and we continue.

    Kotlin 0.6.2052 + SDK 133 builds Kotlin up to the version 0.6.2107. We
    switch again the bootstrap compiler.

    Kotlin 0.6.2107 + SDK 133 works up to Kotlin 0.6.2338.

    And now start the real troubles with Kotlin 0.6.2350. It breaks because
    this is the first version using the SDK 134, but there is no tag nor
    branch for the SDK 134. The 135 branch doesn't work due to the switch to
    ASM 5 (and Kotlin didn't switch to ASM 5 until the version 0.7.391).
    I've located where the version of the SDK changed from 134 and 135
    (commit 4bb7296c2747fa73898247da03e7c9923935cc61) and I've built the SDK
    just before this point. The Kotlin build fails with this error:

    runtime:
    [echo] Cleaning /home/ebourg/kotlin-from-scratch/kotlin.upstream/dist/classes/runtime
    [mkdir] Created dir: /home/ebourg/kotlin-from-scratch/kotlin.upstream/dist/classes/runtime
    [java] exec() finished with INTERNAL_ERROR return codeEXCEPTION: java.lang.IllegalArgumentException: Missing extension point: com.intellij.psi.classFileDecompiler in area null
    [java] at com.intellij.openapi.extensions.impl.ExtensionsAreaImpl.getExtensionPoint(ExtensionsAreaImpl.java:373)
    [java] at com.intellij.openapi.extensions.impl.ExtensionsAreaImpl.getExtensionPoint(ExtensionsAreaImpl.java:37)
    [java] at com.intellij.openapi.extensions.Extensions.getExtensions(Extensions.java:110)
    [java] at com.intellij.openapi.extensions.Extensions.getExtensions(Extensions.java:98)
    [java] at com.intellij.openapi.extensions.ExtensionPointName.getExtensions(ExtensionPointName.java:48)
    [java] at com.intellij.psi.compiled.ClassFileDecompilers.find(ClassFileDecompilers.java:88)
    [java] at com.intellij.psi.ClassFileViewProviderFactory.createFileViewProvider(ClassFileViewProviderFactory.java:33)
    [java] at com.intellij.psi.impl.file.impl.FileManagerImpl.createFileViewProvider(FileManagerImpl.java:202)
    [java] at com.intellij.psi.impl.file.impl.FileManagerImpl.findViewProvider(FileManagerImpl.java:160)
    [java] at com.intellij.psi.impl.file.impl.FileManagerImpl.findFile(FileManagerImpl.java:328)
    [java] at com.intellij.psi.impl.PsiManagerImpl.findFile(PsiManagerImpl.java:188)

    I have no idea how to fix this.

    In a last attempt I tried to build Kotlin 0.7.391 with Kotlin 0.6.2338 +
    SDK 135 but it failed.

    This is where the story halts.

    To summarize:
    Step 1: build Kotlin 0.6.786 with a patched SDK 133
    Step 2: build Kotlin 0.6.1364
    Step 3: build Kotlin 0.6.2052
    Step 4: build Kotlin 0.6.2107 (good up to Kotlin 0.6.2338).
    Step 5: how to build Kotlin 0.6.2350 ???

    If someone figures out the next step, or has any hint about the last
    error, please ping me :)

    Emmanuel Bourg

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Thorsten Glaser@21:1/5 to Emmanuel Bourg on Sat Apr 3 00:10:01 2021
    On Fri, 2 Apr 2021, Emmanuel Bourg wrote:

    Instead of starting with a relatively recent release of Kotlin and a
    binary bootstrap compiler and standard library that are subsequently
    rebuilt, I tried to start further back in time with a release that
    doesn't depend on Kotlin and then build up a path toward the recent
    releases.

    This is the way things “ought” to be done, but… meh :(

    To summarize:
    Step 1: build Kotlin 0.6.786 with a patched SDK 133
    Step 2: build Kotlin 0.6.1364
    Step 3: build Kotlin 0.6.2052
    Step 4: build Kotlin 0.6.2107 (good up to Kotlin 0.6.2338).
    Step 5: how to build Kotlin 0.6.2350 ???

    Wow! Great job getting so far.

    If someone figures out the next step, or has any hint about the last
    error, please ping me :)

    Is there a FOSS contact at Jetbrains we could ask?
    They “ought to” know how to bootstrap/build it, but given
    the “used before it is built” part of your story I doubt
    it is easily possible and they know how…

    bye,
    //mirabilos
    --
    Infrastrukturexperte • tarent solutions GmbH
    Am Dickobskreuz 10, D-53121 Bonn • http://www.tarent.de/
    Telephon +49 228 54881-393 • Fax: +49 228 54881-235
    HRB AG Bonn 5168 • USt-ID (VAT): DE122264941
    Geschäftsführer: Dr. Stefan Barth, Kai Ebenrett, Boris Esser, Alexander Steeg

    *************************************************

    Mit dem tarent-Newsletter nichts mehr verpassen: www.tarent.de/newsletter

    *************************************************

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Emmanuel Bourg@21:1/5 to All on Tue Apr 6 09:50:02 2021
    Le 02/04/2021 à 22:56, Emmanuel Bourg a écrit :

    Step 5: how to build Kotlin 0.6.2350 ???

    I have the next steps, the decompiler code was refactored in the SDK in
    the commit f6c6bec7361fba61ecb3aa955e4145a789d4ed97, and this wasn't
    used until Kotlin 0.6.2443. Kotlin 0.6.2350 builds with Kotlin 0.6.2338
    and the SDK at the preceding commit
    a28195118431b514179fffe75c9ef5ed1247d6a6. And it works up to Kotlin
    0.6.2389.

    Starting with Kotlin 0.6.2407 two new jar files are required, kotlin-jdk-annotations.jar and kotlin-android-sdk-annotations.jar, they
    do no contain any code, just xml files defining the JDK methods that are
    null safe. They can be downloaded from http://teamcity.jetbrains.com
    (the URL are in the update_dependencies.xml file). With these files the
    build climbs to Kotlin 0.6.2415.

    Kotlin 0.6.2422 fails at the "builtins" phase, adding kotlin-runtime.jar
    to the classpath fixes the issue.

    Kotlin 0.6.2433 requires a new version of the SDK, the latest from the
    134 line (commit 1168c7b8cb4dc8318b8d24037b372141730a0d1f) works fine.
    There is also a minor nullable type definition to fix in JavaToKotlinTranslator.kt.

    It continues up to Kotlin 0.6.2500 (except 0.6.2463 which was a short
    lived branch).

    Kotlin 0.6.2516 fails because the jet.* packages were renamed to
    kotlin.*. It turns out the bootstrap compiler must be built with the bootstrap.build.no.tests=true parameter. This embeds the runtime into
    the compiler jar and allows it to build a more recent, incompatible compiler/runtime. I picked Kotlin 0.6.2451 as the bootstrap compiler to
    build Kotlin 0.6.2516, then built Kotlin 0.6.2516 with itself and bootstrap.build.no.tests=false. The resulting compiler is able to build
    Kotlin 0.6.2517.

    And this is the end of the 0.6.x line.

    There are missing symbols when building the first 0.7.x tag, I'm still
    working on it.

    Emmanuel Bourg

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Emmanuel Bourg@21:1/5 to All on Mon Apr 19 16:40:01 2021
    Le 06/04/2021 à 09:40, Emmanuel Bourg a écrit :

    There are missing symbols when building the first 0.7.x tag, I'm still working on it.

    I'm still making progress and I'm now at the version 0.8.409.

    I've started a GitHub project detailing the full build procedure from
    Kotlin 0.6.786 to 0.8.409:

    https://github.com/ebourg/kotlin-bootstrapping

    Emmanuel Bourg

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)