Feature Request: Android App Bundles

It would be great to have EBuild support for the Android App Bundle (.aab) format. From the docs, “the Android App Bundle is Android’s new, official publishing format that offers a more efficient way to build and release your app.” Google Play Store still accepts app submissions in the older .apk format, but the App Bundle format is the official format now and offers several advantages:

  • Smaller app size (reported average ~20% decrease)
  • Larger app size limit (apk limited to 100mb, App Bundle limited to 150mb)
  • Allows app signing by Google Play, which is more efficient than self-signing and often more secure since most of us don’t have the resources to match Google’s security standards in-house
  • Dynamic, modular feature delivery to users

Android Studio of course supports App Bundle, but it also is supported by Unity, Xamarin, and a few other non-standard build toolchains. The format has been available for about 18 months now, and a Google engineer a couple weeks ago said that about 30% of the larger apps on the Play Store are already using it.

I think it should be a fairly straightforward feature add with Google’s bundletool utility. From the docs:

if you don’t want to use Android Studio or Gradle tasks to build bundles—for example, if you use a custom build toolchain—you can use bundletool from the command line to build an app bundle from pre-compiled code and resources

Also it would be nice to be able to generate both the .aab and a .apk from a single build, since other stores (notably the Amazon Appstore) still require the .apk format.

Thanks, logged as bugs://83666

Never a dull day, keeping up with the tool chains. :wink:

Yes, this is definitely something we should look at supporting; it was already on my radar but not formally logged yet, so I did now.

Important tip — I’ll make sure to not make this an either/or option but allow enabling both.

2 Likes

As a way of familiarizing myself with the EBuild code and architecture, I decided to look into what would need to change to support App Bundles, and I figured I should log everything here for future reference, for whoever ends up working on this, even if it’s me. :wink:

TL;DR :upside_down_face:

Steps to get App Bundles working in EBuild:

  1. Register AAPT2 as an Android tool.
  2. Either package the latest Bundletool.jar with EBuild or add logic to download it if absent.
  3. Tweak the GenerateAndroidResources task to use aapt2 compile and aapt2 link when AAPT2 is available.
  4. Tweak the AndroidPack task to use Bundletool to generate an App Bundle and APK when AAPT2 is available.

AAPT2

The larger to-do is supporting AAPT2, which is required for App Bundles. AAPT2 is a significant feature in its own right, since it enables incremental builds by diff-ing binaries of each app resource, which could greatly reduce build times for large Android projects.

A few things to note with AAPT2:

  • Min version: AAPT2 is available in SDK Build Tools 26.0.2 and above, so projects using less than 26.0.2 would need to update their build tools or revert to AAPT.

  • Mutually exclusive: I tried creating an App Bundle with AAPT2, using a dex file created through EBuild (which had compiled the class files using an R.java generated by AAPT) and it builds without issue but immediately crashes at runtime. I think this is the same issue that the docs refer to here:

    If your app has a dependency on a third party library that was built using older versions of the Android SDK Build Tools, your app might crash at runtime without displaying any errors or warnings. This crash might occur because, during the library’s creation, the R.java fields are declared final and, as a result, all of the resource IDs are inlined in the library’s classes. AAPT2 relies on being able to re-assign IDs to library resources when building your app. If the library assumes the IDs to be final and has them inlined in the library dex, there will be a runtime mismatch.

    This is important because it means that if a project is trying to generate both an APK with AAPT and an App Bundle with AAPT2, EBuild cannot share intermediate resources between the two. The R.java is incompatible, and the App Bundle also requires that app resources (drawables, layouts, strings, colors, etc) be in Google’s Protocol Buffer format, which APKs do not support. The format is easily acheived with a --proto-format switch on aapt2 link, but it means that the two cannot share these resources either.

  • Two phases: instead of AAPT’s single aapt package command, AAPT2 has two phases.
    (1) aapt2 compile :: is run for every file in the app’s res directory. This compile phase generates the protocol buffer binaries of the resource files, which are used for intermediary diffing and are added to the final App Bundle.
    (2) aapt2 link :: “merges all the intermediate files generated from the compilation phase such as resource tables, binary XML files, and processed PNG files and packages them into a single APK” (from docs). (Note this APK does not contain the dex files and is not an executable APK but rather an intermediary archive file in the build process, which for some reason they called an apk instead of a zip). The link phase also generates the R.java file.

App Bundle

App Bundles are not executable. An App Bundle is an archive that includes a BundleConfig.pb file (generated by aapt2 link - lists all of the modules in the App Bundle) and a zip file for each module. (Allowing multiple modules would be a good future feature, but for now EBuild could just put everything in the base.zip module). The zip file contains the app manifest, the dex files, the assets folder and native files, and all of the app resources along with their resource tables in protocol buffer (binary) format (see image below, ignoring the “Dynamic Feature” sections for now).

To get an executable APK from an App Bundle, you have to run Google’s Bundletool utility. This is the same tool that Google runs on their servers to generate custom APKs from an uploaded App Bundle.

Note: Since an App Bundle is not executable, this means that if the project is running in Debug mode, it would have to generate a final APK to install on the debug device for testing. Bundletool does have a build-apks command to generate an archive of all possible apks from the App Bundle, and an install-apks command which will automatically install the correct APK for a device connected to the adb bridge. The tool is very easy to use, but I don’t know how well it would cooperate with the debugger as I’ve not tested that aspect of it. Bundletool does offer another command bundletool build-apks --mode=universal which will generate a monolithic APK that can run on any device. This APK could then be installed via adb like Elements does now. This may be the most straightforward way to go about building/installing an APK for debugging.

EBuild

The best solution for EBuild is probably to always use AAPT2 if it is available, and only use AAPT as a fallback if AAPT2 is not available. Under this architecture, if AAPT2 is not available then the EBuild flow would be exactly like it is now, using AAPT and building an APK with apkbuilder. If AAPT2 is available, then EBuild would always build an App Bundle, and then optionally build an APK at the end if (a) the build is in Debug mode, or (b) the user has the project settings configured to generate an APK in Release mode. It is a bit longer process to get a simple APK for debugging, but I don’t think it would add much/any build time, since AAPT2 allows for faster incremental builds.

So… EBuild changes needed when using AAPT2:

GenerateAndroidResources

  1. Recursively loop through res folder and run aapt2 compile for each file.
  2. Run aapt2 link (note, every resource from the previous step is added as a command line argument in this command, so they should be added to an array as the loop runs for the compile phase).

AndroidPack

  1. Extract resources from the intermediary APK created by aapt2 link.
  2. Zip the extracted resources, dex files, manifest, assets, and jni files into a base.zip.
  3. Build the App Bundle: run bundletool build-bundle --modules=base.zip --output=[appname].aab
  4. Optionally generate an APK: bundletool build-apks --mode=universal.
1 Like

Good research! I’ll digest more fully and comment, tomorrow or Thursday. sounds like this is best treated as (at least) two separate tasks: (a) use aapt2 when o tools 26.0.2 or later (possibly optional) and (b) option to generate an app bundle, if (a) is met.

Does D8 figure into his at all, or will aap22 & co work fine with old dex? (EBuild has tentative support for D8, but it’s not publicly exposed yet and largely untested; testing./feedback would be appreciated, iirc the flag for turning it on is <UseD8>True</UseD8>.

1 Like

I’m not sure, I’ll take a look though. I’ll also try testing D8. I did notice a code comment that indicated that EBuild doesn’t work with D8 and multidex together yet. Is that still the case? If so I won’t be able to test with my main, large app since it’s multidexed, but I can try some test projects.

I also git cloned EBuild to open it in Water, but it doesn’t seem to fully load, so I can’t test any code changes for D8 support. Not sure if I’m missing part of the solution, or it could be that I only have the license for Oxygene since that’s all my employer uses for our project, so that could be why? :man_shrugging:t2:

Looks like, yes. it’s been a while, I need to get back to research what they holdup was on that…

Cool.

the CrossBox project (which you probably won’t need) has some C#, so that could be the issue, if you don’'t have a RemObjects C# license. This shouldn’t keep you from working on the other projects, or building the whole solution — you just won’t be able to work on the files in the CrossBox project itself.

I’ll see if can add a Community License to the repo.

1 Like

Looks like there was one, but it was out of date with regard t the actual projects. I’ve updated int, lemme know if this solves the issue for you…

1 Like

Seems compile can take more than one file at a time, tats good, coz spawning it a few hundred times would probably be slow, especially on Windows (on the flip side, Windows has limitations on how many parameters we can pass :().

That said, it seems to fail on this one:

E: aapt: invalid file path ‘/Users/mh/Test Projects/com.remobjects.androidapplication45/res/values/strings.android-xml’.

:frowning:

this fails for me with

D:                   /Users/Shared/Android/sdk/build-tools/29.0.0-rc3/aapt2 link -v -o "/Users/mh/Library/Application Support/RemObjects Software/EBuild/Obj/com.remobjects.androidapplication45-B5CB0D4D7B3898F4CC11143E27D750E3B3F7544B/Release/Cooper-Android/LinkedResources.apk" --manifest "/Users/mh/Library/Application Support/RemObjects Software/EBuild/Obj/com.remobjects.androidapplication45-B5CB0D4D7B3898F4CC11143E27D750E3B3F7544B/Release/Cooper-Android/AndroidManifest.xml" "/Users/mh/Library/Application Support/RemObjects Software/EBuild/Obj/com.remobjects.androidapplication45-B5CB0D4D7B3898F4CC11143E27D750E3B3F7544B/Release/Cooper-Android/res/drawable-hdpi_icon.png.flat" "/Users/mh/Library/Application Support/RemObjects Software/EBuild/Obj/com.remobjects.androidapplication45-B5CB0D4D7B3898F4CC11143E27D750E3B3F7544B/Release/Cooper-Android/res/drawable-ldpi_icon.png.flat" "/Users/mh/Library/Application Support/RemObjects Software/EBuild/Obj/com.remobjects.androidapplication45-B5CB0D4D7B3898F4CC11143E27D750E3B3F7544B/Release/Cooper-Android/res/drawable-mdpi_icon.png.flat" "/Users/mh/Library/Application Support/RemObjects Software/EBuild/Obj/com.remobjects.androidapplication45-B5CB0D4D7B3898F4CC11143E27D750E3B3F7544B/Release/Cooper-Android/res/drawable-xhdpi_icon.png.flat" "/Users/mh/Library/Application Support/RemObjects Software/EBuild/Obj/com.remobjects.androidapplication45-B5CB0D4D7B3898F4CC11143E27D750E3B3F7544B/Release/Cooper-Android/res/layout_main.layout-xml.flat"
H:                   aapt: linking package 'com.remobjects.androidapplication45' using package ID 7f.
H:                   aapt: merging 'drawable/icon' from compiled file /Users/mh/Test Projects/com.remobjects.androidapplication45/res/drawable-hdpi/icon.png.
H:                   aapt: merging 'drawable/icon' from compiled file /Users/mh/Test Projects/com.remobjects.androidapplication45/res/drawable-ldpi/icon.png.
H:                   aapt: merging 'drawable/icon' from compiled file /Users/mh/Test Projects/com.remobjects.androidapplication45/res/drawable-mdpi/icon.png.
H:                   aapt: merging 'drawable/icon' from compiled file /Users/mh/Test Projects/com.remobjects.androidapplication45/res/drawable-xhdpi/icon.png.
H:                   aapt: merging 'layout/main' from compiled file /Users/mh/Test Projects/com.remobjects.androidapplication45/res/layout/main.layout-xml.
H:                   aapt: enabling pre-O feature split ID rewriting.
E:                   aapt: attribute android:versionCode not found. [/Users/mh/Library/Application Support/RemObjects Software/EBuild/Obj/com.remobjects.androidapplication45-B5CB0D4D7B3898F4CC11143E27D750E3B3F7544B/Release/Cooper-Android/AndroidManifest.xml (2)]
E:                   aapt: attribute android:versionName not found. [/Users/mh/Library/Application Support/RemObjects Software/EBuild/Obj/com.remobjects.androidapplication45-B5CB0D4D7B3898F4CC11143E27D750E3B3F7544B/Release/Cooper-Android/AndroidManifest.xml (2)]
E:                   aapt: attribute android:debuggable not found. [/Users/mh/Library/Application Support/RemObjects Software/EBuild/Obj/com.remobjects.androidapplication45-B5CB0D4D7B3898F4CC11143E27D750E3B3F7544B/Release/Cooper-Android/AndroidManifest.xml (7)]
E:                   aapt: attribute android:icon not found. [/Users/mh/Library/Application Support/RemObjects Software/EBuild/Obj/com.remobjects.androidapplication45-B5CB0D4D7B3898F4CC11143E27D750E3B3F7544B/Release/Cooper-Android/AndroidManifest.xml (7)]
E:                   aapt: attribute android:label not found. [/Users/mh/Library/Application Support/RemObjects Software/EBuild/Obj/com.remobjects.androidapplication45-B5CB0D4D7B3898F4CC11143E27D750E3B3F7544B/Release/Cooper-Android/AndroidManifest.xml (7)]
E:                   aapt: attribute android:label not found. [/Users/mh/Library/Application Support/RemObjects Software/EBuild/Obj/com.remobjects.androidapplication45-B5CB0D4D7B3898F4CC11143E27D750E3B3F7544B/Release/Cooper-Android/AndroidManifest.xml (11)]
E:                   aapt: attribute android:name not found. [/Users/mh/Library/Application Support/RemObjects Software/EBuild/Obj/com.remobjects.androidapplication45-B5CB0D4D7B3898F4CC11143E27D750E3B3F7544B/Release/Cooper-Android/AndroidManifest.xml (11)]
E:                   aapt: attribute android:name not found. [/Users/mh/Library/Application Support/RemObjects Software/EBuild/Obj/com.remobjects.androidapplication45-B5CB0D4D7B3898F4CC11143E27D750E3B3F7544B/Release/Cooper-Android/AndroidManifest.xml (15)]
E:                   aapt: attribute android:name not found. [/Users/mh/Library/Application Support/RemObjects Software/EBuild/Obj/com.remobjects.androidapplication45-B5CB0D4D7B3898F4CC11143E27D750E3B3F7544B/Release/Cooper-Android/AndroidManifest.xml (17)]
E:                   aapt: attribute android:targetSdkVersion not found. [/Users/mh/Library/Application Support/RemObjects Software/EBuild/Obj/com.remobjects.androidapplication45-B5CB0D4D7B3898F4CC11143E27D750E3B3F7544B/Release/Cooper-Android/AndroidManifest.xml (22)]
E:                   aapt: failed processing manifest.

it seems to be very picky about the manifest; oddly, most of there entree it complains about are in the manifest ok, though…

<?xml version="1.0" encoding="utf-8"?>
<manifest
	xmlns:android="http://schemas.android.com/apk/res/android"
	package="com.remobjects.androidapplication45"
	android:versionCode="1"
	android:versionName="1.0">
	<application
		android:label="@string/app_name"
		android:icon="@drawable/icon"
		android:debuggable="false">
		<activity
			android:label="@string/app_name"
			android:name="com.remobjects.androidapplication45.MainActivity">
			<intent-filter>
				<action
					android:name="android.intent.action.MAIN"/>
				<category
					android:name="android.intent.category.LAUNCHER"/>
			</intent-filter>
		</activity>
	</application>
	<uses-sdk
		android:targetSdkVersion="29"/>
</manifest>

The first time I ran aapt2 it failed for me on all of EBuild’s custom xml types, and changing them to .xml worked. I did get it working with the custom xml filetypes though. I think I added the --legacy flag to aapt2 compile.

this seems to be by design

Ah good catch, that does seem to work fine

I handles .layout-xml fine though. --legacy doesnt seem to help for this file.

Hmm, pretty sure the structure is what it’s supposed to be, but I guess I’ll compare it against the docs in detail…

Ah, my mistake. I changed my layout xmls back to layout-xml but did not change my string back to android-xml, so when it worked for me it was with strings.xml. I’m seeing the same issue now with strings.android-xml

Yeah, looks like we’ll need to rename those. I’ll handle.

1 Like
E:                   aapt: invalid file path '/Users/mh/Library/Application Support/RemObjects Software/EBuild/Obj/com.remobjects.androidapplication45-B5CB0D4D7B3898F4CC11143E27D750E3B3F7544B/Release/Cooper-Android/temp-res/main.xml'.
E:                   aapt: invalid file path '/Users/mh/Library/Application Support/RemObjects Software/EBuild/Obj/com.remobjects.androidapplication45-B5CB0D4D7B3898F4CC11143E27D750E3B3F7544B/Release/Cooper-Android/temp-res/strings.xml'.

damn :(. it really needs them in the right folder structure, too.

Yeah, fwiw here’s the aapt2 source code for this bit:

static std::string buildIntermediateFilename(const std::string outDir,
                                             const ResourcePathData& data) {
    std::stringstream name;
    name << data.resourceDir;
    if (!data.configStr.empty()) {
        name << "-" << data.configStr;
    }
    name << "_" << data.name << "." << data.extension << ".flat";
    std::string outPath = outDir;
    file::appendPath(&outPath, name.str());
    return outPath;
}

int compile(const std::vector<StringPiece>& args) {
	CompileOptions options;
	Maybe<std::string> product;
	Flags flags = Flags()
			.requiredFlag("-o", "Output path", &options.outputPath)
			.optionalFlag("--product", "Product type to compile", &product)
			.optionalSwitch("-v", "Enables verbose logging", &options.verbose);
	if (!flags.parse("aapt2 compile", args, &std::cerr)) {
		return 1;
	}
	if (product) {
		options.product = util::utf8ToUtf16(product.value());
	}
	CompileContext context;
	std::vector<ResourcePathData> inputData;
	inputData.reserve(flags.getArgs().size());
	// Collect data from the path for each input file.
	for (const std::string& arg : flags.getArgs()) {
		std::string errorStr;
		if (Maybe<ResourcePathData> pathData = extractResourcePathData(arg, &errorStr)) {
			inputData.push_back(std::move(pathData.value()));
		} else {
			context.getDiagnostics()->error(DiagMessage() << errorStr << " (" << arg << ")");
			return 1;
		}
	}
	bool error = false;
	for (ResourcePathData& pathData : inputData) {
		if (options.verbose) {
			context.getDiagnostics()->note(DiagMessage(pathData.source) << "processing");
		}
		if (pathData.resourceDir == u"values") {
			// Overwrite the extension.
			pathData.extension = "arsc";
			const std::string outputFilename = buildIntermediateFilename(
					options.outputPath, pathData);
			if (!compileTable(&context, options, pathData, outputFilename)) {
				error = true;
			}
		} else {
			const std::string outputFilename = buildIntermediateFilename(options.outputPath,
																		 pathData);
			if (const ResourceType* type = parseResourceType(pathData.resourceDir)) {
				if (*type != ResourceType::kRaw) {
					if (pathData.extension == "xml") {
						if (!compileXml(&context, options, pathData, outputFilename)) {
							error = true;
						}
					} else if (pathData.extension == "png" || pathData.extension == "9.png") {
						if (!compilePng(&context, options, pathData, outputFilename)) {
							error = true;
						}
					} else {
						if (!compileFile(&context, options, pathData, outputFilename)) {
							error = true;
						}
					}
				} else {
					if (!compileFile(&context, options, pathData, outputFilename)) {
						error = true;
					}
				}
			} else {
				context.getDiagnostics()->error(
						DiagMessage() << "invalid file path '" << pathData.source << "'");
				error = true;
			}
		}
	}
	if (error) {
		return 1;
	}
	return 0;
}

I’ve made these explicit errors for now, and we can add some IDE toolage to rename these files in the project, once it all works.

If you wanna try it out, I have competed my first round of changes, you can explicitly set “UseAAPT2” to true to enable it (say to see if it compile/link works with you set of files). My nest step (maybe not today) would be to figure out what needs changing in the manifest file…

1 Like

I haven’t run this manually yet, but it seems like aapt2 should work fine with old dex based on this blog post where the author walks through a complete manual build using aapt2 and dx.

The Android Developers blog also seems to indicate that d8 and dx are fully interchangeable, and that even when dx is officially deprecated it will still work with the Android toolchain.

1 Like