Fixes #359 - Adds a Flatpak backend for Linux packaging by freakboy3742 · Pull Request #802 · beeware/briefcase

freakboy3742 · 2022-07-26T06:04:14Z

Adds a Flatpak backend as an alternative for Linux packaging.

To test:

Install flatpak and flatpak-builder as Linux system packages. Installation instructions can be found on the flatpack website. On most systems, this should be as straightforward as apt-get install flatpak flatpack-builder (or whatever the OS-appropriate installer tool is).

Add the following to the linux section of your pyproject.toml:

 template = "https://github.com/beeware/briefcase-linux-flatpak-template"
 template_branch = "dev"

Run briefcase X linux flatpak (where X is create, build, run, or package)

This PR doesn't deal with:

Signing the bundled single-file packages
Publishing packages to flathub or other repositories.

These would both be possible, but require building an integration with GPG, which is a whole integration unto itself.

Thanks to @loganasherjones for their extremely helpful blog post, and @nickzoic for an initial draft PR #754.

PR Checklist:

All new features have been tested
All new features have been documented
I have read the CONTRIBUTING.md file
I will abide by the code of conduct

…of support packages.

rmartin16 · 2022-07-26T16:01:52Z

+The end user will install the Flatpak into their local app repository; this can
+be done by installing directly from a single file `.flatpak` bundle, or by
+installing from a package repository like `Flathub <https://flathub.org>`__.
+Apps can be installed into user-space, or if the user has sufficient priviliges,


typo
priviliges -> privileges

rmartin16 · 2022-07-26T16:02:27Z

+
+    @classmethod
+    def verify(cls, command):
+        """Verify that linuxdeploy tool or plugin is available.


typo
linuxdeploy tool or plugin -> the Flatpak toolchain

rmartin16 · 2022-07-26T16:02:56Z

+            raise BriefcaseCommandError(f"Unable to start app {app_name}.") from e
+
+    def bundle(self, repo_url, bundle, app_name, version, build_path, output_path):
+        """Bubndle a Flatpak for distribution.


typo
Bubndle -> Bundle

rmartin16 · 2022-07-26T16:04:24Z

+    output_format = "flatpak"
+
+    def binary_path(self, app):
+        # Flatpak doesn't really produce an identifiable "binary" as part of it's


more cursed apostrophes
it's -> its

rmartin16 · 2022-07-26T17:49:04Z

+
+        # ``flatpak install`` uses a lot of console animations, and there
+        # doesn't appear to be a way to turn off these animations. Use those
+        # native animations rather than wrapping in a wait_bar.


furthermore, this actually brought up a user prompt for me to clarify which SDK package i wanted to install when I used org.gnome.SDK....not sure input would work with a wait bar.

Interesting... I didn't get that prompt; what was your project configuration to get that prompt? I got an error if I specified:

flatpak_runtime = "org.gnome.Platform" flatpak_sdk = "org.gnome.Sdk"

(because there isn't a gnome version 21.08 runtime); if I add flatpak_runtime_version = "42", it installed without a prompt. Did you use something different?

hmmm....I used the same config. This is also a fairly freshly installed VM too....so I don't think I've added anything crazy to flatpak.

[tool.briefcase.app.flat.linux] template = "https://github.com/beeware/briefcase-linux-flatpak-template" template_branch = "dev" flatpak_runtime = "org.gnome.Platform" flatpak_runtime_version = "42" flatpak_sdk = "org.gnome.SDK"

❯ briefcase build linux flatpak [flat] Ensuring Flatpak runtime for the app is available... Ensuring Flatpak runtime repo is registered... done Looking for matches… Skipping: org.gnome.Platform/x86_64/42 is already installed Similar refs found for ‘org.gnome.SDK’ in remote ‘flathub’ (user): 1) runtime/org.gnome.Sdk.Compat.i386/x86_64/42 2) runtime/org.gnome.Sdk.Compat.aarch64/x86_64/42 3) runtime/org.gnome.Sdk/x86_64/42 4) runtime/org.gnome.Sdk.Docs/x86_64/42 Which do you want to use (0 to abort)? [0-4]: 0 error: No ref chosen to resolve matches for ‘org.gnome.SDK’ Unable to install Flatpak runtime org.gnome.Platform//42 and SDK org.gnome.SDK//42 from repo flathub. Log saved to /home/user/tmp/beeware/flat/briefcase.2022_07_26-21_47_37.build.log

briefcase.2022_07_26-21_47_37.build.log

And, to top it off, it asks me every time I build the flatpak which org.gnome.SDK I want to install....:/

Hrm... I wonder if it might be related to the work you've been doing with i386. If the existence of 2 Platform flatpaks means there's 2 that match org.gnome.SDK//42, perhaps specifying org.gnome.SDK/x86_64/42 will remove the ambiguity...

@rmartin16 I've just pushed an update that will be explicit about architecture when installing the SDK; does that work better for you?

same :/

[flat] Ensuring Flatpak runtime for the app is available... >>> Running Command: >>> flatpak remote-add --user --if-not-exists flathub https://flathub.org/repo/flathub.flatpakrepo >>> Return code: 0 Ensuring Flatpak runtime repo is registered... done >>> Running Command: >>> flatpak install --assumeyes --user flathub org.gnome.Platform/x86_64/42 org.gnome.SDK/x86_64/42 Looking for matches… Skipping: org.gnome.Platform/x86_64/42 is already installed Similar refs found for ‘org.gnome.SDK’ in remote ‘flathub’ (user): 1) runtime/org.gnome.Sdk.Compat.i386/x86_64/42 2) runtime/org.gnome.Sdk.Compat.aarch64/x86_64/42 3) runtime/org.gnome.Sdk/x86_64/42 4) runtime/org.gnome.Sdk.Docs/x86_64/42 Which do you want to use (0 to abort)? [0-4]:

Ah - there's a typo in your config: org.gnome.SDK - > org.gnome.Sdk.

:| i see. thanks. that fixed it.

rmartin16 · 2022-07-26T17:49:32Z

Worked out of the box 🦾

Notes

Building a flatpak for 32bit appears possible....but I gave up actually getting it to work. It doesn't appear possible to actually build the flatpak on a 32bit system but I don't think that's too surprising.

Runtime and SDK versions should probably be updated together but it only fails at attempt to install. This is probably obvious to people more familiar with flatpak but just calling out that only changing flatpak_runtime to org.gnome.Platform and flatpak_runtime_version to 42 bombed out on attempting to install org.freedesktop.SDK//42. A check to ensure all three are set may be over-eager if its potentially valid that not all of them are overridden.

I tried to run tutorial 4 but had to add - --socket=session-bus to finish-args in manifest.yml to get it to run. Also, installed the packaged flatpak (from ubuntu 22.04) in to a system running ubuntu 18.04 and it successfully ran :)

I notice several files are being written without specifying the encoding...while linux is probably much more likely to be using utf-8 than windows, it may be worth explicitly declaring that.

rmartin16 · 2022-07-26T21:05:49Z

+            with bin_path.open("w") as f:
+                f.write(
+                    f"""\
+#!/bin/bash


This may be more portable with #!/bin/sh instead.

mhsmith · 2022-07-26T22:04:59Z

On Debian 10 (Buster) with Flatpak 1.2.5, I was able to run a non-graphical app (Python-support-testbed) with no problems. However, when I tried to run a graphical app, I hit the error message described in flatpak/flatpak#397. This is because I'm forwarding X11 apps over SSH to an X server (VcXsrv) running on on my WIndows desktop.

Unfortunately this can't simply be worked around by passing the DISPLAY environment variable into flatpak run, because PuTTY, which is one of the most popular SSH clients on Windows, also creates an Xauthority file which needs to be available within the sandbox.

This was fixed only a few months ago in flatpak/flatpak#4706, which is recent enough that it won't be included in any stable version of Debian until version 12 (Bookworm) comes out in 2023.

While running X11 over SSH is not that uncommon, it's obviously a lot less common than running a local Linux desktop. So if Flatpak is clearly better than Appimage, I don't think this should block us from making it the default output format on Linux, as long as we keep supporting Appimage as a fallback.

…ation.

freakboy3742 · 2022-07-27T01:45:05Z

Worked out of the box 🦾

W00t!

Building a flatpak for 32bit appears possible....but I gave up actually getting it to work. It doesn't appear possible to actually build the flatpak on a 32bit system but I don't think that's too surprising.

Yeah - I'm happy treating this as a "get it working at all" PR; adding support for 32 bit is something we can tackle later if someone is enthused. As with AppImage, it's likely going to be impeded by CI availability rather than anything technical.

Runtime and SDK versions should probably be updated together but it only fails at attempt to install. This is probably obvious to people more familiar with flatpak but just calling out that only changing flatpak_runtime to org.gnome.Platform and flatpak_runtime_version to 42 bombed out on attempting to install org.freedesktop.SDK//42. A check to ensure all three are set may be over-eager if its potentially valid that not all of them are overridden.

That's a good idea - I'll add a safety around this.

I tried to run tutorial 4 but had to add - --socket=session-bus to finish-args in manifest.yml to get it to run. Also, installed the packaged flatpak (from ubuntu 22.04) in to a system running ubuntu 18.04 and it successfully ran :)

Interesting... Were you able to run any other Toga tutorials? Adding an extra flag to manifest.yml isn't a big deal... except that the Flatpak docs seem to advise against adding that permission unless you really need it

I notice several files are being written without specifying the encoding...while linux is probably much more likely to be using utf-8 than windows, it may be worth explicitly declaring that.

Good catch; I'll add that in.

rmartin16 · 2022-07-27T03:01:06Z

Running in to issues with the requirements.txt file not existing right now. I'll let that stabilize.

Were you able to run any other Toga tutorials?

I saw the same error for the other tutorials as well.

Furthermore, a version check of flatpak-builder may be necessary. When I installed it from the default apt repos on Ubuntu 18.04, I got the error below:

❯ briefcase build linux flatpak -v

>>> Running Command:
>>>     flatpak --version
>>> Command Output:
>>>     Flatpak 1.0.9
>>> Return code: 0

>>> Running Command:
>>>     flatpak-builder --version
>>> Command Output:
>>>     flatpak-builder 0.10.9
>>> Return code: 0

[flat] Ensuring Flatpak runtime for the app is available...

>>> Running Command:
>>>     flatpak remote-add --user --if-not-exists flathub https://flathub.org/repo/flathub.flatpakrepo
>>> Return code: 0
Ensuring Flatpak runtime repo is registered... done

>>> Running Command:
>>>     flatpak install --assumeyes --user flathub org.freedesktop.Platform//21.08 org.freedesktop.Sdk//21.08
<....flatpak install logs omitted...>
>>> Return code: 0

[flat] Building Flatpak...

>>> Running Command:
>>>     flatpak-builder --force-clean --repo repo --install --user build manifest.yml
Can't parse 'manifest.yml': <data>:1:6: Parse error: unexpected identifier `app-id', expected value
>>> Return code: 1
Building...

Error while building app flat.

Log saved to /home/user/tmp/beeware/flat/briefcase.2022_07_26-22_25_48.build.log

I think previous version of flatpak-builder didn't support yaml perhaps.

Also, since I didn't see this explicitly stated, is AppImage being left the default format explicitly...? Or might Flatpak become default in the future?

rmartin16 · 2022-07-27T03:08:57Z

One more.....once I added - --socket=session-bus to manifest.yml for tutorial 3, I got this error (on a variant of Ubuntu 22.04):

>>> Running Command:
>>>     flatpak run com.example.flat
Gtk-Message: 23:03:56.547: Failed to load module "appmenu-gtk-module"
Gtk-Message: 23:03:56.643: Failed to load module "canberra-gtk-module"
/app/lib/python3.10/site-packages/toga_gtk/app.py:40: DeprecationWarning: Gtk.Window.set_wmclass is deprecated
  self.native.set_wmclass(app.interface.name, app.interface.name)
Traceback (most recent call last):
  File "/app/lib/python3.10/site-packages/toga_gtk/app.py", line 102, in gtk_startup
    self.interface.startup()
  File "/app/briefcase/app/flat/app.py", line 9, in startup
    self.webview = toga.WebView(on_webview_load=self.on_webview_loaded, style=Pack(flex=1))
  File "/app/lib/python3.10/site-packages/toga/widgets/webview.py", line 32, in __init__
    self._impl = self.factory.WebView(interface=self)
  File "/app/lib/python3.10/site-packages/toga_gtk/widgets/base.py", line 18, in __init__
    self.create()
  File "/app/lib/python3.10/site-packages/toga_gtk/widgets/webview.py", line 14, in create
    raise RuntimeError(
RuntimeError: Unable to import WebKit2. Ensure that the operating system GTK WebKit bindings (e.g., gir1.2-webkit2-4.0 on Debian/Ubuntu) have been installed.
^C
Aborted by user.

briefcase.2022_07_26-23_07_36.run.log

freakboy3742 · 2022-07-27T03:29:00Z

@rmartin16 Can you provide details about the requirements.txt failure you're seeing? The only reason I can think that would fail is if you're not using the corresponding template modifications from beeware/briefcacse-linux-flatpak-template#2 - you may need to specify template_branch='app-packages'.

As for a version check - looks like you may be right. YAML support wasn't added until 0.10.10. I'll add in a version check.

As for the long term plans for AppImage and Flatpak; for now, we're just adding Flatpak, and AppImage is remaining the default. However, AppImage tooling has proven a bit buggy, is being maintained by a very small team, and doesn't appear to have a huge amount of community support; whereas Flatpak has the full throated support of Redhat, Ubuntu, and others. If Flatpak gives us less headaches than AppImage, we'll likely change the default in future.

The webkit thing, however, is... ugh. I don't know how to fix that. I'm guessing there's an SDK we need to include?

rmartin16 · 2022-07-27T03:43:27Z

Can you provide details about the requirements.txt failure you're seeing? The only reason I can think that would fail is if you're not using the corresponding template modifications from beeware/briefcacse-linux-flatpak-template#2 - you may need to specify template_branch='app-packages'.

Ahh...I didn't realize the template branch changed. That fixed it.

rmartin16 · 2022-07-27T04:01:59Z

Another thought....should we support the flatpak-builder that flatpak can install?

I can run it with flatpak run org.flatpak.Builder. It doesn't looks like flatpaks are intended to be interacted with like "normal" binaries.

freakboy3742 · 2022-07-27T04:19:03Z

I have no idea if using a flatpak of flatpak-builder is a good idea - but given that their own tutorial doesn't mention it, my inclination is to stick with what we know works.

rmartin16 · 2022-07-27T04:28:36Z

I see it referenced here....although the rest of their guide clearly doesn't assume you're using it.

You can also install it as a flatpak with flatpak install flathub org.flatpak.Builder.

As far as getting a current version of flatpak-builder on Ubuntu 18.04, that definitely seemed like the easiest way. Something just to keep in mind if it's out of scope for now.

freakboy3742 · 2022-07-27T04:37:06Z

Looks like the Gnome runtime supports Webkit2 bindings. I guess it makes sense to use Gnome as a base for Toga apps, rather than a more generic FreeDesktop.

mhsmith · 2022-07-27T09:00:26Z

Looks like something's wrong with the CI task "Desktop app verification (ubuntu-latest, toga)": it's been running for over an hour with no output.

freakboy3742 · 2022-07-27T09:24:41Z

@mhsmith Yeah - looks like the GTK Linuxdeploy plugin doesn't like being run under CI...

mhsmith

Commit d592a71 worked fine for me with the current version of the Flatpak template.

The main annoyance I had during this process was having to wait several minutes to recompile Python after every run of briefcase create. It seems like this is something which should be cached outside of the project tree, but I don't know how easy Flatpack would make that.

Of course, this is something that'll affect us developers more than normal users, so it doesn't need to block this PR.

mhsmith · 2022-07-27T10:19:51Z

+    with pytest.raises(
+        BriefcaseConfigError,
+        match=r"",


Missing error message

mhsmith · 2022-07-27T10:20:09Z

+    with pytest.raises(
+        BriefcaseConfigError,
+        match=r"",


Missing error message.

mhsmith · 2022-07-27T10:21:51Z

+def test_custom_runtime_platform_only(first_app_config, tmp_path):
+    """If the user only defines a runtime, accessing the SDK raises an
+    error."""


Shouldn't this be called test_custom_runtime_runtime_only?

mhsmith · 2022-07-27T10:23:18Z

+def test_verify_linux_no_docker(tmp_path):
+    """Verifying on Linux creates an SDK wrapper."""


Docker shouldn't be mentioned here.

mhsmith · 2022-07-27T10:23:34Z

+def test_verify_non_linux_no_docker(tmp_path):
+    """Verifying on non-Linux raises an error."""


Docker shouldn't be mentioned here.

freakboy3742 · 2022-07-27T12:00:25Z

Agreed that the recompilation of Python is annoying; however, I can't see any obvious way to exfiltrate the build products of that step in a way that can be re-used, and Flatpak doesn't appear to have (AFAICT) any concept of "modular" pieces to a build - you either build the entire source tree of every dependency, or you take responsibility for maintaining your own Runtime and SDK.

However, as you note - this is something that we see because we recreate and recompile apps all the time, but most users won't need to rebuild apps as often. If we can find an optimisation in the future, it will be a huge win, but I don't think it's a blocker for now.

codecov · 2022-07-27T12:12:45Z

Codecov Report

Merging #802 (f969a47) into main (33689b3) will increase coverage by 0.16%.
The diff coverage is 100.00%.

Impacted Files	Coverage Δ
src/briefcase/commands/base.py	`98.09% <100.00%> (+0.05%)`	⬆️
src/briefcase/commands/create.py	`99.65% <100.00%> (+0.02%)`	⬆️
src/briefcase/integrations/flatpak.py	`100.00% <100.00%> (ø)`
src/briefcase/platforms/linux/appimage.py	`97.43% <100.00%> (ø)`
src/briefcase/platforms/linux/flatpak.py	`100.00% <100.00%> (ø)`

freakboy3742 added 8 commits July 25, 2022 15:34

Correct a copy/paste error in AppImage backend.

9e40386

Add integration with Flatpak tooling.

06ebbfa

Add an extension point that allows backends to control the unpacking …

c41d441

…of support packages.

Add a flatpak backend.

439f8e0

Add changenote.

b8e3a8b

Disable waitbar when installing flatpak runtimes.

d3b34e9

Add docs for flatpak backend.

34ca844

Make tests resilient to changes in host platform architecture.

01e42e2

rmartin16 reviewed Jul 26, 2022

View reviewed changes

Comment thread src/briefcase/platforms/linux/flatpak.py Outdated

rmartin16 reviewed Jul 26, 2022

View reviewed changes

Make requirements.txt generation a first-class feature of project cre…

f7bca31

…ation.

freakboy3742 added 5 commits July 27, 2022 09:46

Clean up typos and cursed apostrophes.

e5f3523

Ensure files are opened utf-8

721500e

Use a more generic shell for the maker file.

f8c6ec6

Add protection against mismatched platform/SDK combinations.

164105c

Specify architecture when installing runtime/SDK.

23268d0

Add version checks for flatpak tooling.

d592a71

freakboy3742 force-pushed the flatpak branch from cb36ad1 to 93f2c01 Compare July 27, 2022 09:45

mhsmith reviewed Jul 27, 2022

View reviewed changes

freakboy3742 force-pushed the flatpak branch from 93f2c01 to eb43fc9 Compare July 27, 2022 11:51

Run Linux desktop app verification tests outside docker.

b203f8d

freakboy3742 force-pushed the flatpak branch from eb43fc9 to b203f8d Compare July 27, 2022 11:53

Minor cleanups of flatpak test cases.

baa221c

Correct typo in platform name.

f969a47

mhsmith approved these changes Jul 27, 2022

View reviewed changes

freakboy3742 merged commit 0d7c359 into beeware:main Jul 27, 2022

freakboy3742 deleted the flatpak branch July 27, 2022 13:33

freakboy3742 mentioned this pull request Jul 27, 2022

Add Flatpak Packaging #359

Closed

freakboy3742 mentioned this pull request Aug 10, 2022

Add support for Raspberry Pi #449

Open

freakboy3742 mentioned this pull request Dec 5, 2024

Use a consistent bundle ID for all examples. beeware/toga#3020

Merged

4 tasks

		def test_verify_linux_no_docker(tmp_path):
		"""Verifying on Linux creates an SDK wrapper."""

		def test_verify_non_linux_no_docker(tmp_path):
		"""Verifying on non-Linux raises an error."""

Uh oh!

Uh oh!

Conversation

freakboy3742 commented Jul 26, 2022

PR Checklist:

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rmartin16 commented Jul 26, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mhsmith commented Jul 26, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

freakboy3742 commented Jul 27, 2022

Uh oh!

rmartin16 commented Jul 27, 2022

Uh oh!

rmartin16 commented Jul 27, 2022

Uh oh!

freakboy3742 commented Jul 27, 2022

Uh oh!

rmartin16 commented Jul 27, 2022

Uh oh!

rmartin16 commented Jul 27, 2022

Uh oh!

freakboy3742 commented Jul 27, 2022

Uh oh!

rmartin16 commented Jul 27, 2022

Uh oh!

freakboy3742 commented Jul 27, 2022

Uh oh!

mhsmith commented Jul 27, 2022

Uh oh!

freakboy3742 commented Jul 27, 2022

Uh oh!

mhsmith left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

freakboy3742 commented Jul 27, 2022

Uh oh!

codecov Bot commented Jul 27, 2022

Codecov Report

Uh oh!

mhsmith commented Jul 26, 2022 •

edited

Loading