Skip to content

Commit 43ef81a

Browse files
gregjacobussdelliotmitchnegus
authored
feat: ADB Driver (sandialabs#243)
# Add ADB driver support for Android VMs ## Description This PR adds Android Debug Bridge (ADB) support to the VM Resource Handler so FIREWHEEL can manage Android emulator guests without requiring the QEMU Guest Agent. It includes the new `ADBDriver`, VM Resource Handler and minimega API updates needed to discover and use Android-specific connection metadata, and the supporting `adbutils` dependency. It also adds Android system documentation and tutorials for setup, automation, and analysis. ## Type of Change Please select the type of change your pull request introduces: - [ ] Bugfix - [x] New feature - [x] Documentation update - [ ] Other (please describe): ## Checklist - [x] This PR conforms to the process detailed in the [Contributing Guide](https://sandialabs.github.io/firewheel/developer/contributing.html). - [x] I have included no proprietary/sensitive information in my code. - [x] I have performed a self-review of my code. - [x] I have commented my code, particularly in hard-to-understand areas. - [x] I have made corresponding changes to the documentation. - [x] My changes generate no new warnings. - [x] I have tested my code. ## Additional Notes - `adbutils>=2.12.0` is added as a new dependency for Android guest communication. --------- Signed-off-by: gregjacobus <30842019+gregjacobus@users.noreply.github.com> Signed-off-by: Steven Elliott <6461548+sdelliot@users.noreply.github.com> Signed-off-by: Mitch Negus <21086604+mitchnegus@users.noreply.github.com> Co-authored-by: Steven Elliott <6461548+sdelliot@users.noreply.github.com> Co-authored-by: Steven Elliott <selliot@sandia.gov> Co-authored-by: Mitch Negus <21086604+mitchnegus@users.noreply.github.com>
1 parent 2b372ff commit 43ef81a

25 files changed

Lines changed: 2754 additions & 132 deletions

‎docs/source/cli/commands.rst‎

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -42,8 +42,7 @@ config edit
4242

4343
usage: firewheel config edit [-e EDITOR]
4444

45-
Edit the FIREWHEEL configuration with a text editor. The user must set either the VISUAL or EDITOR
46-
environment variable or use the provided flag to override these environment variables.
45+
Edit the FIREWHEEL configuration with a text editor. The user must set either the VISUAL or EDITOR environment variable or use the provided flag to override these environment variables.
4746

4847
options:
4948

‎docs/source/system/architecture.rst‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@ To accomplish this, FIREWHEEL's *Control* system interprets user defined experim
1717
FIREWHEEL Software Architecture
1818

1919
FIREWHEEL's software architecture, as shown in :numref:`Figure %s <fw_sw_arch>`, consists of a Command Line Interface (CLI), Experiment Models, and three other major components (*Control*, minimega, and the *VM Resource Manager*) that convert experiment models into running experiments, and manage FIREWHEEL's VM deployment, virtual network and VM provisioning, and run time execution capabilities.
20-
These FIREWHEEL software components are executed within a Python Virtual Environment, which we will refer to as ``fwpy``, and interact with the Hardware Virtualization components, QEMU/KVM and Open vSwitch (OVS), to instantiate experiment models as configured VMs communicating via virtual networks.
20+
These FIREWHEEL software components are executed within a Python Virtual Environment, which we will refer to as ``fwpy``, and interact with hardware virtualization and networking components, including QEMU/KVM, the Android Emulator when Android VMs are used, and Open vSwitch (OVS), to instantiate experiment models as configured VMs communicating via virtual networks.
2121
We'll next discuss each of FIREWHEEL's major software components.
2222

2323
.. note::
@@ -103,7 +103,8 @@ FIREWHEEL uses `NetworkX <https://networkx.org/>`_ [#netx]_, a Python library fo
103103
minimega
104104
========
105105

106-
`minimega <https://www.sandia.gov/minimega/>`__, is the name of the virtualization management component that's currently available with FIREWHEEL. minimega will instantiate an experiment using QEMU/KVM for VMs, and OVS networks.
106+
`minimega <https://www.sandia.gov/minimega/>`__, is the name of the virtualization management component that's currently available with FIREWHEEL.
107+
minimega will instantiate an experiment using supported VM backends, such as QEMU/KVM for conventional VMs and native Android emulator support for Android VMs, and OVS networks.
107108
minimega receives an experiment graph from FIREWHEEL's *Control* system, then determines and sends the appropriate set of instructions to the QEMU/KVM hypervisor and OVS virtual networking systems that are needed to instantiate the experiment model as an emulated computer network.
108109
At this stage the experiment network's topology has been fully instantiated on the virtualization platform, but has yet to have any of its scheduled actions applied to it -- which may further configure the compute systems and network devices as needed before beginning the intended cyber experiment.
109110

Lines changed: 124 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,124 @@
1+
.. _adb-driver:
2+
3+
Android Debug Bridge Driver
4+
===========================
5+
6+
The `Android Debug Bridge (ADB) <https://developer.android.com/tools/adb>`_ driver enables the VM Resource Handler to communicate with Android guests.
7+
It is selected using the VM Resource Handler engine name ``ADB``.
8+
9+
The ADB driver is intended for Android VMs launched through minimega's native ``android`` VM type.
10+
It does not use the QEMU Guest Agent or a QGA socket.
11+
12+
Driver Selection
13+
----------------
14+
15+
Android VM Resource Handler process configurations should use:
16+
17+
.. code-block:: json
18+
19+
{
20+
"engine": "ADB",
21+
"adb_serial": "emulator-5554",
22+
"android_console_port": 5554,
23+
"android_adb_port": 5555,
24+
"require_root": true
25+
}
26+
27+
The ``adb_serial`` value is the primary identifier used by the driver. If it is
28+
not provided, the driver can fall back to ``emulator-<android_console_port>``.
29+
30+
ADB, Console Ports, and Serial Names
31+
------------------------------------
32+
33+
Android emulators normally use two sequential ports:
34+
35+
* an emulator console port, for example ``5554``;
36+
* an ADB daemon port, typically ``5555``.
37+
38+
The ADB serial uses the console port:
39+
40+
.. code-block:: text
41+
42+
emulator-5554
43+
44+
minimega is authoritative for the actual assigned port values.
45+
If the requested port pair is unavailable, minimega may assign a different port pair.
46+
FIREWHEEL should use the actual values reported by minimega before launching the VM Resource Handler.
47+
48+
Runtime Paths
49+
-------------
50+
51+
The ADB driver uses Android-native FIREWHEEL runtime paths:
52+
53+
.. code-block:: text
54+
55+
/data/local/tmp/firewheel/launch
56+
/data/local/tmp/firewheel/proc
57+
58+
Generated call scripts use:
59+
60+
.. code-block:: sh
61+
62+
#!/system/bin/sh
63+
64+
This differs from QGA/Linux VMs, which traditionally use ``/var/launch`` and ``/bin/bash``.
65+
66+
Execution Model
67+
---------------
68+
69+
The ADB driver executes VM resources by writing a small runner script under:
70+
71+
.. code-block:: text
72+
73+
/data/local/tmp/firewheel/proc
74+
75+
The runner records:
76+
77+
* stdout
78+
* stderr
79+
* exit code
80+
* a started marker
81+
82+
This avoids long-lived ADB stream parsing and provides reliable exit-code capture for Android VM resources.
83+
84+
Android Shell Limitations
85+
-------------------------
86+
87+
Android guests should not be assumed to provide:
88+
89+
* ``/bin/bash``
90+
* ``/bin/sh``
91+
* Python
92+
* systemd
93+
94+
VM resources intended for Android should generally use:
95+
96+
.. code-block:: sh
97+
98+
#!/system/bin/sh
99+
100+
Android model components may provide optional Bash or Python runtimes, but these are not required by the ADB driver.
101+
102+
Root Access
103+
-----------
104+
105+
The ADB driver can require root by setting ``require_root`` to ``true``.
106+
This is the default for FIREWHEEL Android experiments that need to configure networking or system-level state.
107+
108+
If ``require_root`` is enabled, the Android image must support ``adb root``.
109+
110+
If ``require_root`` is false, the driver can connect to non-rooted devices, but VM resources that require privileged operations such as network configuration may fail.
111+
112+
File Transfer
113+
-------------
114+
115+
The ADB driver uses ADB sync operations to push and pull files. File transfers are timestamp-filtered where possible using Android ``stat``/``toybox stat``.
116+
117+
Reboot Behavior
118+
---------------
119+
120+
The ADB driver can reboot Android guests through ADB.
121+
Runtime network state such as IP addresses, routes, policy rules, and firewall rules is generally lost on Android reboot.
122+
Installed APKs typically persist across a normal Android reboot.
123+
124+
Android-specific reboot and runtime-state limitations are documented in the Android model component repository.

‎docs/source/system/vm_resource_manager/index.rst‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,3 +35,4 @@ However, there are several benefits to dynamic, automated configuration of VMs o
3535
vmr_in_experiment
3636
vm_resource_handler
3737
qga
38+
adb

‎docs/source/system/vm_resource_manager/qga.rst‎

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -4,10 +4,15 @@ QEMU Guest Agent
44
================
55

66
The `QEMU Guest Agent (QGA) <https://wiki.qemu.org/Features/GuestAgent>`_ is a binary file located within a VM which receives QEMU Machine Protocol (QMP) messages and can be used to perform actions within the virtual machine.
7-
The QGA is the primary way in which the :ref:`vm-resource-handler` communicates with VMs and enables the completion of VM resources.
8-
Therefore, in order for a VM image to be "FIREWHEEL-compatible" the QGA should be installed on the system and set to run as a service on system start.
7+
The QGA is the primary way in which the :ref:`vm-resource-handler` communicates with QEMU/KVM VMs and enables the completion of VM resources.
8+
Therefore, in order for a QEMU/KVM VM image to be "FIREWHEEL-compatible" the QGA should be installed on the system and set to run as a service on system start.
99

10-
When a VM is launched, a virtual serial port that is added which enables the QGA to communicate to FIREWHEEL via the :py:class:`QemuGuestAgentDriver <firewheel.vm_resource_manager.drivers.qemu_guest_agent_driver.QemuGuestAgentDriver>`, which then works with the :ref:`vm-resource-handler` for scheduling and executing VMRs within the VM.
10+
.. note::
11+
12+
Android guests use the ADB driver rather than the QEMU Guest Agent. See
13+
:ref:`adb-driver`.
14+
15+
When a QEMU/KVM VM is launched with QGA support, a virtual serial port that is added which enables the QGA to communicate to FIREWHEEL via the :py:class:`QemuGuestAgentDriver <firewheel.vm_resource_manager.drivers.qemu_guest_agent_driver.QemuGuestAgentDriver>`, which then works with the :ref:`vm-resource-handler` for scheduling and executing VMRs within the VM.
1116

1217
.. seealso::
1318

‎docs/source/system/vm_resource_manager/vm_resource_handler.rst‎

Lines changed: 10 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -7,13 +7,19 @@ To facilitate the execution of VMRs, FIREWHEEL uses a :py:class:`VM Resource Han
77
In addition to being in charge of loading VMRs on to a VM, it also facilitates executing commands on VMs and extracting information out of the environment (e.g. :ref:`helper_pull_file`).
88

99
Practically, the :py:class:`VM Resource Handler <firewheel.vm_resource_manager.vm_resource_handler.VMResourceHandler>` dynamically loads in drivers which perform the actual interactions between the VM and the :py:class:`VM Resource Handler <firewheel.vm_resource_manager.vm_resource_handler.VMResourceHandler>`.
10-
Currently, there is only one driver (:py:class:`QemuGuestAgentDriver <firewheel.vm_resource_manager.drivers.qemu_guest_agent_driver.QemuGuestAgentDriver>`) which handles VMs of type ``QemuVM`` (see :ref:`qemu-guest-agent`).
11-
However, this dynamic loading function enables future expansion of drivers that will work without changing the :py:class:`VM Resource Handler <firewheel.vm_resource_manager.vm_resource_handler.VMResourceHandler>` [#f1]_.
10+
FIREWHEEL currently supports multiple VM Resource Handler communication drivers, including:
11+
12+
* :py:class:`QemuGuestAgentDriver <firewheel.vm_resource_manager.drivers.qemu_guest_agent_driver.QemuGuestAgentDriver>` for QEMU/KVM VMs using the QEMU Guest Agent. This driver is selected with engine ``QemuVM``. See :ref:`qemu-guest-agent`.
13+
* :py:class:`ADBDriver <firewheel.vm_resource_manager.drivers.adb_driver.ADBDriver>` for Android guests using the Android Debug Bridge. This driver is selected with engine ``ADB``. See :ref:`adb-driver`.
14+
15+
.. note::
16+
17+
Additional drivers may be added in the future for other guest communication mechanisms, such as `miniccc <https://www.sandia.gov/minimega/module-28-miniccc-and-the-cc-api/>`_.
18+
19+
This dynamic loading mechanism allows FIREWHEEL to support different guest communication mechanisms without changing the VM Resource Handler's scheduling logic.
1220

1321
.. _vm-resource-logs:
1422

1523
VM Resource Logs
1624
----------------
1725
The :py:class:`VM Resource Handler <firewheel.vm_resource_manager.vm_resource_handler.VMResourceHandler>` also facilitates logging VMR data onto the physical host.
18-
19-
.. [#f1] For example, we hope to eventually create a driver for `miniccc <https://www.sandia.gov/minimega/module-28-miniccc-and-the-cc-api/>`_.

‎docs/source/system/vm_resource_manager/vmr_in_experiment.rst‎

Lines changed: 60 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -101,7 +101,7 @@ These get passed to the program on the command line.
101101
Finally, the ``vm_resource`` parameter is an optional Boolean that indicates if ``program`` is the name of a script that needs to be loaded on to the VM before execution.
102102
If ``vm_resource`` is ``True`` then the specified program name is assumed to be the local filename of the file (i.e. not the full path, just the filename) to load on to the VM.
103103
In this case, like :py:meth:`drop_file <base_objects.VMEndpoint.drop_file>`, the file that corresponds to ``program`` must be part of the ``vm_resources`` list in a model component's ``MANIFEST`` file (see :ref:`vm_resources_field`).
104-
The user does not need to specify where the file will be dropped on the VM since it will be placed in the appropriate ``/var/launch`` or ``C:\launch`` location (see :ref:`vmr-location`).
104+
The user does not need to specify where the file will be dropped on the VM since it will be placed in the appropriate launch directory for the VM's communication driver. For QGA Linux guests this is ``/var/launch``; for Windows guests this is ``C:\launch``; for Android guests using the ADB driver this is ``/data/local/tmp/firewheel/launch`` (see :ref:`vmr-location`).
105105

106106
Examples
107107
""""""""
@@ -183,13 +183,22 @@ Another concrete example of how this can be done is shown in the example below w
183183
184184
Finally, the :ref:`tests.reboot_mc` has various examples of Python VMRs requesting a reboot.
185185

186+
.. note::
187+
188+
Guest reboot effects are OS-specific. On Android, runtime network configuration
189+
such as IP addresses, routes, policy routing rules, and firewall rules is
190+
generally lost after reboot. Installed APKs typically persist across a normal
191+
Android reboot. Android resources that need networking after reboot should
192+
rerun the Android network configuration resources.
193+
186194
.. [#] For more details about why this is the case see https://www.howtogeek.com/182817/htg-explains-why-does-windows-want-to-reboot-so-often/
187195
188196
.. _vmr-env:
189197

190198
VMRs In-Experiment Environment
191199
------------------------------
192-
Because of how the VM Resources are executed within the VMs by the :ref:`QEMU Guest Agent <qga-driver>`, users should not make any assumptions about the `environment variables <https://en.wikipedia.org/wiki/Environment_variable>`_ which are available.
200+
Because VM Resources are executed through a VM Resource Handler driver, users should not make assumptions about the `environment variables <https://en.wikipedia.org/wiki/Environment_variable>`_ available inside the VM.
201+
The exact environment depends on the guest OS and driver. For example, :ref:`QEMU Guest Agent <qga-driver>` Linux execution and Android ADB execution expose different shell environments.
193202
That is, users should **NOT** assume that standard Environment variables (e.g. ``$HOME``, ``$USER``, ``$SHELL``, etc.) are available.
194203
Some software may make assumptions that these common environment variables exist, which may result in odd failures when attempting to run the software.
195204
Below are some examples of environments which *might* be available.
@@ -211,13 +220,19 @@ Example output from :py:meth:`run_executable <base_objects.VMEndpoint.run_execut
211220
OLDPWD=/
212221
_=/usr/bin/printenv
213222

223+
Android ADB-based VM resources should generally assume ``/system/bin/sh`` is available, but should **not** assume Bash or Python are installed.
214224

215225
.. _vmr-location:
216226

217227
Location of VMRs within the VM
218228
------------------------------
219229
Prior to the start of :ref:`schedule-negative-time`, VMRs are uploaded onto the VM.
220-
FIREWHEEL uses the ``/var/launch`` directory on Linux based VMs and the ``C:\launch`` directory on Windows based VMs.
230+
FIREWHEEL uses driver- and OS-specific launch directories. Common locations include:
231+
232+
* ``/var/launch`` for QGA Linux guests
233+
* ``C:\launch`` for Windows guests
234+
* ``/data/local/tmp/firewheel/launch`` for Android guests using the ADB driver
235+
221236
Inside these directories is a series of directories with :ref:`start times <start-time>` that are used by the scheduled VMRs.
222237

223238
For example, if there are VMRs scheduled at -300, -250, -100, and 5 then ``/var/launch`` would look like:
@@ -227,7 +242,7 @@ For example, if there are VMRs scheduled at -300, -250, -100, and 5 then ``/var/
227242
$ ls /var/launch
228243
-100 -250 -300 5
229244
230-
Inside these ref:`<start-time>` directories is a directory for each VMR which is scheduled at the given time.
245+
Inside these :ref:`start time <start-time>` directories is a directory for each VMR which is scheduled at the given time.
231246

232247
For example, if ``set_hostname.sh`` and ``get_stat.py`` both occurred at ``-250`` it would look like:
233248

@@ -239,16 +254,17 @@ For example, if ``set_hostname.sh`` and ``get_stat.py`` both occurred at ``-250`
239254
.. note::
240255
To access negative time folders you will likely need to use either a full path (e.g. ``/var/launch/-100`` or a specific relative path ``./-100``. This is because the negative sign (i.e. the hyphen) is typically used to express a CLI option for most shell programs.
241256

242-
Each of the folders with the VMR name contains a file called ``call_arguments.sh`` which is a dynamically-generated script which executes the VMR.
257+
Each VMR directory contains a dynamically-generated call-arguments file which executes the VMR.
258+
On Linux and Android guests this is typically ``call_arguments.sh``; on Windows guests this is typically ``call_arguments.bat``.
243259
Additionally, if there is other data, scripts, etc. which need to be executed for the given VMR, they are also located in this directory.
244-
For example, here is the ``/var/launch/-250/set_hostname.sh`` directory:
260+
For example, on a QGA Linux guest, here is the ``/var/launch/-250/set_hostname.sh`` directory:
245261

246262
.. code-block:: bash
247263
248264
$ ls /var/launch/-250/set_hostname.sh
249265
call_arguments.sh set_hostname.sh
250266
251-
Here is an example ``call_arguments.sh`` file for ``set_hostname.sh``:
267+
Here is an example ``call_arguments.sh`` file on a QGA Linux guest for ``set_hostname.sh``:
252268

253269
.. code-block:: bash
254270
:caption: Example ``call_arguments.sh``
@@ -258,14 +274,36 @@ Here is an example ``call_arguments.sh`` file for ``set_hostname.sh``:
258274
cd /var/launch/-250/set_hostname.sh
259275
/var/launch/-250/set_hostname.sh/set_hostname.sh host.root.net
260276
261-
To re-run the VMR (for debugging purposes) a user can simply re-execute ``call_arguments.sh`` as the root user:
277+
Here is an example Android ADB ``call_arguments.sh`` file:
278+
279+
.. code-block:: sh
280+
281+
#!/system/bin/sh
282+
CURRENT_DIR="$(dirname "$0")"
283+
cd /data/local/tmp/firewheel/launch/-250/set_hostname_android.sh
284+
/data/local/tmp/firewheel/launch/-250/set_hostname_android.sh/set_hostname_android.sh android-0
285+
286+
To re-run a VMR (for debugging purposes) a user can simply re-execute ``call_arguments.sh`` as the root user.
287+
For example, on a Linux VM might it might look like:
262288

263289
.. code-block:: bash
264290
265291
$ sudo /var/launch/-250/set_hostname.sh/call_arguments.sh
266292
293+
On an Android guest using the ADB driver, a user can inspect or re-run the
294+
generated script with ADB:
295+
296+
.. code-block:: bash
297+
298+
$ adb -s emulator-5554 shell /system/bin/sh /data/local/tmp/firewheel/launch/-250/set_hostname_android.sh/call_arguments.sh
299+
267300
Becoming familiar with the locations of VMRs with the VMs is useful for developing and debugging new VMRs.
268301

302+
.. note::
303+
304+
Android VMRs should generally use ``/system/bin/sh`` and should **not** assume
305+
``/bin/bash``, ``/bin/sh``, or Python are installed.
306+
269307
.. _vmr-output:
270308

271309
Extracting VMR data
@@ -300,6 +338,20 @@ This could include running a script that modifies the entire network topology or
300338
The primary method for scheduling host-based VM resources is through the use of the :py:meth:`run_host_mm_command <base_objects.VMEndpoint.run_host_mm_command>` method, which creates a :py:class:`RunHostExecutableScheduleEntry <base_objects.RunHostExecutableScheduleEntry>` for executing Minimega commands.
301339
Additional interfaces into host-based scheduling may be added in the future to facilitate common tasks.
302340

341+
Host-side VM resources receive several FIREWHEEL-specific environment variables:
342+
343+
* ``FIREWHEEL_VM_NAME``
344+
* ``FIREWHEEL_VM_UUID``
345+
* ``FIREWHEEL_VM_CONFIG_JSON``
346+
* ``FIREWHEEL_PYTHON``
347+
348+
For Android/ADB-backed VMs, the environment may also include:
349+
350+
* ``FIREWHEEL_ADB_SERIAL``
351+
* ``FIREWHEEL_ANDROID_CONSOLE_PORT``
352+
353+
These can be useful for host-side resources that need access to information about the scheduling VM.
354+
303355
:py:meth:`run_host_mm_command <base_objects.VMEndpoint.run_host_mm_command>`
304356
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
305357
The :py:meth:`run_host_mm_command <base_objects.VMEndpoint.run_host_mm_command>` method allows a user to schedule Minimega commands directly on the host.

0 commit comments

Comments
 (0)